| *** Qiming has quit IRC | 00:00 | |
| *** amitgandhinz has quit IRC | 00:01 | |
| *** sabari has left #openstack-infra | 00:02 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update EnvInject plugin to use convert xml https://review.openstack.org/324140 | 00:04 |
|---|---|---|
| *** amrith is now known as _amrith_ | 00:05 | |
| *** nwkarsten has joined #openstack-infra | 00:08 | |
| EmilienM | anteaya, pabelanger: if you have time tomorrow, can you look https://review.openstack.org/#/c/323870/ please? | 00:10 |
| *** zeih has joined #openstack-infra | 00:11 | |
| *** Swami has quit IRC | 00:11 | |
| *** amotoki has joined #openstack-infra | 00:12 | |
| *** nwkarsten has quit IRC | 00:13 | |
| *** zeih has quit IRC | 00:15 | |
| *** xarses has quit IRC | 00:17 | |
| *** itsuugo has quit IRC | 00:18 | |
| *** sdake_ is now known as sdake | 00:19 | |
| *** fitoduarte has quit IRC | 00:20 | |
| *** ayoung has joined #openstack-infra | 00:22 | |
| *** sree has joined #openstack-infra | 00:29 | |
| *** oanson has joined #openstack-infra | 00:29 | |
| *** amotoki has quit IRC | 00:32 | |
| *** sree has quit IRC | 00:34 | |
| *** r-mibu has quit IRC | 00:36 | |
| *** oanson has quit IRC | 00:36 | |
| *** r-mibu has joined #openstack-infra | 00:36 | |
| *** kurtmartin has quit IRC | 00:40 | |
| *** zz_dimtruck is now known as dimtruck | 00:45 | |
| *** thorst has joined #openstack-infra | 00:46 | |
| *** bpokorny_ has joined #openstack-infra | 00:51 | |
| *** Julien-zte has joined #openstack-infra | 00:52 | |
| EmilienM | pabelanger: thx, I replied | 00:52 |
| *** bpokorny_ has quit IRC | 00:54 | |
| *** bpokorny has quit IRC | 00:56 | |
| *** Julien-zte has quit IRC | 00:56 | |
| asselin | mordred, nibalizer (or anyone else familiar) what's the status of this spec? working? great? done? https://specs.openstack.org/openstack-infra/infra-specs/specs/ansible_puppet_apply.html | 00:56 |
| *** xarses has joined #openstack-infra | 00:57 | |
| *** bryan_att has quit IRC | 01:03 | |
| *** pvaneck has quit IRC | 01:04 | |
| *** zhurong has joined #openstack-infra | 01:05 | |
| pabelanger | jeblair: notmorgan: I see this exception from nodepool every so often too: http://paste.openstack.org/show/507074/ | 01:08 |
| *** kien-ha has quit IRC | 01:08 | |
| *** woodster_ has quit IRC | 01:08 | |
| pabelanger | #status log nodepool.o.o restarted to fix jenkins01.o.o (wasn't launching jobs) | 01:09 |
| openstackstatus | pabelanger: finished logging | 01:09 |
| notmorgan | pabelanger: oh interesting | 01:10 |
| notmorgan | pabelanger: looks like we have some issues with bad types sneaking through | 01:11 |
| notmorgan | pabelanger: that tells me we aren't validating things quite right. :( | 01:11 |
| notmorgan | (or shade isn't) | 01:11 |
| *** Qiming has joined #openstack-infra | 01:12 | |
| pabelanger | notmorgan: Ya, something odd is going on. I'd expect node.id to be populated, I think we automagicly get it from sqlalchemy | 01:12 |
| pabelanger | I look more into it tomorrow and see what is happening to the node before that | 01:12 |
| *** Sukhdev has quit IRC | 01:13 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:13 | |
| *** Sukhdev has joined #openstack-infra | 01:14 | |
| *** amotoki has joined #openstack-infra | 01:15 | |
| *** akshai has joined #openstack-infra | 01:15 | |
| *** Julien-zte has joined #openstack-infra | 01:16 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Refactor side-effect functions https://review.openstack.org/320225 | 01:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Patch glean.cmd.open for unit testing https://review.openstack.org/320224 | 01:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Use mock patching for most functions https://review.openstack.org/318464 | 01:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Add selinux context manager for writing files https://review.openstack.org/304357 | 01:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Remove mock of os.stat() https://review.openstack.org/324153 | 01:17 |
| *** rhallisey has quit IRC | 01:17 | |
| ianw | mordred / SpamapS / yolanda : ^ i hope we can get that in before i have to rebase again. the os.stat() thing is i think left-over code but is wrong | 01:17 |
| *** Sukhdev has quit IRC | 01:19 | |
| *** fitoduarte has joined #openstack-infra | 01:20 | |
| *** amotoki_ has joined #openstack-infra | 01:24 | |
| *** sree has joined #openstack-infra | 01:24 | |
| *** amotoki has quit IRC | 01:26 | |
| *** Daisy has joined #openstack-infra | 01:26 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack-infra/project-config: Don't run puppet tests on rst/unit test changes https://review.openstack.org/324158 | 01:26 |
| oomichi | EmilienM: ^^^ done | 01:27 |
| *** psilvad has quit IRC | 01:27 | |
| *** sree has quit IRC | 01:29 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: layout: cleanup redundant rules https://review.openstack.org/324159 | 01:29 |
| EmilienM | oomichi: could you rebase it on top of https://review.openstack.org/#/c/323870/ please? | 01:30 |
| EmilienM | oomichi: I did some refactoring the last days and it's going to land soon, thanks | 01:30 |
| *** Apoorva has quit IRC | 01:30 | |
| oomichi | EmilienM: ah, cool. thanks. will do it soon :) | 01:31 |
| EmilienM | thx a lot | 01:31 |
| EmilienM | oomichi: I think we can reduce it | 01:31 |
| EmilienM | to make it very generic | 01:31 |
| EmilienM | oomichi: actually, I have an idea | 01:32 |
| oomichi | EmilienM: yeah, I prefer that | 01:32 |
| EmilienM | oomichi: try to find: | 01:33 |
| EmilienM | name: ^gate-(puppet-.*-(syntax|unit|beaker|integration)-|tripleo-ci-f22-).*$ | 01:33 |
| EmilienM | and use it! | 01:33 |
| EmilienM | just add openstack/tempest project and set the files you want and we're good | 01:33 |
| EmilienM | no need to rebase | 01:33 |
| *** jerryz has quit IRC | 01:34 | |
| *** bbroad has quit IRC | 01:34 | |
| EmilienM | oomichi: does it make sense? | 01:35 |
| oomichi | EmilienM: oh, a newer idea seems good for me :) | 01:38 |
| *** Julien-zte has quit IRC | 01:38 | |
| oomichi | as generic way | 01:38 |
| EmilienM | yeah | 01:38 |
| EmilienM | oomichi: so don't rebase, and reuse the line I told you, and it will work & be simple | 01:38 |
| * EmilienM afk | 01:39 | |
| *** amit213 has joined #openstack-infra | 01:39 | |
| oomichi | EmilienM: yeah, thanks for your advice | 01:39 |
| *** zeih has joined #openstack-infra | 01:39 | |
| oomichi | will do it soon | 01:39 |
| *** yanyanhu has joined #openstack-infra | 01:39 | |
| *** amit213 has quit IRC | 01:41 | |
| *** zeih has quit IRC | 01:43 | |
| *** amit213 has joined #openstack-infra | 01:44 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack-infra/project-config: Don't run puppet tests on rst/unit test changes https://review.openstack.org/324158 | 01:45 |
| oomichi | EmilienM: ^^^ thanks, done again | 01:46 |
| *** sdake has quit IRC | 01:49 | |
| *** dkaigarodsev has quit IRC | 01:50 | |
| *** tlbr has quit IRC | 01:50 | |
| *** SergK has quit IRC | 01:51 | |
| *** mattymo has quit IRC | 01:51 | |
| *** _degorenko|afk has quit IRC | 01:51 | |
| *** skolekonov has quit IRC | 01:51 | |
| *** dkaigarodsev has joined #openstack-infra | 01:51 | |
| *** tlbr has joined #openstack-infra | 01:51 | |
| openstackgerrit | Ari proposed openstack-infra/jenkins-job-builder: Adds support multijob support to the CopyArtifact plugin https://review.openstack.org/323495 | 01:52 |
| *** kaisers__ has joined #openstack-infra | 01:52 | |
| *** skolekonov has joined #openstack-infra | 01:52 | |
| *** kaisers_ has quit IRC | 01:54 | |
| *** degorenko has joined #openstack-infra | 01:55 | |
| *** banix has quit IRC | 01:55 | |
| *** SergK has joined #openstack-infra | 01:55 | |
| *** mattymo has joined #openstack-infra | 01:57 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul: Pass OS_*_CAPTURE env variables https://review.openstack.org/324165 | 01:57 |
| *** amitgandhinz has joined #openstack-infra | 01:58 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Cherrypick nova for bug 1585641 https://review.openstack.org/323516 | 01:58 |
| openstack | bug 1585641 in tripleo "CI: jobs failing with Message: No valid host was found. There are not enough hosts available., Code: 500"" [High,Triaged] https://launchpad.net/bugs/1585641 - Assigned to James Slagle (james-slagle) | 01:58 |
| *** PollyZ_ has joined #openstack-infra | 01:59 | |
| *** sputnik13 has quit IRC | 02:00 | |
| *** kzaitsev_mb has quit IRC | 02:01 | |
| *** kaisers__ has quit IRC | 02:01 | |
| *** roxanaghe has joined #openstack-infra | 02:04 | |
| *** amitgandhinz has quit IRC | 02:04 | |
| *** erlon has quit IRC | 02:06 | |
| *** moravec has joined #openstack-infra | 02:06 | |
| *** zeih has joined #openstack-infra | 02:06 | |
| *** roxanaghe has quit IRC | 02:09 | |
| *** banix has joined #openstack-infra | 02:09 | |
| *** EricGonczer_ has quit IRC | 02:11 | |
| *** zeih has quit IRC | 02:11 | |
| *** pahuang has joined #openstack-infra | 02:11 | |
| *** yamahata has quit IRC | 02:11 | |
| *** sdake has joined #openstack-infra | 02:16 | |
| *** sdake has quit IRC | 02:18 | |
| mordred | ianw: they look great to me - thanks! +2 to the stack | 02:18 |
| *** dingyichen has joined #openstack-infra | 02:23 | |
| *** watersoul has quit IRC | 02:26 | |
| *** watersoul has joined #openstack-infra | 02:28 | |
| *** itisha has quit IRC | 02:30 | |
| *** Daisy has quit IRC | 02:31 | |
| *** Daisy has joined #openstack-infra | 02:31 | |
| *** watersoul has joined #openstack-infra | 02:32 | |
| *** dingyichen has quit IRC | 02:32 | |
| Shrews | look, it's the mythical mordred creature. o/ | 02:34 |
| *** Daisy has quit IRC | 02:36 | |
| *** pt_15 has quit IRC | 02:36 | |
| *** Daisy has joined #openstack-infra | 02:39 | |
| *** anteaya has quit IRC | 02:48 | |
| mordred | Shrews: o hai | 02:51 |
| *** yuanying has quit IRC | 02:51 | |
| *** claudiub has quit IRC | 02:52 | |
| openstackgerrit | Merged openstack-infra/glean: Remove mock of os.stat() https://review.openstack.org/324153 | 02:56 |
| *** kzaitsev_mb has joined #openstack-infra | 02:57 | |
| *** amitgandhinz has joined #openstack-infra | 03:01 | |
| *** roxanaghe has joined #openstack-infra | 03:04 | |
| *** julim has joined #openstack-infra | 03:05 | |
| *** akshai has quit IRC | 03:05 | |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: [Monasca] Add Tempest Jobs as non-voting for monasca repos https://review.openstack.org/316293 | 03:06 |
| *** amitgandhinz has quit IRC | 03:07 | |
| greghaynes | ianw: left you a comment on https://review.openstack.org/#/c/318464/3 | 03:08 |
| *** pahuang has quit IRC | 03:09 | |
| *** roxanaghe has quit IRC | 03:09 | |
| *** yamamoto_ has joined #openstack-infra | 03:10 | |
| *** Wei_Liu has joined #openstack-infra | 03:11 | |
| *** stevebaker has quit IRC | 03:27 | |
| *** yamahata has joined #openstack-infra | 03:28 | |
| *** stevebaker has joined #openstack-infra | 03:28 | |
| *** Daisy has quit IRC | 03:30 | |
| *** Daisy has joined #openstack-infra | 03:31 | |
| *** Daisy has quit IRC | 03:34 | |
| *** zeih has joined #openstack-infra | 03:35 | |
| *** links has joined #openstack-infra | 03:35 | |
| *** Daisy has joined #openstack-infra | 03:35 | |
| *** kzaitsev_mb has quit IRC | 03:35 | |
| *** julim has quit IRC | 03:36 | |
| *** zeih has quit IRC | 03:39 | |
| *** Daisy has quit IRC | 03:40 | |
| *** dimtruck is now known as zz_dimtruck | 03:40 | |
| *** xarses_ has joined #openstack-infra | 03:40 | |
| *** xarses_ has quit IRC | 03:41 | |
| *** xarses_ has joined #openstack-infra | 03:41 | |
| *** yamamoto_ has quit IRC | 03:42 | |
| *** xarses_ has quit IRC | 03:42 | |
| *** xarses has quit IRC | 03:43 | |
| *** xarses_ has joined #openstack-infra | 03:43 | |
| *** tlian has quit IRC | 03:44 | |
| *** yuanying has joined #openstack-infra | 03:48 | |
| *** Sukhdev has joined #openstack-infra | 03:48 | |
| *** elo has quit IRC | 03:48 | |
| *** elo has joined #openstack-infra | 03:49 | |
| *** kaisers_ has joined #openstack-infra | 03:49 | |
| *** liusheng has quit IRC | 03:52 | |
| *** andreww has joined #openstack-infra | 03:52 | |
| *** liusheng has joined #openstack-infra | 03:52 | |
| *** kaisers_ has quit IRC | 03:53 | |
| ianw | greghaynes: hmm, not sure it matters for these system calls we're patchign out | 03:53 |
| greghaynes | ianw: which part? MagicMock should be avoided like the plague in all cases :) | 03:54 |
| mordred | jhesketh: do you think get_job_log_location itself should be named differently? or just comments? | 03:54 |
| mordred | jhesketh: (I'm having a tough time coming up with a name for that that makes any sense) | 03:54 |
| jhesketh | mordred: it's more a preference thing, but I'd go for something like get_running_server_by_job_uuid | 03:55 |
| *** xarses_ has quit IRC | 03:55 | |
| ianw | greghaynes: i guess i don't get the argument for not magic mock ... are you going to define implementations for all the standard library mocked calls? | 03:58 |
| *** thorst has quit IRC | 03:58 | |
| *** thorst has joined #openstack-infra | 03:58 | |
| greghaynes | ianw: Either it shouldnt matter that youre using mock vs magicmock or youre hitting the trap of relying on calling arbitrary methods silently succeeding | 04:00 |
| greghaynes | ianw: so no, you shouldnt need to define implementations for all of them - the current code doesnt and doesnt use magicmock | 04:01 |
| *** zeih has joined #openstack-infra | 04:02 | |
| greghaynes | ianw: fwiw, theres a ton of fear about using mock in general because of how easy it is to foot-gun due to that | 04:02 |
| *** zhurong has quit IRC | 04:03 | |
| greghaynes | I think infra mostly swears it off | 04:03 |
| ianw | i totally get that mock can be used dangerously. i'm not sure that MagicMock makes anything worse | 04:03 |
| ianw | in fact, if you want an example of how *not* to use mock, this is probably a good one | 04:03 |
| greghaynes | ianw: it absolutely does - you wont know if youre depending on code calling random methods in your mocked object | 04:04 |
| *** zhurong has joined #openstack-infra | 04:04 | |
| greghaynes | as opposed to failing loudly if that happens | 04:04 |
| ianw | to my understanding, MagicMock just implements some of the, well, "magic" methods (__ stuff) | 04:04 |
| ianw | unless you're building objects with a "spec" argument | 04:04 |
| ianw | which i think does what you say and restricts the called methods, etc | 04:05 |
| *** roxanaghe has joined #openstack-infra | 04:05 | |
| *** kaisers_ has joined #openstack-infra | 04:05 | |
| greghaynes | ianw: hrm, let me re-page that in. When I originally wrote this code I had to battle out making it work with mock rather than a magicmock but let me test... | 04:06 |
| *** zeih has quit IRC | 04:06 | |
| mordred | jhesketh: ossum. it also returns a port ... how about get_job_log_stream_address ? | 04:07 |
| *** thorst has quit IRC | 04:07 | |
| ianw | greghaynes: you're probably remembering issues with open() and it being a context manager | 04:09 |
| *** roxanaghe has quit IRC | 04:09 | |
| greghaynes | ianw: ah, yes, its all the magic methods | 04:13 |
| greghaynes | ianw: so ya, context manager is a good example | 04:13 |
| greghaynes | ianw: and yes, a spec would be even better, but less surface area is better than more | 04:13 |
| greghaynes | ianw: now I just think we really really should spec these | 04:14 |
| *** itsuugo has joined #openstack-infra | 04:15 | |
| ianw | greghaynes: tbh, i really don't think it's worth much heroic effort here. if you want to do this properly, then spend the time using the vfs stuff that presents a mocked up filesystem and drop all this mocking together | 04:16 |
| greghaynes | ianw: sure, so to just not regress on Mock vs Magic its just one argument in the decorators though? | 04:16 |
| ianw | it's just tending towards implementing containers in test | 04:16 |
| *** banix has quit IRC | 04:17 | |
| *** roxanaghe has joined #openstack-infra | 04:17 | |
| ianw | greghaynes: i'm really not sure i've ever seen this argument against magicmock, or seen code that avoids it | 04:17 |
| greghaynes | I think in infra mostly mock is just avoided wholesale due to this argument ;) | 04:18 |
| greghaynes | so I really dont want this to be the great example of why thats the right solution | 04:20 |
| ianw | greghaynes: i'm really not sure what the argument is? you want to re-implement the __ methods (probably badly?) | 04:21 |
| greghaynes | ianw: no, they shouldnt be implemented at all, they should raise errors and our mocked objects should only allow being called in the ways we intend | 04:21 |
| ianw | greghaynes: that's not going to happen... | 04:22 |
| openstackgerrit | xiaodongwang991481 proposed openstack-infra/project-config: add networking-sfc gate for functional tests and tempest tests https://review.openstack.org/324135 | 04:22 |
| *** ayoung has quit IRC | 04:22 | |
| *** kdas__ has joined #openstack-infra | 04:22 | |
| ianw | greghaynes: that's why MagicMock implements the functions, so they do something | 04:22 |
| *** gtmanfred has quit IRC | 04:23 | |
| greghaynes | ianw: yes, what I am saying is - theres a huge list of bugs that come from using mocked objects that implement more functions than the objects they are mocking | 04:23 |
| *** erikmwilson has quit IRC | 04:24 | |
| *** dmellado has quit IRC | 04:24 | |
| greghaynes | ianw: the most common being calling .assert_called_once(something) | 04:24 |
| *** cschwede has quit IRC | 04:24 | |
| greghaynes | and thinking youre actually asserting something | 04:24 |
| *** cschwede has joined #openstack-infra | 04:25 | |
| *** erikmwilson has joined #openstack-infra | 04:25 | |
| *** Wei_Liu has quit IRC | 04:25 | |
| *** gtmanfred has joined #openstack-infra | 04:25 | |
| greghaynes | so it seems like an antipattern to use a mock that implements more methods than necessary... | 04:25 |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: [Monasca] Add Tempest Jobs as non-voting for monasca repos https://review.openstack.org/316293 | 04:27 |
| *** dmellado has joined #openstack-infra | 04:27 | |
| *** gildub has quit IRC | 04:27 | |
| *** Wei_Liu has joined #openstack-infra | 04:27 | |
| *** yamamoto_ has joined #openstack-infra | 04:29 | |
| *** csomerville has joined #openstack-infra | 04:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fixes: use six.moves.configparser https://review.openstack.org/323020 | 04:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: WIP: Add web-based console log streaming https://review.openstack.org/320563 | 04:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fix: use proper octal notation https://review.openstack.org/323017 | 04:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fix: use bytesIO instead of cStringIO https://review.openstack.org/323016 | 04:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fixes: use six.moves.urllib.parse instead of urlparse https://review.openstack.org/323019 | 04:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fix: Absolute Imports https://review.openstack.org/323018 | 04:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fix: use six.reraise https://review.openstack.org/323015 | 04:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fixes: Use print() not print https://review.openstack.org/322238 | 04:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fix: use six.moves.urillib https://review.openstack.org/323014 | 04:31 |
| mordred | notmorgan: I rebased your patchset and fixed a minor niggle in the print one that I found in the pep8 tests for the streaming - because I need python working-ish in the tree for the log streaming code | 04:32 |
| *** kdas__ is now known as kushal | 04:33 | |
| *** kushal has joined #openstack-infra | 04:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Refactor side-effect functions https://review.openstack.org/320225 | 04:35 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Patch glean.cmd.open for unit testing https://review.openstack.org/320224 | 04:35 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Use mock patching for most functions https://review.openstack.org/318464 | 04:35 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Add selinux context manager for writing files https://review.openstack.org/304357 | 04:35 |
| *** kzaitsev_mb has quit IRC | 04:36 | |
| ianw | greghaynes: ^ i have never seen any code like that, but there you go... | 04:36 |
| greghaynes | ianw: yea, I think thats a combination of what you said - we probably just shouldnt be mocking, and that mocks are generally not safely used | 04:36 |
| greghaynes | like how we use them :( | 04:36 |
| ianw | greghaynes: can you please take a quick look at dib-run-parts patches https://review.openstack.org/317216 & https://review.openstack.org/314263 | 04:38 |
| ianw | they are well tested, and i'll add a job to run the func-test | 04:39 |
| greghaynes | ianw: sure | 04:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fixes: use six.moves.configparser https://review.openstack.org/323020 | 04:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: WIP: Add web-based console log streaming https://review.openstack.org/320563 | 04:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fix: use proper octal notation https://review.openstack.org/323017 | 04:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fixes: use six.moves.urllib.parse instead of urlparse https://review.openstack.org/323019 | 04:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fix: Absolute Imports https://review.openstack.org/323018 | 04:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Python 3 Fixes: Replace missing builtins https://review.openstack.org/324205 | 04:46 |
| *** psachin has joined #openstack-infra | 04:47 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Cleanup source-repositories output https://review.openstack.org/321961 | 04:47 |
| *** Sukhdev has quit IRC | 04:48 | |
| *** csomerville has quit IRC | 04:48 | |
| *** jamielennox is now known as jamielennox|away | 04:48 | |
| *** mrmartin has joined #openstack-infra | 04:48 | |
| *** gildub has joined #openstack-infra | 04:51 | |
| *** sree has joined #openstack-infra | 04:52 | |
| greghaynes | ianw: its pretty annoying that we dont run dib functests on dib-utils patches | 04:53 |
| ianw | greghaynes: we do? | 04:53 |
| greghaynes | ianw: oh, its just called something wierd | 04:53 |
| greghaynes | ianw: nvm, right | 04:54 |
| ianw | greghaynes: 314263 introduces some tests for drp which are separate, i can add them | 04:54 |
| greghaynes | welp, that went from annoying to awesome | 04:54 |
| jeblair | mordred: i noticed an ansible problem with zl.... | 04:56 |
| jeblair | mordred: well, maybe more than one | 04:56 |
| mordred | jeblair: uhoh | 04:56 |
| mordred | jeblair: it's not fashion problem is it? like, it's wore white after labor day? | 04:57 |
| jeblair | mordred: it's mostly going okay, but i noticed a job that was running too long (~3 hours) | 04:57 |
| mordred | oh good | 04:57 |
| *** sree has quit IRC | 04:57 | |
| mordred | that's just what we've always wanted it to do | 04:57 |
| jeblair | mordred: first -- it looks like the thing the zuul_runner was running finished, but zuul_runner *appeared* to still be waiting for it to exit. i suspect it may have hung indefinitely waiting for the next line to be output on stdout/err or something | 04:58 |
| jeblair | mordred: even though that was the underlying cause, i'm not too worried about that right this second... i can probably dig into that in simulations... | 04:58 |
| mordred | kk | 04:58 |
| jeblair | mordred: the next thing kind of has me stumped | 04:58 |
| mordred | that has me worried | 04:58 |
| jeblair | mordred: zuul_runner is an async task with a timeout (10800 in this case -- or 3 hours) and a 5 second poll interval | 04:59 |
| *** furlongm_ is now known as furlongm | 04:59 | |
| mordred | nod | 04:59 |
| jeblair | mordred: i happened to notice this at about 5-10 minutes shy of 3 hours | 04:59 |
| *** sree has joined #openstack-infra | 05:00 | |
| jeblair | mordred: everything is as expected -- there are ansible procs on zl, and when i log into the host, there's an ansible async task running | 05:00 |
| jeblair | mordred: 3 hour timeout passes, and the ansible async process on the host stops | 05:00 |
| jeblair | (so far so good) | 05:00 |
| mordred | \o/ | 05:00 |
| jeblair | mordred: but i don't see any change on the controller (zl) | 05:00 |
| *** flwang has quit IRC | 05:00 | |
| mordred | hrm. | 05:01 |
| jeblair | mordred: and when i strace the farthest descendent child process related to that | 05:01 |
| jeblair | mordred: i see it wake up every 5 seconds and do something something async something | 05:01 |
| *** Daisy has joined #openstack-infra | 05:01 | |
| jeblair | mordred: i killed it in the hopes of getting more information once it returned (and zuul-launcher logged ansible-playbook stdout/err), but that didn't tell me anything else | 05:02 |
| mordred | it does seem like it's polling every 5 seconds - one would wonder why it's not aborting when the task isn't there | 05:02 |
| jeblair | mordred: yeah, it *seems* like the async timeout didn't do the right thing on the controller end, even though maybe it did on the host end | 05:02 |
| mordred | jeblair: have you seen the async timeout do the right thing in other contexts? | 05:03 |
| mordred | like - have we tested that async timeout does in fact work? | 05:03 |
| *** Daisy_ has joined #openstack-infra | 05:03 | |
| jeblair | mordred: i'm pretty sure i tested this more or less when i wrote the timeout callback | 05:03 |
| mordred | darn. I was hoping you just hadn't bothered, becaues otherwise it sounds like a thing that might be hard to reproduce | 05:04 |
| jeblair | mordred: (the callback is used to subtract elapsed time from total playbook timeout after each task -- so it's just doing math, not actually enforcing timeouts or anything hairy) | 05:04 |
| *** jamielennox|away is now known as jamielennox | 05:04 | |
| jeblair | mordred: it was several weeks ago :) | 05:04 |
| *** amitgandhinz has joined #openstack-infra | 05:04 | |
| *** thorst has joined #openstack-infra | 05:05 | |
| *** Daisy has quit IRC | 05:05 | |
| jeblair | mordred: it's my bedtime now; i can try to reproduce some or all of this tomorrow (should be doable without zuul-launcher -- just a playbook with the callback and zuul_runner), but if this sounds interesting, i'd appreciate any help (either in understanding it or ideas on how to debug) | 05:07 |
| mordred | jeblair: cool. I'll try to poke at it some - although today is a conference day so I may or may not get very far | 05:07 |
| * jeblair dreams of ansible fairies | 05:07 | |
| mordred | mmmm. fairies | 05:08 |
| *** Daisy_ has quit IRC | 05:08 | |
| *** amitgandhinz has quit IRC | 05:09 | |
| *** kushal has quit IRC | 05:09 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add get_runs_counts_by_run_metadata() https://review.openstack.org/323231 | 05:10 |
| *** thorst has quit IRC | 05:12 | |
| *** jtomasek has joined #openstack-infra | 05:16 | |
| *** amotoki_ is now known as amotoki | 05:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add dib-utils func test https://review.openstack.org/324213 | 05:19 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix variable unbound error while REPOREF="*" https://review.openstack.org/320859 | 05:20 |
| *** qb_ has joined #openstack-infra | 05:20 | |
| jhesketh | mordred: sure, that works for me | 05:21 |
| mordred | jhesketh: woot. I updated the WIP patch based on your feedback with either fixes or todos - thanks for the eyes! | 05:23 |
| jhesketh | no worries, I'll take another look | 05:24 |
| *** maxskew has joined #openstack-infra | 05:24 | |
| *** abregman has joined #openstack-infra | 05:25 | |
| *** Wei_Liu has quit IRC | 05:26 | |
| *** Wei_Liu has joined #openstack-infra | 05:26 | |
| openstackgerrit | Merged openstack-infra/shade: Add magnum services call to shade https://review.openstack.org/313583 | 05:30 |
| *** hdaniel has joined #openstack-infra | 05:30 | |
| *** zeih has joined #openstack-infra | 05:30 | |
| *** maxskew has quit IRC | 05:34 | |
| *** zeih has quit IRC | 05:35 | |
| *** maxskew has joined #openstack-infra | 05:35 | |
| *** bhavik has joined #openstack-infra | 05:38 | |
| *** roxanaghe has quit IRC | 05:38 | |
| *** roxanaghe has joined #openstack-infra | 05:38 | |
| *** bhavik has quit IRC | 05:41 | |
| *** abregman has quit IRC | 05:41 | |
| *** bhavik has joined #openstack-infra | 05:42 | |
| *** roxanaghe has quit IRC | 05:43 | |
| *** Daisy has joined #openstack-infra | 05:47 | |
| *** mrmartin has quit IRC | 05:47 | |
| *** rcernin has joined #openstack-infra | 05:48 | |
| *** Daisy has quit IRC | 05:51 | |
| *** Daisy has joined #openstack-infra | 05:52 | |
| *** Daisy_ has joined #openstack-infra | 05:53 | |
| *** Daisy has quit IRC | 05:53 | |
| *** bhavik has quit IRC | 05:54 | |
| *** bhavik has joined #openstack-infra | 05:55 | |
| *** Daisy_ has quit IRC | 05:55 | |
| *** Daisy has joined #openstack-infra | 05:55 | |
| *** Daisy has quit IRC | 05:56 | |
| *** Daisy has joined #openstack-infra | 05:56 | |
| *** savihou has joined #openstack-infra | 05:56 | |
| *** zeih has joined #openstack-infra | 05:57 | |
| *** abregman has joined #openstack-infra | 05:58 | |
| *** greghaynes has quit IRC | 05:58 | |
| *** infra-red has joined #openstack-infra | 05:58 | |
| *** flwang has joined #openstack-infra | 05:58 | |
| *** Daisy has quit IRC | 05:59 | |
| *** Daisy has joined #openstack-infra | 05:59 | |
| *** ggillies has quit IRC | 05:59 | |
| *** aeng has quit IRC | 05:59 | |
| *** camunoz has quit IRC | 05:59 | |
| *** camunoz has joined #openstack-infra | 06:00 | |
| *** sflanigan has quit IRC | 06:00 | |
| *** sflanigan has joined #openstack-infra | 06:00 | |
| *** sflanigan has joined #openstack-infra | 06:00 | |
| *** jamielennox is now known as jamielennox|away | 06:01 | |
| *** ggillies has joined #openstack-infra | 06:01 | |
| *** zeih has quit IRC | 06:02 | |
| *** Daisy has quit IRC | 06:04 | |
| *** Daisy has joined #openstack-infra | 06:04 | |
| *** infra-red has quit IRC | 06:04 | |
| *** flwang has quit IRC | 06:05 | |
| *** amitgandhinz has joined #openstack-infra | 06:05 | |
| *** infra-red has joined #openstack-infra | 06:05 | |
| *** greghaynes has joined #openstack-infra | 06:09 | |
| *** thorst has joined #openstack-infra | 06:10 | |
| *** amitgandhinz has quit IRC | 06:10 | |
| *** aeng has joined #openstack-infra | 06:13 | |
| yolanda | good morning | 06:14 |
| *** e0ne has joined #openstack-infra | 06:15 | |
| *** abregman has quit IRC | 06:15 | |
| *** zhurong has quit IRC | 06:16 | |
| *** jamielennox|away is now known as jamielennox | 06:16 | |
| *** thorst has quit IRC | 06:17 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add bay certificates to shade https://review.openstack.org/313308 | 06:19 |
| *** zhurong has joined #openstack-infra | 06:19 | |
| *** zul has quit IRC | 06:21 | |
| *** hdaniel has quit IRC | 06:22 | |
| *** nadya has joined #openstack-infra | 06:28 | |
| *** andreas_s has joined #openstack-infra | 06:30 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:32 | |
| *** mikelk has joined #openstack-infra | 06:34 | |
| *** zul has joined #openstack-infra | 06:36 | |
| *** kzaitsev_mb has quit IRC | 06:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Be more specific in Puppet OpenStack Zuul layout https://review.openstack.org/323870 | 06:38 |
| notmorgan | mordred: cool | 06:38 |
| notmorgan | mordred: thanks! | 06:38 |
| *** e0ne has quit IRC | 06:38 | |
| notmorgan | mordred: also... Oh hai, hows the beach? | 06:38 |
| *** roxanaghe has joined #openstack-infra | 06:39 | |
| *** esikachev has joined #openstack-infra | 06:43 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add get_runs_counts_by_run_metadata() https://review.openstack.org/323231 | 06:43 |
| *** roxanaghe has quit IRC | 06:43 | |
| *** infra-red has quit IRC | 06:44 | |
| senk_ | Anyone able to review this? https://review.openstack.org/310823 | 06:45 |
| *** senk_ is now known as senk | 06:46 | |
| mordred | notmorgan: it's quite lovely! we did some kayaking at the speaker's dinner last night | 06:47 |
| *** nadya has quit IRC | 06:47 | |
| mordred | morning yolanda | 06:47 |
| *** aeng has quit IRC | 06:48 | |
| mordred | yolanda, notmorgan: if you get a sec: https://review.openstack.org/#/c/323703/ passed tests | 06:49 |
| *** _vs has joined #openstack-infra | 06:50 | |
| yolanda | hi mordred , good morning | 06:51 |
| yolanda | looking | 06:51 |
| notmorgan | mordred: looking | 06:51 |
| notmorgan | mordred: /jealous of kayaking | 06:52 |
| notmorgan | mordred: but i did have whiskey with devananda tonight, which was good. | 06:52 |
| yolanda | mordred, i added a comment on the README, what do you think? | 06:53 |
| *** tesseract- has joined #openstack-infra | 06:53 | |
| *** arxcruz has joined #openstack-infra | 06:54 | |
| openstackgerrit | Merged openstack-infra/gear: Add py34 to tox.ini envlist https://review.openstack.org/181097 | 06:54 |
| senk | good morning everyone o/ | 06:55 |
| openstackgerrit | Merged openstack-infra/gear: Remove shebang as the script is managed by an entry_point https://review.openstack.org/186889 | 06:55 |
| senk | mordred: good morning, can you please look at this one https://review.openstack.org/#/c/310823/ it's quite a time open now | 06:56 |
| *** sdake has joined #openstack-infra | 06:57 | |
| *** jlanoux has joined #openstack-infra | 06:57 | |
| openstackgerrit | Merged openstack-infra/gear: Remove argparse from requirements https://review.openstack.org/270412 | 06:58 |
| *** _vs has quit IRC | 06:59 | |
| notmorgan | yolanda, mordred: I also added a comment on the readme | 07:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/gear: Bump version requirement for pbr https://review.openstack.org/301222 | 07:01 |
| openstackgerrit | Merged openstack-infra/gear: Add a test for worker termination https://review.openstack.org/252555 | 07:02 |
| *** jaosorior has joined #openstack-infra | 07:02 | |
| *** Daisy has quit IRC | 07:02 | |
| *** Daisy has joined #openstack-infra | 07:03 | |
| *** flwang has joined #openstack-infra | 07:03 | |
| *** _vsaienko has joined #openstack-infra | 07:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/gear: Switch to six for configparser https://review.openstack.org/280876 | 07:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/gear: Remove py26 from tox.ini https://review.openstack.org/324252 | 07:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/gear: Remove vestigal hook entry from setup.cfg https://review.openstack.org/324253 | 07:04 |
| mordred | notmorgan, yolanda I now go look | 07:04 |
| *** sputnik13 has joined #openstack-infra | 07:05 | |
| *** dizquierdo_afk has joined #openstack-infra | 07:05 | |
| *** amitgandhinz has joined #openstack-infra | 07:06 | |
| *** dizquierdo_afk is now known as dizquierdo | 07:06 | |
| *** Daisy has quit IRC | 07:07 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Reword the entries in the README a bit https://review.openstack.org/323704 | 07:07 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Add shade constructor helper method https://review.openstack.org/323703 | 07:07 |
| *** flwang has quit IRC | 07:07 | |
| mordred | senk: looks great | 07:08 |
| mordred | notmorgan, yolanda how's that? | 07:08 |
| *** kaisers_ has quit IRC | 07:08 | |
| yolanda | ++ | 07:08 |
| *** sputnik13 has quit IRC | 07:09 | |
| *** flepied has quit IRC | 07:09 | |
| senk | mordred: thanks :) | 07:09 |
| *** camunoz has quit IRC | 07:09 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Added missing Characters indicating placeholder https://review.openstack.org/310823 | 07:10 |
| mordred | senk: than you! | 07:11 |
| *** hdaniel has joined #openstack-infra | 07:11 | |
| *** amitgandhinz has quit IRC | 07:11 | |
| *** sputnik13 has joined #openstack-infra | 07:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/gear: Bump version requirement for pbr https://review.openstack.org/301222 | 07:12 |
| mordred | yolanda: I was clearing through gear patches - there are 4 trivial patches here: https://review.openstack.org/#/q/status:open+project:openstack-infra/gear+branch:master+topic:cleanup | 07:13 |
| yolanda | ok i don't tend to review gear because i'm not very familiar, but it they are trivial i'll take a look | 07:14 |
| *** esikachev has quit IRC | 07:15 | |
| mordred | yolanda: yah - they're all not real gear things | 07:15 |
| *** thorst has joined #openstack-infra | 07:15 | |
| *** _vsaienko has quit IRC | 07:15 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: [translations] Fix path for firstapp https://review.openstack.org/324259 | 07:17 |
| *** _vs has joined #openstack-infra | 07:17 | |
| openstackgerrit | Merged openstack-infra/gear: Switch to six for configparser https://review.openstack.org/280876 | 07:19 |
| *** Wei_Liu has quit IRC | 07:20 | |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add network quotas support https://review.openstack.org/319144 | 07:20 |
| *** hashar has joined #openstack-infra | 07:21 | |
| openstackgerrit | John Schwarz proposed openstack-infra/project-config: Fix neutron-scenario branch criteria https://review.openstack.org/324263 | 07:22 |
| *** thorst has quit IRC | 07:22 | |
| *** abregman has joined #openstack-infra | 07:23 | |
| *** yaume has joined #openstack-infra | 07:24 | |
| *** jlanoux has quit IRC | 07:25 | |
| *** vincentll has joined #openstack-infra | 07:25 | |
| *** infra-red has joined #openstack-infra | 07:26 | |
| *** _vs has quit IRC | 07:26 | |
| *** nbloom has joined #openstack-infra | 07:27 | |
| *** fbo has joined #openstack-infra | 07:27 | |
| *** vikram_ has joined #openstack-infra | 07:28 | |
| nbloom | Hi all, we've been having problem lately with python34 jobs, functional.api.test_versions.TestVersions and functional.api.v1.test_authentication.TestAuthentications fail due to "No module named 'Vine'", see http://logs.openstack.org/38/322738/5/check/gate-vitrage-python34/d493fd6/testr_results.html.gz. Does anyone know the reason for it and how it | 07:29 |
| nbloom | can be fixed? | 07:29 |
| *** samuelBartel has joined #openstack-infra | 07:31 | |
| openstackgerrit | Merged openstack/os-client-config: Add shade constructor helper method https://review.openstack.org/323703 | 07:31 |
| *** savihou has quit IRC | 07:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:34 | |
| openstackgerrit | Merged openstack/os-client-config: Reword the entries in the README a bit https://review.openstack.org/323704 | 07:34 |
| *** ramishra_ has joined #openstack-infra | 07:34 | |
| *** zeih has joined #openstack-infra | 07:35 | |
| *** ramishra has quit IRC | 07:36 | |
| *** Daisy has joined #openstack-infra | 07:36 | |
| *** ramishra_ has quit IRC | 07:36 | |
| *** Daisy has quit IRC | 07:36 | |
| *** ramishra has joined #openstack-infra | 07:37 | |
| *** jtomasek_ has joined #openstack-infra | 07:37 | |
| *** Daisy has joined #openstack-infra | 07:38 | |
| *** kzaitsev_mb has quit IRC | 07:38 | |
| *** Wei_Liu has joined #openstack-infra | 07:39 | |
| *** Daisy_ has joined #openstack-infra | 07:39 | |
| *** sdake has quit IRC | 07:39 | |
| *** hashar has quit IRC | 07:39 | |
| *** hashar has joined #openstack-infra | 07:39 | |
| *** Daisy_ has quit IRC | 07:39 | |
| *** Daisy_ has joined #openstack-infra | 07:40 | |
| *** roxanaghe has joined #openstack-infra | 07:40 | |
| *** amoralej|off is now known as amoralej | 07:42 | |
| *** jaosorior has quit IRC | 07:42 | |
| *** Daisy has quit IRC | 07:42 | |
| *** jtomasek_ has quit IRC | 07:44 | |
| *** roxanaghe has quit IRC | 07:44 | |
| mordred | yolanda: do you happen to remember where the etherpad is for the project renames for tomorrow? | 07:45 |
| mordred | yolanda: and/or if there is a topic I'm supposed to use? | 07:46 |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Create separate ACL groups for murano-apps project https://review.openstack.org/323340 | 07:46 |
| *** flwang has joined #openstack-infra | 07:48 | |
| *** flepied has joined #openstack-infra | 07:49 | |
| yolanda | mordred, sorry, i haven't followed that closely. Have you checked on the meeting logs? | 07:49 |
| *** jlanoux has joined #openstack-infra | 07:49 | |
| mordred | yah - just found it | 07:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Rename ansible-puppet to ansible-role-puppet https://review.openstack.org/324273 | 07:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Update .gitreview to ansible-role-puppet https://review.openstack.org/324275 | 07:51 |
| *** kzaitsev_mb has joined #openstack-infra | 07:51 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Consume renamed ansible-role-puppet https://review.openstack.org/324276 | 07:51 |
| *** sflanigan has quit IRC | 07:55 | |
| *** flwang has quit IRC | 07:55 | |
| *** jaosorior has joined #openstack-infra | 07:56 | |
| *** vgridnev has joined #openstack-infra | 07:57 | |
| *** dtantsur|afk is now known as dtantsur | 07:58 | |
| *** matrohon has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zeih has quit IRC | 08:01 | |
| *** claudiub has joined #openstack-infra | 08:02 | |
| *** zzzeek has joined #openstack-infra | 08:02 | |
| *** esikachev has joined #openstack-infra | 08:04 | |
| mordred | fungi: whence you awaken, I'd love to get https://review.openstack.org/#/c/324253 landed and then get a new gear release cut - there is a fix in it (https://review.openstack.org/#/c/322352/) that's needed for the python3 support work in zuul, and that work has become needed by the web-console-streaming work | 08:05 |
| mordred | fungi: best I can tell, there are not any other particularly interesting/dangerous changes - however, requirements do change a smidge, so it seems like 0.6.0 might be appropriate? | 08:06 |
| *** amitgandhinz has joined #openstack-infra | 08:07 | |
| *** shashank_hegde has quit IRC | 08:07 | |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config: [magnum] Fix job for magnum in rally https://review.openstack.org/324283 | 08:08 |
| strigazi | jhesketh: ^^ | 08:08 |
| strigazi | yolanda: ^^ | 08:09 |
| openstackgerrit | Merged openstack-infra/project-config: Use ES2.x for searchlight tests https://review.openstack.org/317000 | 08:10 |
| yolanda | hi strigazi , ah i see that neutron was needed. But were you able to sort the problems with zuul-cloner and magnum? | 08:10 |
| strigazi | hi yolanda | 08:10 |
| strigazi | It looks like the fix didn't take effect immediately | 08:11 |
| *** sputnik13 has quit IRC | 08:11 | |
| yolanda | ah, so the macro was not properly updated on all jenkins masters? | 08:11 |
| strigazi | I guess | 08:11 |
| *** amitgandhinz has quit IRC | 08:12 | |
| strigazi | but after two rechecks it worked | 08:12 |
| *** sputnik13 has joined #openstack-infra | 08:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Add ironic-staging-drivers-dsvm job https://review.openstack.org/299318 | 08:12 |
| *** e0ne has joined #openstack-infra | 08:13 | |
| *** jaosorior has quit IRC | 08:14 | |
| strigazi | yolanda before merging this, I have a question | 08:14 |
| yolanda | tell me | 08:14 |
| strigazi | In magnum's devstack settings many services are enabled: https://github.com/openstack/magnum/blob/master/devstack/settings | 08:15 |
| *** jaosorior has joined #openstack-infra | 08:15 | |
| strigazi | the same services are enabled in jenkins/jobs/rally.yaml | 08:15 |
| *** ifarkas has joined #openstack-infra | 08:15 | |
| strigazi | Is this going to create a problem? | 08:15 |
| yolanda | strigazi, are they conflicting in some way, or magnum services are overriding the rally ones? | 08:16 |
| strigazi | I mean heat is enabled in the job and in the and in magnum/devstack/settings | 08:17 |
| yolanda | strigazi, not 100% sure about it, but should be fine, as long as there is no conflict between services you don't want to enable | 08:18 |
| strigazi | yoalnda: I think there is no such conflict | 08:18 |
| *** chem has joined #openstack-infra | 08:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Publish python-swiftclient docs on every patch https://review.openstack.org/323501 | 08:20 |
| openstackgerrit | Merged openstack-infra/project-config: Adding new repository trove-image-builder https://review.openstack.org/312805 | 08:20 |
| openstackgerrit | Merged openstack-infra/project-config: Add octavia periodic jobs with oslo master https://review.openstack.org/320151 | 08:20 |
| *** thorst has joined #openstack-infra | 08:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Add check-requirements and publish-to-pypi to renderspec/pymod2pkg https://review.openstack.org/321473 | 08:20 |
| *** ihrachys has joined #openstack-infra | 08:21 | |
| strigazi | yolanda: it there a way to tests this kind of changes locally? | 08:21 |
| *** sputnik13_ has joined #openstack-infra | 08:22 | |
| yolanda | strigazi, there is a reproduce.sh script , generated in the logs of the job, that you can reuse in a local vm | 08:22 |
| *** yaume has quit IRC | 08:22 | |
| *** BobBall_AWOL is now known as BobBall | 08:23 | |
| *** yaume has joined #openstack-infra | 08:23 | |
| *** sputnik13 has quit IRC | 08:23 | |
| *** Daisy_ has quit IRC | 08:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Retire monitoring-for-openstack https://review.openstack.org/323751 | 08:24 |
| *** Daisy has joined #openstack-infra | 08:24 | |
| strigazi | yolanda: I mean before pushing changes, not after a gate job has failed | 08:24 |
| yolanda | strigazi, you could reuse the reproduce.sh generated, and tweak with your new settings | 08:25 |
| *** prometheanfire has quit IRC | 08:25 | |
| strigazi | yolanda: ok, thanks | 08:26 |
| *** thorst has quit IRC | 08:27 | |
| *** mrmartin has joined #openstack-infra | 08:28 | |
| vsaienko | hello team, please help to merge 1 line patch https://review.openstack.org/#/c/315496 it allows to pass provision/unprovision timeout to ironic grenade job | 08:28 |
| *** Daisy has quit IRC | 08:29 | |
| *** eezhova has joined #openstack-infra | 08:29 | |
| *** markvoelker has quit IRC | 08:29 | |
| *** markvoelker has joined #openstack-infra | 08:30 | |
| *** eezhova has quit IRC | 08:30 | |
| *** prometheanfire has joined #openstack-infra | 08:30 | |
| *** mrmartin has quit IRC | 08:33 | |
| *** sputnik13_ has quit IRC | 08:35 | |
| *** Daisy has joined #openstack-infra | 08:37 | |
| *** sputnik13 has joined #openstack-infra | 08:38 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Only publish latest image to magnum https://review.openstack.org/324294 | 08:38 |
| strigazi | jhesketh: are you there? | 08:38 |
| yolanda | mordred, ^ | 08:39 |
| *** zeih has joined #openstack-infra | 08:40 | |
| mordred | yolanda: lgtm | 08:42 |
| yolanda | thanks for the ultra-fast reviews :) | 08:43 |
| *** mrmartin has joined #openstack-infra | 08:43 | |
| strigazi | @mordred: can you also have a look at this: https://review.openstack.org/#/c/324283/ | 08:43 |
| *** psachin has quit IRC | 08:45 | |
| *** prometheanfire has quit IRC | 08:46 | |
| *** vdrok has joined #openstack-infra | 08:47 | |
| *** zeih has quit IRC | 08:52 | |
| *** sputnik13 has quit IRC | 08:53 | |
| *** kaisers_ has joined #openstack-infra | 08:56 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet/beaker/experimental: export puppet version on Xenial https://review.openstack.org/324111 | 08:58 |
| *** dtantsur is now known as dtantsur|bbl | 09:00 | |
| *** psachin has joined #openstack-infra | 09:01 | |
| *** psachin has quit IRC | 09:01 | |
| *** psachin has joined #openstack-infra | 09:01 | |
| *** kaisers_ has quit IRC | 09:01 | |
| *** markusry has joined #openstack-infra | 09:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Only publish latest image to magnum https://review.openstack.org/324294 | 09:05 |
| *** nijaba has quit IRC | 09:05 | |
| *** sputnik13 has joined #openstack-infra | 09:07 | |
| *** dizquierdo has quit IRC | 09:07 | |
| *** amitgandhinz has joined #openstack-infra | 09:07 | |
| *** dtardivel has joined #openstack-infra | 09:08 | |
| jhesketh | strigazi: hello | 09:09 |
| jhesketh | strigazi: if it was 324283 you were asking about it lgtm | 09:09 |
| strigazi | jhesketh: yeah, thanks | 09:10 |
| *** pbourke_ has left #openstack-infra | 09:10 | |
| *** nijaba has joined #openstack-infra | 09:11 | |
| *** nijaba has joined #openstack-infra | 09:11 | |
| *** amitgandhinz has quit IRC | 09:12 | |
| *** hughhalf has quit IRC | 09:12 | |
| *** sdake has joined #openstack-infra | 09:13 | |
| *** yamahata has quit IRC | 09:14 | |
| *** sputnik13_ has joined #openstack-infra | 09:14 | |
| *** sputnik13_ has quit IRC | 09:16 | |
| openstackgerrit | Merged openstack-infra/project-config: Improve skipped jobs for Heat gate https://review.openstack.org/319898 | 09:17 |
| openstackgerrit | Merged openstack-infra/project-config: [magnum] Fix job for magnum in rally https://review.openstack.org/324283 | 09:23 |
| *** sputnik13 has quit IRC | 09:24 | |
| *** thorst has joined #openstack-infra | 09:25 | |
| *** flwang has joined #openstack-infra | 09:29 | |
| *** thorst has quit IRC | 09:32 | |
| *** mrmartin has quit IRC | 09:35 | |
| *** _amrith_ is now known as amrith | 09:37 | |
| *** kzaitsev_mb has quit IRC | 09:37 | |
| *** haaj has joined #openstack-infra | 09:40 | |
| *** sdake has quit IRC | 09:40 | |
| *** e0ne has quit IRC | 09:41 | |
| *** e0ne has joined #openstack-infra | 09:41 | |
| *** roxanaghe has joined #openstack-infra | 09:42 | |
| *** Daisy has quit IRC | 09:42 | |
| *** Daisy has joined #openstack-infra | 09:42 | |
| *** Daisy_ has joined #openstack-infra | 09:45 | |
| *** Daisy_ has quit IRC | 09:46 | |
| *** roxanaghe has quit IRC | 09:46 | |
| *** Daisy_ has joined #openstack-infra | 09:46 | |
| *** Daisy has quit IRC | 09:47 | |
| *** hughhalf has joined #openstack-infra | 09:48 | |
| rcarrillocruz | mordred: heya, let me know what you think about https://review.openstack.org/#/c/322823/ | 09:50 |
| rcarrillocruz | i have still to put the same mechanism for per-profile servers, but you can get the idea... | 09:50 |
| *** Daisy_ has quit IRC | 09:51 | |
| *** zeih has joined #openstack-infra | 09:51 | |
| *** ociuhandu has joined #openstack-infra | 09:51 | |
| *** sambetts|afk is now known as sambetts | 09:51 | |
| strigazi | jhesketh yolanda: I tested the reproduce script in a vm and seems to work. Hasn't finished yet. BUT | 09:54 |
| strigazi | jhesketh yolanda: to get it working I installed some dependencies (for cffi libs) before | 09:54 |
| yolanda | strigazi, yes, reproduce script needs some previous deps | 09:55 |
| *** zeih has quit IRC | 09:56 | |
| strigazi | yolanda: so it won't be a problem when jenkins runs it | 09:57 |
| yolanda | it will be fine | 09:57 |
| *** Qiming has quit IRC | 09:59 | |
| *** amotoki has quit IRC | 10:01 | |
| *** dtantsur|bbl is now known as dtantsur | 10:02 | |
| *** rhallisey has joined #openstack-infra | 10:02 | |
| *** amotoki has joined #openstack-infra | 10:04 | |
| *** derekh has joined #openstack-infra | 10:05 | |
| *** chem has quit IRC | 10:06 | |
| *** amitgandhinz has joined #openstack-infra | 10:08 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Update gate-ironic-staging-drivers-dsvm-nv job https://review.openstack.org/324340 | 10:08 |
| *** zhurong has quit IRC | 10:10 | |
| *** amitgandhinz has quit IRC | 10:13 | |
| *** ociuhandu has quit IRC | 10:16 | |
| *** sputnik13 has joined #openstack-infra | 10:16 | |
| *** sree has quit IRC | 10:20 | |
| *** rhallisey has quit IRC | 10:21 | |
| *** sputnik13 has quit IRC | 10:23 | |
| *** nadya has joined #openstack-infra | 10:27 | |
| *** thorst has joined #openstack-infra | 10:30 | |
| *** ociuhandu has joined #openstack-infra | 10:36 | |
| *** electrofelix has joined #openstack-infra | 10:36 | |
| *** thorst has quit IRC | 10:37 | |
| *** haaj has quit IRC | 10:37 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:37 | |
| *** amotoki has quit IRC | 10:37 | |
| *** haaj has joined #openstack-infra | 10:37 | |
| *** amotoki has joined #openstack-infra | 10:39 | |
| *** rodrigods has quit IRC | 10:39 | |
| *** rodrigods has joined #openstack-infra | 10:40 | |
| *** amrith is now known as _amrith_ | 10:40 | |
| vsaienko | please review https://review.openstack.org/#/c/324340/, the patch adds ironic ironic-staging-drivers to $PROJECTS on ironic-staging-drivers-dsvm job | 10:41 |
| *** haaj has quit IRC | 10:42 | |
| *** roxanaghe has joined #openstack-infra | 10:42 | |
| *** bhavik has quit IRC | 10:46 | |
| *** sdague has joined #openstack-infra | 10:46 | |
| *** kaisers_ has joined #openstack-infra | 10:46 | |
| *** roxanaghe has quit IRC | 10:47 | |
| *** trown|outtypewww is now known as trown | 10:51 | |
| *** kaisers_ has quit IRC | 10:51 | |
| *** yanyanhu has quit IRC | 10:51 | |
| *** ikhudoshyn has quit IRC | 10:55 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Update gate-ironic-staging-drivers-dsvm-nv job https://review.openstack.org/324340 | 10:55 |
| *** lezbar has quit IRC | 10:56 | |
| *** lezbar has joined #openstack-infra | 10:56 | |
| *** Qiming has joined #openstack-infra | 10:56 | |
| *** weshay has quit IRC | 10:57 | |
| *** zeih has joined #openstack-infra | 10:59 | |
| *** oanson has joined #openstack-infra | 10:59 | |
| *** rhallisey has joined #openstack-infra | 11:00 | |
| *** trown is now known as trown|outtypewww | 11:00 | |
| *** dizquierdo has joined #openstack-infra | 11:00 | |
| *** sputnik13 has joined #openstack-infra | 11:03 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Report job status 'QUEUED' for queued jobs https://review.openstack.org/272967 | 11:04 |
| *** nadya has quit IRC | 11:05 | |
| *** psachin has quit IRC | 11:06 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Create Merger Git repos using connection names https://review.openstack.org/282660 | 11:07 |
| *** amotoki has quit IRC | 11:07 | |
| *** sputnik13 has quit IRC | 11:08 | |
| *** amitgandhinz has joined #openstack-infra | 11:09 | |
| *** amitgandhinz has quit IRC | 11:13 | |
| *** dizquierdo has quit IRC | 11:14 | |
| *** sputnik13 has joined #openstack-infra | 11:14 | |
| *** amotoki has joined #openstack-infra | 11:15 | |
| *** vdrok has quit IRC | 11:19 | |
| *** yujunz has joined #openstack-infra | 11:20 | |
| *** jlanoux has quit IRC | 11:22 | |
| *** sputnik13 has quit IRC | 11:22 | |
| *** yujunz has quit IRC | 11:23 | |
| *** esikachev has quit IRC | 11:23 | |
| *** banix has joined #openstack-infra | 11:24 | |
| *** ccamacho has joined #openstack-infra | 11:25 | |
| *** thorst has joined #openstack-infra | 11:26 | |
| senk | nibalizer: Hey! | 11:28 |
| *** haaj has joined #openstack-infra | 11:29 | |
| senk | nibalizer: do you have plans to add content to your last talk "better together ansible and puppet" cuz http://docs.openstack.org/infra/publications/ first link is broken | 11:29 |
| *** julim has joined #openstack-infra | 11:31 | |
| *** deadnull_ has joined #openstack-infra | 11:31 | |
| *** weshay has joined #openstack-infra | 11:32 | |
| strigazi | yolanda: there is something wrong with jenkins. I got the same error as yesterday | 11:33 |
| yolanda | strigazi, so you saw that running successfully, and then failing again? | 11:33 |
| strigazi | yolanda: yes | 11:34 |
| strigazi | yolanda: senlin should not be there | 11:34 |
| yolanda | can you check the logs, to see if the failed/successful jobs are on different jenkins masters? | 11:34 |
| *** esikachev has joined #openstack-infra | 11:34 | |
| strigazi | where I can see on which master it run? | 11:35 |
| *** ldnunes has joined #openstack-infra | 11:36 | |
| rcarrillocruz | at the top of the log, it should print which master it ran on | 11:37 |
| strigazi | rcarrillocruz: thanks saw it | 11:37 |
| *** maxskew has quit IRC | 11:38 | |
| strigazi | Today failed on ubuntu-trusty-ovh-gra1-1363673 | 11:38 |
| strigazi | Yesterday was ubuntu-trusty-rax-ord-1352277 | 11:39 |
| yolanda | strigazi, can you paste the logfile? | 11:39 |
| yolanda | i think jjb may not have run properly on all jenkins masters | 11:39 |
| *** lezbar has quit IRC | 11:39 | |
| strigazi | today: http://logs.openstack.org/46/309546/6/check/gate-rally-dsvm-magnum-rally-nv/2072e62/console.html | 11:40 |
| strigazi | yesterday: http://logs.openstack.org/46/309546/6/check/gate-rally-dsvm-magnum-rally-nv/b7b9508/console.html | 11:40 |
| *** mikelk has quit IRC | 11:40 | |
| *** lezbar has joined #openstack-infra | 11:40 | |
| *** haaj has quit IRC | 11:41 | |
| *** mikelk has joined #openstack-infra | 11:41 | |
| strigazi | the masters are jenkins03 and jenkins06 ? | 11:42 |
| *** banix has quit IRC | 11:42 | |
| *** ifarkas has quit IRC | 11:42 | |
| *** jlanoux has joined #openstack-infra | 11:42 | |
| *** ifarkas has joined #openstack-infra | 11:42 | |
| *** baoli has joined #openstack-infra | 11:43 | |
| *** roxanaghe has joined #openstack-infra | 11:43 | |
| *** baoli has quit IRC | 11:44 | |
| *** baoli has joined #openstack-infra | 11:44 | |
| *** zhurong has joined #openstack-infra | 11:45 | |
| *** baoli_ has joined #openstack-infra | 11:46 | |
| *** Daisy has joined #openstack-infra | 11:47 | |
| *** roxanaghe has quit IRC | 11:48 | |
| *** baoli has quit IRC | 11:49 | |
| *** mikelk has quit IRC | 11:49 | |
| openstackgerrit | Merged openstack-infra/zuul: Pass OS_*_CAPTURE env variables https://review.openstack.org/324165 | 11:49 |
| yolanda | strigazi, which job was the successful? i see failures on all | 11:51 |
| strigazi | yolanda: This job http://logs.openstack.org/46/309546/6/check/gate-rally-dsvm-magnum-rally-nv/b7b9508/console.html included magnum as it was suppose to do | 11:52 |
| *** Daisy has quit IRC | 11:52 | |
| *** cdent has joined #openstack-infra | 11:52 | |
| *** pradk has joined #openstack-infra | 11:52 | |
| strigazi | today's job didn't include magnum in env var PROJECTS | 11:52 |
| *** _amrith_ is now known as amrith | 11:52 | |
| *** yaume has quit IRC | 11:52 | |
| *** esikachev has quit IRC | 11:53 | |
| fungi | mordred: seems fine, though jeblair has been the one to tag almost all prior gear releases so i was going to let him do that again unless he wants me to take care of it | 11:53 |
| yolanda | strigazi, i see a finished: failure on that | 11:53 |
| *** raildo-afk is now known as raildo | 11:53 | |
| strigazi | yolonda: yesterday's job failed because of barbican which I remmoved today | 11:54 |
| yolanda | anyway, going to check if jjb has run properly on masters, it is likely to be the problem | 11:54 |
| strigazi | yolanda: Both jobs failed, yesterday and today. BUT, today failed for the wrong reason :) | 11:55 |
| *** furlongm has quit IRC | 11:56 | |
| *** furlongm has joined #openstack-infra | 11:56 | |
| *** yujunz has joined #openstack-infra | 11:56 | |
| yolanda | strigazi, ok, i saw in devstacklog now. i'm checking jjb runs on masters, to ensure that everything was updated | 11:57 |
| *** dprince has joined #openstack-infra | 12:02 | |
| *** bhavik has joined #openstack-infra | 12:03 | |
| *** yaume has joined #openstack-infra | 12:05 | |
| *** yaume is now known as yaume[away] | 12:05 | |
| *** psilvad has joined #openstack-infra | 12:06 | |
| *** amitgandhinz has joined #openstack-infra | 12:10 | |
| *** dtardivel has quit IRC | 12:10 | |
| *** trown|outtypewww is now known as trown | 12:10 | |
| *** amotoki has quit IRC | 12:11 | |
| *** psilvad has quit IRC | 12:11 | |
| *** psachin has joined #openstack-infra | 12:11 | |
| *** kzaitsev_mb has quit IRC | 12:13 | |
| *** moravec has quit IRC | 12:14 | |
| *** aysyd has joined #openstack-infra | 12:14 | |
| *** amitgandhinz has quit IRC | 12:14 | |
| yolanda | infra-root, seeing that when running jjb: You have a macro ('tox') defined for 'builder' component type that is masking an inbuilt definition | 12:16 |
| yolanda | shall we worry about it? | 12:16 |
| strigazi | looking into it | 12:17 |
| *** esikachev has joined #openstack-infra | 12:17 | |
| *** cdent has left #openstack-infra | 12:17 | |
| openstackgerrit | amrith proposed openstack-infra/infra-manual: Update section on Gerrit permissions for shared permissions https://review.openstack.org/312893 | 12:17 |
| *** jcoufal has joined #openstack-infra | 12:18 | |
| *** amotoki has joined #openstack-infra | 12:18 | |
| strigazi | yolanda: I haven't defined this macro | 12:18 |
| *** amotoki has quit IRC | 12:18 | |
| fungi | yolanda: that's normal. after two times of jjb adding the "tox" plugin and clobbering our configuration we convinced them to let configured names shadow built-in names rather than the other way around | 12:18 |
| yolanda | strigazi, yes, not related to your changes | 12:19 |
| fungi | it's been like this since some time last year i think | 12:19 |
| *** sree has joined #openstack-infra | 12:19 | |
| yolanda | fungi, no idea why it raised my attention today then ... maybe a long time without running jjb updates | 12:19 |
| *** sree has quit IRC | 12:20 | |
| *** sree has joined #openstack-infra | 12:21 | |
| yolanda | fungi, also seeing several "not deleting unmanaged jenkins job" alerts on jenkins01 | 12:21 |
| openstackgerrit | Rodrigo Duarte proposed openstack-infra/project-config: Make keystone functional tests job voting https://review.openstack.org/321890 | 12:21 |
| fungi | yolanda: sounds like it's possible someone created some extra jobs there by hand to test something | 12:21 |
| *** esikachev has quit IRC | 12:22 | |
| *** amoralej is now known as amoralej|lunch | 12:22 | |
| beisner | hi yolanda :) | 12:23 |
| *** haaj has joined #openstack-infra | 12:23 | |
| sambetts | Hi, I'm maintaining the third party CI for the Cisco Ironic drivers, and I'm seeing a weird issue with my Zuul where it doesn't seem to be reading the Depends-On correcty, I'm even seeing the logs "Change <change reference> depends on changes [] | 12:24 |
| sambetts | " | 12:24 |
| sambetts | any ideas what might be causing it? | 12:24 |
| sambetts | and I missing some configuration? | 12:24 |
| sambetts | am I * | 12:25 |
| *** gema has quit IRC | 12:26 | |
| *** yaume[away] is now known as yaume | 12:26 | |
| *** thiagop has joined #openstack-infra | 12:29 | |
| *** mhickey has joined #openstack-infra | 12:29 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:29 | |
| EmilienM | I see a lot of mirror failures this morning | 12:30 |
| EmilienM | http://logs.openstack.org/89/324089/1/check/gate-puppet-tripleo-puppet-syntax-3-centos-7/8247ed5/console.html#_2016-06-02_12_08_23_668 | 12:30 |
| EmilienM | example ^ but there is more | 12:30 |
| *** yujunz has quit IRC | 12:32 | |
| *** kaisers_ has joined #openstack-infra | 12:34 | |
| *** psilvad has joined #openstack-infra | 12:35 | |
| *** amotoki has joined #openstack-infra | 12:36 | |
| *** vikram_ has quit IRC | 12:36 | |
| *** rlandy has quit IRC | 12:38 | |
| *** Goneri has joined #openstack-infra | 12:39 | |
| *** kaisers_ has quit IRC | 12:39 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| *** julim has quit IRC | 12:41 | |
| *** tpsilva has joined #openstack-infra | 12:44 | |
| *** roxanaghe has joined #openstack-infra | 12:44 | |
| *** binbincong has quit IRC | 12:46 | |
| *** gordc has joined #openstack-infra | 12:46 | |
| *** roxanaghe has quit IRC | 12:48 | |
| *** kgiusti has joined #openstack-infra | 12:49 | |
| *** haaj has quit IRC | 12:49 | |
| *** tlian has joined #openstack-infra | 12:50 | |
| *** haaj has joined #openstack-infra | 12:50 | |
| *** chem has joined #openstack-infra | 12:50 | |
| *** markusry has quit IRC | 12:52 | |
| *** amotoki has quit IRC | 12:53 | |
| *** Wei_Liu has quit IRC | 12:54 | |
| *** amitgandhinz has joined #openstack-infra | 12:56 | |
| *** jlanoux_ has joined #openstack-infra | 12:57 | |
| *** jlanoux has quit IRC | 12:57 | |
| *** yamamoto_ has quit IRC | 12:58 | |
| *** icey_ has joined #openstack-infra | 12:58 | |
| *** dizquierdo has joined #openstack-infra | 12:59 | |
| *** abregman is now known as abregman|mtg | 12:59 | |
| *** amrith is now known as _amrith_ | 13:00 | |
| *** annegentle has joined #openstack-infra | 13:00 | |
| *** amitgandhinz has quit IRC | 13:01 | |
| *** binbincong has joined #openstack-infra | 13:02 | |
| *** gildub has quit IRC | 13:02 | |
| *** amitgandhinz has joined #openstack-infra | 13:02 | |
| *** icey has quit IRC | 13:02 | |
| *** burgerk has joined #openstack-infra | 13:03 | |
| *** abregman|mtg is now known as abregman | 13:03 | |
| strigazi | yolanda: Were all jenkins masters updated? | 13:03 |
| *** sorantis has joined #openstack-infra | 13:04 | |
| sorantis | Hi | 13:04 |
| sorantis | Is this normal ? https://github.com/openstack/requirements/blob/master/global-requirements.txt#L223 | 13:04 |
| *** amitgandhinz has quit IRC | 13:06 | |
| *** amitgandhinz has joined #openstack-infra | 13:07 | |
| *** akshai has joined #openstack-infra | 13:08 | |
| *** _amrith_ is now known as amrith | 13:09 | |
| *** esikachev has joined #openstack-infra | 13:11 | |
| *** njohnston is now known as njohnston|afk | 13:11 | |
| *** mdrabe has joined #openstack-infra | 13:11 | |
| yolanda | jenkins04 still pending | 13:12 |
| *** haaj_ has joined #openstack-infra | 13:12 | |
| *** Julien-zte has joined #openstack-infra | 13:13 | |
| openstackgerrit | Mário Santos proposed openstack-infra/shade: Add support for changing metadata of compute instances https://review.openstack.org/319395 | 13:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Support subscription to worklists in the API https://review.openstack.org/324493 | 13:13 |
| *** mriedem has joined #openstack-infra | 13:14 | |
| *** julim has joined #openstack-infra | 13:14 | |
| *** haaj has quit IRC | 13:15 | |
| *** mrmartin has joined #openstack-infra | 13:15 | |
| *** yamahata has joined #openstack-infra | 13:16 | |
| *** esikachev has quit IRC | 13:16 | |
| yolanda | strigazi, you may want to try again. You may not have the bad luck to fall on jenkins04, that stil has not been updated | 13:17 |
| *** nadya has joined #openstack-infra | 13:17 | |
| *** Goneri has quit IRC | 13:17 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout: fix a typo for centos jobs https://review.openstack.org/324496 | 13:17 |
| strigazi | yolanda: fingers crossed | 13:18 |
| *** anteaya has joined #openstack-infra | 13:18 | |
| *** PollyZ_ has quit IRC | 13:18 | |
| EmilienM | yolanda, pabelanger: I made a typo in my zuul/layout work yesterday... /me autoslap - very quick review https://review.openstack.org/324496 | 13:19 |
| EmilienM | are you aware about centos mirror issues on different clouds? this one is on ovh: http://logs.openstack.org/89/324089/2/gate/gate-puppet-tripleo-puppet-syntax-4-centos-7/e6b68e1/console.html#_2016-06-02_12_41_39_455 | 13:21 |
| *** matt-borland has joined #openstack-infra | 13:21 | |
| *** amoralej|lunch is now known as amoralej | 13:21 | |
| *** moravec has joined #openstack-infra | 13:22 | |
| *** zeih has quit IRC | 13:22 | |
| *** mixos has joined #openstack-infra | 13:23 | |
| *** _ari_ has joined #openstack-infra | 13:24 | |
| *** eharney has joined #openstack-infra | 13:24 | |
| fungi | yolanda: if one of the masters is taking a long time to update, it's reasonable to put it in "prepare for shutdown" mode so that it won't run more jobs | 13:24 |
| yolanda | EmilienM, +W | 13:24 |
| EmilienM | yolanda: thx | 13:24 |
| *** e0ne has quit IRC | 13:25 | |
| yolanda | fungi, finished now | 13:26 |
| yolanda | all jenkins masters shall be ready | 13:26 |
| *** e0ne has joined #openstack-infra | 13:27 | |
| _ari_ | hashar: greetings I answered your question on the review - https://review.openstack.org/#/c/323495/ | 13:29 |
| *** edmondsw has joined #openstack-infra | 13:29 | |
| _ari_ | hashar: the helper already handles specific selectors and how to handle sub options. Everything else is the same | 13:30 |
| *** vdrok has joined #openstack-infra | 13:30 | |
| *** esikachev has joined #openstack-infra | 13:30 | |
| *** xyang1 has joined #openstack-infra | 13:31 | |
| *** zeih has joined #openstack-infra | 13:34 | |
| *** PollyZ_ has joined #openstack-infra | 13:34 | |
| *** isaacb has joined #openstack-infra | 13:34 | |
| *** crinkle has quit IRC | 13:35 | |
| *** crinkle has joined #openstack-infra | 13:36 | |
| *** hdaniel has quit IRC | 13:38 | |
| *** abregman has quit IRC | 13:39 | |
| pleia2 | https://etherpad.openstack.org/p/repo-renaming-2016-06-03 has been updated to include the other rename too (thanks for the change, mordred!) | 13:41 |
| *** tonytan4ever has joined #openstack-infra | 13:41 | |
| *** savihou has joined #openstack-infra | 13:41 | |
| *** mrmartin has quit IRC | 13:42 | |
| *** nadya_ has joined #openstack-infra | 13:43 | |
| *** rbrndt has joined #openstack-infra | 13:44 | |
| *** nadya has quit IRC | 13:45 | |
| *** roxanaghe has joined #openstack-infra | 13:45 | |
| *** nadya has joined #openstack-infra | 13:47 | |
| andreaf | I see a lot of grenade jobs in various pipelines that appear to be stuck / running indefinitely | 13:47 |
| andreaf | any idea what could be wrong? is it possible to abort them? | 13:47 |
| phschwartz | looking at openstack-infra/project-config I am failing to see the flow followed for updating the projects/acl's in gerrit when a new project is proposed. I see things to normalize the project list and such, but not the others. Can someone point me in the right direction | 13:48 |
| andreaf | 'gate-grenade-dsvm-multinode' to be precise | 13:48 |
| strigazi | yolanda: thanks | 13:48 |
| yolanda | strigazi, were your tests successful? | 13:48 |
| *** nwkarsten has joined #openstack-infra | 13:49 | |
| *** nadya_ has quit IRC | 13:49 | |
| *** roxanaghe has quit IRC | 13:49 | |
| strigazi | 2 out 3, the 3rd that failed was unrelated to jenkins | 13:49 |
| strigazi | yolanda: http://logs.openstack.org/46/309546/6/check/gate-rally-dsvm-magnum-rally-nv/711b389/rally-plot/results.html.gz | 13:50 |
| *** nadya has quit IRC | 13:51 | |
| strigazi | yolanda: We'll need an ubuntu-mesos in the near future | 13:51 |
| strigazi | yolanda: I meant, an ubuntu-mesos image | 13:52 |
| *** Goneri has joined #openstack-infra | 13:52 | |
| *** nadya_ has joined #openstack-infra | 13:52 | |
| yolanda | strigazi, generated with diskimage-builder as we did for atomic? what are the requirements, what needs to have added in top uf minimal ubuntu? | 13:53 |
| pabelanger | EmilienM: it looks like the mirror we rsync is out of date | 13:53 |
| pabelanger | EmilienM: try a recheck now | 13:53 |
| EmilienM | ok | 13:53 |
| strigazi | yolanda: I don't know. Will have to investigate, I mostly work on atomic. | 13:54 |
| openstackgerrit | Merged openstack-infra/project-config: zuul/layout: fix a typo for centos jobs https://review.openstack.org/324496 | 13:54 |
| *** tmorin has joined #openstack-infra | 13:55 | |
| *** mrmartin has joined #openstack-infra | 13:55 | |
| *** timl0101 has joined #openstack-infra | 13:56 | |
| *** andreww has quit IRC | 13:56 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Switch to mirrors.kernel.org for centos / epel rsync https://review.openstack.org/324542 | 13:57 |
| pabelanger | yolanda: pleia2: We seem to be having some sync issues on our current EPEL / CentOS mirrors. ^ Do you mind reviewing as we are pinning to mirrors.kernel.org | 13:58 |
| pleia2 | sure thing | 13:58 |
| pleia2 | lgtm | 13:58 |
| yolanda | sure | 13:59 |
| *** oanson has quit IRC | 14:00 | |
| *** mrmartin has quit IRC | 14:00 | |
| *** ddieterly has joined #openstack-infra | 14:00 | |
| *** yujunz has joined #openstack-infra | 14:04 | |
| *** burgerk has quit IRC | 14:04 | |
| *** ayoung has joined #openstack-infra | 14:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Simplify translation logic https://review.openstack.org/324548 | 14:06 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Allow the Subscribe directive to work without a list of subscriptions https://review.openstack.org/324549 | 14:06 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add a subscribe button to worklists https://review.openstack.org/324550 | 14:06 |
| *** jaosorior has quit IRC | 14:06 | |
| *** Na3iL has joined #openstack-infra | 14:06 | |
| *** nadya_ has quit IRC | 14:07 | |
| *** yamamoto has joined #openstack-infra | 14:07 | |
| fungi | pabelanger: where did mirror.sfo12.us.leaseweb.net come from? | 14:07 |
| fungi | i guess the official centos mirrors list? | 14:08 |
| fungi | but yeah, getting both from mirrors.kernel.org makes more sense | 14:08 |
| *** nadya has joined #openstack-infra | 14:08 | |
| pleia2 | it did seem like an odd choice :) | 14:09 |
| pleia2 | (yay kernel.org) | 14:09 |
| fungi | our linfo friends run that, so it should be nice and solid | 14:09 |
| *** sorantis has quit IRC | 14:10 | |
| fungi | actually, that's an ambiguous abbrev since linfo.org also exists (and is "the linux information project") | 14:11 |
| *** links has quit IRC | 14:12 | |
| fungi | i guess they just prefer "lf" | 14:12 |
| pleia2 | yeah | 14:12 |
| *** claudiub has quit IRC | 14:13 | |
| *** amitgandhinz has quit IRC | 14:13 | |
| *** _vsaienko has joined #openstack-infra | 14:14 | |
| *** andreww has joined #openstack-infra | 14:15 | |
| *** yamamoto has quit IRC | 14:16 | |
| *** woodster_ has joined #openstack-infra | 14:17 | |
| *** zz_dimtruck is now known as dimtruck | 14:19 | |
| *** fedexo has joined #openstack-infra | 14:19 | |
| *** ayoung has quit IRC | 14:19 | |
| *** nadya has quit IRC | 14:19 | |
| *** nadya has joined #openstack-infra | 14:20 | |
| *** banix has joined #openstack-infra | 14:20 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in ceilometer https://review.openstack.org/315224 | 14:20 |
| nwkarsten | it looks like the change to the puppet-zuul module here https://github.com/openstack-infra/puppet-zuul/commit/c21307271328b5e49962b7fbb4dee5393fe81d67 breaks the single_node_ci class in the openstackci module. http://paste.openstack.org/show/507294/ apologies if this was already known | 14:22 |
| *** kaisers_ has joined #openstack-infra | 14:23 | |
| *** banix_ has joined #openstack-infra | 14:24 | |
| openstackgerrit | Sahdev Zala proposed openstack-infra/project-config: Add TOSCA-Parser job for Heat-Translator project https://review.openstack.org/324556 | 14:24 |
| *** yamamoto has joined #openstack-infra | 14:24 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Implement propose_rdo_promote CI periodic job https://review.openstack.org/324557 | 14:24 |
| EmilienM | flepied, dmsimard: fyi^ | 14:25 |
| fungi | nwkarsten: got it, so we need puppet-zuul and puppet-nodepool to both conditionally install build-essential i guess? | 14:25 |
| clarkb | fungi: or push build essential into a dep of both of th | 14:26 |
| nwkarsten | fungi: that's how i got puppet to work on my machine | 14:26 |
| *** mixos has quit IRC | 14:26 | |
| clarkb | not sure I like having a new dep just for buipd essential I would use paclage install guards | 14:26 |
| *** claudiub has joined #openstack-infra | 14:26 | |
| fungi | clarkb: good point, i wonder if both of them depend on a common class somewhere | 14:26 |
| hashar | _ari_: sorry I finally looked again at the copyartifcat/multijob code | 14:27 |
| fungi | nwkarsten: were you working on submitting patches for that yet? or should i do it now? | 14:27 |
| nwkarsten | fungi: i won't have time today | 14:27 |
| hashar | _ari_: and indeed it looks all good. I must have been confused somehow yesterday :D | 14:27 |
| *** banix has quit IRC | 14:27 | |
| clarkb | we almost need a bindeo modupe to do all package installs for us | 14:27 |
| _ari_ | hashar: no worries | 14:27 |
| *** bswartz has quit IRC | 14:27 | |
| fungi | nwkarsten: no problem, thanks for pointing it out. i'll get it up here in a bit | 14:27 |
| _ari_ | hashar: I did add the example to the docs as well just so people see it | 14:28 |
| *** Julien-zte has quit IRC | 14:28 | |
| fungi | clarkb: yeah, that's another way of saying "puppet's package management is terrible, we should replace it with some other tool" (which i don't entirely disagree with) | 14:28 |
| *** _vsaienko has quit IRC | 14:28 | |
| _ari_ | hashar: thanks again for the review | 14:28 |
| *** kaisers_ has quit IRC | 14:28 | |
| sdague | clarkb / fungi - https://review.openstack.org/#/c/320925/ - reviews on getting rid of gate64 hack? | 14:28 |
| thiagop | Guys, good morning! I'm with a couple of problems with my instance of zuul on a third party ci and I have no clues about what's happening | 14:29 |
| hashar | _ari_: yeah that looks all fine. And the more doc the more people can easily use jjb :) | 14:29 |
| pabelanger | fungi: random mirror I selected during testing | 14:30 |
| _ari_ | hashar: indeed | 14:30 |
| pabelanger | fungi: but official | 14:30 |
| fungi | sdague: does https://review.openstack.org/324098 need merging first? | 14:30 |
| fungi | mriedem: ^ | 14:30 |
| *** banix_ has quit IRC | 14:30 | |
| *** banix has joined #openstack-infra | 14:30 | |
| *** cbader has quit IRC | 14:30 | |
| _ari_ | hashar: I am worried jjb will be obsolete though now that pipeline is out | 14:30 |
| sdague | fungi: it has a depends-on | 14:30 |
| mriedem | fungi: yeah | 14:30 |
| sdague | so it should all do the right things | 14:30 |
| fungi | sdague: right, rhetorical question | 14:30 |
| thiagop | problem 1 is that my queues aren't appearing on zuul. I'm seeing nothing on zuul.log or debug.log | 14:31 |
| clarkb | fungi: remember to patch the zuul and nodepool modules not just the zuul one | 14:31 |
| fungi | just pointing out the d-g change won't merge without that landing | 14:31 |
| fungi | clarkb: yep, that's what i've got cooking now | 14:31 |
| fungi | clarkb: since you can't guarantee order | 14:31 |
| *** haaj has joined #openstack-infra | 14:31 | |
| *** rellerreller has joined #openstack-infra | 14:31 | |
| *** _vsaienko has joined #openstack-infra | 14:31 | |
| pabelanger | fungi: any objections for me testing out the CNAME entry for mirror.tripleo-test-cloud-rh1? | 14:32 |
| clarkb | sdague: what about liberty for nova? | 14:32 |
| sdague | clarkb: we removed those tests instead | 14:32 |
| *** yaume_ has joined #openstack-infra | 14:32 | |
| sdague | it wasn't deemed a security bug | 14:32 |
| *** haaj__ has joined #openstack-infra | 14:32 | |
| fungi | pabelanger: no objection, other than echoing jeblair's sentiment that if their environment can't support the same mechanisms we embed in our other providers, it's really better suited as a third-party ci | 14:32 |
| *** yaume has quit IRC | 14:33 | |
| *** haaj__ has quit IRC | 14:33 | |
| *** haaj_ has quit IRC | 14:33 | |
| mriedem | clarkb: yeah https://review.openstack.org/#/c/324107/ | 14:33 |
| *** haaj_ has joined #openstack-infra | 14:34 | |
| rellerreller | Could someone review this patch, https://review.openstack.org/#/c/307820/ ? I believe I addressed all of the concerns of Andreas Jaeger. This seems to have fallen off the radar a bit, probably due to the Memorial Day holiday. | 14:34 |
| vsaienko | please review https://review.openstack.org/#/c/324340/, the patch fixes configuration for ironic-staging-drivers job by adding ironic and ironic-staging-drivers to $PROJECT | 14:34 |
| pabelanger | fungi: I agree. In that case, I believe infra should have some sort of official statement on the situation. Maybe some sort of imposed migration date. | 14:34 |
| DuncanT | Seeing a few gate failures due to bluebox mirror issues... can I recheck them all or should I wait for something to be fixed? | 14:34 |
| clarkb | ok, I do find it important to point out a big reason it is non voting is we keep letting regressions like this sneak in | 14:34 |
| clarkb | it did work really well at loberty release iirc | 14:34 |
| *** ajmiller has joined #openstack-infra | 14:35 | |
| *** amotoki has joined #openstack-infra | 14:35 | |
| pabelanger | fungi: because both projects had differing opinions on the current status quo. | 14:35 |
| clarkb | DuncanT: what kind of errors and which mirror? examples may help | 14:35 |
| DuncanT | https://review.openstack.org/#/c/324361 is an example | 14:35 |
| *** ayoung has joined #openstack-infra | 14:36 | |
| *** haaj has quit IRC | 14:36 | |
| DuncanT | W: Failed to fetch http://mirror.regionone.bluebox-sjc1.openstack.org/ubuntu/dists/trusty/InRelease | 14:36 |
| fungi | confirmed, e.g. http://mirror.regionone.bluebox-sjc1.openstack.org/ubuntu/dists/trusty/Release.gpg is a 404 | 14:36 |
| *** haaj_ has quit IRC | 14:36 | |
| clarkb | http://logs.openstack.org/61/324361/2/check/gate-python-cinderclient-pep8/ba889df/console.html#_2016-06-02_10_59_10_058 is a log link for those curious | 14:37 |
| *** haaj has joined #openstack-infra | 14:37 | |
| *** dizquierdo has quit IRC | 14:37 | |
| *** nwkarsten has quit IRC | 14:37 | |
| clarkb | according to the logs it couodnt connect at all? we may need to chexk if this is persisting in bluebox | 14:37 |
| pabelanger | What is InRelease? | 14:38 |
| *** banix_ has joined #openstack-infra | 14:38 | |
| *** ajmiller_ has quit IRC | 14:38 | |
| *** banix has quit IRC | 14:39 | |
| *** banix_ is now known as banix | 14:39 | |
| *** jbernard has quit IRC | 14:40 | |
| krotscheck | fungi: Did you ever get anywhere with those coredumps? | 14:40 |
| pabelanger | wonder if that is something reprepro does, since it doesn't exist upstream: http://archive.ubuntu.com/ubuntu/dists/trusty/ | 14:40 |
| fungi | krotscheck: no, got sidetracked by other broken | 14:40 |
| pleia2 | pabelanger: InRelease is Release + Release.gpg, instead it's signed inline | 14:41 |
| *** yaume_ has quit IRC | 14:41 | |
| *** yaume has joined #openstack-infra | 14:41 | |
| fungi | pleia2: except we don't sign these anyway | 14:41 |
| * pleia2 nods | 14:41 | |
| pleia2 | so, broadly speaking :) | 14:41 |
| pabelanger | pleia2: TIL | 14:41 |
| krotscheck | fungi: Urgh. And of course now I've gotten sidetracked by an intermittent horizon timeout issue. | 14:41 |
| fungi | so i think this means the mirror is fine, and we've lost our apt configuration to stop checking repo digs? | 14:41 |
| pabelanger | It must be reprepro doing some magic, since we rebuild the indexes | 14:41 |
| clarkb | fungi: oh maybe | 14:42 |
| DuncanT | fungi: I'll hold off any more rechecks then. | 14:42 |
| fungi | s/digs/sigs/ | 14:42 |
| *** mtanino has joined #openstack-infra | 14:42 | |
| fungi | clarkb: at least i confirmed that the files it's complaining about aren't present in any of our mirrors (they're all consistent) | 14:42 |
| clarkb | ya I only see "Release" | 14:43 |
| fungi | so afs seems to be working as intended here | 14:43 |
| *** _vsaienko has quit IRC | 14:43 | |
| *** yamamoto has quit IRC | 14:43 | |
| *** icey_ has quit IRC | 14:44 | |
| *** nwkarst__ has joined #openstack-infra | 14:44 | |
| *** david-lyle has joined #openstack-infra | 14:44 | |
| *** devananda has quit IRC | 14:45 | |
| *** roxanaghe has joined #openstack-infra | 14:45 | |
| tmorin | lifeless ? could you help on a pylint/astroid issue (upper-constrained apprently preventing the use of an astroid release working with recent pylint) ? | 14:46 |
| *** haaj has quit IRC | 14:46 | |
| tmorin | lifeless: asking you because you did https://review.openstack.org/#/c/251600/ | 14:47 |
| *** esikachev has quit IRC | 14:47 | |
| *** esikachev has joined #openstack-infra | 14:47 | |
| *** icey has joined #openstack-infra | 14:47 | |
| clarkb | tmorin: lifeless wont be around for another 4 hours probably. ehat is the issue? | 14:47 |
| tmorin | clarkb: I have a job that uses pylint, which is working fine if upper-constraints.txt isn't used, and failing if upper-constraints.txt is used | 14:49 |
| phschwartz | for nodepool, if I want 1 image with multiple labels associated to different size nodes, can I add the min-ram to the label instead of the image in the config? | 14:49 |
| clarkb | tmorin: so you need a newer version of astroid? you can confirm by updating the version of astroid in the constraints file when you test | 14:50 |
| tmorin | clarkb: it fails on pylint, and the only difference is that the working job uses astroid 1.4.5 which upper-constraints.txt prevents from using (upper-constraints says astroid==1.3.8) | 14:50 |
| clarkb | phschwartz: no, you will need two different logical images unfortunately | 14:50 |
| *** roxanaghe has quit IRC | 14:50 | |
| tmorin | clarkb: I think that anyone using pylint and no pinning it to a former release would have the same issue | 14:51 |
| *** bswartz has joined #openstack-infra | 14:51 | |
| phschwartz | clarkb: so I was thinking that, looking at the nodepool config info it says that I can only define max servers on a provider, not on an image | 14:51 |
| *** nwkarsten has joined #openstack-infra | 14:51 | |
| phschwartz | clarkb: is there a way around this? I have 60 max in my pool and don't want these larger to scale that far up | 14:51 |
| clarkb | tmorin: also note that the change you linked is for a stable release which should also have pylint pinned | 14:52 |
| *** _vsaienko has joined #openstack-infra | 14:52 | |
| tmorin | clarkb: do you mean something like edit upper-constraints.txt via a sed command in tox.ini ? | 14:52 |
| clarkb | phschwartz: no the idea is you set the minimum ready for each then job demand dictates what gets built up to the max server number | 14:52 |
| *** vgridnev has quit IRC | 14:52 | |
| tmorin | clarkb: correct, but the master upper-constraints.txt also pins astroid | 14:52 |
| clarkb | phschwartz: in theory removing the need for micro management | 14:53 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-nodepool: Conditionally install build-essential https://review.openstack.org/324580 | 14:53 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-zuul: Conditionally install build-essential https://review.openstack.org/324581 | 14:53 |
| *** vgridnev_ has joined #openstack-infra | 14:53 | |
| phschwartz | clarkb: ah, ok. I will have to make sure we are careful then | 14:53 |
| *** isaacb has quit IRC | 14:53 | |
| tmorin | clarkb: although I can't find the source for this change (done by the bot, but I wasn't able to tell based on what) | 14:53 |
| *** vgridnev_ has quit IRC | 14:53 | |
| *** claudiub|2 has joined #openstack-infra | 14:53 | |
| fungi | nwkarsten: clarkb: https://review.openstack.org/324580 and https://review.openstack.org/324581 should take care of the coinstallability issue | 14:53 |
| *** nwkarst__ has quit IRC | 14:53 | |
| clarkb | tmorin: upper constraints is based on global requirements being installed and passing | 14:54 |
| fungi | phschwartz: different tenants with their own quotas, and relegate each to one flavor? | 14:55 |
| clarkb | tmorin: so the change to global-requirements which you linked results in the upper constraints change | 14:55 |
| *** ajmiller has quit IRC | 14:55 | |
| clarkb | tmorin: if you want to change the version of astroid you will need to update the version in global-requirements. Then the bot will propose and update to constraints (or you can go ahead and put that in your global-requirements change to shorten turn around time) | 14:55 |
| clarkb | fungi: phschwartz if you want to go that route they can even use the same tenant as long as you do the maths correctly. But I do like the logical separation of discrete quotas there | 14:56 |
| *** claudiub has quit IRC | 14:56 | |
| *** devananda has joined #openstack-infra | 14:56 | |
| *** mdrabe has quit IRC | 14:56 | |
| *** vgridnev has joined #openstack-infra | 14:56 | |
| *** zhurong has quit IRC | 14:57 | |
| clarkb | tmorin: anyways the thing to do is ensure that pylint and astroid work together at some version and update global requirements to match. Keep in mind that the stable maintainers are not likely to want to bump versions for either thing on the stable branches so you may need to pin pylint to a version that works with older astroid | 14:57 |
| phschwartz | clarkb: fungi: at the moment I will have to put them as one, but separate tenents will be my long term goal | 14:57 |
| *** abregman has joined #openstack-infra | 14:57 | |
| *** andreykurilin__ has joined #openstack-infra | 14:57 | |
| *** bhavik has quit IRC | 14:58 | |
| tmorin | clarkb: in fact, global-requirements.txt does not force anything anymore on astroid, only upper-constraints.txt does | 14:58 |
| *** zeih has quit IRC | 14:58 | |
| *** yamamoto has joined #openstack-infra | 14:58 | |
| *** ajmiller has joined #openstack-infra | 14:59 | |
| clarkb | tmorin: https://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt?h=stable/kilo#n6 | 14:59 |
| fungi | tmorin: that seems like a questionable process choice | 14:59 |
| *** EricGonczer_ has joined #openstack-infra | 14:59 | |
| fungi | tmorin: if you're not reasonably in sync with global requirements, i can imagine the upper constraints list would quickly become nonsensical in that context | 15:00 |
| tmorin | clarkb: I meant that *on master* global-requirements.txt does not force anything anymore on astroid, only upper-constraints.txt does | 15:00 |
| clarkb | tmorin: fungi bit it is listed,see the link above | 15:00 |
| *** njohnston|afk is now known as njohnston | 15:00 | |
| clarkb | tmorin: ok, but the change you linked was for kilo /me is confused | 15:00 |
| *** csomerville has joined #openstack-infra | 15:00 | |
| tmorin | clarkb: yes, well, I shouldn't have linked this change, just wanted to explain lifeless why I was pinging him | 15:01 |
| *** nadya_ has joined #openstack-infra | 15:01 | |
| clarkb | here is master https://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n6 | 15:01 |
| tmorin | fungi: sorry, I don't follow you | 15:01 |
| krotscheck | infra-core: I'm trying to track down why we have intermittent gate failures on horizon's karma test suite- it appears that in some cases, either writing the coverage report, or detaching from chrome, hangs indefinitely. Could someone recheck https://review.openstack.org/#/c/324074/ for me and capture whatever node gets the npm-run-test job? | 15:01 |
| *** kzaitsev_mb has quit IRC | 15:01 | |
| *** nadya has quit IRC | 15:01 | |
| *** amrith is now known as _amrith_ | 15:02 | |
| tmorin | clarkb: sorry, my mistake, wasn't looking at a fresh tree :-/ | 15:02 |
| *** kzaitsev_mb has joined #openstack-infra | 15:02 | |
| *** nwkarste_ has joined #openstack-infra | 15:02 | |
| fungi | tmorin: upper-constraints.txt is a precomputed dependency resolution of the contents of global-requirements.txt. if your requirements aren't basically a subset of global-requirements.txt then upper-constraints.txt isn't especially meaningful | 15:04 |
| *** Kaiyan has joined #openstack-infra | 15:04 | |
| *** fitoduarte has quit IRC | 15:04 | |
| clarkb | ok lets reset the discussion. Astroid is listed in global requirements as is pylink | 15:05 |
| *** nwkarsten has quit IRC | 15:05 | |
| clarkb | *pylint | 15:05 |
| tmorin | clarkb: my job is using pylint 1.5.5, which not consistent with global-requirements.txt -- so I guess the issue in that my pep8 job does not in fact properly draws pylint | 15:05 |
| clarkb | if you want to bump the version of either of them in use you will need to update global requirements | 15:05 |
| *** hongbin has joined #openstack-infra | 15:05 | |
| tmorin | clarkb: ^^ | 15:05 |
| clarkb | tmorin: yes if you are not following global requirements but are using our upper constraints file then you will run into the problem fungi describes | 15:05 |
| *** tpsilva has quit IRC | 15:05 | |
| clarkb | you should not do this | 15:05 |
| clarkb | either use global requirements or do not use upper constraints | 15:06 |
| *** maximov has quit IRC | 15:06 | |
| *** markusry has joined #openstack-infra | 15:06 | |
| *** _vsaienko has quit IRC | 15:06 | |
| *** dham1 has quit IRC | 15:07 | |
| *** tpsilva has joined #openstack-infra | 15:07 | |
| *** Swanson has quit IRC | 15:07 | |
| *** ildikov has quit IRC | 15:07 | |
| *** rackertom has quit IRC | 15:07 | |
| *** mgkwill has quit IRC | 15:07 | |
| *** vdrok-afk has quit IRC | 15:07 | |
| *** zeih has joined #openstack-infra | 15:07 | |
| pabelanger | jeblair: do you think we can update our javascript on status.o.o adding the real-time links for zl01.o.o? | 15:08 |
| *** maximov has joined #openstack-infra | 15:08 | |
| *** patrickeast has quit IRC | 15:08 | |
| *** spredzy has joined #openstack-infra | 15:08 | |
| pabelanger | for the moment, I'm just manually checking nodepool.o.o for zl01 managers | 15:08 |
| tmorin | clarkb/fungi: yes, in fact, I now understand what happened : our pylint/astroid dependency is declared in deps= in our tox.ini, instead of doing that in test-requirements | 15:08 |
| *** _amrith_ is now known as amrith | 15:08 | |
| *** spredzy has left #openstack-infra | 15:08 | |
| *** ildikov has joined #openstack-infra | 15:09 | |
| *** rackertom has joined #openstack-infra | 15:09 | |
| *** mgkwill has joined #openstack-infra | 15:09 | |
| *** dham1 has joined #openstack-infra | 15:09 | |
| *** patrickeast has joined #openstack-infra | 15:09 | |
| *** amrith is now known as _amrith_ | 15:09 | |
| *** bhavik has joined #openstack-infra | 15:09 | |
| Kaiyan | Hi all, I have a question regarding to a failed gate-project-config-layout job with error "ERROR:jenkins_jobs.parser:Duplicate definitions for job 'gate-tempest-dsvm-monasca-java-full-nv' specified". I defined a job-group in monasca.yaml looks like this: - 'gate-tempest-dsvm-monasca-java-full{job-suffix}': job-suffix: '' - 'gate-tempest-dsvm-monasca-java-full{job-suffix}': job-suffix: '-nv' Is this where the duplicate came f | 15:09 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/release-tools: unset custom GREP_OPTIONS https://review.openstack.org/324605 | 15:10 |
| *** vdrok-afk has joined #openstack-infra | 15:10 | |
| *** zeih has quit IRC | 15:11 | |
| *** mriedem1 has joined #openstack-infra | 15:11 | |
| *** mriedem has quit IRC | 15:12 | |
| *** mdrabe has joined #openstack-infra | 15:12 | |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: [Monasca] Add Tempest Jobs as non-voting for monasca repos https://review.openstack.org/316293 | 15:12 |
| *** burgerk has joined #openstack-infra | 15:12 | |
| *** nadya has joined #openstack-infra | 15:13 | |
| *** _amrith_ is now known as amrith | 15:14 | |
| *** kzaitsev_mb has quit IRC | 15:14 | |
| *** nadya_ has quit IRC | 15:15 | |
| *** EricGonczer_ has quit IRC | 15:15 | |
| *** denisra has joined #openstack-infra | 15:17 | |
| jeblair | pabelanger: i would like to do that after all the jenkins are gone, and we can send an announcement -- i'm preparing the announcement today | 15:17 |
| *** asselin_ has joined #openstack-infra | 15:17 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:17 | |
| *** mriedem has joined #openstack-infra | 15:17 | |
| pabelanger | jeblair: No problem at all! | 15:18 |
| clarkb | krotscheck: so bumping the log level on that change hasn't helped? | 15:19 |
| clarkb | krotscheck: give me a few to load keys up and I can try capturing the host for you | 15:19 |
| krotscheck | clarkb: All we know is that the failure is somewhere between "Start writing coverage reports" and "Browsers have all been shut down" | 15:19 |
| clarkb | Kaiyan: can you provide a link to the change? | 15:19 |
| krotscheck | clarkb: Much obliged. | 15:19 |
| *** mriedem1 has quit IRC | 15:20 | |
| krotscheck | clarkb: I already rechecked it, it's in the queue. | 15:20 |
| krotscheck | (doesn' thave a host yet) | 15:20 |
| clarkb | ah | 15:20 |
| *** claudiub|2 has quit IRC | 15:21 | |
| * krotscheck sometimes looks at the gate queue and channels seinfeld's soup episode. | 15:22 | |
| krotscheck | You- END OF THE LINE! | 15:22 |
| clarkb | careful zuul might say "NO GATE FOR YOU" | 15:23 |
| Kaiyan | Hi Clark, the link is http://logs.openstack.org/93/316293/13/check/gate-project-config-layout/1d2cdb3/console.html for the console log and the review is https://review.openstack.org/#/c/316293/ Thank you! | 15:23 |
| phschwartz | clarkb: seems that I have an erorr. http://paste.openstack.org/show/507319/ I used the same diskimage for the 2 provider images, could that be the cause of that | 15:24 |
| krotscheck | clarkb++ | 15:24 |
| krotscheck | clarkb: I have a node for gate-horizon-npm-run-test on 324074 | 15:25 |
| krotscheck | clarkb: Would you be so kind to grab that for me? :D | 15:25 |
| asselin_ | mordred, nibalizer (or anyone else familiar) what's the status of this spec? working? great? done? https://specs.openstack.org/openstack-infra/infra-specs/specs/ansible_puppet_apply.html | 15:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/gear: Bump version requirement for pbr https://review.openstack.org/301222 | 15:25 |
| openstackgerrit | Samuel Cassiba proposed openstack-infra/project-config: Add mitaka to gerritbot for OpenStack-Chef https://review.openstack.org/324612 | 15:25 |
| nibalizer | asselin_: yes we run with puppet apply from ansible now | 15:26 |
| *** vhosakot has joined #openstack-infra | 15:26 | |
| nibalizer | have for a few months | 15:26 |
| nibalizer | there are some places we need to do some cleanup but its working | 15:26 |
| asselin_ | what kind of cleanup? | 15:26 |
| clarkb | krotscheck: can you tell me the node name while I get keys out | 15:26 |
| krotscheck | clarkb: I don't think so, I can't link through to the log anymore | 15:26 |
| clarkb | oh right | 15:27 |
| clarkb | gah | 15:27 |
| nibalizer | senk: hrm | 15:28 |
| thiagop | Folks, what can I do (besides crying) when the publish-to-pypi job fails? | 15:28 |
| krotscheck | thiagop: Sacrifice a goat. | 15:28 |
| *** hashar is now known as hasharAway | 15:28 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add a subscribe button to worklists https://review.openstack.org/324550 | 15:28 |
| krotscheck | Make sure it's a tofu goat. | 15:28 |
| pabelanger | clarkb: fungi: Did we ever figure out the conditions where openstacklocal was being appended to DNS queries on slaves? | 15:29 |
| thiagop | krotscheck: in favor of which god? Murphy? | 15:29 |
| jeblair | thiagop: why did it fail? | 15:29 |
| krotscheck | clarkb: Maybe ubuntu-xenial-rax-ord-1371625 | 15:29 |
| fungi | pabelanger: nova-agent may do that? | 15:29 |
| nibalizer | senk: that index is generated by a script, i'll have to debug it later thogh | 15:29 |
| clarkb | krotscheck: I think we missed it because it completed, I need to find a quick way of getting the node name while the job is running now that we don't have the jenkins links | 15:30 |
| krotscheck | thiagop: Anoia. http://wiki.lspace.org/mediawiki/Anoia | 15:30 |
| jeblair | clarkb: it's in the status.json | 15:30 |
| thiagop | jeblair: not very clear: http://logs.openstack.org/72/7270afc4cbd8e0d9b8cdc0cc11c0a7ffcd9155f1/release/python-oneviewclient-tarball/261fc13/console.html | 15:30 |
| fungi | pabelanger: i think it's something that ends up set in nova/neutron, and we mostly get to ignore it because we blow away that configuration | 15:30 |
| *** zeih has joined #openstack-infra | 15:30 | |
| *** sree has quit IRC | 15:30 | |
| openstackgerrit | Merged openstack-infra/gear: Remove py26 from tox.ini https://review.openstack.org/324252 | 15:31 |
| clarkb | jeblair: ty | 15:31 |
| openstackgerrit | Merged openstack-infra/gear: Remove vestigal hook entry from setup.cfg https://review.openstack.org/324253 | 15:31 |
| clarkb | krotscheck: I am now prepared for the next time around... | 15:31 |
| krotscheck | clarkb: Gotcha | 15:31 |
| thiagop | jeblair: it did timeout building the venv, that's as far as I can tell by these logs unfortunately | 15:31 |
| *** lezbar has quit IRC | 15:31 | |
| pabelanger | fungi: possible, let me look into that | 15:31 |
| krotscheck | clarkb: To be honest, any rax node will do, we've only seen failures on those. | 15:31 |
| *** yamahata has quit IRC | 15:31 | |
| krotscheck | clarkb: I'll ping you as soon as the guild passes and I recheck | 15:31 |
| thiagop | krotscheck: lol | 15:32 |
| *** lezbar has joined #openstack-infra | 15:32 | |
| fungi | krotscheck: interestingly, we're also only seeing the apache segfaults in rax, and nowhere else | 15:32 |
| jeblair | clarkb: hrm. it may not be in the status.json until it completes. i have no idea why that would have been written that way. | 15:32 |
| *** nwkarsten has joined #openstack-infra | 15:32 | |
| *** nbloom has quit IRC | 15:33 | |
| jeblair | clarkb: if it runs on zl01, you'll get an ip address in status.json you can match up | 15:33 |
| jeblair | clarkb: otherwise, zuul log should tell you | 15:33 |
| krotscheck | fungi: ....huhn. That.... is that a hypervisor thing? | 15:33 |
| clarkb | jeblair: I was looking in zuul's log and it isn't immediately clear what I should be grepping on. if I grep change number and job name it doesn't say. If I take the build number from that first grep and grep that it doesn't say | 15:34 |
| fungi | krotscheck: that's what i've been wondering. maybe something to do with system clock stepping or... who knows | 15:34 |
| *** nwkarste_ has quit IRC | 15:34 | |
| *** iurygregory has joined #openstack-infra | 15:34 | |
| *** vdrok has quit IRC | 15:34 | |
| fungi | krotscheck: i'm kinda worried that once i do get a coredump and load it in gdb, the backtrace will at best tell me which of the (very few, completely standard) apache modules is freaking out | 15:34 |
| krotscheck | fungi: Ya know, if we can reproduce it consistently using the npm tooling (since thats' where the npm install mirror failures came from), we'd have something to escalate to support. | 15:35 |
| clarkb | on <Worker Unknown> is what the zuul log says | 15:35 |
| *** deadnull_ has quit IRC | 15:35 | |
| *** zeih has quit IRC | 15:35 | |
| jeblair | clarkb: the build uuid should get you a line like this: | 15:35 |
| jeblair | 2016-06-02 13:45:18,220 DEBUG zuul.Gearman: Build <gear.Job 0x7f6fe657cb50 handle: H:127.0.0.1:526552 name: build:gate-neutron-dsvm-functional unique: dd5b03e7571c4d54b55436b3da2c5fa3> update {u'name': u'gate-neutron-dsvm-functional', u'url': u'https://jenkins03.openstack.org/job/gate-neutron-dsvm-functional/3622/', u'worker': u'ubuntu-trusty-rax-ord-1368855_exec-0', u'number': 3622, u'node_name': u'', u'manager': ... | 15:35 |
| jeblair | ... u'jenkins03.openstack.org', u'node_labels': [u'master']} | 15:35 |
| fungi | clarkb: once you get to the log entry with the job parameters, you'll also have the build node name | 15:35 |
| jeblair | and that has 'worker' in it | 15:36 |
| fungi | or there, yeah | 15:36 |
| clarkb | Kaiyan: the test log says that you have defined the job twice. | 15:36 |
| clarkb | jeblair: oh I see it now | 15:36 |
| Kaiyan | Yeah but I couldn't find anywhere. The only place i can think of is in the monasca.yaml, the job-group part. Because I defined once for -nv and another for '' | 15:38 |
| iurygregory | Hi people, sorry to bother you guys. I have a patch for puppet-openstack-guide repo merged (https://review.openstack.org/#/c/319519/) but the docs didn't get update http://docs.openstack.org/developer/puppet-openstack-guide/ . In the zuul i don't see any job running anymore. Can someone help me with this? | 15:38 |
| *** kzaitsev_mb has quit IRC | 15:39 | |
| Kaiyan | @clarkb i also checked in zuul/layout.yaml when i specify which jobs each project should run, i didn't see any duplicates either hmmm | 15:41 |
| *** psachin has quit IRC | 15:41 | |
| *** vincentll has quit IRC | 15:42 | |
| clarkb | Kaiyan: the zuul layout.yaml shouldn't care about duplicates, it will just run the job multiple times iirc. Its the jjb config failing there | 15:42 |
| krotscheck | fungi: Checking kibana- it's not just restricted to rax. | 15:43 |
| fungi | krotscheck: which isn't | 15:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: DO NOT MERGE https://review.openstack.org/324620 | 15:43 |
| krotscheck | fungi: Horizon's timeout issues | 15:44 |
| fungi | krotscheck: the npm timeouts downloading from our mirrors you mean? | 15:44 |
| krotscheck | fungi: No, I'm talking about the thing that clarkb and I are working on. There's an intermittet timeout in npm-run-test for horizon, failing sonewhere between "write coverage test" and "release captured browsers" | 15:44 |
| Kaiyan | Hmm thanks Clark! Can you check the monasca.yaml when you get a chance then? I defined the jobs there and also added a job-group, but I didn't see any duplicates though | 15:45 |
| krotscheck | So, likely unrelated to coredumps | 15:45 |
| fungi | krotscheck: okay, just confirming' | 15:45 |
| *** andreas_s has quit IRC | 15:45 | |
| clarkb | it is really tricky to find nodes to hold now that we don't have the live job run time dashboard | 15:46 |
| jeblair | clarkb: so close :) | 15:46 |
| pabelanger | I am excited | 15:47 |
| clarkb | ok I think I managed to hold an ubuntu-xenial before the job completed on it | 15:47 |
| pabelanger | if I haven't said that already | 15:47 |
| clarkb | krotscheck: ^ as soon as the job finishes there I can give you access, what public IP can I use for you | 15:47 |
| jeblair | clarkb: you can use a variation on either the script i gave fungi or the zuul-changes script to skim the json for you | 15:47 |
| jeblair | pabelanger: did you see what i was talking with mordred about last night? (timeout not working) | 15:48 |
| clarkb | jeblair: I did nodepool list t oget a couple candidates then checked in zuul log for whether or not it had completed yet then held and confirm that the host doesn't go away after due to races | 15:48 |
| jeblair | it doesn't appear that any ansible fairies fixed that problem while i slept | 15:48 |
| jeblair | clarkb: that works too :) | 15:48 |
| *** andreww is now known as xarses | 15:49 | |
| fungi | jeblair: not sure if you saw earlier, but mordred is interested in having a new gear release. wasn't sure if you wanted to keep tagging those so i held off | 15:49 |
| *** Goneri has quit IRC | 15:50 | |
| thiagop | any hint on how to trigger the publish job again or should I send an e-mail? | 15:50 |
| jeblair | fungi: i saw. i approved/refreshed a couple more changes that i think can go in first. i'll tag it after it lands. | 15:50 |
| jeblair | i also agree 0.6.0 is warranted. | 15:50 |
| fungi | thiagop: which commit should have triggered it the first time? | 15:51 |
| fungi | jeblair: yep, saw some merging which is what reminded me to check | 15:51 |
| pabelanger | jeblair: Ya, I see the discussion, but haven't looked into it. I think we have the same issue today on puppetmaster.o.o. The SSH timeout values doesn't appear to work all the time and ansible-playbook will hang around long after the timeout has expired. IIRC, we see that if we power off a node doing an ansible run | 15:51 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/puppet: bring back scenario003 on master https://review.openstack.org/324627 | 15:51 |
| jeblair | pabelanger: oh, that's helpful and saddening. | 15:51 |
| clarkb | Kaiyan: I figured it out, each time you add the job template to your projects in projects.yaml you are making a new copy of that job | 15:51 |
| thiagop | fungi: tag for 2.2.1 version, this one: https://github.com/openstack/python-oneviewclient/releases/tag/2.2.1 | 15:51 |
| clarkb | Kaiyan: you only need to create the jobs once in projects.yaml since the jobs are not project specific and instead are integration jobs shared among the projects | 15:52 |
| fungi | thiagop: oh, there are no post jobs for a tag | 15:52 |
| jeblair | pabelanger: if there really is an ansible issue, we may need to have zuul watchdog ansible until we can get it sorted :/ | 15:52 |
| pabelanger | jeblair: it also happens a lot when we OOM on puppetmaster too | 15:52 |
| fungi | thiagop: is your concern that the version number didn't get updated? | 15:52 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Make is_superuser a public field for users https://review.openstack.org/324629 | 15:52 |
| *** yamahata has joined #openstack-infra | 15:52 | |
| thiagop | fungi: the publish-to-pypi job failed by timeout | 15:52 |
| *** Apoorva has joined #openstack-infra | 15:52 | |
| fungi | thiagop: got it. i can retry the upload manually in that case | 15:53 |
| thiagop | fungi: http://logs.openstack.org/72/7270afc4cbd8e0d9b8cdc0cc11c0a7ffcd9155f1/release/python-oneviewclient-tarball/261fc13/console.html | 15:53 |
| *** arxcruz has quit IRC | 15:53 | |
| fungi | thiagop: that's not the pypi upload job log | 15:53 |
| pabelanger | jeblair: I think a watchdog is a good idea. Since we are just launching the ansible-playbook binary, maybe adding timeout is something to consider? | 15:53 |
| mtreinish | jeblair: I found another weird issue with the new zuul stuff: http://logs.openstack.org/33/323233/3/check/gate-openstack-health-python27/b944ffc/console.html | 15:53 |
| *** eharney has quit IRC | 15:53 | |
| mtreinish | jeblair: this time it failed but zuul reported the job as successful | 15:53 |
| thiagop | fungi: it does build the tarball first, then upload, right? | 15:54 |
| pabelanger | jeblair: that's what we did on puppetmaster | 15:54 |
| fungi | thiagop: but i can retrigger all the jobs for that tag | 15:54 |
| fungi | thiagop: should `tox -evenv python setup.py sdist` normally run quickly for that project? | 15:54 |
| jeblair | pabelanger: well, that's about as bad an error as we can get... :| | 15:54 |
| fungi | thiagop: are your branch-tarball uploads working? | 15:54 |
| thiagop | fungi: yes for the first question, didn't understand the second | 15:55 |
| *** eharney has joined #openstack-infra | 15:55 | |
| clarkb | krotscheck: I have node ready now, I just need an ssh public key | 15:55 |
| thiagop | fungi: maybe just ignorante, what is a branch-tarball? | 15:55 |
| thiagop | ignorance* | 15:56 |
| Kaiyan | @clarkb just saw your comments! Thanks! | 15:56 |
| fungi | thiagop: the job which creates python-oneviewclient-master.tar.gz at http://tarballs.openstack.org/python-oneviewclient/ | 15:56 |
| fungi | thiagop: and it looks recent (matching your most recent merge into master) so i think so | 15:56 |
| clarkb | phschwartz: ok now that I am less distracted | 15:56 |
| *** ddieterly is now known as ddieterly[away] | 15:57 | |
| thiagop | fungi: got it | 15:57 |
| clarkb | phschwartz: there are 3 "layers" in your nodepool config for image stuff. The label which maps booted instances onto targets/jenkins, the provider images which map image objects in glance to provider flavors and credentials for booting, and the dib images which create the things to put into glance | 15:57 |
| *** matrohon has quit IRC | 15:57 | |
| fungi | thiagop: http://logs.openstack.org/ae/ae3414cd5bce512abaf30d3c7e103edec7deb88f/post/python-oneviewclient-branch-tarball/bfa43ec/console.html | 15:58 |
| *** zeih has joined #openstack-infra | 15:58 | |
| fungi | thiagop: for reference (looks like it built quickly) | 15:58 |
| clarkb | phschwartz: multiple labels can share a single provider image, multiple provider images can share a single disk image | 15:58 |
| fungi | thiagop: so anyway, i'll reenqueue the tag ref in zuul and all the release jobs should then rerun for it | 15:58 |
| *** tesseract- has quit IRC | 15:59 | |
| phschwartz | clarkb: I had 2 provider images using the same dib image and it would not work. The moment I added a new disk image it started building | 15:59 |
| clarkb | phschwartz: I believe that errors means that you are attempting to upload something called bare-trusty-lg but none of your provider images define what that is | 15:59 |
| thiagop | fungi: right, thank you very much. | 15:59 |
| clarkb | phschwartz: so nodepool doesn't know what glance to talk to and with what credentials | 15:59 |
| *** infra-red has quit IRC | 15:59 | |
| clarkb | phschwartz: ok, we definitely have shared disk images here | 15:59 |
| jeblair | pabelanger: http://paste.openstack.org/show/507326/ | 16:00 |
| jeblair | pabelanger: i wonder if the problem is that new block thing i did? | 16:00 |
| jeblair | Shrews: around ^? | 16:00 |
| phschwartz | clarkb: here is the config I was using that was failing with that error http://paste.openstack.org/show/507327/ | 16:00 |
| clarkb | we don't appear to do it currnetly though, but in the past we would make a test image with diskimage builder, then when it works flip the production image over to it too | 16:00 |
| clarkb | then sometime later delete the test image | 16:00 |
| fungi | thiagop: okay, it's in the release pipeline again | 16:01 |
| *** rcernin has quit IRC | 16:01 | |
| clarkb | phschwartz: oh I know, you probably didn't restart the builder after updating your config right? the builder doens't dynamically load new configs | 16:01 |
| strigazi | yolanda: \o/ My other rally job comleted successfully as well http://logs.openstack.org/20/309620/5/check/gate-rally-dsvm-magnum-rally-nv/b2051df/rally-plot/results.html.gz | 16:02 |
| clarkb | which had the effect of erroring with "I don't know what provider to upload this to" | 16:02 |
| phschwartz | clarkb: O.o that I did not | 16:02 |
| *** ddieterly[away] is now known as ddieterly | 16:02 | |
| jeblair | pabelanger: i guess if rescue runs successfully the play is successful? | 16:03 |
| *** zeih has quit IRC | 16:03 | |
| phschwartz | clarkb: I would expect it to auto load like nodepool does. | 16:03 |
| clarkb | infra-root: if I end up not around when krotscheck provides an ssh public key, the held node is ubuntu-xenial-rax-ord-1372303 | 16:03 |
| clarkb | phschwartz: me too, patches welcome | 16:03 |
| fungi | clarkb: thanks | 16:04 |
| *** yujunz has quit IRC | 16:05 | |
| *** nwkarsten has quit IRC | 16:06 | |
| *** e0ne has quit IRC | 16:06 | |
| *** nadya has quit IRC | 16:06 | |
| *** hrybacki is now known as hrybacki|food | 16:06 | |
| thiagop | fungi: It appears that it failed again... wth... | 16:06 |
| notmorgan | jeblair: thats odd :( | 16:07 |
| krotscheck | clarkb: Oh nice, thanks | 16:07 |
| krotscheck | Sorry | 16:07 |
| notmorgan | oh look it's a krotscheck ! | 16:07 |
| *** Kaiyan_ has joined #openstack-infra | 16:08 | |
| *** gyee has joined #openstack-infra | 16:08 | |
| EmilienM | fungi: when you have time, I proposed a new periodic job for our CI, https://review.openstack.org/#/c/324557/ - I'll let you give feedback & vote | 16:08 |
| *** ayoung has quit IRC | 16:09 | |
| *** Kaiyan has quit IRC | 16:09 | |
| clarkb | krotscheck: jenkins@104.239.192.192 | 16:09 |
| krotscheck | clarkb: Danke | 16:09 |
| fungi | thiagop: timeout again? seems too quick to have been a timeout | 16:09 |
| *** ayoung has joined #openstack-infra | 16:09 | |
| clarkb | phschwartz: its one of the lsat remaining issues with the builder that I would like to see fixed I just havne't had time to mess with it and its been semi low priority since once you reach steady ish state the number of times you need to restart hte service approaches 0 | 16:10 |
| *** nadya has joined #openstack-infra | 16:10 | |
| thiagop | fungi: the tarball job completed, but the publish couldn't get the tarball to upload to pypi | 16:10 |
| *** jlanoux_ has quit IRC | 16:10 | |
| thiagop | fungi: http://logs.openstack.org/72/7270afc4cbd8e0d9b8cdc0cc11c0a7ffcd9155f1/release/python-oneviewclient-tarball/19a27ac/console.html | 16:10 |
| *** amitgandhinz has joined #openstack-infra | 16:10 | |
| thiagop | fungi: http://logs.openstack.org/72/7270afc4cbd8e0d9b8cdc0cc11c0a7ffcd9155f1/release/python-oneviewclient-pypi-both-upload/224db92/console.html | 16:10 |
| clarkb | thiagop: that file is a 404 fwiw | 16:11 |
| jeblair | notmorgan: yeah, i'm going to try a quick repro test case | 16:11 |
| *** tphummel has joined #openstack-infra | 16:11 | |
| clarkb | I think the _ is breaking things | 16:11 |
| *** Qiming has quit IRC | 16:11 | |
| thiagop | clarkb: but it should have been built by the tarball job that suceeded, right? | 16:12 |
| *** kaisers_ has joined #openstack-infra | 16:13 | |
| fungi | thiagop: jeblair i wonder if uploads aren't working correctly from zuul-launcher? | 16:13 |
| *** breton_ is now known as breton | 16:13 | |
| clarkb | ya and comparing that log it uses _ too which I guess is canonicalized version naming | 16:13 |
| clarkb | fungi: ya I don't see anything in the log about copying the tarball | 16:13 |
| fungi | right, the filenames seem to match afaict | 16:13 |
| jeblair | fungi, thiagop: i'll check the zl logs | 16:14 |
| *** samuelBartel has quit IRC | 16:14 | |
| * thiagop thought releases were easy... until today | 16:14 | |
| fungi | http://tarballs.openstack.org/python-oneviewclient/ confirms the tarballs were not uploaded | 16:14 |
| fungi | thiagop: yeah, you've hit two seemingly unrelated bugs here | 16:14 |
| *** tmorin has quit IRC | 16:14 | |
| sambetts | Is it expected that the last released version of Zuul doesn | 16:15 |
| jeblair | under dist: http://tarballs.openstack.org/python-oneviewclient/dist/ | 16:15 |
| fungi | oho | 16:15 |
| thiagop | fungi: seeing the logs of the previous release, I saw that it has some SCP at the end to send the file... | 16:15 |
| clarkb | jeblair: its not preserving the strip path behavior of the jenkins jobs then | 16:15 |
| jeblair | maybe that was depending on the 'flatten' feature? | 16:15 |
| fungi | jeblair: i wonder if the job interpreter doesn't support the keep-hierarchy: false behavior | 16:16 |
| jeblair | fungi: well, rsync has no such thing :) | 16:16 |
| *** kushal has joined #openstack-infra | 16:16 | |
| clarkb | sambetts: you cut off at "Zuul doesn" | 16:16 |
| sambetts | Is it expected that the last released version of Zuul (2.1.0) doesn't work with the current version of gerrit we use for Depends-On processing?? | 16:16 |
| sambetts | clarkb: ^ | 16:16 |
| fungi | we do depend on that in lots of places | 16:16 |
| openstackgerrit | Ken Giusti proposed openstack-infra/project-config: Add pushMerge for oslo.messaging core for feature branches https://review.openstack.org/319452 | 16:16 |
| jeblair | fungi: so far, i hadn't seen any that mattered :) | 16:16 |
| pabelanger | jeblair: I haven't tried the new block logic yet. So not sure of the answer | 16:17 |
| *** kaisers_ has quit IRC | 16:17 | |
| fungi | jeblair: any of the scp publishers that don't specify keep-hierarchy: true were getting their file trees flattened | 16:17 |
| pabelanger | jeblair: but I would have expected ansible to have failed=1 | 16:17 |
| clarkb | sambetts: git says 2.0.0-367-g96698e2 is when we added the depends on stuff so it should work in 2.1.0 | 16:17 |
| *** vgridnev has quit IRC | 16:17 | |
| *** vgridnev has joined #openstack-infra | 16:18 | |
| jeblair | okay, there are three independent problems with zuul-launcher. | 16:18 |
| jeblair | who is available to work on them? | 16:18 |
| clarkb | sambetts: that said not sure if it would work with current version of gerrit | 16:19 |
| clarkb | sambetts: but generally the feature was in place at the time | 16:19 |
| clarkb | sambetts: specific error info or behavior would probably help | 16:19 |
| sambetts | clarkb: its not working in my setup and I've isolated the problem down to the fact the the gerrit query its making no longer returns a key 2.1.0 expects to exist in the returned data | 16:19 |
| clarkb | sambetts: I think you have answered your own question then :) | 16:20 |
| jeblair | mordred, Shrews, notmorgan, pabelanger, jhesketh, rcarrillocruz, clarkb, fungi: are any of you available to tackle a zuul-launcher problem today? | 16:20 |
| fungi | jeblair: what are the other two bugs? | 16:20 |
| clarkb | jeblair: no, I have to take twins to doctor shortly and its vaccination day so will likely be dealing with cranky infants all afternoon | 16:20 |
| sambetts | clarkb: right, but surely we should release a new version of Zuul, or are we expecting CIs to run trunk Zuul? | 16:20 |
| notmorgan | jeblair: i am here. i am getting my coffee but yes i can help | 16:20 |
| fungi | i can look into workarounds for squashing file trees before calling rsync | 16:20 |
| *** amotoki has quit IRC | 16:20 | |
| clarkb | sambetts: we CD zuul and I think generally suggest that others do too | 16:20 |
| jeblair | i'm writing a story now :) | 16:20 |
| anteaya | clarkb: sounds like a fun day planned, hope things go well | 16:20 |
| clarkb | sambetts: we try to make releases when major features like the depends on support are merged | 16:21 |
| notmorgan | no guarantees i can help, but i'm here | 16:21 |
| pabelanger | jeblair: I have some bandwidth to help | 16:21 |
| sambetts | clarkb: but thats now broken because gerrit updated | 16:21 |
| anteaya | sambetts: most third party CI operators run trunk that I talk to | 16:21 |
| sambetts | anteaya: oh? | 16:21 |
| anteaya | yes | 16:21 |
| * notmorgan ^5's pabelanger | 16:22 | |
| clarkb | sambetts: yes, I don't think anyone is arguing that, but its a non issue if you CD. We can definitely consider making a release now that we know there is an issue wit hthe previous one | 16:22 |
| notmorgan | :) | 16:22 |
| krotscheck | clarkb, fungi: Can one of you either A) give the jenkins user on 104.239.192.192 passwordless sudo, or B) figure out why process 11803 is hung? | 16:22 |
| *** amotoki has joined #openstack-infra | 16:22 | |
| * krotscheck can't strace | 16:22 | |
| *** nadya has quit IRC | 16:22 | |
| clarkb | krotscheck: there is a bad joke about karma in answering the B question | 16:22 |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config: [magnum] Use keystone v2.0 for rally job https://review.openstack.org/324651 | 16:23 |
| clarkb | krotscheck: I will readd the passwordless sudo bits | 16:23 |
| jeblair | pabelanger, notmorgan, fungi: https://storyboard.openstack.org/#!/story/2000619 | 16:23 |
| krotscheck | clarkb: Danke | 16:23 |
| sambetts | clarkb: well I guess the relase isn't broken, its just that it only supports up to a certain version of gerrit 2.9 I think, I don't know if thats documented anywhere | 16:23 |
| *** dtantsur is now known as dtantsur|afk | 16:23 | |
| *** david-lyle has quit IRC | 16:24 | |
| clarkb | krotscheck: if you log in again you should have jenkins sudo powers | 16:24 |
| fungi | sambetts: if you look in the master history since that tag there's probably a change to make zuul's commit message queries support gerrit 2.11 | 16:24 |
| clarkb | sambetts: I don't think anyone was aware of this so likely not documented | 16:24 |
| fungi | sambetts: there is likely a new zuul release coming along any day now | 16:24 |
| *** vgridnev has quit IRC | 16:24 | |
| krotscheck | MWA HA HA | 16:25 |
| fungi | sambetts: probably the change in question would have merged around december since that's when we upgraded | 16:25 |
| jeblair | pabelanger, notmorgan, fungi: i'm adding task notes to those -- there's a dark red icon for those | 16:25 |
| *** weshay is now known as weshay_brb | 16:26 | |
| notmorgan | jeblair: looking through the things. | 16:26 |
| sambetts | fungi, clarkb: makes sense, I've just been very confused all day trying to work out why Depends-On wasn't working, I could see it processing the commit message etc fine but then it just said it had no dependencies, and it was because its just dropping the gerrit responses | 16:27 |
| sambetts | seems like a backward incompatible change from the gerrit side, with only a minor version bump though :/ | 16:27 |
| *** Swami has joined #openstack-infra | 16:27 | |
| fungi | sambetts: we were likely similarly confused ~6 months ago, and probably should have tagged a point release once we worked out the gerrit 2.11 support needs | 16:27 |
| jeblair | pabelanger, notmorgan, fungi: okay, notes added to all of those, let me know if that makes sense. | 16:28 |
| fungi | sambetts: they probably don't consider that api "stable" or something | 16:28 |
| notmorgan | ok looking | 16:28 |
| *** nwkarsten has joined #openstack-infra | 16:28 | |
| *** dizquierdo has joined #openstack-infra | 16:29 | |
| sambetts | fungi: I guess not, ah well at least I know whats going on now, thanks for the insight, I'll probably update to trunk for now, thanks :D | 16:29 |
| notmorgan | jeblair: oh wow that rescue block looks icky | 16:30 |
| sambetts | anteaya: Do you know if any of the info about zuul versions etc for third party CI is documented anywhere? | 16:30 |
| jeblair | hrm | 16:30 |
| notmorgan | jeblair: it makes sense that it's a problem though | 16:30 |
| jeblair | the notes i wrote for keep-hierarchy don't seem to be there | 16:30 |
| *** timello has joined #openstack-infra | 16:30 | |
| jeblair | does anyone else see notes for that item? | 16:30 |
| notmorgan | jeblair: nope | 16:30 |
| jeblair | ugh | 16:30 |
| jeblair | that was a lot of typing | 16:30 |
| asselin_ | sambetts, currently third party ci is using master. you can pin it though | 16:30 |
| notmorgan | jeblair: :( | 16:30 |
| fungi | sambetts: i think a lot of them are using puppet-openstackci, which does continuous deployment from master | 16:30 |
| notmorgan | jeblair: sorry! | 16:30 |
| anteaya | sambetts: no it isn't, it is word of mouth based on discussions in meetings | 16:31 |
| jeblair | Zara, SotK: i just added notes to the first task in https://storyboard.openstack.org/#!/story/2000619 but they disappeared | 16:31 |
| jeblair | Zara, SotK: fungi assigned the task to himself, maybe that triggered it? | 16:31 |
| asselin_ | sambetts, http://git.gozer.hpcloud.net/cgit/openstack-infra/puppet-openstackci/tree/contrib/single_node_ci_data.yaml#n28 | 16:31 |
| *** nadya has joined #openstack-infra | 16:31 | |
| anteaya | asselin_: thank you | 16:31 |
| phschwartz | I have a job that is in a zuul pipeline and when I do an echo of status from gear it is there build:build-kernel:bare-trusty-lg 0 0 2 but when I try to run a review that should hit it, it fails with a NOT_REGISTERED. What else can I check to see why it is not seen? | 16:32 |
| *** amotoki has quit IRC | 16:32 | |
| SotK | jeblair: that sounds possible | 16:32 |
| anteaya | asselin_: that link is not rendering for me, do you need to be on the hp network to read it? | 16:32 |
| SotK | fungi: did you notice if the task had notes visible in your browser when you assigned yourself to it? | 16:32 |
| fungi | phschwartz: look in the zuul log, it may be looking for build:build-kernel:some-other-node-label | 16:33 |
| *** nwkarste_ has joined #openstack-infra | 16:33 | |
| sambetts | anteaya: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/single_node_ci_data.yaml#n28 <- same data I guess | 16:33 |
| anteaya | sambetts: great | 16:33 |
| asselin_ | anteaya, sambetts yes, sorry internal mirror. try this: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/single_node_ci_data.yaml#n28 | 16:33 |
| fungi | SotK: there wern't any at the time, no. i think i assigned it to myself before he started adding notes | 16:33 |
| anteaya | asselin_: thanks | 16:33 |
| anteaya | a race perhaps? | 16:33 |
| SotK | I expect it was a race, yeah | 16:34 |
| sambetts | thanks everyone, I need to have a chat with my team, and possibly look into switching to puppet-openstackci :) | 16:34 |
| *** docaedo2 has quit IRC | 16:34 | |
| *** nwkarsten has quit IRC | 16:34 | |
| anteaya | SotK: any way of confirming via logs, do you think? | 16:34 |
| asselin_ | sambetts, docs are here: http://docs.openstack.org/infra/openstackci/ | 16:35 |
| *** shashank_hegde has joined #openstack-infra | 16:35 | |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config: [magnum] Use keystone v2.0 for rally job https://review.openstack.org/324651 | 16:35 |
| fungi | jeblair: SotK: anteaya: i'm happy to pull logs around that to confirm what order they happened in | 16:35 |
| anteaya | that would be awesome, thank you | 16:36 |
| *** Goneri has joined #openstack-infra | 16:36 | |
| *** johnny has joined #openstack-infra | 16:36 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul: Conditionally install build-essential https://review.openstack.org/324581 | 16:36 |
| jeblair | fungi: i re-wrote the task notes | 16:36 |
| SotK | the events timeline on the story suggests it was a race | 16:36 |
| *** johnny is now known as Guest1281 | 16:36 | |
| jeblair | fungi: though i suspect you understand the problem, i did throw out some brainstormy solution ideas | 16:36 |
| *** nadya has quit IRC | 16:37 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:37 | |
| *** nwkarste_ has quit IRC | 16:37 | |
| *** nadya has joined #openstack-infra | 16:37 | |
| clarkb | krotscheck: let us know when you are done with that instance so it can be deleted. Curious to know what you find too | 16:37 |
| *** nwkarsten has joined #openstack-infra | 16:38 | |
| krotscheck | clarkb: Indeed. Righ tnow I'm installing nodejs-dbg, because strace really just told me that it's hung on epoll_wait | 16:38 |
| jeblair | notmorgan, pabelanger: either of you want to claim the success-on-rescue task? | 16:38 |
| notmorgan | jeblair: looking at that one now. | 16:38 |
| notmorgan | i'll claim it. | 16:38 |
| krotscheck | Somewhere during the coverage-report generation process, which annoys me. | 16:38 |
| jeblair | notmorgan: cool | 16:38 |
| notmorgan | unless i get reallllly stuck :) | 16:38 |
| *** weshay_brb is now known as weshay | 16:38 | |
| jeblair | pabelanger: either of the timeout tasks sound fun to you? | 16:39 |
| jeblair | notmorgan: (are you going to assign it to yourself in storyboard, or should i? | 16:39 |
| notmorgan | jeblair: i just was logging in | 16:40 |
| jeblair | cool | 16:40 |
| strigazi | hi notmorgan, thoughts on my comments on https://review.openstack.org/#/c/324651/2 | 16:40 |
| jeblair | SotK, anteaya: is there a way to add all the tasks in a story to a board? | 16:40 |
| notmorgan | jeblair: found a new bug in storyboard where logging in drops you back to the main page not where you were (so had to re-find the task) | 16:40 |
| notmorgan | strigazi: i think that is the wrong approach tbh. i would drive at figuring out why there is a timeout over merging that. | 16:41 |
| anteaya | not yet I don't think, you could try creating an automatic worklist and see if that catches all the tasks within a story | 16:41 |
| *** nadya has quit IRC | 16:41 | |
| anteaya | jeblair: do you know automatic worklists yet? | 16:42 |
| notmorgan | strigazi: just a personal opinion. moving to 2.0 as a "fix" isn't really great :( | 16:42 |
| pleia2 | anteaya: btw, updated the etherpad with the 2nd project rename (we have a change now) | 16:42 |
| jeblair | anteaya: no | 16:42 |
| jeblair | anteaya: i'm exploring that now | 16:42 |
| SotK | jeblair: if you click "add card" and change "tasks" to "stories", then search for the story title, you can expand the task list and add them all | 16:42 |
| anteaya | pleia2: yay thank you | 16:42 |
| *** nadya has joined #openstack-infra | 16:42 | |
| anteaya | SotK: ah wonderful | 16:42 |
| jeblair | SotK: ok, if i do that, and add new tasks, i'll have to add new cards, right? | 16:42 |
| notmorgan | strigazi: i mean i can't force you to do v3 only, but i worry this is yet-another-case of "oh well this didn't work, back to v2" and we don't get directed feedback/work to find what is wrong. | 16:42 |
| notmorgan | strigazi: if that makes sense. | 16:43 |
| SotK | jeblair: yeah | 16:43 |
| *** shashank_hegde has quit IRC | 16:43 | |
| notmorgan | strigazi: like i said, i can't force it. but i would encourage strongly to figure out why there is a timeout over changing to v2.0 (and work with the folks in #openstack-keystone ) so we can keep things moving forward. | 16:44 |
| *** sdake has joined #openstack-infra | 16:44 | |
| pabelanger | jeblair: I can try my hand at timeout if zuul_runner hangs | 16:44 |
| strigazi | notmorgan: It's not a fix. It will help to invetigate the problem. In magnum we use v3 and in our production env as well | 16:44 |
| *** infra-red has joined #openstack-infra | 16:44 | |
| *** fedexo has quit IRC | 16:44 | |
| *** tesseract has joined #openstack-infra | 16:44 | |
| *** fedexo has joined #openstack-infra | 16:45 | |
| notmorgan | strigazi: right. this is exactly why chasing the timeout down is the right course of action. | 16:45 |
| anteaya | SotK: thanks I was able to get that to work for me: https://storyboard-dev.openstack.org/#!/worklist/12 | 16:46 |
| *** EricGonczer_ has joined #openstack-infra | 16:46 | |
| anteaya | pleia2: do you have the link for tomorrow's rename etherpad handy? | 16:46 |
| *** roxanaghe has joined #openstack-infra | 16:47 | |
| jeblair | anteaya: do i still need to add users to boards? | 16:47 |
| jeblair | https://storyboard.openstack.org/#!/board/25 | 16:47 |
| jeblair | like, that's hidden? | 16:47 |
| *** claudiub|2 has joined #openstack-infra | 16:48 | |
| *** nadya_ has joined #openstack-infra | 16:48 | |
| anteaya | jeblair: I can see that board but I can't edit it | 16:48 |
| jeblair | oh that's how it works | 16:48 |
| *** jogo has quit IRC | 16:48 | |
| anteaya | jeblair: so you have to add users if you want them to be able to create a new list and so on | 16:48 |
| pleia2 | anteaya: https://etherpad.openstack.org/p/repo-renaming-2016-06-03 | 16:48 |
| anteaya | pleia2: thanks | 16:48 |
| pleia2 | thanks for reviewing it :) | 16:49 |
| *** nadya has quit IRC | 16:49 | |
| JayF | https://review.openstack.org/#/c/313800/ I'd love to get core reviews on this; it's been outstanding a long time, and already has +1s from 2 cores and the PTL of Ironic | 16:49 |
| anteaya | pleia2: thanks for creating it | 16:49 |
| *** flepied has quit IRC | 16:49 | |
| lucasagomes | hi all, can I get some eyes on https://review.openstack.org/#/c/315646/ please? Already has a +2 and been there since May 12th (I've been rebasing it along) | 16:49 |
| anteaya | jeblair: yeah so your board team can use the board everyone else can view it | 16:49 |
| SotK | jeblair: if you want one that can't be seen by people who aren't users/owners for whatever reason, tick the `private` box | 16:50 |
| lucasagomes | thanks in advance | 16:50 |
| jeblair | anteaya: thx | 16:50 |
| JayF | lucasagomes: yours will conflict with mine | 16:50 |
| JayF | lucasagomes: so we should pick one to be in front | 16:50 |
| anteaya | SotK: which gets me to thinking, what happens if someone creates a worklist of private stories? | 16:50 |
| JayF | lucasagomes: and maybe just form a chain together? | 16:50 |
| lucasagomes | JayF, oh, urs is more important I think | 16:50 |
| lucasagomes | JayF, I can rebase on top of it | 16:50 |
| jeblair | clarkb, fungi, notmorgan, pabelanger, mordred, Shrews: https://storyboard.openstack.org/#!/board/25 | 16:50 |
| *** sambetts is now known as sambetts|afk | 16:50 | |
| openstackgerrit | Paul Hummer proposed openstack-infra/project-config: Add nova-lxd to openstack https://review.openstack.org/324667 | 16:50 |
| JayF | I wasn't going to say that :) but I'm glad you did | 16:50 |
| anteaya | SotK: can you make a worklist and board private? | 16:50 |
| JayF | lol | 16:50 |
| lucasagomes | JayF, hah fair | 16:50 |
| SotK | anteaya: you can | 16:50 |
| lucasagomes | JayF, ok, lemme do it | 16:50 |
| anteaya | SotK: awesome thank you | 16:50 |
| openstackgerrit | Merged openstack-infra/project-config: Setup OSA os_aodh core group https://review.openstack.org/323815 | 16:50 |
| *** yaume has quit IRC | 16:51 | |
| SotK | board permissions should override worklist permissions for worklists which are in boards though | 16:51 |
| *** jogo has joined #openstack-infra | 16:51 | |
| *** roxanaghe has quit IRC | 16:51 | |
| SotK | and if you end up with private stories in a public board, people who can't see those stories should just not see the cards representing them in the board | 16:51 |
| *** julim has quit IRC | 16:51 | |
| jeblair | fungi: i'm going to pull zuul-launcher out of service since we have identified a false-result bug | 16:51 |
| *** tesseract has quit IRC | 16:51 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add timestamps to logs for indexing in logstash https://review.openstack.org/320443 | 16:52 |
| *** fedexo has quit IRC | 16:52 | |
| fungi | jeblair: yep, that's prudent | 16:52 |
| fungi | jeblair: i'll retrigger thiagop's release again and clean up the errant dists directory for now | 16:52 |
| anteaya | pleia2: in step 2 didn't their used to be an action for things that used storyboard? like ensure that old names will be cleaned up in storyboard? | 16:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Revert "Revert "Revert "Revert "Revert "Start using the Zuul Ansible launcher""""" https://review.openstack.org/324668 | 16:53 |
| openstackgerrit | Merged openstack-infra/project-config: Setup OSA os_cloudkitty core group https://review.openstack.org/323813 | 16:53 |
| *** EricGonczer_ has quit IRC | 16:53 | |
| *** EricGonc_ has joined #openstack-infra | 16:53 | |
| anteaya | SotK: thank you | 16:53 |
| openstackgerrit | Merged openstack-infra/project-config: Setup OSA os_sahara core group https://review.openstack.org/323811 | 16:53 |
| pleia2 | anteaya: these projects don't use storyboard, so I didn't include it | 16:53 |
| anteaya | SotK: perhaps that would be a good thing for tristanC to test, what happens if you have private stories on a public board? | 16:53 |
| anteaya | pleia2: ah okay, can you link to where you drew the instructions from? | 16:54 |
| openstackgerrit | Merged openstack-infra/project-config: Setup OSA os_zaqar core group https://review.openstack.org/323806 | 16:54 |
| anteaya | pleia2: I just go by last times etherpad myself to generational shift is something I watch for | 16:54 |
| pleia2 | anteaya: the last one, https://etherpad.openstack.org/p/repo-renaming-2016-02-12 | 16:54 |
| anteaya | s/to/so | 16:54 |
| openstackgerrit | Merged openstack-infra/project-config: Setup OSA os_ceilometer core group https://review.openstack.org/323823 | 16:54 |
| krotscheck | Whaa.... when did tarball upload start uploading to the dist/ dir rather than the project dir? | 16:55 |
| *** regXboi has joined #openstack-infra | 16:55 | |
| *** Kaiyan_ has quit IRC | 16:55 | |
| anteaya | pleia2: right which is why I am concerned about the generational shift, I don't want the storyboard steps to get lost for when we do need them | 16:55 |
| pleia2 | anteaya: I adjusted it a bit to be more clear, so we don't miss disabling ansible/puppet early again :) | 16:55 |
| jeblair | krotscheck: it had a 1/8 chance of doing it since yesterday afternoon | 16:55 |
| anteaya | pleia2: that is good | 16:55 |
| pleia2 | I looked at the one from September too | 16:55 |
| fungi | krotscheck: that's a zuul-launcher bug i'm working on fixing now | 16:55 |
| krotscheck | jeblair: Just my luck... http://tarballs.openstack.org/eslint-config-openstack/ | 16:55 |
| jeblair | krotscheck: fungi is performing a similar cleanup right now | 16:55 |
| pleia2 | happy to add back the unneeded storyboard step if that's better | 16:55 |
| *** cbader has joined #openstack-infra | 16:55 | |
| anteaya | I think my point is I have no frame of reference for instructions taht include storyboard if we leave them off this etherpad | 16:56 |
| pleia2 | ok | 16:56 |
| krotscheck | Ok, I'll leave y'all alone until it's fixed. | 16:56 |
| anteaya | thanks | 16:56 |
| anteaya | just strikethrough them after they are added | 16:56 |
| anteaya | thanks | 16:56 |
| jeblair | anteaya: we have documentation for reference | 16:56 |
| anteaya | jeblair: yes, that was what I was thinking we have | 16:56 |
| fungi | krotscheck: do you need any jobs depending on that tarball retriggered, or do you just want the tarball copied into the parent dir? | 16:56 |
| pleia2 | anteaya: oh, it is there, step 15 | 16:57 |
| jeblair | anteaya: i mean http://docs.openstack.org/infra/system-config/gerrit.html#renaming-a-project | 16:57 |
| *** nadya_ has quit IRC | 16:57 | |
| anteaya | http://docs.openstack.org/infra/system-config/gerrit.html#renaming-a-project ? | 16:57 |
| *** eharney has quit IRC | 16:57 | |
| anteaya | jeblair: great thank you :) | 16:57 |
| notmorgan | jeblair: it looks like the error block needs an explicit false return code (possibly) chasing down specifics internally now on how to do that right. | 16:57 |
| krotscheck | fungi: Well, should I fix the npm-publish-tarball to download from dist first? I'd like to know what the eventual solution will be before I ask for any retriggers. | 16:57 |
| *** EricGonczer_ has joined #openstack-infra | 16:58 | |
| krotscheck | fungi: (The answer is- yes I need a retrigger, but until the preconditions are valid, there's no point) | 16:58 |
| fungi | krotscheck: no, that dist directory is a bug (for now anyway) | 16:58 |
| openstackgerrit | Merged openstack-infra/project-config: Setup OSA security core group https://review.openstack.org/323825 | 16:58 |
| fungi | krotscheck: i'll reenqueue your tag then and we'll get a new tarball, and i'll blow that dist subtree away in the meantime | 16:58 |
| openstackgerrit | Merged openstack-infra/project-config: Setup OSA os_designate core group https://review.openstack.org/323830 | 16:58 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing Openstack Juju charm https://review.openstack.org/323716 | 16:58 |
| krotscheck | fungi: Gotcha. In that case, yes- I need the content of the dist directory copied into parent, and then eslint-config-openstack-npm-upload retriggered on the release slave. | 16:58 |
| pleia2 | oops, I'm in trouble, I used one of my nephew's jurassic park band-aids | 16:59 |
| *** EricGonc_ has quit IRC | 16:59 | |
| krotscheck | Oh good | 16:59 |
| openstackgerrit | Merged openstack-infra/project-config: Add tempest charm project to Juju Charms https://review.openstack.org/324096 | 16:59 |
| krotscheck | fungi: That works too. | 16:59 |
| openstackgerrit | Merged openstack-infra/project-config: Remove kilo bitrot job definition and usage https://review.openstack.org/324091 | 16:59 |
| notmorgan | jeblair: is it safe to assume ansible 2.0? or | 16:59 |
| SotK | anteaya: sure | 16:59 |
| *** prometheanfire has joined #openstack-infra | 16:59 | |
| fungi | pleia2: are the collectibles? | 16:59 |
| notmorgan | jeblair: because i think 2.0 handles some of this differently internally. | 16:59 |
| anteaya | pleia2: oh no | 16:59 |
| pleia2 | fungi: no, but he's three, so much "omg my band-aid!" :) | 16:59 |
| anteaya | I got star trek postage stamps yesterday | 17:00 |
| jeblair | notmorgan: yes | 17:00 |
| pleia2 | anteaya: nice | 17:00 |
| anteaya | not sure if they make star trek band-aids | 17:00 |
| anteaya | pleia2: I was happy | 17:00 |
| notmorgan | jeblair: ok cool. let me find the correct way to exit with a false status (hopefully without calling out to shell: /bin/false | 17:00 |
| *** lucasagomes is now known as lucas-afk | 17:00 | |
| *** EricGonc_ has joined #openstack-infra | 17:00 | |
| anteaya | fungi: did the storyboard logs produce anything of value? | 17:00 |
| notmorgan | jeblair: but that appears to be why error block (rescue) is causing success | 17:00 |
| jeblair | notmorgan: if it gets to that point, we control the zuul_log module which is the only reason we use that -- we could pass in an argument to tell it how to exit | 17:01 |
| jeblair | notmorgan: (though i would prefer a more ansibly way of doing it if there is one) | 17:01 |
| fungi | anteaya: apparently they at least did make star trek band-aids at one time https://www.etsy.com/listing/109638500/vintage-star-trek-collectible-band-aid | 17:01 |
| *** eharney has joined #openstack-infra | 17:01 | |
| fungi | anteaya: and no, i didn't check the logs because SotK indicated that the event timeline for the story was sufficient to indicate what order we'd done things in | 17:02 |
| notmorgan | jeblair: oh absolutely, i'm going to see if i have a nice | 17:02 |
| *** PollyZ_ has quit IRC | 17:02 | |
| notmorgan | "heyyyy this failed" mechanism intetrnally to ansible | 17:02 |
| anteaya | fungi: wow, how could they have stopped production on those? | 17:02 |
| openstackgerrit | Merged openstack-infra/project-config: layout: cleanup redundant rules https://review.openstack.org/324159 | 17:02 |
| *** EricGonczer_ has quit IRC | 17:02 | |
| anteaya | fungi: ah okay thank you | 17:02 |
| anteaya | SotK: you have what you need then to tackle addressing the race? | 17:03 |
| *** esikachev has quit IRC | 17:03 | |
| SotK | anteaya: I do | 17:03 |
| anteaya | SotK: marvelous, thank you | 17:03 |
| *** esikachev has joined #openstack-infra | 17:04 | |
| *** penguinolog has quit IRC | 17:05 | |
| * SotK confirms that it is indeed a race to begin with | 17:05 | |
| *** pvaneck has joined #openstack-infra | 17:05 | |
| anteaya | SotK: oh, good start | 17:06 |
| *** hrybacki|food is now known as hrybacki | 17:07 | |
| fungi | jeblair: as luck would have it, thiagop and krotscheck were the only ones to tag releases of things which got bit by the keep-hierarchy issue (everything else needing cleanup is just branch tarballs) | 17:07 |
| anteaya | pleia2: do you want to take over andeas' -2 on this patch? https://review.openstack.org/#/c/299192/6 | 17:07 |
| jeblair | fungi: whew, thanks | 17:07 |
| pabelanger | jeblair: I think I could a job from zl01 that is hung | 17:07 |
| pabelanger | jeblair: nc 66.187.229.31 8088 | 17:07 |
| pabelanger | jeblair: job fails with 'exit 137' but looks like zl01 is still polling the server | 17:08 |
| krotscheck | clarkb: Current working theory is that ulimit -n == 1024 is a problem. | 17:08 |
| pleia2 | anteaya: done | 17:08 |
| anteaya | pleia2: this patch does use storyboard: https://review.openstack.org/#/c/324273/1/gerrit/projects.yaml | 17:08 |
| *** esikachev has quit IRC | 17:09 | |
| anteaya | pleia2: so we have to ensure the old project name is cleaned up | 17:09 |
| fungi | thiagop: your release is up now https://pypi.python.org/pypi/python-oneviewclient | 17:09 |
| pleia2 | anteaya: ah, interesting! | 17:09 |
| anteaya | and thank you | 17:09 |
| jeblair | pabelanger: hrm, it ran the final "job complete" task; that's weird (the one i saw ended with 'task exit code' but no 'job complet' | 17:09 |
| *** ayoung has quit IRC | 17:09 | |
| anteaya | pleia2: we haven't been good at cleaning up old project names on storyboard in the past, which is why I am looking to change that this time | 17:09 |
| pabelanger | jeblair: okay, so it just cleaned up | 17:09 |
| pleia2 | anteaya: can you update the etherpad with that? | 17:09 |
| pleia2 | great | 17:09 |
| anteaya | pleia2: sure, do you recall what etherpad last had the storyboard steps included on it? | 17:10 |
| pabelanger | jeblair: okay, I think this is an issue with the triplo-ci code | 17:10 |
| jeblair | pabelanger: or actually, maybe not even that far -- maybe like this one: nc 63.251.114.228 8088 | 17:10 |
| pabelanger | not zl01 | 17:10 |
| anteaya | pleia2: I can go looking, was just wondering if you remembered | 17:10 |
| jeblair | pabelanger: how so? | 17:11 |
| anteaya | pleia2: I'll find it | 17:11 |
| pleia2 | anteaya: I can look, not the September one | 17:11 |
| *** jerryz has joined #openstack-infra | 17:11 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/bindep: Add Archlinux support https://review.openstack.org/324716 | 17:11 |
| anteaya | pleia2: I'll look and skip september, thanks | 17:12 |
| pabelanger | jeblair: you connected just at the right time to see the job finish. If you look at the timestamp on 'exit 137' and the following command, that was the delay I was seeing | 17:12 |
| notmorgan | jeblair: so, it looks like the easiest way is going to be making the zuul_log module fail explicitly if a failure param is passed | 17:12 |
| mtreinish | fungi: ^^^ I'm not sure that's something we care about | 17:12 |
| jeblair | pabelanger: nc 158.69.82.205 8088 is another | 17:12 |
| fungi | this is weird... http://git.openstack.org/cgit/openstack/eslint-config-openstack shows a 2.0.0 tag but git remote update isn't fetching it | 17:12 |
| notmorgan | jeblair: it seems to be the "ansible" way. | 17:12 |
| mtreinish | but I was hitting errors trying to run it on my machines | 17:12 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Filter non-public fields in users when updating https://review.openstack.org/324717 | 17:12 |
| jeblair | pabelanger: ah, yeah, it's another ansible task, so it's slow | 17:12 |
| notmorgan | jeblair: afaict. unless we shell to bin/false (useful if doing yaml-y-things directly) | 17:12 |
| fungi | mtreinish: it might be, though i have a feeling it's untestable in our ci without archlinux workers | 17:12 |
| jeblair | pabelanger: but 63.251.114.228 and 158.69.82.205 look like the thing i was seeing | 17:12 |
| jeblair | pabelanger: they both have stopped running zuul_runner, but ansible is still running | 17:13 |
| fungi | mtreinish: you wouldn't happen to want to hack on that next? ;) | 17:13 |
| mtreinish | fungi: I could, is there a doc on how to set that up? | 17:13 |
| jeblair | notmorgan: okay, sounds good. you want to write that patch? | 17:13 |
| mtreinish | if there's an easy guide I can chug away in my spare time | 17:13 |
| mtreinish | assuming I have any :) | 17:13 |
| notmorgan | jeblair: yep, starting to work on it now. | 17:14 |
| fungi | mtreinish: not really, step #1 is getting a working element for it in diskimage-builder | 17:14 |
| notmorgan | jeblair: going to make "fail" an optional param that just calls out module.fail_json instead of .exit_json | 17:14 |
| fungi | mtreinish: the rest is fairly simple once that's done | 17:14 |
| jeblair | notmorgan: ++ | 17:14 |
| rcarrillocruz | jeblair: not really here, out of town till Monday | 17:14 |
| mtreinish | fungi: ok, I'll put that on the list of things to look at | 17:14 |
| fungi | mtreinish: though #2 (or #0.5?) is probably getting archilinux support into glean | 17:14 |
| jeblair | rcarrillocruz: ok, we'll have fun without you :) | 17:14 |
| jeblair | pabelanger: 149.202.177.22 is another | 17:15 |
| rcarrillocruz | Happy to help next week if you have not solved it by then :/ | 17:15 |
| fungi | mtreinish: er, archlinux | 17:15 |
| jeblair | rcarrillocruz: if so, we'll find something new for you :) | 17:15 |
| *** AJaeger has joined #openstack-infra | 17:15 | |
| mtreinish | fungi: I think we need to fork archlinux and make archilinux, it's a better name | 17:15 |
| rcarrillocruz | ++ | 17:16 |
| *** e0ne has joined #openstack-infra | 17:16 | |
| fungi | mtreinish: archie linux | 17:17 |
| *** tonytan4ever has quit IRC | 17:17 | |
| fungi | mtreinish: the archie ftp search tool should play a predominant role | 17:17 |
| mtreinish | haha | 17:18 |
| *** penguinolog has joined #openstack-infra | 17:18 | |
| *** trown is now known as trown|lunch | 17:18 | |
| notmorgan | jeblair: sooo question on tagging the task in the commit message... i haven't done a storyboard managed bug fix yet | 17:18 |
| *** savihou has quit IRC | 17:18 | |
| mtreinish | fungi: does any other project use pylibmc? I figured having the libmemcached headers would be enough: http://logs.openstack.org/94/324094/7/check/gate-openstack-health-docs/e232fd6/console.html | 17:18 |
| notmorgan | jeblair: is it the same as LP? witht he ID? or? | 17:18 |
| fungi | mtreinish: check codesearch.o.o? | 17:18 |
| mtreinish | why can't they just package wheels, sigh | 17:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix translation proposals https://review.openstack.org/324723 | 17:18 |
| notmorgan | fwiw, i think i have this fix ready. | 17:19 |
| *** sputnik13 has joined #openstack-infra | 17:19 | |
| jeblair | notmorgan: i'm not sure where we are on that, though i don't think we have automation. I'd just do "Story: ###" for now unless someone has better info | 17:19 |
| notmorgan | jeblair: WFM :) | 17:19 |
| mtreinish | fungi: everyone else does it optionally it's not in their requirements.txt | 17:19 |
| fungi | jeblair: notmorgan: or do the normal bug headers and just ignore that they're going to link to something invalid/incorrect in lp for now | 17:19 |
| notmorgan | fungi: ok cool, so it's tagged to the story not the task? | 17:20 |
| fungi | mtreinish: that's disappointing | 17:20 |
| fungi | mtreinish: maybe there's a fork of it with better packaging? | 17:20 |
| notmorgan | fungi: i think i like Story: #### personally | 17:20 |
| notmorgan | fungi: for now. | 17:21 |
| fungi | notmorgan: we've been doing the story number, mainly because that's what we imported from lp, but in the future our automation may be better off keying on tasks (so that a change can update a specific task in a multi-task story) | 17:21 |
| *** derekh has quit IRC | 17:21 | |
| AJaeger | Hi team, is there any urgent work that needs help? or is everything fine? | 17:22 |
| persia | So, while "Story: ####" is appealing, there aren't any restrictions on a story containing two tasks against the same project. | 17:22 |
| mtreinish | fungi: not that I've found. I could try one of the other memcached connectors I guess | 17:22 |
| persia | So there would need to be some task-id based key. | 17:22 |
| *** ifarkas has quit IRC | 17:22 | |
| mtreinish | I don't remember why I used pylibmc | 17:22 |
| fungi | AJaeger: are you looking for excuses to not be on vacation? | 17:22 |
| AJaeger | fungi, vacation ended on Monday - and I'm slowly getting up to speed again. Took me a bit longer than expected... | 17:23 |
| fungi | AJaeger: i think we have stuff mostly under control at the moment anyway, so take your time catching back up | 17:23 |
| AJaeger | fungi, glad to hear! | 17:23 |
| JayF | clarkb: sdague: fungi: I've been told you all hate bad Ironic CI. https://review.openstack.org/#/c/313800/ is one of the changes we're doing to make it faster and more reliable, I'd really appreciate core reviews on it if possible. | 17:23 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Filter non-public fields when searching for users https://review.openstack.org/324725 | 17:23 |
| AJaeger | in that case, I'd like to ask project-config cores for reviews of https://review.openstack.org/#/c/324723/ and https://review.openstack.org/#/c/324259/ - two fixes for translation jobs that I noticed while checking current status | 17:24 |
| *** e0ne has quit IRC | 17:25 | |
| AJaeger | team, are you aware that according to http://status.openstack.org/zuul/ networking-sfc is since 153 hours (!) in the post queue waiting for networking-sfc-docs to finish? | 17:26 |
| anteaya | AJaeger: here there | 17:26 |
| *** rbrndt has quit IRC | 17:26 | |
| anteaya | pleia2: done my pass on the etherpad, nice job | 17:26 |
| pleia2 | anteaya: thank you | 17:26 |
| anteaya | thank you! | 17:26 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Call :func:`AnsibleModule.fail_json` on failure https://review.openstack.org/324726 | 17:26 |
| anteaya | AJaeger: when you have a minute can you remove your -2 from: https://review.openstack.org/#/c/299192/6 | 17:27 |
| notmorgan | jeblair, Shrews, pabelanger: ^ I *think* that will do it. (no tests yet, passes pep8) | 17:27 |
| anteaya | AJaeger: pleia2 has it covered with her -2 | 17:27 |
| fungi | no clue why remote update wasn't finding the 2.0.0 tag on eslint-config-openstack, but i blew it away and recloned from the same origin and it's there now | 17:27 |
| *** Daisy has joined #openstack-infra | 17:27 | |
| jeblair | notmorgan: yeah, we have no tests of ansible launcher right now (which i think is ok for the moment) | 17:27 |
| krotscheck | clarkb: Looks like it's definitely file descriptor limits. Xenial's default is 1024, which is a bit low anyway. Which would you prefer- patch to system-config to update it globally, or projects-config to somehow reload that during the build.... | 17:28 |
| krotscheck | fungi: ^^ | 17:28 |
| jeblair | pabelanger: i just noticed that ansible async logs to syslog, on the worker node -- you might see if there are any clues there | 17:29 |
| *** sputnik13 has quit IRC | 17:29 | |
| AJaeger | anteaya: on my way... Happy to pass on the -2 ;) | 17:29 |
| krotscheck | ALso, this might be the same root cause of the npm install issues. | 17:29 |
| pabelanger | jeblair: ya, I am still tracing the logs on 63.251.114.228 | 17:29 |
| pabelanger | jeblair: it would be nice if our subprocess.Popen calls did live output vs waiting for the process to finish | 17:30 |
| *** e0ne has joined #openstack-infra | 17:30 | |
| *** shashank_hegde has joined #openstack-infra | 17:30 | |
| pabelanger | curious to see which task is hung | 17:30 |
| jeblair | pabelanger: i'm starting to think so too | 17:30 |
| anteaya | AJaeger: thank you | 17:30 |
| anteaya | AJaeger: nice to see you back | 17:30 |
| AJaeger | thanks, anteaya | 17:30 |
| clarkb | krotscheck: probably fine to bake into images if kept to a reasonable 4k or 8k | 17:30 |
| anteaya | AJaeger: did you have a nice vacation? | 17:30 |
| krotscheck | clarkb: I'll switch to 4K. Anything larger and we can yell at the project to stop opening so many files. | 17:31 |
| *** flepied has joined #openstack-infra | 17:31 | |
| notmorgan | jeblair: i figure we just will replicate the error message to .fail_json, it seems "correct" | 17:31 |
| *** Daisy has quit IRC | 17:32 | |
| notmorgan | fungi: ok i can just do bugid: | 17:32 |
| *** ddieterly is now known as ddieterly[away] | 17:32 | |
| notmorgan | fungi: easy to edit the thigns | 17:32 |
| jeblair | pabelanger: i suspect the zuul_runner task based on my investigation last night | 17:32 |
| *** amotoki has joined #openstack-infra | 17:32 | |
| jeblair | pabelanger: (even though it has actually been killed) | 17:32 |
| notmorgan | persia: we need a way to identify specific tasks imo | 17:33 |
| AJaeger | anteaya: it was great and relaxing! Had everything from snow to warm sunshine in the Dolomites - with some nice walks and action with my family! | 17:33 |
| anteaya | oh lovely | 17:33 |
| anteaya | that sounds divine | 17:33 |
| anteaya | I'm so glad you and your family had a nice relaxing time | 17:33 |
| anteaya | :) | 17:33 |
| notmorgan | persia: which i don't see an easy way (as in tasks are not clearly identifiable) | 17:33 |
| notmorgan | persia: i'd like something like "STORY-TASK: ####" but yeah :( | 17:34 |
| notmorgan | persia: not there yt | 17:34 |
| notmorgan | yet* | 17:34 |
| persia | notmorgan: There was a patch to show the task ID on patches, but I think it was ugly. I suspect that something like that will land before we can usefully have automation. | 17:34 |
| AJaeger | anteaya: thanks. And it was pre-tourist season, so we were sometimes the only ones walking - more marmots than people ;) | 17:34 |
| persia | Yes, not yet :) | 17:34 |
| notmorgan | persia: ++ | 17:34 |
| anteaya | jeblair notmorgan fungi yes so right now we are still using Story: <story-id> for commit messages | 17:35 |
| anteaya | at the last storyboard mid-cycle we agreed that we wanted to move to task-id for gerrit integration | 17:35 |
| notmorgan | anteaya: ok cool so i'm following current "best practices" | 17:36 |
| notmorgan | anteaya: i did "partial-story: ####" | 17:36 |
| anteaya | which Zara is working on implementing, currently she has created a bot script and is now looking at the gerrit storyboard plugin | 17:36 |
| anteaya | notmorgan: well we never had a partial-story tag, just story: story-d | 17:36 |
| anteaya | story-id | 17:36 |
| anteaya | so you are making something up | 17:36 |
| *** csomerville has quit IRC | 17:37 | |
| *** sputnik13 has joined #openstack-infra | 17:37 | |
| notmorgan | anteaya: hehe, woo, i like making things up | 17:37 |
| anteaya | but yes, the current best practice is Story: <story-id> | 17:37 |
| notmorgan | anteaya: anyone can use | 17:37 |
| notmorgan | "current" ideas | 17:37 |
| notmorgan | :P | 17:37 |
| notmorgan | anteaya: ^_^ | 17:37 |
| *** derekh has joined #openstack-infra | 17:37 | |
| anteaya | notmorgan: you are great at that :) | 17:37 |
| anteaya | and yes having something more fine grained and trackable is the goal and we are working on it | 17:37 |
| notmorgan | jeblair: i am not sure if we need an else block. I *think* fail_json exits. | 17:38 |
| *** amotoki has quit IRC | 17:38 | |
| krotscheck | clarkb: I'm done with the node, thanks! | 17:38 |
| anteaya | currently anyone who feels like helping get the storyboard gerrit plugin working on the review-dev server would really help progress here as Zara can then just figure out how to get the plugin working | 17:38 |
| anteaya | rather than figure out how to stand up a gerrit | 17:38 |
| notmorgan | anteaya: i have some things to file about usability around storyboard from today and if i learn enough JS i'll try and fix too. | 17:39 |
| anteaya | notmorgan: awesome thank you | 17:39 |
| jeblair | notmorgan: yeah, see zuul_runner for reference | 17:39 |
| anteaya | notmorgan: do you want to toss your thoughts in an etherpad first so I can see if there are any in progress stories that cover your discoveries? | 17:39 |
| *** gyee has quit IRC | 17:40 | |
| anteaya | notmorgan: or you can just make stories, your call | 17:40 |
| *** electrofelix has quit IRC | 17:40 | |
| notmorgan | jeblair: ah ok | 17:40 |
| *** amitgandhinz has quit IRC | 17:41 | |
| notmorgan | anteaya: it's things like "notes" should be expandable from the task, not just a modal | 17:41 |
| notmorgan | anteaya: so i can see notes for all tasks in a story at once (if I want) | 17:41 |
| anteaya | ah yes, we have discussed that | 17:41 |
| notmorgan | or see it w/o "edit" rights to the box (no accidental changes) | 17:41 |
| *** amitgandhinz has joined #openstack-infra | 17:41 | |
| anteaya | how far should notes go | 17:41 |
| *** Na3iL has quit IRC | 17:42 | |
| *** tonytan4ever has joined #openstack-infra | 17:42 | |
| notmorgan | anteaya: and a bug - login takes me to the home page, not back to where i was | 17:42 |
| anteaya | if you are not logged in to storyboard you can see the notes without being able to edit the notes | 17:42 |
| notmorgan | anteaya: so the redirect isn't being propagated to the OID login. | 17:42 |
| notmorgan | anteaya: i was looking from a logged in perspective. | 17:42 |
| notmorgan | anteaya: because i assume i'd be logged in most of the time. | 17:42 |
| anteaya | but if you want only certain people to be able to edit notes then you have the concept of permissions on stories, which goes against one of the main design principles | 17:42 |
| fungi | krotscheck: i've got your tag waiting to go back into the release pipeline (for the past ~15 minutes so far) but zuul is busy with a glut of events/results for a little longer. i'll let you know once it goes through | 17:43 |
| anteaya | in that anyone can create a story, but permissions are on worklists and boards | 17:43 |
| notmorgan | nope, i want it just a "read only view unless i explicitly want to edit" | 17:43 |
| fungi | also i cleaned up all the "dist" cruft on tarballs.o.o | 17:43 |
| notmorgan | i accidently deleted text and didn't want to save it. | 17:43 |
| anteaya | notmorgan: yes the login does not reinstate your location is a known issue, let me find the story on that | 17:43 |
| *** yamamoto has quit IRC | 17:43 | |
| jeblair | fungi: it looks like zuul is taking a long time to report changes | 17:43 |
| jeblair | fungi: i wonder if gerrit is slow | 17:44 |
| notmorgan | anteaya: that should be a super easy fix. i chased down how that worked for gerrit through OID. | 17:44 |
| anteaya | notmorgan: right, so again stories don't have the concept of permissions and that is on purpose | 17:44 |
| notmorgan | anteaya: happy to at least help dig out what attributes to set/consume. | 17:44 |
| *** SumitNaiksatam has joined #openstack-infra | 17:44 | |
| anteaya | notmorgan: thanks, I'll see if I can dig up the story | 17:44 |
| notmorgan | anteaya: and i'm not talking about permission i'm talking about strict UX. | 17:44 |
| anteaya | notmorgan: okay maybe I don't understand then | 17:44 |
| notmorgan | anteaya: if i (or any logged in person) clicks edit it should let them | 17:44 |
| notmorgan | anteaya: if i am viewing the note, i don't want it to be "edit" mode - no accidental changes to the content | 17:45 |
| fungi | jeblair: interestingly, the java gc doesn't seem to be active (yet) according to javamelody | 17:45 |
| notmorgan | anteaya: but anyone should be able to "edit" | 17:45 |
| persia | notmorgan: anteaya: I have opinions about notes: please also let me know the story if it becomes one. tl;dr is that I think notes should only contain URLs and other references with text in the discussion. | 17:45 |
| *** ihrachys has quit IRC | 17:45 | |
| jeblair | fungi: replication could also be slow (i think zuul still waits for that) | 17:45 |
| notmorgan | persia: sure. however that is done | 17:45 |
| *** derekh has quit IRC | 17:45 | |
| Shrews | jeblair: notmorgan: back from lunch. playing catchup | 17:45 |
| persia | notmorgan: When someone creates the story, highlight me with the number :) | 17:45 |
| notmorgan | persia: i personally like the LP thing with comments that are non-editable once submitted. | 17:45 |
| persia | notmorgan: I also | 17:45 |
| krotscheck | fungi: Thanks! | 17:46 |
| persia | (this is why I have my opinion about notes) | 17:46 |
| notmorgan | persia: :) | 17:46 |
| anteaya | notmorgan: ah I see, opening the note should not default to edit mode, yes, makes sense | 17:46 |
| notmorgan | anteaya: exactly ^_^. | 17:46 |
| fungi | jeblair: gerrit show-queue doesn't have a bunch of replication showing at least | 17:47 |
| *** PollyZ__ has joined #openstack-infra | 17:47 | |
| jeblair | notmorgan: well, notes aren't comments. | 17:47 |
| notmorgan | jeblair: ok will gate with the else. | 17:47 |
| anteaya | persia: sure, right now I am listening to morgan to the best of my ability, once I feel I have listened to him to his satisfaction, I can listen to you, thanks | 17:47 |
| AJaeger | jeblair, fungi, are you aware that according to http://status.openstack.org/zuul/ networking-sfc is since 153 hours (!) in the post queue waiting for networking-sfc-docs to finish? | 17:47 |
| notmorgan | jeblair: sure. :) | 17:47 |
| notmorgan | jeblair: but in either case, non-edit by default. | 17:47 |
| persia | anteaya: No rush: I only want to be part of the discussion once your listening to morgan becomes an artifact. | 17:47 |
| jeblair | storyboard has comments, but their utility is limited by the fact that they are almost never displayed in practice | 17:47 |
| fungi | AJaeger: yes, zuul is waiting for that gearman task to report, and it never will because the slave is gone and the jenkins master has been restarted since then. we're basically waiting for a zuul restart to blow that away | 17:48 |
| AJaeger | fungi: ok | 17:48 |
| *** chem has quit IRC | 17:48 | |
| anteaya | persia: absolutely, thank you | 17:48 |
| *** akshai has quit IRC | 17:48 | |
| anteaya | persia: do you know of a story that exists that captures the no redirect to my original page upon login current functionality? | 17:49 |
| notmorgan | persia: i'll try and make a story or find one as well. but chasing some things down atm. | 17:49 |
| notmorgan | persia: for zuul ansible launcher :) | 17:49 |
| *** akshai has joined #openstack-infra | 17:49 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Call :func:`AnsibleModule.fail_json` on failure https://review.openstack.org/324726 | 17:49 |
| notmorgan | jeblair: ^ with the gate now. | 17:50 |
| anteaya | jeblair: yes, the display discussion around comments thus far that I am aware of has come down to chronological order or reverse chronological or toggable, any choice hides many due to pagination | 17:50 |
| jeblair | anteaya: they should not paginate, and they should not display uninteresting timeline entries by default (and some of the only moderately interesting ones should be visually very tiny) | 17:51 |
| notmorgan | OOH story: #xxxx is a link in gerrit | 17:51 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Increased open file limits on debian systems https://review.openstack.org/324735 | 17:51 |
| notmorgan | so we're def doing it right :) | 17:51 |
| fungi | phschwartz: are you aware your philip-schwartz account has left an average of 35.5 gerrit comments a minute between 17:40 and 17:50 utc alone? | 17:52 |
| krotscheck | clarkb, fungi: ^^ | 17:52 |
| anteaya | jeblair: okay I can create a story with those thoughts and see what comes out of that | 17:52 |
| notmorgan | fungi: gertty backlog? | 17:52 |
| persia | anteaya: I do not know of such a story | 17:52 |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Create separate ACL groups for murano-apps project https://review.openstack.org/323340 | 17:52 |
| anteaya | persia: great thank you | 17:52 |
| jeblair | anteaya: launchpad displays comments (and timeline entries) perfectly | 17:53 |
| fungi | notmorgan: well, it was >3k comments since 17:00 utc, so doubtful | 17:53 |
| anteaya | notmorgan: I cannot find, nor can persia find the redirect to oringinal page on login story, shall I create it? | 17:53 |
| notmorgan | fungi: holy crap. | 17:53 |
| phschwartz | fungi: no, I haven't. Are you able to see the ip the connections are coming from? | 17:53 |
| notmorgan | anteaya: sure. | 17:53 |
| fungi | phschwartz: yep, just a sec | 17:53 |
| anteaya | jeblair: thank you, I'll add that | 17:53 |
| phschwartz | fungi: I haven't pushed anything out or made a comment since monday so it is coming from somewhere, but not me directly | 17:53 |
| notmorgan | anteaya: and i'll get some time later today to find the attribute passthroughs so we can make it work. | 17:54 |
| anteaya | notmorgan: awesome, thank you | 17:54 |
| fungi | phschwartz: nevermind, bad pattern match, it's queries | 17:54 |
| persia | anteaya: To be clear: I have not looked: storyboard doesn't happen to work with the ancient browsers on my laptop currently. I just don't remember any discussions of such a story. | 17:54 |
| anteaya | persia: let me capture these stories then I can listen | 17:54 |
| fungi | phschwartz: not submitting comments | 17:54 |
| notmorgan | anteaya: it likely is very straight forward with a minor csrf guard | 17:54 |
| anteaya | persia: wonderful thank you | 17:54 |
| anteaya | notmorgan: I believe you | 17:54 |
| phschwartz | fungi: still worrysome. can you still provide the ip so I can track it down | 17:54 |
| notmorgan | anteaya: i'm hoping at least ;) | 17:54 |
| fungi | phschwartz: yeah, 159.122.223.80 | 17:55 |
| *** kzaitsev_mb has quit IRC | 17:56 | |
| *** pradk has quit IRC | 17:56 | |
| phschwartz | fungi: ty, that is an IBM server in London that is running a thirdparty CI, I will see why it has gone nuts quering with my creds of all else. | 17:56 |
| jeblair | fungi: that would certainly distract zuul a bit, but i'd be surprised if that were causing the reporting slowdown | 17:56 |
| jeblair | fungi: (i can't think of a better explanation) | 17:56 |
| notmorgan | phschwartz: they co-opted your account! | 17:56 |
| *** links has joined #openstack-infra | 17:57 | |
| fungi | jeblair: yep, just mining gerrit logs looking for anything that could be causing it to be "slow" | 17:57 |
| phschwartz | notmorgan: more like took over a dev server and most likely turned it back on with my creds still cached from openstackci | 17:57 |
| jeblair | fungi: it seems to have just picked up the pace | 17:57 |
| *** PollyZ__ has quit IRC | 17:57 | |
| anteaya | jeblair: https://storyboard.openstack.org/#!/story/2000620 | 17:58 |
| JayF | AJaeger: thanks for that +2, but that patch depends on https://review.openstack.org/#/c/313800 which is more urgent in terms of Ironic. If you could point a review there as well I'd be very grateful. | 17:59 |
| AJaeger | JayF: slowly going through all reviews ;) Will tackle that one next... | 18:00 |
| amrith | hi, commits https://review.openstack.org/#/c/312806/ and https://review.openstack.org/#/c/312805/ registered the new trove-image-builder repository. The change to zuul/layout.yaml in https://review.openstack.org/#/c/312805/7/zuul/layout.yaml provides what is described as the minimal information to get zuul to trigger jobs. Yet a test commit https://review.openstack.org/#/c/324713/ is sitting with "Needs Verified" for a | 18:00 |
| amrith | while now. Is there something additional that I need to do on the infra side before commits to this repo will merge? | 18:00 |
| anteaya | notmorgan: https://storyboard.openstack.org/#!/story/2000621 | 18:00 |
| JayF | thanks AJaeger! Woo | 18:00 |
| *** annegentle has quit IRC | 18:00 | |
| notmorgan | anteaya: tagging it in my browser so i can look later today | 18:00 |
| anteaya | persia: okay thanks for your patience do you want to talk here or storyboard? | 18:00 |
| *** Kaiyan has joined #openstack-infra | 18:00 | |
| AJaeger | amrith: check the zuul status on status.openstack.org/zuul | 18:00 |
| anteaya | notmorgan: awesome, thank you | 18:00 |
| amrith | AJaeger, I did. nothing there | 18:00 |
| AJaeger | amrith: "Queue lengths: 2298 events, 2273 results." | 18:01 |
| amrith | but I do see some jobs that've been running 154hr | 18:01 |
| fungi | amrith: some or just one? | 18:01 |
| Kaiyan | @clarkb Thank you for your help! https://review.openstack.org/#/c/316293/ just passed the check jobs | 18:01 |
| AJaeger | amrith: as long as the queue is not 0, your job might be one of those queued ones - please wait. | 18:01 |
| persia | anteaya: I have nothing you have not addressed: I just wanted the pointer to the story about notes (so I could get it on my phone easily), and to be clear that I had not searched for anything when you asked. Thank you. | 18:01 |
| AJaeger | amrith: the one in the post queue is known | 18:01 |
| anteaya | Kaiyan: you don't need to use @ in front of someone's nickname in irc | 18:01 |
| amrith | AJaeger, I have no context in which to understand that. | 18:01 |
| amrith | ah, OK | 18:01 |
| anteaya | Kaiyan: that is for twitter | 18:02 |
| amrith | now that I know what "queue length" is (jobs not showing up yet) I'm happy to wait :) | 18:02 |
| anteaya | persia: okay as for notes, I don't have a reference story | 18:02 |
| *** e0ne has quit IRC | 18:02 | |
| Kaiyan | oh haha good to know! | 18:02 |
| anteaya | persia: just a personal memory of a conversation that took place at the last mid-cycle but might never made it to the artifact stage | 18:02 |
| persia | anteaya: No worries. If I read correctly, one may be created later, once there is time for the concerned party to think about it more. | 18:02 |
| *** annegentle has joined #openstack-infra | 18:02 | |
| fungi | amrith: zuul watches an event stream from gerrit looking for events that should trigger jobs (for example, new changes/patchsets or recheck comments, or approvals), and sometimes it gets behind in processing the incoming events (especially because it generates quite a few internal events of its own which end up in the same event queue) | 18:03 |
| *** infra-red has quit IRC | 18:03 | |
| anteaya | persia: I think so, yes, there has been discussion about it and Zara had some thoughts on it she posted in the storyboard channel | 18:03 |
| anteaya | persia: but the discussion hasn't been exhaustive and as far as I know just tendencies have been expressed | 18:03 |
| *** kzaitsev_mb has joined #openstack-infra | 18:04 | |
| amrith | fungi, thanks. the queue is growing quite fast (now that I know where to look). It is almost 110 more than just 2min ago | 18:04 |
| *** amoralej is now known as amoralej|off | 18:04 | |
| anteaya | persia: notmorgan but I can create a story about what morgan noticed in that when opening a note is should not be in edit mode by default, I'll create that story now | 18:05 |
| AJaeger | amrith: that's slow IMHO - fungi and jeblair are currently checking whether there's a problem | 18:05 |
| fungi | amrith: yeah, that will also spike up pretty quickly when changes merge, because it needs to generate merge-check jobs and enqueue them looking for open changes which may now conflict with the target branch tip state | 18:05 |
| notmorgan | anteaya: added myself for the redirect as the owner for now. I'll unassign if i can't fix once i have the mechanism documented | 18:05 |
| *** rhallisey has quit IRC | 18:05 | |
| *** csomerville has joined #openstack-infra | 18:06 | |
| amrith | AJaeger, fungi thanks for the answers. hope you find and are able to fix whatever ails the thing. | 18:06 |
| *** degorenko is now known as _degorenko|afk | 18:06 | |
| anteaya | notmorgan: persia https://storyboard.openstack.org/#!/story/2000622 | 18:06 |
| pleia2 | JayF: +2A | 18:07 |
| anteaya | notmorgan: awesome thank you, any activity or notes you can offer on the story are wonderful! | 18:07 |
| *** rhallisey has joined #openstack-infra | 18:07 | |
| JayF | pleia2: Thanks a bunch! I really appreciate it and will monitor the ironic gate to make sure it was effective. | 18:07 |
| *** e0ne has joined #openstack-infra | 18:08 | |
| * anteaya steps away for a bit, thanking folks for the stories will be back for more shortly | 18:08 | |
| *** david-lyle has joined #openstack-infra | 18:09 | |
| *** rbrndt has joined #openstack-infra | 18:09 | |
| fungi | jeblair: looks like there was a zuul reconfiguration triggered at 17:47:40 (according to the debug log), though that seems after things got slow | 18:10 |
| *** regXboi has quit IRC | 18:11 | |
| fungi | but it's still in the midst of that for now, so it could explain additional slowness at least | 18:11 |
| jeblair | fungi: yeah, could be multiple issues | 18:11 |
| jeblair | reporting seems to have slowed down a bit again i think | 18:12 |
| jeblair | (we should log that time) | 18:12 |
| fungi | as in log the duration from when the review call is made to gerrit until the response comes back? | 18:13 |
| jeblair | yep | 18:13 |
| fungi | log and/or emit stats | 18:13 |
| AJaeger | sdake: is https://review.openstack.org/#/c/311574/ now ready to merge? Could you rebase in that case, please | 18:13 |
| jeblair | cause right now, it's 'count how long between when i see a report and when i see a log line that i know is from the same thread' | 18:13 |
| jeblair | which is not very scientific | 18:13 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Revert "Revert "Revert "Revert "Start using the Zuul Ansible launcher""""" https://review.openstack.org/324668 | 18:13 |
| jeblair | pabelanger, fungi, notmorgan: so on the task i have assigned to me -- figuring out why zuul_launcher hangs -- i have replicated it by runnig 'sleep 30' in zuul launcher, and then killing the running bash script out of band | 18:15 |
| *** maishsk has joined #openstack-infra | 18:15 | |
| *** kzaitsev_mb has quit IRC | 18:15 | |
| *** pradk has joined #openstack-infra | 18:16 | |
| jeblair | pabelanger, fungi, notmorgan: that apparently causes nothing actionable to happen to the proc.stdout.readline() call, so it sits there | 18:16 |
| *** amitgandhinz has quit IRC | 18:17 | |
| *** csomerville has quit IRC | 18:18 | |
| *** amitgandhinz has joined #openstack-infra | 18:18 | |
| *** cody-somerville has joined #openstack-infra | 18:18 | |
| pabelanger | jeblair: nice find | 18:19 |
| Shrews | notmorgan: jeblair: it would actually be simpler to just add a 'command: /bin/false' to the end of error block | 18:19 |
| *** kzaitsev_mb has joined #openstack-infra | 18:19 | |
| notmorgan | Shrews: it is fine to shell out, that was the alternative. | 18:19 |
| jeblair | Shrews: that's a local command, right? | 18:20 |
| *** dizquierdo is now known as dizquierdo_afk | 18:20 | |
| notmorgan | Shrews: i'm ok with making it do the shell bin/false instead of calling .fail_json in our module | 18:20 |
| notmorgan | Shrews: if it's best practices to do so. | 18:21 |
| Shrews | jeblair: remote iirc | 18:21 |
| sdake | AJaeger my outstanding review queue is 2 pages long with half of them in a needs rebase state | 18:21 |
| AJaeger | sdake: fun ;) | 18:22 |
| sdake | AJaeger but I will really try to get to it today | 18:22 |
| sdake | I have a bunch of governance changes i'd like to get to first | 18:22 |
| Shrews | notmorgan: i don't think there's any established "best practice" yet | 18:22 |
| notmorgan | Shrews: it *seems* like the module should be self contained though? | 18:22 |
| notmorgan | Shrews: just feels more or less "this is how other modules do it?" | 18:22 |
| fungi | jeblair: is there no signal caught by the parent? | 18:22 |
| *** trown|lunch is now known as trown | 18:22 | |
| AJaeger | sdake: thanks. It's not urgent, just wanted to have "my" queue cleaner and this one looked forgotten... | 18:23 |
| Shrews | notmorgan: what we're doing (checking the value of the ansible-playbook executable) is really outside the realm of playbook best practices | 18:23 |
| *** julim has joined #openstack-infra | 18:23 | |
| Shrews | notmorgan: your way should work fine. just providing a less-code-alternative | 18:24 |
| notmorgan | Shrews: hehe. yeah it would be a inject task(shell) afaict in ansible launcher | 18:24 |
| notmorgan | Shrews: for what we're doing. | 18:24 |
| notmorgan | Shrews: so. *shrug*? | 18:24 |
| Shrews | notmorgan: wha? | 18:24 |
| Shrews | notmorgan: error_block.append(dict(command='/bin/false')) .... was what i was thinking | 18:25 |
| *** pt_15 has joined #openstack-infra | 18:25 | |
| notmorgan | Shrews: yes. it's a single line. | 18:25 |
| *** dizquierdo_afk has quit IRC | 18:26 | |
| notmorgan | Shrews: sorry it's not a lot less code (-1,+6 vs -1,+1) ;) | 18:26 |
| notmorgan | Shrews: i'm good with either :) | 18:26 |
| *** savihou has joined #openstack-infra | 18:27 | |
| jeblair | fungi: hrm, not that it has clued me in about. i will try running it outside of the ansible context and see what happens | 18:27 |
| fungi | jeblair: is there a grandchild process holding stdout open? | 18:27 |
| fungi | jeblair: oh, rigt, ansible is the child process, and the bash your sleep runs in is a grandchild? | 18:27 |
| fungi | jeblair: so ansible isn't terminating? | 18:27 |
| pabelanger | jeblair: odd, I could not reproduce the readline() issue locally using a simple python script. | 18:28 |
| pabelanger | so must be something specific to zuul | 18:28 |
| jeblair | fungi: ansible -> zuul_runner -> bash -> sleep | 18:28 |
| jeblair | fungi: and yes, i think sleep may be holding stdout open | 18:28 |
| fungi | jeblair: okay, and yes that makes sense | 18:28 |
| jeblair | fungi: when i kill both, it cleans up | 18:28 |
| fungi | jeblair: so we need some way for termination of the direct child to cascade to the grandchildren i guess | 18:29 |
| jeblair | fungi: oh! so maybe what's going on in jobs is that something in devstack/grenade land is holding stdout open while the bash script exits? | 18:29 |
| fungi | jeblair: yes, i see behavior like that when running devstack remotely on a vm too. it starts some children which inherit standard file descriptors, so the noninteractive ssh session hangs after the tests complete | 18:30 |
| *** esikachev has joined #openstack-infra | 18:31 | |
| *** hasharAway has quit IRC | 18:31 | |
| fungi | s/devstack/devstack-gate/ | 18:31 |
| jeblair | fungi: so first thing -- i'll dig and see if we're getting signals or what... and second thing, i'll investigate maybe doing some kind of select loop where i poll for whether the child has exited or something... | 18:32 |
| *** jvrbanac_ is now known as jvrbanac | 18:32 | |
| nikhil | hi, I'm trying to make sense of this https://github.com/openstack/glance/blob/master/tox.ini#L50 , https://github.com/openstack-infra/project-config/blob/aa831be9f499b13df81f930e885487670467c840/zuul/layout.yaml#L6585 and https://review.openstack.org/#/c/298850/4/tox.ini@32 | 18:32 |
| fungi | jeblair: yeah, poking around at examples it looks like you may be able to register a handler to trap sigchld | 18:32 |
| nikhil | AJaeger: if you are around :-) (the above has a note from you) | 18:32 |
| jeblair | fungi: cool. sounds like a plan then. this will take a bit, so probably will be well after lunch | 18:33 |
| fungi | jeblair: it's possible sigchld is set to sig_ign by default and so missed? | 18:33 |
| *** amotoki has joined #openstack-infra | 18:34 | |
| fungi | jeblair: but yeah, the signal route hopefully avoids you needing a separate thread to monitor teh process for termination at least | 18:34 |
| fungi | not sure if that's a concern or not | 18:34 |
| AJaeger | nikhil: let me read... | 18:34 |
| nikhil | particularly, glance doesn't have a post job (?) yet there are some values overridden. (or what I said isn't true) | 18:34 |
| jeblair | fungi: heh, i hadn't thought about using a thread; i was thinking select with timeout | 18:34 |
| nikhil | AJaeger: ^ (in case you clicked already) | 18:34 |
| *** savihou has quit IRC | 18:35 | |
| fungi | jeblair: oh, yep that would probably be a saner alternative | 18:35 |
| *** david-lyle has quit IRC | 18:35 | |
| AJaeger | nikhil: I clicked but now I'm confused. Could you restate the problem you have, please? | 18:35 |
| nikhil | I noticed that while filing a review for glance_store https://review.openstack.org/324755 | 18:35 |
| nikhil | AJaeger: surely.. | 18:36 |
| notmorgan | jeblair: ok going to step off for lunch and to help devananda with a couple things. will check back shortly and if needed can re-spin the success fix. | 18:36 |
| nikhil | AJaeger: I read you comments on the trove patch | 18:36 |
| AJaeger | nikhil: I see that glance does not have a cover job. But if it would have one, it would be run in post queue most likely ;) | 18:36 |
| nikhil | AJaeger: it looks like the override of the install_commands is needed in case of post jobs as upstream infra doesn't support those? | 18:36 |
| pabelanger | looks like somebody patch bomb zuul again, 40 patches this time | 18:37 |
| pabelanger | openstack/bareon | 18:37 |
| nikhil | AJaeger: the project-config doesn't have any post job for glance :) | 18:37 |
| *** ddieterly[away] is now known as ddieterly | 18:37 | |
| *** kzaitsev_mb has quit IRC | 18:37 | |
| AJaeger | nikhil: indeed. | 18:37 |
| nikhil | AJaeger: so even releasenotes may not need the override? | 18:37 |
| nikhil | https://github.com/openstack/glance/blob/master/tox.ini#L15 | 18:37 |
| AJaeger | nikhil: glance uses postjobs via templates - like releasenotes, branch tarballs, translation sync, ... | 18:37 |
| fungi | pabelanger: jeblair: perhaps that's what bogged down zuul for a while? putting together all the bareon check queues? | 18:38 |
| *** amotoki has quit IRC | 18:39 | |
| *** oanson has joined #openstack-infra | 18:39 | |
| pabelanger | fungi: jeblair: this is what happened last time somebody uploaded a large patch series. You can see the silent queue is trying to process all 40 patches | 18:39 |
| fungi | pabelanger: jeblair: they were all uploaded at 17:09, which seems like around when stuff started to bog down (or could just be coincidence) | 18:39 |
| nikhil | AJaeger: aha .. that's what I was missing. How can I read up on which are the post jobs for glance using template? Any link would really help? (sorry for being a noob in reading the zull configs) | 18:39 |
| AJaeger | nikhil: glance uses releasenotes job in post - it uses release-notes-jobs template http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n607 | 18:39 |
| jeblair | fungi: it didn't look like that to me, but maybe? | 18:39 |
| *** links has quit IRC | 18:39 | |
| fungi | pabelanger: at least this isn't as large as last time which was some several hundred, if memory serves | 18:39 |
| AJaeger | nikhil: you can look at the zuul debug output like http://logs.openstack.org/59/324259/1/check/gate-project-config-layout/8499057/console.html but that is really for the expert. | 18:40 |
| AJaeger | nikhil: Otherwise: Check each template and see what runs | 18:40 |
| AJaeger | nikhil: or check status.openstack.org/zuul and monitor a change run in the post queue and see what it uses ;) | 18:40 |
| pabelanger | fungi: right, zuul will do its job, but downside is nodepool will decrease launching new nodes | 18:40 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool to see the result | 18:40 |
| AJaeger | nikhil: Or trust me: venv, releasenotes, and cover tox environments are used in post queue | 18:41 |
| notmorgan | jeblair: i meant that to be a kwarg. let me fix it. | 18:41 |
| nikhil | AJaeger: thanks for that info... | 18:41 |
| nikhil | AJaeger: yeah, I was really hoping not the 3rd option. I ran into this confusion for my glance_store (fix) | 18:41 |
| jeblair | notmorgan: ++ | 18:41 |
| AJaeger | nikhil: shall I -1 your glance_store fix or do you now know what to do? | 18:41 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add tempest tests to periodic and experimental job https://review.openstack.org/321019 | 18:41 |
| nikhil | AJaeger: https://review.openstack.org/324755 is where I need to indentify which on to override | 18:41 |
| fungi | pabelanger: well, to be fair, nodepool stopped launching new nodes because zuul was busy going through its outstanding events/results queues and wasn't adding new jobs to gearman, so demand fell to ~0 | 18:42 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Call :func:`AnsibleModule.fail_json` on failure https://review.openstack.org/324726 | 18:42 |
| AJaeger | nikhil: let me -1 ;) | 18:42 |
| notmorgan | jeblair: fixed | 18:42 |
| nikhil | AJaeger: -1 appreciated (however, weird that may sound) | 18:42 |
| *** micole has joined #openstack-infra | 18:42 | |
| fungi | pabelanger: you can see it's picking up again now that zuul has added job tasks in gearman | 18:42 |
| pabelanger | fungi: Yup, we should have a big uptick now | 18:42 |
| notmorgan | jeblair: i realize... this might in-fact duplicate the failure line. | 18:42 |
| notmorgan | jeblair: if it does, i'll sweep back through and make it only log once. | 18:43 |
| notmorgan | (the log message) | 18:43 |
| *** sarob has joined #openstack-infra | 18:43 | |
| AJaeger | nikhil: done ;) If you have questions, just ask... | 18:43 |
| *** ddieterly has quit IRC | 18:43 | |
| Shrews | notmorgan: offered yet another alternative in the review | 18:43 |
| nikhil | AJaeger: much appreciated.. thanks! | 18:43 |
| Shrews | minus the shelling portion | 18:44 |
| *** yamamoto has joined #openstack-infra | 18:44 | |
| micole | I was wondering if anyone had a best practices for the Jenkins Job Builder. I'm trying to create a bunch of very similar jobs, but a lot of them all have special axes in the matrix project-type. Not sure how I can use the job-template and pass in the different axes (sometimes one axis, sometimes multiple depending on the job) | 18:44 |
| *** harlowja has quit IRC | 18:44 | |
| jeblair | Shrews: notmorgan i kind of like the fail module option | 18:45 |
| AJaeger | project-config cores, would be great to have soon fixes for translation jobs merged: https://review.openstack.org/#/c/324723/ and https://review.openstack.org/#/c/324259/, please. | 18:45 |
| hongbin | Hi, I want to add a pypi jobs for Magnum, but found there is already a pypi package called "magnum" . Could you suggest a work around? | 18:45 |
| AJaeger | micole: see how ironic does it. You really need to pass in all parameter | 18:45 |
| AJaeger | hongbin: we do not release server packages on pypi | 18:45 |
| notmorgan | Shrews: ah the fail module? | 18:45 |
| hongbin | AJaeger: ack | 18:46 |
| notmorgan | Shrews: hmm. i looked at the fail module... it... was empty? | 18:46 |
| notmorgan | Shrews: on github at least... | 18:46 |
| AJaeger | hongbin: So, just ignore the conflict | 18:46 |
| hongbin | ack | 18:47 |
| * AJaeger signs off again and waves good-bye | 18:47 | |
| notmorgan | Shrews: https://github.com/ansible/ansible-modules-core/blob/devel/utilities/logic/fail.py that... unless it's magic? | 18:47 |
| anteaya | AJaeger: have a good evening | 18:47 |
| notmorgan | Shrews: looks a lot of empty. | 18:47 |
| anteaya | AJaeger: nice to have you back | 18:47 |
| fungi | jeblair: worried i'm short on context as to what "ant" is in the comment at http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/ansiblelaunchserver.py#n657 | 18:47 |
| notmorgan | Shrews: i might just be missing that it's got magic in it from inside ansible | 18:47 |
| *** javeriak has joined #openstack-infra | 18:47 | |
| notmorgan | Shrews: or the docs are pointing at the wrong place :P | 18:48 |
| Shrews | notmorgan: http://docs.ansible.com/ansible/fail_module.html | 18:48 |
| notmorgan | Shrews: yes. and that if you chanse goes to that link | 18:48 |
| notmorgan | which... looks like it does nothing? | 18:48 |
| notmorgan | or is that just a namespace thing | 18:49 |
| Shrews | notmorgan: i don't know enough about the internals to explain why it works. i just know it does | 18:50 |
| notmorgan | Shrews: well then i can respin, but you see why i was like "uhhhhh does this even do the thing?" | 18:50 |
| nikhil | AJaeger: thanks again.. I've quitely added you to client fix :) https://review.openstack.org/#/c/324660/ but then thought might be better to mention here as well. | 18:53 |
| *** yamamoto has quit IRC | 18:53 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:55 | |
| *** jbernard has joined #openstack-infra | 18:56 | |
| notmorgan | Shrews: ok i'll re-spin post lunch then and we'll have winnings | 18:56 |
| Shrews | notmorgan: yay winning and tiger blood! | 18:56 |
| *** shamail has joined #openstack-infra | 18:56 | |
| notmorgan | Shrews: lol :) | 18:57 |
| *** nadya has joined #openstack-infra | 18:59 | |
| jeblair | fungi: the java build tool. jenkins uses its globbing syntax for things like the scp and ftp module (** for any number of dirs, etc) | 19:00 |
| *** EricGonczer_ has joined #openstack-infra | 19:00 | |
| anteaya | fungi: how are you feeling about the task tracker spec? https://review.openstack.org/#/c/314185/ | 19:02 |
| fungi | jeblair: oh, hah right now i know what ant you meant. thanks | 19:02 |
| anteaya | do you feel it is ready to merge? | 19:02 |
| fungi | anteaya: you waited until 19:02 to remind me--you're off your game ;) | 19:02 |
| anteaya | I am so | 19:02 |
| anteaya | was finishing an email | 19:02 |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: [Monasca] Add Tempest Jobs as non-voting for monasca repos https://review.openstack.org/316293 | 19:03 |
| fungi | anteaya: i hadn't gotten a chance to look at the votes on it today, but wow that's quite the turn-out | 19:03 |
| *** EricGonc_ has quit IRC | 19:03 | |
| anteaya | fungi: folks showed up indeed | 19:03 |
| fungi | 9 rc votes? that may be an infra council record ;) | 19:03 |
| anteaya | wooooo | 19:04 |
| *** kaisers_ has joined #openstack-infra | 19:04 | |
| fungi | though i'm not exactly flush with time to run the historical numbers right now and find out | 19:04 |
| openstackgerrit | Merged openstack-infra/project-config: zuul/puppet: bring back scenario003 on master https://review.openstack.org/324627 | 19:04 |
| openstackgerrit | Merged openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs https://review.openstack.org/313800 | 19:04 |
| anteaya | fungi: understood | 19:05 |
| *** kaisers_ has quit IRC | 19:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Put inspector basic tempest job to check pipeline https://review.openstack.org/323708 | 19:05 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack-infra/project-config: Don't run puppet tests on rst/unit test changes https://review.openstack.org/324158 | 19:05 |
| anteaya | submitting +W suits my needs just fine right now | 19:05 |
| openstackgerrit | Merged openstack-infra/system-config: Add openstack-powervm to meetbot and statusbot channels https://review.openstack.org/264307 | 19:05 |
| *** cody-somerville has quit IRC | 19:06 | |
| *** cody-somerville has joined #openstack-infra | 19:06 | |
| anteaya | fungi: thank you | 19:08 |
| fungi | dmsimard: we got consensus that having small projects (whose contributors are well aware of the current feature gap) start using storyboard is something we should encourage | 19:08 |
| anteaya | yay \o/ | 19:09 |
| dmsimard | fungi: yay :) | 19:09 |
| openstackgerrit | Prakash Ramchandran proposed openstack-infra/project-config: add new project meghdwar https://review.openstack.org/319466 | 19:09 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: create release-check high priority queue https://review.openstack.org/324768 | 19:09 |
| dmsimard | fungi: I put the ara project creation on hold momentarily with a comment, however | 19:09 |
| *** e0ne has quit IRC | 19:09 | |
| dmsimard | There's significant interest from the Ansible community as well and I'd like to give them a chance to discuss where the project should live first | 19:09 |
| fungi | dmsimard: wfm, we're just as likely to run it whether we're hosting the code in our gerrit or not | 19:10 |
| fungi | so put it where it makes sense for you | 19:10 |
| * dmsimard nods | 19:10 | |
| anteaya | I think I am going to go for a walk and enjoy some sunshine, back after a while | 19:10 |
| tristanC | thanks you all for that common task tracker effort, exciting time ahead! | 19:11 |
| dmsimard | sunshine is good | 19:11 |
| dmsimard | rainy here today | 19:11 |
| *** nadya has quit IRC | 19:11 | |
| *** nadya has joined #openstack-infra | 19:12 | |
| *** nadya has quit IRC | 19:12 | |
| *** nadya has joined #openstack-infra | 19:12 | |
| fungi | yeah, rainy here too, but hard to complain. a rainy beach is still a beach | 19:15 |
| fungi | also, what's a little more water? | 19:15 |
| openstackgerrit | Merged openstack-infra/infra-specs: A common task tracker for OpenStack https://review.openstack.org/314185 | 19:16 |
| jeblair | fungi: i don't know why the nodepool config hasn't updated with the zl revert | 19:16 |
| *** tpsilva has quit IRC | 19:17 | |
| jeblair | fungi: oh, pabelanger put nodepool in emergency | 19:18 |
| fungi | jeblair: some things i try running locally on nodepool tell me i have too many open files too | 19:18 |
| jeblair | pabelanger: would you mind performing the nodepool config update and also database cleanup | 19:18 |
| fungi | or that there are too many open files | 19:18 |
| jeblair | fungi: i 'sudo su - nodepool' | 19:18 |
| fungi | jeblair: um, i was trying to tail syslog | 19:18 |
| *** _vsaienko has joined #openstack-infra | 19:19 | |
| *** oanson has quit IRC | 19:19 | |
| jeblair | pabelanger: once the config is in place, you will need to: update node set target_name='jenkins01' where target_name='zuul'; | 19:19 |
| jeblair | in order to convince nodepool to delete those without error | 19:19 |
| jeblair | i have to run; i'm late for lunch | 19:19 |
| fungi | have a good lunch, jeblair | 19:20 |
| *** nadya has quit IRC | 19:21 | |
| fungi | i have a feeling root has reached its max open files because of diskimage-builder | 19:22 |
| jeblair | if pabelanger isn't around, some other infra-root needs to do that -- we've been passing 1/8 of all changes for hours because that hasn't updated | 19:22 |
| openstackgerrit | Fei Long Wang proposed openstack-infra/project-config: Add new project python-distilclient https://review.openstack.org/318367 | 19:23 |
| *** linuxmaniac_ has joined #openstack-infra | 19:23 | |
| *** linuxmaniac has quit IRC | 19:23 | |
| *** nadya has joined #openstack-infra | 19:23 | |
| *** david-lyle has joined #openstack-infra | 19:24 | |
| *** ams__ has joined #openstack-infra | 19:26 | |
| pabelanger | jeblair: sorry, I thought I deleted it | 19:26 |
| *** esikachev has quit IRC | 19:27 | |
| pabelanger | appologies | 19:27 |
| fungi | pabelanger: jeblair: i've manually done the config revert and cleanup on nodepool.o.o just now | 19:27 |
| pabelanger | fungi: jeblair: removed | 19:27 |
| fungi | thanks pabelanger | 19:27 |
| *** nadya has quit IRC | 19:28 | |
| *** Sukhdev has joined #openstack-infra | 19:28 | |
| fungi | there were 84 rows matched in that update query, so we should get back that many here shortly | 19:28 |
| *** Jeffrey4l_ has quit IRC | 19:30 | |
| *** Jeffrey4l_ has joined #openstack-infra | 19:31 | |
| *** nadya has joined #openstack-infra | 19:32 | |
| *** esikachev has joined #openstack-infra | 19:33 | |
| *** pramchan has joined #openstack-infra | 19:34 | |
| *** amotoki has joined #openstack-infra | 19:34 | |
| *** sdake has quit IRC | 19:35 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: [WIP] Add mulitnode CI job support https://review.openstack.org/324777 | 19:35 |
| *** zeih has joined #openstack-infra | 19:35 | |
| *** SumitNaiksatam has quit IRC | 19:36 | |
| pramchan | hey guys, I'm trying to add a new project to the openstack/governance repo, but when I try to set the gerrit code review remote to ssh://pramchan@review.openstack.org:29418/openstack/governance.git, I get the following error: | 19:36 |
| JayF | what does NOT_REGISTERED mean in a gate job? | 19:37 |
| *** nadya has quit IRC | 19:37 | |
| JayF | I suspect it means my patch was broken, but I want to try and pinpoint exactly how so I can fix | 19:37 |
| pramchan | fatal: Repository not found | 19:37 |
| pramchan | what's the proper repo for gerrit code review on the openstack/governance repo? | 19:37 |
| *** sdake has joined #openstack-infra | 19:38 | |
| *** ociuhandu has quit IRC | 19:38 | |
| *** _vsaienko has quit IRC | 19:38 | |
| pabelanger | JayF: it usually means that zuul cannot find the configured jenkins job. Which review are you seeing it on? | 19:38 |
| *** amotoki has quit IRC | 19:39 | |
| JayF | https://review.openstack.org/#/c/313800/ was just merged changing the config | 19:39 |
| *** _vsaienko has joined #openstack-infra | 19:39 | |
| *** openstackstatus has quit IRC | 19:39 | |
| *** openstack has joined #openstack-infra | 19:40 | |
| *** sdake_ has joined #openstack-infra | 19:40 | |
| notmorgan | jeblair, Shrews: ^ | 19:41 |
| notmorgan | jeblair, Shrews: now with Fail module instead of .fail_json | 19:41 |
| notmorgan | let me fix the commit message | 19:41 |
| JayF | pabelanger: jroll said it could possibly be that the new jobs aren't configured in jenkins yet? That zuul picks up new jobs faster than jenkins? | 19:41 |
| *** sdake has quit IRC | 19:43 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: When a playbook fails, don't succeed in the error_block https://review.openstack.org/324726 | 19:43 |
| *** markusry has quit IRC | 19:44 | |
| JayF | Is there anywhere I can look to troubleshoot these NOT_REGISTERED jobs? | 19:47 |
| *** _vsaienko has quit IRC | 19:47 | |
| pabelanger | JayF: correct, that is possible | 19:49 |
| JayF | woo, that's much better than the other option: I broke a gate | 19:49 |
| *** nadya has joined #openstack-infra | 19:49 | |
| JayF | recheck it in about 30m or so/ | 19:49 |
| JayF | ? | 19:49 |
| pabelanger | 30mins should work | 19:50 |
| pabelanger | every 15mins JJB runs on our masters to update the jobs | 19:50 |
| JayF | ah so it's just a little racey | 19:50 |
| JayF | that works, thanks :) | 19:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: DO NOT MERGE https://review.openstack.org/324620 | 19:55 |
| *** micole has quit IRC | 19:55 | |
| *** rellerreller has quit IRC | 19:56 | |
| *** david-lyle has quit IRC | 19:57 | |
| *** shamail has left #openstack-infra | 19:58 | |
| *** flwang has quit IRC | 19:59 | |
| anteaya | I can't stop laughing | 20:00 |
| anteaya | at the end of my driveway there is an assumed gravel road | 20:00 |
| anteaya | some contractor got himself the job of sweeping gravel for the afternoon | 20:01 |
| anteaya | we looked at each other and giggled to ourselves quietly | 20:01 |
| anteaya | I have never seen him before | 20:01 |
| anteaya | kudos to him for convincing some city person to pay him to sweep sand | 20:01 |
| anteaya | I'm tempted to see if I can get the customer's number and sell them a contract to bail the lake | 20:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add configure_mirror.sh back to tripleo-centos-7 https://review.openstack.org/324790 | 20:03 |
| *** zeih has joined #openstack-infra | 20:03 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:04 | |
| anteaya | s/assumed/unassumed | 20:04 |
| *** zeih has quit IRC | 20:07 | |
| *** amrith is now known as _amrith_ | 20:07 | |
| *** dprince has quit IRC | 20:08 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Refactor: remove unused functions https://review.openstack.org/324792 | 20:08 |
| *** Sukhdev has quit IRC | 20:09 | |
| *** Sukhdev has joined #openstack-infra | 20:09 | |
| fungi | hrm, need to break for an early dinner. so far most of the answers i've come up with for keep-hierarchy=false on the launcher side would involve using ansible's shell module instead of the command module | 20:09 |
| fungi | or running an additional ansible command | 20:10 |
| fungi | (and feeding output from that into the new command arguments) | 20:10 |
| pramchan | I'm getting the following error when attempting to send the openstack/governance repo for a git review | 20:11 |
| pramchan | fatal: ICLA contributor agreement requires current contact information. | 20:11 |
| pramchan | Anybody seen this error before? | 20:11 |
| pramchan | I checked the ICLA agreement on my account and it is valid | 20:11 |
| *** sputnik13_ has joined #openstack-infra | 20:11 | |
| anteaya | fungi: enjoy dinner | 20:11 |
| anteaya | and I beat the rain home | 20:12 |
| *** Sukhdev has quit IRC | 20:13 | |
| anteaya | pramchan: https://ask.openstack.org/en/question/3413/icla-contributor-agreement-requires-current-contact-information-error-while-using-git-review/ | 20:13 |
| pramchan | thanks anteaya | 20:13 |
| anteaya | pramchan: and I suggest sending your first patch to the sandbox repo | 20:13 |
| pramchan | sorry about that, will google more thoroughly next time | 20:13 |
| anteaya | to practice | 20:13 |
| anteaya | no need to apologizer | 20:13 |
| pramchan | what is the sandbox repo? | 20:13 |
| *** tqtran has joined #openstack-infra | 20:14 | |
| anteaya | pramchan: https://review.openstack.org/#/q/project:openstack-dev/sandbox | 20:14 |
| pramchan | thanks | 20:14 |
| anteaya | pramchan: a place for new contributors to test sending their first patch | 20:14 |
| anteaya | welcome | 20:14 |
| *** sputnik13 has quit IRC | 20:14 | |
| anteaya | you likely will not know what to expect from your actions yet | 20:14 |
| anteaya | you might be embarassed if your first patch is to the governance repo and it doesn't appear the way you like | 20:15 |
| anteaya | and we don't delete patches | 20:15 |
| pramchan | ok, I'll test it with the sandbox first | 20:15 |
| anteaya | good idea | 20:16 |
| anteaya | have you read the developer's manual yet? http://docs.openstack.org/infra/manual/developers.html | 20:16 |
| anteaya | might be a good place to begin | 20:16 |
| anteaya | and depending on what you want to send to governance, reviewers might be looking for you to already have a history of collaborative work within openstack | 20:17 |
| anteaya | you might consider offering a patch to an existing project as a way to meet people and learn the workflow | 20:17 |
| *** ociuhandu has joined #openstack-infra | 20:18 | |
| *** rhallisey has quit IRC | 20:19 | |
| *** rhallisey has joined #openstack-infra | 20:20 | |
| *** rcernin has joined #openstack-infra | 20:20 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Print installed tempest-plugins after tempest run https://review.openstack.org/319465 | 20:24 |
| *** harlowja has joined #openstack-infra | 20:24 | |
| openstackgerrit | Prakash Ramchandran proposed openstack-infra/project-config: add new project meghdwar https://review.openstack.org/319466 | 20:24 |
| *** esikachev has quit IRC | 20:24 | |
| *** moravec has quit IRC | 20:26 | |
| *** nwkarsten has quit IRC | 20:26 | |
| *** bhavik has quit IRC | 20:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Create DIB element for preinstall packages https://review.openstack.org/323513 | 20:27 |
| pramchan | @anteaya: I have worked with openstack in the past with Neutron, I have read the developer's manual, and followed all the steps in the wiki for project creation. I will wait on repo creation by infra team | 20:29 |
| *** moravec has joined #openstack-infra | 20:29 | |
| *** banix has quit IRC | 20:29 | |
| anteaya | pramchan: oh I'm just surprised you encountered the gerrit error you did if you had experience contributing before | 20:29 |
| anteaya | the error you encountered is usually only experienced by folks who have never contributed to openstack before | 20:30 |
| *** nwkarsten has joined #openstack-infra | 20:30 | |
| *** gyee has joined #openstack-infra | 20:31 | |
| *** rbrndt has quit IRC | 20:31 | |
| *** kzaitsev_mb has quit IRC | 20:31 | |
| *** piet_ has joined #openstack-infra | 20:34 | |
| *** e0ne has joined #openstack-infra | 20:34 | |
| *** amotoki has joined #openstack-infra | 20:35 | |
| *** nadya has quit IRC | 20:35 | |
| jeblair | pabelanger, fungi: thanks | 20:37 |
| *** pramchan has quit IRC | 20:39 | |
| *** amotoki has quit IRC | 20:40 | |
| *** tonytan4ever has quit IRC | 20:41 | |
| *** e0ne has quit IRC | 20:41 | |
| *** nwkarsten has quit IRC | 20:42 | |
| *** jcoufal has quit IRC | 20:43 | |
| *** nwkarste_ has joined #openstack-infra | 20:45 | |
| *** e0ne has joined #openstack-infra | 20:45 | |
| *** nwkarst__ has joined #openstack-infra | 20:47 | |
| krotscheck | fungi: Would you mind adding https://review.openstack.org/#/c/314312/ to your review queue? Not super critical, but it'd be nice to have it ready to go. | 20:48 |
| *** nwkars___ has joined #openstack-infra | 20:48 | |
| openstackgerrit | Sahdev Zala proposed openstack-infra/project-config: Add TOSCA-Parser job for Heat-Translator project https://review.openstack.org/324556 | 20:48 |
| *** nwkarste_ has quit IRC | 20:49 | |
| *** roxanagh_ has joined #openstack-infra | 20:50 | |
| openstackgerrit | Sahdev Zala proposed openstack-infra/project-config: Add TOSCA-Parser job for Heat-Translator project https://review.openstack.org/324556 | 20:50 |
| *** moravec has quit IRC | 20:51 | |
| *** joaotargino has quit IRC | 20:51 | |
| *** trown is now known as trown|outtypewww | 20:52 | |
| *** nwkarst__ has quit IRC | 20:52 | |
| *** _ari_ has quit IRC | 20:52 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Introduce `exit` phase https://review.openstack.org/324811 | 20:52 |
| *** julim has quit IRC | 20:53 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:54 | |
| *** roxanagh_ has quit IRC | 20:55 | |
| *** eharney has quit IRC | 20:55 | |
| *** raildo is now known as raildo-afk | 20:56 | |
| *** ayoung has joined #openstack-infra | 20:59 | |
| openstackgerrit | Merged openstack-infra/system-config: Switch to mirrors.kernel.org for centos / epel rsync https://review.openstack.org/324542 | 21:01 |
| *** aeng has joined #openstack-infra | 21:01 | |
| clarkb | krotscheck: ok I will delete the instance shortly | 21:01 |
| clarkb | and done | 21:02 |
| *** kzaitsev_mb has quit IRC | 21:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr: Updated from global requirements https://review.openstack.org/324819 | 21:03 |
| *** thorst has quit IRC | 21:03 | |
| *** _vsaienko has joined #openstack-infra | 21:04 | |
| *** baoli_ has quit IRC | 21:05 | |
| *** thorst has joined #openstack-infra | 21:05 | |
| *** abregman has quit IRC | 21:06 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add missing release jobs for instack & instack-undercloud https://review.openstack.org/324837 | 21:07 |
| *** rhallisey has quit IRC | 21:08 | |
| *** thorst_ has joined #openstack-infra | 21:08 | |
| *** _vsaienko has quit IRC | 21:08 | |
| *** psilvad has quit IRC | 21:09 | |
| *** thorst has quit IRC | 21:10 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:12 | |
| *** thorst_ has quit IRC | 21:12 | |
| *** maxskew has joined #openstack-infra | 21:12 | |
| *** aysyd has quit IRC | 21:13 | |
| *** esikachev has joined #openstack-infra | 21:13 | |
| *** mriedem has quit IRC | 21:15 | |
| *** ldnunes has quit IRC | 21:18 | |
| anteaya | clarkb: how are the babies doing? | 21:18 |
| clarkb | anteaya: very unhappy, but that was expected | 21:18 |
| anteaya | glad they didn't have to do a return to the doctor's/hospital | 21:19 |
| anteaya | and hopefully their time of unhappiness will be brief | 21:19 |
| *** ffrank has joined #openstack-infra | 21:19 | |
| *** kzaitsev_mb has quit IRC | 21:19 | |
| *** claudiub|2 has quit IRC | 21:21 | |
| *** yamahata has quit IRC | 21:21 | |
| clarkb | apparently as they get older the vaccines get nastier and less fun | 21:22 |
| anteaya | oh, :( | 21:23 |
| anteaya | glad you are taking the time to hold them when they are unhappy | 21:23 |
| *** kzaitsev_mb has joined #openstack-infra | 21:25 | |
| *** ayoung has quit IRC | 21:25 | |
| *** maxskew has quit IRC | 21:25 | |
| *** flwang1 has joined #openstack-infra | 21:27 | |
| flwang1 | AJaeger: ping re the distil client :) | 21:27 |
| *** mhickey has quit IRC | 21:27 | |
| *** nwkars___ has quit IRC | 21:28 | |
| *** flwang has joined #openstack-infra | 21:29 | |
| *** thiagop has quit IRC | 21:29 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: [WIP] Add mulitnode CI job support https://review.openstack.org/324777 | 21:30 |
| *** _amrith_ is now known as amrith | 21:30 | |
| *** thorst has joined #openstack-infra | 21:30 | |
| clarkb | flwang1: it is really late in the evening where AJaeger is, anything we can help you with? | 21:31 |
| flwang1 | clarkb: ah, i just wanna make sure if only the TC approved Big tent projects can use the PyPi jobs | 21:31 |
| *** matt-borland has quit IRC | 21:33 | |
| anteaya | flwang1: no anyone can submit releases to pypi | 21:33 |
| anteaya | only big tent can use docs.o.o though | 21:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: Write console in thread https://review.openstack.org/324889 | 21:34 |
| flwang1 | anteaya: i mean a job like this https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n9255 | 21:34 |
| *** thorst has quit IRC | 21:35 | |
| anteaya | right that is pypi-jobs | 21:35 |
| EmilienM | I still see some mirror issues on bluebox http://logs.openstack.org/55/323955/4/check/gate-puppet-tripleo-puppet-unit-3.8-centos-7/335b136/console.html#_2016-06-02_21_04_45_231 | 21:35 |
| EmilienM | pabelanger: anything we can do? | 21:35 |
| anteaya | which as I said any repo can use to submit releases to pypi | 21:35 |
| *** amotoki has joined #openstack-infra | 21:36 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: [WIP] Add mulitnode CI job support https://review.openstack.org/324777 | 21:36 |
| openstackgerrit | Fei Long Wang proposed openstack-infra/project-config: Add new project python-distilclient https://review.openstack.org/318367 | 21:37 |
| *** e0ne has quit IRC | 21:37 | |
| *** e0ne has joined #openstack-infra | 21:38 | |
| flwang1 | anteaya: then i don't understand why AJaeger said my pypi job is still wrong? | 21:38 |
| flwang1 | https://review.openstack.org/#/c/318367/3/jenkins/jobs/projects.yaml | 21:38 |
| *** rcernin_ has joined #openstack-infra | 21:38 | |
| *** e0ne has quit IRC | 21:38 | |
| *** javeriak has quit IRC | 21:39 | |
| clarkb | flwang1: "https://pypi.python.org/pypi/python-distilclient is not owned by openstackci, openstacker is the wrong group." | 21:39 |
| anteaya | flwang1: yes what clarkb said, andreas comment from June 1 | 21:40 |
| flwang1 | clarkb: yep, because now the python-distilclient is maintained by us | 21:40 |
| flwang1 | if we want to use the pypi job, could we just delete the lib from pypi and let the pypi-job take over? | 21:41 |
| clarkb | flwang1: no, just update the existing package to include our user | 21:41 |
| clarkb | flwang1: this step is not automated, the hand over has to happen by one of the package maintainers | 21:42 |
| clarkb | we can't just take over and you don't want to deltee any existing packages | 21:42 |
| *** amotoki has quit IRC | 21:42 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Python 3 Fix: use six.moves.urillib https://review.openstack.org/323014 | 21:42 |
| cloudnull | ping mordred clarkb fungi pabelanger -- I've talked to folks within the osic and we've opened up the GATEWAY_NET to be a shared provider network so from within infra there's no longer a need to use a float on instances. | 21:43 |
| *** ams__ has quit IRC | 21:43 | |
| *** PollyZ has joined #openstack-infra | 21:43 | |
| cloudnull | I'm not sure what will need to be updated to allow infra to use a provider net instead of a float, but its ready now. | 21:44 |
| mwhahaha | looks like we have some inconsistencies with the centos mirrors, http://logs.openstack.org/70/324770/3/check/gate-fuel-library-puppet-unit-3.4-centos-7/24996c5/console.html#_2016-06-02_21_37_37_219 | 21:44 |
| clarkb | cloudnull: \o/ that is awesome news, let me whip up the config change for that | 21:44 |
| *** kzaitsev_mb has quit IRC | 21:44 | |
| flwang1 | clarkb: then what's the user i need to add? :) | 21:45 |
| clarkb | flwang1: the one in AJaeger's comment 'openstackci' | 21:45 |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Run Ironic Inspector jobs with TinyIPA https://review.openstack.org/323994 | 21:46 |
| openstackgerrit | Merged openstack-infra/project-config: New project: js-openstack-lib https://review.openstack.org/314312 | 21:47 |
| jeblair | Shrews: can you look at my reply in https://review.openstack.org/324137 ? | 21:47 |
| Shrews | jeblair: certainly | 21:48 |
| *** e0ne has joined #openstack-infra | 21:48 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use provider net on OSIC https://review.openstack.org/324893 | 21:48 |
| clarkb | cloudnull: ^ that should do it | 21:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: Write console in thread https://review.openstack.org/324889 | 21:48 |
| jeblair | anteaya: is there a way to load a story given its number? | 21:49 |
| jeblair | searching for it does not work | 21:49 |
| jeblair | (also, typing numbers into the search box at the top produces a stream of errors) | 21:50 |
| *** kzaitsev_mb has joined #openstack-infra | 21:50 | |
| pabelanger | cloudnull: woah, that is awesome | 21:50 |
| notmorgan | jeblair: https://storyboard.openstack.org/#!/story/<id> ? | 21:50 |
| jeblair | notmorgan: i can never remember the punctuation for that :) | 21:50 |
| notmorgan | jeblair: not a great option, just ... a way to do it? | 21:50 |
| Shrews | jeblair: yeah. fwiw, i couldn't see anyway the current code would actually fail, but i was thinking ahead if somebody added something else that may attempt to use that flag. i like how you changed it. | 21:50 |
| notmorgan | jeblair: i happened to have one open so... :) | 21:51 |
| notmorgan | jeblair: not that i knew off the top of my head. | 21:51 |
| pabelanger | clarkb: +2 | 21:51 |
| jeblair | Shrews: cool, i'll type that into an editor then :) | 21:51 |
| pabelanger | EmilienM: fix should be landing now | 21:51 |
| EmilienM | cool | 21:52 |
| EmilienM | thx! | 21:52 |
| anteaya | jeblair: so on this story: https://storyboard.openstack.org/#!/story/2000621 | 21:52 |
| pabelanger | EmilienM: might take about 60mins to refresh the mirrors | 21:52 |
| pabelanger | I cannot do it now, have to run | 21:52 |
| anteaya | jeblair: ah searching | 21:52 |
| flwang1 | clarkb: cool, got it, thanks | 21:52 |
| anteaya | jeblair: I do get errors but I also get a drop down menu with the story with the id in front if I copy paste the id | 21:53 |
| anteaya | jeblair: which id are you trying and I can try too | 21:53 |
| jeblair | pabelanger: while i was testing the zuul_runner hanging, i tried to reproduce the ansible-failing-to-timeout behavior with some simple tests (including by killing the immediate child of zuul_runner without zuul_runner itself dying). i was unable to do so. | 21:53 |
| *** esikachev has quit IRC | 21:54 | |
| jeblair | anteaya: i do now. i can no longer say whether i saw that before or not. | 21:54 |
| jeblair | anteaya: 2000619 | 21:54 |
| *** shashank_hegde has quit IRC | 21:55 | |
| jeblair | anteaya: it's possible i missed the dropdown, or it's possible that it wasn't there. several things have changed since then (i have logged in, and i have modified the story) | 21:55 |
| anteaya | https://storyboard.openstack.org/#!/story/2000619 | 21:55 |
| jeblair | yeah, i got there :) | 21:55 |
| anteaya | fair enough, also boatloads of errors | 21:55 |
| jeblair | yep | 21:56 |
| anteaya | it tries to search on every character as one types, kind of a hinderance to seeing the drop down bit once it arrives | 21:56 |
| anteaya | I'll craft a story | 21:56 |
| *** timello has quit IRC | 21:57 | |
| *** markvoelker has quit IRC | 21:58 | |
| *** tlian has quit IRC | 21:58 | |
| *** burgerk has quit IRC | 21:58 | |
| *** tlian has joined #openstack-infra | 21:58 | |
| *** xyang1 has quit IRC | 21:59 | |
| *** kzaitsev_mb has quit IRC | 21:59 | |
| anteaya | https://storyboard.openstack.org/#!/story/2000623 | 21:59 |
| *** shashank_hegde has joined #openstack-infra | 22:01 | |
| *** timello has joined #openstack-infra | 22:01 | |
| *** yujunz has joined #openstack-infra | 22:02 | |
| *** gordc has quit IRC | 22:02 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: create release-check high priority queue https://review.openstack.org/324768 | 22:03 |
| clarkb | any other infra-root or project-config core want to give https://review.openstack.org/#/c/324893/ a go? it has cloudnull and pabelanger's positive votes | 22:03 |
| *** megm has quit IRC | 22:03 | |
| clarkb | should help us better utilize the osic cloud (which would be great because yay more cloud) | 22:03 |
| anteaya | dhellmann: we have had the release priority queue discussion in the past, but we can have it again | 22:04 |
| anteaya | dhellmann: mostly it comes down to the fact that we only have 3 settings for priority | 22:05 |
| jeblair | clarkb: +2, do you want to +3 and babysit? | 22:05 |
| *** kzaitsev_mb has joined #openstack-infra | 22:05 | |
| clarkb | jeblair: yup will do | 22:05 |
| anteaya | dhellmann: and noone wants to take low, so everyone wants high | 22:05 |
| dhellmann | anteaya : this would only apply to the openstack/releases repo, for requests to do releases. the other jobs are already in high priority queues | 22:05 |
| *** megm has joined #openstack-infra | 22:05 | |
| anteaya | right fungi and I had this discussion a year ago | 22:06 |
| *** rcernin_ has quit IRC | 22:06 | |
| anteaya | with me being on the side of releases | 22:06 |
| openstackgerrit | xiaodongwang991481 proposed openstack-infra/project-config: add networking-sfc gate for functional tests and tempest tests https://review.openstack.org/324135 | 22:06 |
| anteaya | dhellmann: but that is okay, maybe you will have greater success than I | 22:06 |
| mtreinish | AJaeger: did you ever land the patch which added subunit output to the translation jobs? | 22:06 |
| dhellmann | ok, I don't know the history, I just know that we couldn't finish processing releases today | 22:06 |
| anteaya | as you can tell I was unsuccessful | 22:06 |
| anteaya | dhellmann: yup, I understand | 22:07 |
| mtreinish | AJaeger: I don't see any results in the post queue, wanted to make sure you didn't get stuck | 22:07 |
| jeblair | anteaya, dhellmann: i'm not taking sides here -- but i would like to say that i believe one of the reasons there is tension around this is because of the 'clean check' gate requirement. | 22:07 |
| dhellmann | jeblair : I'm not sure what that means? | 22:08 |
| dhellmann | "clean check"? | 22:08 |
| jeblair | it means that we added a requirement to the gate pipeline some time ago that in order to enter it, a change must have passed check | 22:08 |
| dhellmann | jeblair : if there's another way to let us process requests more quickly, I'm open to that for sure, this was just a knob I knew was available for turning | 22:08 |
| jeblair | my personal vision of how zuul should be used does not involve that | 22:08 |
| *** yujunz has quit IRC | 22:09 | |
| jeblair | dhellmann: i would be interested to know whether you thought being able to skip check and go straight to gate would be helpful in this case, or whether you want check results first | 22:09 |
| dhellmann | ah ok. I may have to give that some thought. I'm not sure how I feel. We've found the check queue quite valuable, but I suppose if we had a way to bypass it for a revision of a patch that might be ok | 22:09 |
| dhellmann | I'd be worried about missing something about an update to a patch (like if someone had 2 things to fix but only fixed one and the reviewer didn't notice) | 22:10 |
| dhellmann | or if they introduced a new issue, I guess | 22:10 |
| dhellmann | although that would be caught in gate | 22:10 |
| jeblair | (it's certainly helpful for some infra work; i could see it going either way for this use case, so yeah, if you cipher on it a bit, let me know) | 22:10 |
| dhellmann | yeah, I'll give that a bit of thought, thanks for suggesting it | 22:10 |
| *** moravec has joined #openstack-infra | 22:11 | |
| jeblair | (my hope is that in zuulv3, we can avoid that for infra projects, even if openstack projects still want it) | 22:11 |
| *** sputnik13_ has quit IRC | 22:11 | |
| *** ayoung has joined #openstack-infra | 22:12 | |
| *** banix has joined #openstack-infra | 22:13 | |
| *** PollyZ has quit IRC | 22:14 | |
| *** PollyZ has joined #openstack-infra | 22:19 | |
| *** shashank_hegde has quit IRC | 22:19 | |
| *** openstackgerrit has quit IRC | 22:19 | |
| *** Julien-zte has joined #openstack-infra | 22:20 | |
| *** openstackgerrit has joined #openstack-infra | 22:20 | |
| *** edmondsw has quit IRC | 22:28 | |
| *** mdrabe has quit IRC | 22:29 | |
| *** akshai has quit IRC | 22:29 | |
| *** dimtruck is now known as zz_dimtruck | 22:29 | |
| *** IlyaG has joined #openstack-infra | 22:30 | |
| jlk | What's the repo I need to make a review for to turn off the blue box cloud while we rebuild it? | 22:33 |
| *** weshay has quit IRC | 22:33 | |
| clarkb | jlk: openstack-infra/project-config | 22:33 |
| clarkb | jlk: the nodepool/nodepool.yaml file specifically | 22:33 |
| *** abregman has joined #openstack-infra | 22:35 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/project-config: Turn off bluebox while the cluster is rebuilt https://review.openstack.org/324908 | 22:38 |
| openstackgerrit | Merged openstack-infra/project-config: Use provider net on OSIC https://review.openstack.org/324893 | 22:44 |
| *** akshai has joined #openstack-infra | 22:45 | |
| *** vhosakot has quit IRC | 22:46 | |
| *** shashank_hegde has joined #openstack-infra | 22:46 | |
| *** akshai_ has joined #openstack-infra | 22:46 | |
| EmilienM | hey, I have an outstanding request for releasing tripleo we need this patch in project-config, https://review.openstack.org/#/c/324837/ - if someone can have a look, thanks a lot | 22:48 |
| *** akshai has quit IRC | 22:50 | |
| craige | o/ | 22:50 |
| clarkb | osic nodes are going ready and nodepool.yaml is updated. Not sure if they have stopped using fips yet as there are openstack client issues on the puppetmaster | 22:51 |
| clarkb | oh I may have the cloud name wrong | 22:51 |
| clarkb | | addresses | GATEWAY_NET=172.99.106.224 | 22:52 |
| *** roxanagh_ has joined #openstack-infra | 22:52 | |
| clarkb | I think this means we owe cloudnull beers or other preferred beverage | 22:52 |
| fungi | cloudnull: clarkb: awesome! that is of course the ultimate fix for floating ips in clouds | 22:53 |
| *** Adri2000 has quit IRC | 22:53 | |
| fungi | maybe the aforementioned bluebox rebuild will yield similar success | 22:54 |
| *** Adri2000 has joined #openstack-infra | 22:54 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add release command https://review.openstack.org/324137 | 22:55 |
| clarkb | I should check that our FIP consumption drops to zero over the next hour or so | 22:55 |
| jeblair | Shrews: er, i think your suggestion about the lock was the best after all ^ :) | 22:55 |
| clarkb | its currently at 3 FIPs used | 22:55 |
| notmorgan | jeblair: that makes a lot of sense. | 22:56 |
| notmorgan | jeblair: it's just playing it super safe | 22:56 |
| *** roxanagh_ has quit IRC | 22:56 | |
| jeblair | clarkb, fungi: can you approve this at your leisure: https://review.openstack.org/323594 it's not urgent, but it will be nice to have it in before we productionize zl01 again | 22:57 |
| notmorgan | so, question... you want that to block on the lock? or fail if it can't accquire the lock? | 22:57 |
| *** markvoelker has joined #openstack-infra | 22:58 | |
| Shrews | jeblair: coolio | 23:00 |
| jeblair | notmorgan: block i think -- the release function starts by 'pausing' which means that by the time it acquires the lock, we should no longer be waiting for a job | 23:00 |
| Shrews | fungi: clarkb: there was a FIP fix in osic? | 23:01 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add documentation about openstack-health deployment https://review.openstack.org/237268 | 23:01 |
| clarkb | jeblair: vcsrepo may update after which file copy? | 23:01 |
| clarkb | Shrews: yes we stopped using FIPs | 23:01 |
| Shrews | clarkb: WOO! | 23:01 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack-infra/project-config: Add manila-specs to specs.yaml https://review.openstack.org/324941 | 23:01 |
| Shrews | great fix | 23:01 |
| *** gouthamr has joined #openstack-infra | 23:01 | |
| clarkb | see the in use graph at http://grafana.openstack.org/dashboard/db/nodepool-osic to see how awesomely that affected things | 23:02 |
| jeblair | notmorgan: (getJob() is the only thing inside a critical section that could block -- and pausing gets us out of that) | 23:02 |
| *** markvoelker has quit IRC | 23:03 | |
| clarkb | cloudnull: see above grafana link. Really illustrates how this helped | 23:03 |
| *** ayoung has quit IRC | 23:04 | |
| *** pahuang has joined #openstack-infra | 23:06 | |
| *** ociuhandu has quit IRC | 23:06 | |
| *** tonytan4ever has joined #openstack-infra | 23:06 | |
| *** ociuhandu_ has joined #openstack-infra | 23:06 | |
| *** sulo has quit IRC | 23:06 | |
| clarkb | jeblair: the 3 FIP instances in osic are all assocatied with zl01 and in used state for a little over 4 hours, can those all be safely deleted? | 23:06 |
| jeblair | clarkb: yes | 23:07 |
| *** r-daneel has joined #openstack-infra | 23:07 | |
| clarkb | jeblair: should I look for them in other providers too? | 23:07 |
| *** sulo has joined #openstack-infra | 23:07 | |
| jeblair | clarkb: yes (they will get cleaned up, but if you want to recover more quota, yes) | 23:07 |
| jeblair | clarkb: i do not understand the behavior i describe in the 2nd paragraph in that commit message, though i am fairly certain i saw it. | 23:07 |
| clarkb | I mostly just wanted to make sure we were off the FIPs and didn't leave a mess behind with the switch. Will let the other proviers do their own zl01 cleanup | 23:08 |
| jeblair | clarkb: it seems that the reload trigger should happen only if we copied files, and we should only copy files after vcs repo updates, and even though zuul points to the vcs repo, not the copy target, that should be okay. | 23:09 |
| *** Qiming has joined #openstack-infra | 23:09 | |
| jeblair | clarkb: oh wait, i think i get it | 23:09 |
| *** rcernin has quit IRC | 23:10 | |
| *** pahuang has quit IRC | 23:10 | |
| jeblair | clarkb: the file copy isn't subscribed to the vcs repo update, it's independent. so puppet might check whether files need to be copied, decide nothing needs to be done, then do a vcs repo update. next puppet pass checks the files firest, does the copy, then updates zuul. | 23:11 |
| clarkb | its really neat to compare the performance graphs in grafana with this, time to ready drops, errors drop, building and used graphs inverted | 23:12 |
| jeblair | (meanwhile, zuul is pointing at the old location; it's not an error, it's just that if you inspect the filesystem, what you think zuul is using is not what it's using) | 23:12 |
| jeblair | (it would be best if we could also make the File resource get evaluated after the Vcsrepo resource....) | 23:13 |
| clarkb | jeblair: so by using the copied location we get consistent results that may be a puppet cycle behind | 23:13 |
| jeblair | clarkb: yep. | 23:13 |
| jeblair | clarkb: so that change gets us "zuul is always using /etc/jenkins_jobs/config". we should write another change that eliminates the cycle behind thing. | 23:14 |
| clarkb | ya | 23:14 |
| jhesketh | Morning | 23:15 |
| anteaya | morning jhesketh | 23:15 |
| jeblair | jhesketh: good morning; zl01 is out of production, here are the problems we're working on: https://storyboard.openstack.org/#!/story/2000619 | 23:15 |
| anteaya | jhesketh: so it appears all your election bits are merged | 23:15 |
| *** pt_15 has quit IRC | 23:16 | |
| jhesketh | anteaya: cool, thanks | 23:16 |
| anteaya | jhesketh: has anyone tested the path for the documenation do you know? | 23:16 |
| jeblair | clarkb: i kind of suspect we have this problem with jjb on jenkins masters today | 23:16 |
| jeblair | clarkb: at least, potentially -- maybe puppet is coincidentally sorting resources into the 'correct' order there | 23:17 |
| notmorgan | jeblair: okie | 23:18 |
| jhesketh | anteaya: not sure sorry | 23:18 |
| *** PollyZ has quit IRC | 23:18 | |
| *** IlyaG has quit IRC | 23:18 | |
| anteaya | jhesketh: no worries, I just think we should test the path well before elections start again | 23:19 |
| clarkb | jeblair: for those tasks that say review how do I get the change numbers? | 23:20 |
| *** david-lyle has joined #openstack-infra | 23:20 | |
| jeblair | clarkb: i made sure they all have "topic:v25" | 23:21 |
| jeblair | clarkb: i did not manually add them to storyboard tho | 23:21 |
| jeblair | jhesketh, mordred: if you have time, you might see if fungi or pabelanger can hand their tasks off to you during their evenings. | 23:22 |
| openstackgerrit | Merged openstack-infra/project-config: Turn off bluebox while the cluster is rebuilt https://review.openstack.org/324908 | 23:22 |
| *** hongbin has quit IRC | 23:23 | |
| jhesketh | jeblair: sure. fungi, pabelanger let me know how you go | 23:23 |
| *** pahuang has joined #openstack-infra | 23:24 | |
| *** abregman is now known as abregman|afk | 23:24 | |
| clarkb | can we maybe not use the same topic for python3 work? | 23:25 |
| mtreinish | jeblair: here's yet another example of the new zuul stuff being weird: http://logs.openstack.org/62/324662/1/gate/gate-grenade-dsvm-multinode/63cf2f6/console.html | 23:26 |
| clarkb | jeblair: is https://review.openstack.org/#/c/324889/ the fix for zuul_runner hanging? | 23:26 |
| mtreinish | this time it was in the gate queue, and reported success on a failure (it also took 4hrs...) | 23:26 |
| *** Kaiyan has quit IRC | 23:28 | |
| pabelanger | jeblair: Ya, I didn't make much head way on the timeout issue this afternoon. Was trying to find the best way to live stream our ansible-playbook subprocess. Have some code going locally, but need to push it up for review. | 23:29 |
| *** zhurong has joined #openstack-infra | 23:36 | |
| pabelanger | clarkb: wow, looking at http://grafana.openstack.org/dashboard/db/nodepool-osic is dramatic | 23:36 |
| clarkb | pabelanger: yup its night and day | 23:36 |
| pabelanger | clarkb: going to update grafyaml now | 23:37 |
| bkero | Apparently nobody needs floating IPs | 23:37 |
| pabelanger | bkero: friends don't let friends deploy clouds using FIPs ;) | 23:37 |
| jlk | sorry :( | 23:37 |
| bkero | pabelanger: just make a v6 only cloud and give every host a permanent IP :) | 23:38 |
| pabelanger | jlk: All good! | 23:38 |
| pabelanger | bkero: ++ | 23:38 |
| *** amitgandhinz has quit IRC | 23:38 | |
| *** EricGonczer_ has quit IRC | 23:38 | |
| *** amotoki has joined #openstack-infra | 23:39 | |
| *** Qiming has quit IRC | 23:40 | |
| *** zhurong has quit IRC | 23:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Remove FIPs graphing from OSIC https://review.openstack.org/324947 | 23:41 |
| pabelanger | clarkb: ^ | 23:41 |
| clarkb | pabelanger: +2 you can probably just self approve with one +2 since its correcting a minor UI thing | 23:42 |
| pabelanger | clarkb: WFM | 23:44 |
| *** EricGonczer_ has joined #openstack-infra | 23:44 | |
| clarkb | jeblair: comment on https://review.openstack.org/#/c/324889/2 its a minor issue but worried about confusion if we go back to productio nwithout addressing it so I left a -1 | 23:44 |
| *** amotoki has quit IRC | 23:44 | |
| *** xarses has quit IRC | 23:46 | |
| pabelanger | clarkb: do you mind reviewing: 323865. Updates our skip tests to include other-requirements.txt file for project-config | 23:47 |
| clarkb | pabelanger: /me grumbles about the test skips in general | 23:48 |
| *** EricGonczer_ has quit IRC | 23:49 | |
| mtreinish | clarkb, pabelanger, fungi, jhesketh: if you get a sec: https://review.openstack.org/#/c/324103/1 will prevent 500s on health.o.o | 23:49 |
| pabelanger | clarkb: I like the idea in general, but gets complicated fast | 23:49 |
| clarkb | pabelanger: yes I think we should remove all of them they require too much handholding or we can just run the tests and not worry | 23:49 |
| mtreinish | and a related change: https://review.openstack.org/323658 | 23:49 |
| *** Julien-zte has quit IRC | 23:49 | |
| *** Guest1281 has quit IRC | 23:50 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Part two of kolla-mesos retirement https://review.openstack.org/311574 | 23:50 |
| fungi | jeblair: jhesketh: any feel for whether running rsync under the shell module (so it can include a command substitution) would be better or worse than having ansible run an additional command and try to collect its stdout to pass into the argument list for the rsync? | 23:50 |
| sdake_ | AJaeger there ya go, should be ready to merge now :) | 23:50 |
| pabelanger | clarkb: I actually think once we land bindep support on project-config, we'll shave a good amount of time of our jobs. | 23:50 |
| pabelanger | clarkb: So, I'd be happy to revisit the skip-test logic | 23:51 |
| jeblair | mtreinish: thanks -- we have identified both problems and pulled it out of prod; we have a fix for the false-success issue, and half a fix for the timeout issue; still working on the other half. | 23:51 |
| jeblair | clarkb: yes, 889 is the zuul_runner hang fix | 23:51 |
| clarkb | pabelanger: its not a matter of revisiting the logic, its an issue of anytime we make a change to our repos those skips break | 23:52 |
| clarkb | pabelanger: then projects start complaining not realizing they broke themselves, then someone has to fix it and it generates more reviews which btw runs more tests :) where the alternative is to just run the tests consistently and not worr yabout it | 23:52 |
| *** roxanaghe has joined #openstack-infra | 23:52 | |
| clarkb | so my recommendation would be to just delete those skips and not worry about it | 23:53 |
| clarkb | rather than editing them every few weeks to accomodate $newthing | 23:53 |
| *** kzaitsev_mb has quit IRC | 23:53 | |
| jeblair | clarkb: responded on 889; i'll respin it if you still think we need it | 23:53 |
| pabelanger | clarkb: I don't have a strong opinion to disagree. Let me update the patch | 23:53 |
| clarkb | jeblair: oh thats a good point, readline should read until newline or EOF | 23:54 |
| *** sdake has joined #openstack-infra | 23:54 | |
| jeblair | fungi: i think rsync under shell (instead of synchronize i'm guessing?) is okay, and the fewer tasks the better. we're already doing rsync in the second stage because we can't use synchronize. | 23:56 |
| *** sdake_ has quit IRC | 23:56 | |
| *** roxanaghe has quit IRC | 23:57 | |
| *** sdake has quit IRC | 23:57 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Remove skip test logic for project-config https://review.openstack.org/323865 | 23:57 |
| pabelanger | clarkb: ^ | 23:57 |
| *** sdake has joined #openstack-infra | 23:57 | |
| *** sdague has quit IRC | 23:58 | |
| *** Goneri has quit IRC | 23:58 | |
| jeblair | fungi: (if we decide to use another task, i might suggest 'synchronize from node to controller; modify tree on controller; rsync controller to target' as an alternative to 'collect file list from node; synchronize from node to controller using file list; rsync controller to target' -- which is i think what you were suggesting as the alternative to switching to "rsync `find`" or something...? | 23:58 |
| *** dingyichen has joined #openstack-infra | 23:59 | |
| fungi | jeblair: expanding the list of sources with a single find | 23:59 |
| fungi | but close, yes | 23:59 |
| *** xarses has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!