| jhesketh | clarkb: so, the ones it fails on appear to reach the retry limit | 00:00 |
|---|---|---|
| clarkb | mtreinish: so really probably what it needs is a non boolean flag that determines how far along you get | 00:00 |
| *** abhirc has quit IRC | 00:00 | |
| clarkb | mtreinish: no tempest is a service you have to enable | 00:00 |
| *** abhirc has joined #openstack-infra | 00:00 | |
| jhesketh | clarkb: and during that time continues to fetch the same number of bytes each time | 00:00 |
| clarkb | jhesketh: yup | 00:00 |
| jhesketh | (rather than -c continuing the download) | 00:00 |
| jhesketh | so I'm dubious this will fix it, but it's worth a try | 00:00 |
| clarkb | jhesketh: gotcha | 00:01 |
| mtreinish | clarkb: sure, but it's default on iirc. Where in d-g.sh does it set that in the localrc for devstack if it's needed? | 00:01 |
| armax | mtreinish: posting a patch now to unveil the failure mode I was observing | 00:01 |
| clarkb | mtreinish: via features.yaml | 00:01 |
| *** dimtruck is now known as zz_dimtruck | 00:01 | |
| clarkb | which dumps tempest in enable_services | 00:01 |
| fungi | so so much teh scrollbax | 00:02 |
| mtreinish | clarkb: ah, that's what I was forgetting sry. | 00:03 |
| *** dangers is now known as dangers_away | 00:03 | |
| mtreinish | clarkb: was the comment about tempest-lib in response to my question regarding spinning up functional testing in projects? | 00:03 |
| clarkb | mtreinish: yes the only reason we have this problem is tempest is being misuesd | 00:04 |
| mtreinish | armax: nah, I got what was missing now, clarkb showed me what I was missing | 00:04 |
| clarkb | the black box integration tests are being used to also run functional tests | 00:04 |
| clarkb | which is probably expedient but I thought tempest-lib exists to solve this more properly | 00:04 |
| armax | clarkb: my udnerstanding is that tempestlib is not quite there yet | 00:05 |
| *** ddieterly has quit IRC | 00:05 | |
| *** oomichi has joined #openstack-infra | 00:05 | |
| fungi | morganfainberg: you are single-handedly restoring my faith in humanity | 00:06 |
| armax | mtreinish: ya, I think that was what I found out while playing with d-g, but I must have blanked out | 00:06 |
| armax | :) | 00:06 |
| mtreinish | clarkb: yeah tempest-lib is there for breaking out the common bits so people can write there own functional tests using the bits from tempest they need | 00:06 |
| mtreinish | but it's still getting rolling, we haven't broken out all the common pieces yet. The only thing people can really do with it now is the cli testing | 00:07 |
| clarkb | I see | 00:07 |
| mtreinish | (which jogo and I started porting novaclient to use the other day) | 00:07 |
| clarkb | jogo: neutron aiopcpu does pass so its likely my networking changes that broke it | 00:07 |
| clarkb | (my networking changes are there just for nova network) | 00:07 |
| fungi | Ryan_Lane: long time no ping! hope things are well | 00:08 |
| Ryan_Lane | :) | 00:08 |
| Ryan_Lane | they are | 00:08 |
| Ryan_Lane | I got tired of waiting for barbican, so I'm writing a credential manager | 00:08 |
| *** jgrimm is now known as zz_jgrimm | 00:09 | |
| *** fandi has quit IRC | 00:09 | |
| Ryan_Lane | alas, it's pretty heavily based on AWS services | 00:09 |
| *** mriedem has joined #openstack-infra | 00:10 | |
| *** amotoki has joined #openstack-infra | 00:10 | |
| fungi | eek | 00:10 |
| Ryan_Lane | :D | 00:10 |
| Ryan_Lane | that said, it's all python! :D | 00:10 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/devstack-gate: Add variable to allow Tempest to disable the test execution https://review.openstack.org/148052 | 00:10 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Allow JJB to set connection timeout to jenkins server https://review.openstack.org/151472 | 00:11 |
| mtreinish | clarkb: but tempest-lib probably won't solve the setup issue. People will probably still need to setup separate devstack envs to run the tests with | 00:11 |
| fungi | Ryan_Lane: i hear barbican is coming along, at least. if you have done similar things now, they might appreciate reviews/patches ;) | 00:11 |
| clarkb | mtreinish: well tempest isn't solving that either | 00:11 |
| clarkb | mtreinish: the issue is literally need a working tmepest | 00:11 |
| clarkb | mtreinish: whcih tempest-lib will solve | 00:11 |
| Ryan_Lane | well, I'm only a week into mine. the scope of mine is also much, much smaller | 00:11 |
| *** craigbr has quit IRC | 00:11 | |
| *** ashleighfarnham has quit IRC | 00:11 | |
| openstackgerrit | Merged openstack-infra/puppet-project_config: Add missing LICENSE file https://review.openstack.org/151448 | 00:11 |
| *** tteggel_ is now known as tteggel | 00:14 | |
| *** tteggel has quit IRC | 00:14 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: Add missing LICENSE file https://review.openstack.org/151447 | 00:14 |
| *** tteggel has joined #openstack-infra | 00:14 | |
| *** amotoki has quit IRC | 00:15 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-httpd: Add missing LICENSE file https://review.openstack.org/151432 | 00:16 |
| *** jaypipes has joined #openstack-infra | 00:17 | |
| *** shivharis has joined #openstack-infra | 00:18 | |
| clarkb | mtreinish: fungi I just approved the subunit worker revert. Once that gets in I will restart the workers | 00:18 |
| *** tjones1 has quit IRC | 00:18 | |
| mtreinish | clarkb: ok cool, thanks | 00:19 |
| mtreinish | IIRC we were only running one process before | 00:19 |
| clarkb | mtreinish: ok | 00:20 |
| mtreinish | I probably should update the policy to indicate that with 4 it seems to eat all the rams | 00:20 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-vcsrepo: Add missing LICENSE file https://review.openstack.org/151454 | 00:20 |
| fungi | i'm not sure that ^ one actually makes sense to approve | 00:20 |
| openstackgerrit | Merged openstack-infra/puppet-yum: Add missing LICENSE file https://review.openstack.org/151456 | 00:21 |
| fungi | the original puppet-vcsrepo seems to be gpl v2+ not apl2 | 00:21 |
| clarkb | fungi: there was a conversation that happened where puppet said "oh ya thats the wrong license" | 00:21 |
| * mtreinish flight's landing, is out for the night | 00:21 | |
| clarkb | fungi: I think it was just missed when they did their mass relicensing | 00:21 |
| openstackgerrit | Merged openstack-infra/puppet-storyboard: Add missing LICENSE file https://review.openstack.org/151452 | 00:22 |
| openstackgerrit | Merged openstack-infra/puppet-redis: Add missing LICENSE file https://review.openstack.org/151449 | 00:22 |
| *** mwagner_lap has joined #openstack-infra | 00:23 | |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Revert "Add support to the subunit workers to reuse zuul uuids" https://review.openstack.org/151133 | 00:25 |
| *** fandi has joined #openstack-infra | 00:25 | |
| *** sabeen has quit IRC | 00:27 | |
| *** otter768 has joined #openstack-infra | 00:27 | |
| fungi | clarkb: 151455 isn't going to pass without a rakefile. i'll add one | 00:28 |
| openstackgerrit | Merged openstack-infra/puppet-ssh: Add missing LICENSE file https://review.openstack.org/151451 | 00:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-vinz: Add missing LICENSE file https://review.openstack.org/151455 | 00:29 |
| clarkb | fungi: sorry I am mostly mindlessly going "license file? recheck" | 00:29 |
| fungi | heh | 00:29 |
| clarkb | I thought I could get away with that | 00:29 |
| jeblair | clarkb, fungi, jhesketh: http://paste.openstack.org/show/163891/ | 00:29 |
| openstackgerrit | Ilya Alekseyev (irc:adiantum) proposed openstack-infra/system-config: Add list for discussions in russian openstack-ru https://review.openstack.org/151483 | 00:29 |
| *** crc32 has quit IRC | 00:30 | |
| jeblair | clarkb, fungi, jhesketh: given that git tree structure, if zuul enqueued the * change, which order do you think is correct? | 00:30 |
| fungi | clarkb: keep in mind that i added them to all openstack-infra/puppet-.* projects even if they predated the module split, so some (like puppet-vinz) might be plain broken | 00:30 |
| *** david-lyle is now known as david-lyle_afk | 00:31 | |
| jhesketh | jeblair: by git tree structure do you mean dependencies? | 00:31 |
| jeblair | jhesketh: yep | 00:31 |
| jhesketh | ie either through gerrit or the depends-on commit msg | 00:31 |
| jeblair | jhesketh: yeah, either one, though i don't have more than one parent here, so it's safe to assume just git structure for now | 00:31 |
| fungi | jeblair: 1 and 2 are in a dependent pipeline together? if so, how? | 00:31 |
| jhesketh | jeblair: surely the two scenarios there are the same except with different names? | 00:32 |
| jeblair | clarkb, fungi, jhesketh: (2) is the shortest path to getting C morged, but (1) keeps the children closest to their parents | 00:32 |
| jeblair | fungi: 1 and 2 are alternate scenarios | 00:32 |
| fungi | aha | 00:32 |
| clarkb | jeblair: I would do a breadth first search | 00:32 |
| *** otter768 has quit IRC | 00:32 | |
| *** dims has quit IRC | 00:32 | |
| clarkb | jeblair: because that should drop out merge conflicts earlier? | 00:32 |
| clarkb | or maybe it will make more merge conflicts | 00:32 |
| fungi | i still not understanding the question. in 1 the * change is D and in 2 the * change is C. each has its own parentage and so depending on whether you enqueued D or C you would use the history associated with that commit | 00:33 |
| jhesketh | jeblair: I'm confused by the two scenarios... were you meant to * C in both of them? | 00:33 |
| jeblair | clarkb: i _think_ 2 reduces potential for merge conflicts for the actual change added. of course, this is actually a bit weird because it's very hard to get into that state since you have to ignore events in order to do it | 00:34 |
| *** jaypipes has quit IRC | 00:34 | |
| jeblair | jhesketh: nope, enqueuing the commit in the same position regardless -- only the _outcome_ of the linearization changes in the two scenarios | 00:34 |
| jeblair | fungi: yeah ^ the same commit is being enqueued both times -- i'm just asking which ordered outcome is correct | 00:35 |
| jeblair | fungi: (the zuul queue is ABCDEFG in both cases, so you see which commits those point to changes) | 00:35 |
| openstackgerrit | Merged openstack-infra/puppet-tmpreaper: Add puppet module files for tmpreaper https://review.openstack.org/151469 | 00:35 |
| *** waynr has quit IRC | 00:36 | |
| *** fandi has quit IRC | 00:36 | |
| fungi | jeblair: i assumed each of a, b, c, d, e, f and g would just be cherry-picked onto the branch tip in series in a dependent pipeline in that case | 00:36 |
| jeblair | i'm starting to think that (1) is better because it matches what you would get if you enqueued A, then B, then D | 00:36 |
| fungi | jeblair: so the history would look like neither of those and would just be linear | 00:36 |
| *** zz_dimtruck is now known as dimtruck | 00:37 | |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Add missing LICENSE file https://review.openstack.org/151453 | 00:37 |
| jeblair | fungi: yes it would, but the order of the commits is different | 00:37 |
| fungi | i'm very clearly missing a key part of the problem statement | 00:37 |
| jeblair | fungi: let me redraw | 00:37 |
| jhesketh | yeah I'm not sure I follow either, but I just thought that was me being silly | 00:37 |
| fungi | i thought zuul enqueued them chronologically in trigger event order | 00:37 |
| fungi | oh, though i guess it does skip events and pull changes in immediately for gerrit dependencies | 00:38 |
| marun | who to talk to about storyboard? | 00:38 |
| jeblair | fungi: yep. easiest way to imagine this is a cold start with approvals on everything. then someone adds C or D | 00:38 |
| jeblair | fungi, clarkb, jhesketh: http://paste.openstack.org/show/163892/ | 00:38 |
| *** derekh has joined #openstack-infra | 00:39 | |
| *** salv-orlando has quit IRC | 00:39 | |
| *** salv-orlando has joined #openstack-infra | 00:39 | |
| jeblair | marun: i'm happy to talk about it, or may further direct your call as needed :) | 00:39 |
| marun | jhesketh: thanks for the review on skip-if. I'll have a patch up shortly with scenario testing working. | 00:39 |
| clarkb | jeblair: ya so its basically breadth first or depth first after B | 00:39 |
| *** tjones1 has joined #openstack-infra | 00:39 | |
| *** markmcclain has joined #openstack-infra | 00:39 | |
| jhesketh | marun: no trouble, it looks cool. I'd also like to see it as an equeue-if scenario too, (replacing the file match thingy) but we can figure that out later | 00:40 |
| clarkb | jeblair: my initial reaction is still to go with C to cut off long patch chains early rather than late | 00:40 |
| *** wenlock has quit IRC | 00:40 | |
| fungi | jeblair: ahh, got it, so green field no events zuul constructing a test order out of the blue | 00:40 |
| *** markmcclain has quit IRC | 00:40 | |
| jeblair | clarkb: you mean (1) | 00:40 |
| clarkb | jeblair: yes | 00:40 |
| jeblair | fungi: yep, but starting with D as the triggering change | 00:40 |
| marun | jeblair: I think there is a real need to be able to manage reviews in a similar way to managing the bugs and blueprints they relate to. | 00:40 |
| marun | jeblair: i.e. assignment, priority, etc. | 00:41 |
| fungi | jeblair: this is to work out the enqueueing order for cross-repo dependencies i guess? | 00:41 |
| marun | jeblair: I've always thought of this as its own thing, but I think that's me accepting the status quo that is launchpad. | 00:41 |
| jhesketh | jeblair: if D is the one to be queued, I would expect the test order to be ABD (ie #2 scenario) | 00:41 |
| jhesketh | because if the right side of the tree has more commits, where would you stop? | 00:42 |
| jhesketh | as far as gerrit is concerned, dependencies are rather linear | 00:42 |
| marun | jeblair: I'd like to take a look at where storyboard is at with an eye to seeing if adding this sort of capability makes sense. | 00:42 |
| jeblair | fungi: actually i'm just trying to correct the current situation with single repos | 00:42 |
| *** tmcpeak has quit IRC | 00:42 | |
| jeblair | jhesketh: i thought so too at first, but then when i think about the fact that if you re-enqueued everything, you might get a different order, i am being swayed toward (1) | 00:43 |
| fungi | jeblair: i think i'm with clark on 1 mainly because that increases the chances that changes closest to the branch tip will make it in earlier, regardless of the fact that it lengthens the change series ahead of the triggering change | 00:43 |
| openstackgerrit | Merged openstack-infra/puppet-elastic_recheck: Add puppet module files for elastic_recheck https://review.openstack.org/151460 | 00:43 |
| clarkb | jhesketh: (1) should avoid the infinite depth problem too | 00:43 |
| jeblair | marun: https://wiki.openstack.org/wiki/StoryBoard https://wiki.openstack.org/wiki/StoryBoard/Vision https://wiki.openstack.org/wiki/StoryBoard/Roadmap | 00:44 |
| clarkb | since in my head its breadth first | 00:44 |
| marun | jeblair: awesome, thank you! | 00:44 |
| jhesketh | jeblair: I'm not sure how you'd get a different order on re-enqueue unless something has merged in the mean time. If that's the case then it'll be a different scenario anyway | 00:44 |
| jeblair | marun: so i think we want very good tie in with storyboard and gerrit; i think we are expecting a story task for every commit in gerrit that links to a story | 00:44 |
| *** signed8bit is now known as signed8bit_ZZZzz | 00:44 | |
| openstackgerrit | Shiv Haris proposed openstack-infra/project-config: As a next step: Adding <project>.config file https://review.openstack.org/151487 | 00:44 |
| jhesketh | fungi: I do think that point is valid though (re easier to get changes up the branch) | 00:45 |
| marun | jeblair: neat | 00:45 |
| jeblair | marun: i don't think that storyboard will take on being an alternate interface to gerrit, but i think there will probably be enough data and cross-linking to help with that | 00:45 |
| jhesketh | but what about where we have a long set of patches relating to the one issue. Wouldn't a depth first make more sense from the patch submitters point of view? | 00:45 |
| openstackgerrit | Merged openstack-infra/puppet-zanata: Add missing LICENSE file https://review.openstack.org/151457 | 00:45 |
| *** annegent_ has joined #openstack-infra | 00:45 | |
| shivharis | fungi: hi | 00:45 |
| *** spandhe has quit IRC | 00:46 | |
| openstackgerrit | Merged openstack-infra/puppet-asterisk: Add missing LICENSE file https://review.openstack.org/151420 | 00:46 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Add missing LICENSE file https://review.openstack.org/151445 | 00:46 |
| jeblair | marun: krotscheck and ttx are other big-picture thinkers on storyboard; the infra or dev lists are good for discussion, and there's an irc meeting | 00:47 |
| fungi | shivharis: what's broken? | 00:48 |
| marun | jeblair: cool | 00:48 |
| shivharis | fungi: nothing, just that I was asked to contact infra team when I am ready to add core members to the project | 00:49 |
| marun | jeblair: You've given me what I'll need to engage, thank you. | 00:49 |
| jeblair | jhesketh: if you had (2), and then cleared the queue and re-enqueued everything in queue order, you'd end up with (1) | 00:49 |
| jeblair | jhesketh: (because you would enqueue A to start with instead of D) | 00:49 |
| jhesketh | jeblair: so maybe we need to maintain the order when re-equeuing? | 00:50 |
| jeblair | jhesketh: i think it's easier to just make it stable and go with the algorithm that produces (1) | 00:50 |
| shivharis | fungi: i need to add a group networking-brocade-core group, i have the corresponding .config file ready | 00:50 |
| jhesketh | jeblair: sure, that's probably true | 00:51 |
| jeblair | (which is, incidentally, what we would usually see anyway since the only way to get (2) is to miss events) | 00:51 |
| shivharis | fungi: patchset https://review.openstack.org/151487 | 00:51 |
| jhesketh | I don't think I fully understand the consequences of re-enqueuing, but I think there are enough reasons to do #2 | 00:51 |
| jhesketh | *#1 | 00:51 |
| *** bdpayne has quit IRC | 00:52 | |
| *** wuhg has joined #openstack-infra | 00:52 | |
| fungi | shivharis: i've added you as the initial member of the two groups created by that change. you should be able to make further membership edits to those as needed | 00:52 |
| shivharis | fungi: great, thanks much, doing this for the first time ever so treading very carefully | 00:52 |
| shivharis | fungi: bug thanks | 00:53 |
| shivharis | fungi: s/bug/big/ | 00:53 |
| jeblair | clarkb, fungi, jhesketh: thanks! | 00:53 |
| fungi | shivharis: for some reason, networking-brocade-core and networking-brocade-release groups already exist | 00:54 |
| fungi | shivharis: even though that change you linked hasn't merged yet | 00:54 |
| fungi | looking into it | 00:54 |
| shivharis | fungi: surprising, who happens to be in the group | 00:54 |
| fungi | shivharis: they were empty | 00:55 |
| jeblair | marun, jhesketh: i haven't had a chance to do a real review yet, but just letting it rattle around in my brain, i have a small concern that we may find it hard to control the behavior of skip-if across a large number of projects...(i will elaborate) | 00:55 |
| fungi | shivharis: oh, there's an existing stackforge/networking-brocade project | 00:55 |
| marun | jeblair: maybe we need to mandate the project field? | 00:55 |
| jeblair | marun, jhesketh: since it applies to a job, and we use, say, 'gate-tempest-dsvm' on many projects, if we applied skip-if to that, then it might skip the run if someone made changes only to files in the tests/ directory in tempest | 00:56 |
| openstackgerrit | Merged openstack-infra/puppet-bup: Add puppet module files for bup https://review.openstack.org/151459 | 00:56 |
| *** bradjones has quit IRC | 00:56 | |
| shivharis | fungi: that project was merged three days ago, I am adding a new file to it, gerrit/acls/... | 00:56 |
| *** carl_baldwin has quit IRC | 00:56 | |
| fungi | shivharis: did you mean to edit the acl for that project rather than adding one for openstack/networking-brocade (which doesn't seem to be a project)? | 00:56 |
| jeblair | (i don't know offhand what the dir structure of tempest is, but one might imagine that the intention to not run jobs on neutron unit test changes might conflict with the tempest dir structure) | 00:57 |
| marun | jeblair: my intent was that the scope be limited based on how dangerous the potential filter could be. | 00:57 |
| jhesketh | jeblair: interesting thought... I might have to rethink on the re-review, but I think it's avoidable by tightening the filter params... will look at it later | 00:57 |
| marun | jeblair: so anything that wasn't universal (and enforced) would have to be project-specific | 00:57 |
| marun | jeblair: e.g. x/tests/.* is always safe to skip tempest jobs on | 00:58 |
| shivharis | i meant to add the file to the existing project at stackforge/networking-brocade | 00:58 |
| jeblair | marun: yeah, and i don't have fully-formed thoughts around this yet -- after all, the existing files filter has the same problem, though it's a bit easier if you're saying "only run this on a requirements.txt change"; it's inverting that where it gets a little more vague | 00:58 |
| shivharis | fungi: i meant to add the file to the existing project at stackforge/networking-brocade | 00:58 |
| clarkb | fyi I am going to rerun the aipcpu experimental jobs and hold the nodes running the nova networking test. Grepping through logs I don't see why my shift of the IPs would break so going to manually investigate. | 00:58 |
| jeblair | marun: why is "x/tests/.*" always safe? | 00:58 |
| *** Sukhdev has quit IRC | 00:58 | |
| marun | jeblair: I made the feature generic because it made for a cleaner implementation, but I didn't intend for its use to be unlimited. | 00:58 |
| jeblair | (is x standing for something important?) | 00:59 |
| fungi | shivharis: you'll want to update your change to adjust the correct acl then | 00:59 |
| marun | jeblair: x -> project under test | 00:59 |
| *** annegent_ has quit IRC | 00:59 | |
| *** dannywilson has quit IRC | 00:59 | |
| *** bradjones has joined #openstack-infra | 00:59 | |
| marun | jeblair: I just mean that unless a certain path means the same thing in every project targeted by a job, we can't skip on it. | 00:59 |
| jeblair | marun: ok, yeah, makes sense. | 00:59 |
| marun | jeblair: without adding a project specifier, anyway | 01:00 |
| *** fifieldt has quit IRC | 01:00 | |
| jeblair | marun: indeed -- that was the thought i was having. i'm glad we share it. | 01:00 |
| jeblair | marun: i don't have anything else to add right now, but we've achieved a shared context, so that will be helpful. :) | 01:00 |
| marun | jeblair: I'm happy to incorporate that kind of limitation into the feature if you don't think that a convention could be safely enforced. | 01:00 |
| shivharis | fungi: ok will fix it | 01:00 |
| marun | jeblair: cool :) | 01:01 |
| *** avozza is now known as zz_avozza | 01:05 | |
| openstackgerrit | Merged openstack-infra/puppet-meetbot: Add missing LICENSE file https://review.openstack.org/151443 | 01:07 |
| openstackgerrit | Merged openstack-infra/puppet-statusbot: Add puppet module files for Statusbot https://review.openstack.org/151465 | 01:08 |
| openstackgerrit | Merged openstack-infra/puppet-sudoers: Add puppet module files for sudoers https://review.openstack.org/151467 | 01:09 |
| openstackgerrit | Shiv Haris proposed openstack-infra/project-config: File updated gerrit/acls/stackforge/networking-brocade.config https://review.openstack.org/151487 | 01:10 |
| *** derekh has quit IRC | 01:10 | |
| *** armax has quit IRC | 01:12 | |
| *** zhiwei has joined #openstack-infra | 01:13 | |
| fungi | so as asselin_ said, we | 01:16 |
| *** ddieterly has joined #openstack-infra | 01:16 | |
| fungi | 're now just missing polish commits on puppet-ulimit, unattended_upgrades, unbound, user and zuul | 01:17 |
| *** amitgandhinz has joined #openstack-infra | 01:17 | |
| *** adalbas has quit IRC | 01:18 | |
| *** hdd has quit IRC | 01:18 | |
| *** SumitNaiksatam has quit IRC | 01:19 | |
| *** Sukhdev has joined #openstack-infra | 01:19 | |
| *** achanda_ has joined #openstack-infra | 01:19 | |
| *** ddieterly has quit IRC | 01:21 | |
| *** achanda has quit IRC | 01:23 | |
| clarkb | ok I can pingthe 172.24.5.1 floating ip given to a vm | 01:24 |
| *** achanda_ has quit IRC | 01:24 | |
| clarkb | will let tempest get further along before I much with it too much | 01:24 |
| *** zz_avozza is now known as avozza | 01:26 | |
| clarkb | fungi: watch it will be a security group thing where tempest assumes 172.24.4.0/24 is the range to open | 01:26 |
| fungi | hah | 01:26 |
| clarkb | hrm shelve instance just failed we shall wait to see why | 01:26 |
| clarkb | it may be less work overall to just rewrite my comments from scratch if thats the case | 01:27 |
| clarkb | since 172.24.4.0/24 seems fairly embedded | 01:27 |
| openstackgerrit | Merged openstack-infra/puppet-snmpd: Add missing LICENSE file https://review.openstack.org/151450 | 01:27 |
| fungi | it's unfortunate that there's not just one variable somewhere all other uses consult | 01:27 |
| *** amitgandhinz has quit IRC | 01:28 | |
| clarkb | ya another ssh time out so at least its consistent now I can work on debugging it | 01:28 |
| fungi | i think i'm going to knock off for the night, but enjoy your evening west-coasters | 01:31 |
| *** tiswanso has joined #openstack-infra | 01:32 | |
| *** tiswanso has quit IRC | 01:34 | |
| *** markvoelker_ has quit IRC | 01:34 | |
| *** tiswanso has joined #openstack-infra | 01:35 | |
| *** markvoelker has joined #openstack-infra | 01:35 | |
| clarkb | hrm there are no rules in the default security group | 01:36 |
| *** mtanino has quit IRC | 01:39 | |
| *** hdd has joined #openstack-infra | 01:39 | |
| *** yamahata has joined #openstack-infra | 01:39 | |
| clarkb | hrm I can't even ping the addr now | 01:40 |
| clarkb | jogo: ^ possibly tempest tests regressed? | 01:40 |
| *** markvoelker has quit IRC | 01:40 | |
| *** ayoung has joined #openstack-infra | 01:41 | |
| *** tjones1 has quit IRC | 01:42 | |
| clarkb | ok tempest cleans up all of its security groups (probably a good thing) which makes it slightly hard to review them after the fact | 01:44 |
| *** whayutin_ has quit IRC | 01:44 | |
| *** annegent_ has joined #openstack-infra | 01:45 | |
| *** ayoung has quit IRC | 01:46 | |
| jogo | clarkb: very possible something regressed | 01:47 |
| jogo | clarkb: which job? | 01:47 |
| clarkb | jogo: nova net aiopcpu | 01:47 |
| clarkb | jogo: the shelve server test fails due to an ssh client timeout | 01:47 |
| clarkb | jogo: I am on the two nodes post tempest fail. I have booted a node, added a floating ip to it, updated security group rules, and both ping + ssh work | 01:47 |
| clarkb | so I have at least convinced my self that the base set of netowrking is sound. Going to try replicating what the test does now | 01:48 |
| *** ChuckC_ has quit IRC | 01:48 | |
| jogo | clarkb: not a regression, its a known issue | 01:48 |
| jogo | clarkb: the shelve issue is: | 01:48 |
| clarkb | jogo: ok? does that mean we cannot merge aiopcpu right now? | 01:48 |
| jogo | shelve instance on machine A, if unshelve moves instance to machine B it fails | 01:49 |
| jogo | else success | 01:49 |
| *** ChuckC_ has joined #openstack-infra | 01:49 | |
| jogo | clarkb: so its a race condition. We cannot make this job voting anywhere | 01:49 |
| clarkb | I see | 01:49 |
| clarkb | why don't we stop testing shelve if its broken | 01:49 |
| jogo | but we can get the basic bits in place to make this experimental on tempest and nova so we can better debug | 01:49 |
| clarkb | also why can't we make it voting? | 01:50 |
| jogo | clarkb: that is an option, I was hoping to try fixing it first | 01:50 |
| clarkb | the whole point of this exercise is to have the test actually run and provide feedback | 01:50 |
| jogo | clarkb: well we haven't run this job enough to see how stable it is | 01:50 |
| jogo | so first step non-voting experimental on a few repos | 01:50 |
| jogo | and go from there | 01:50 |
| clarkb | sure but to flat out say we can't make it voting anywhere is premature | 01:50 |
| clarkb | its already experimental now | 01:51 |
| clarkb | I want to change the status qu | 01:51 |
| jogo | clarkb: well its only experimental on devstack-gate right? | 01:51 |
| *** annegent_ has quit IRC | 01:51 | |
| marun | jhesketh: can you please help me interpret the following? | 01:51 |
| marun | jhesketh: <Completed build, result: SUCCESS name: project-test-skip-if #21 changes: 5,2 6,2> | 01:51 |
| clarkb | jogo: I think it may also be on devstack | 01:51 |
| clarkb | jogo: I would have to check since most of the owrk so far has been on d-g | 01:51 |
| jogo | clarkb: actually come to think of it, enough patches have landed in the d-g patch series to get this to pass most of the time on master d-g | 01:52 |
| clarkb | jogo: but the stability won't change if you move it to devstack/nova/etc experimental... | 01:52 |
| jogo | it wouldn't have multihost though | 01:52 |
| clarkb | correct | 01:52 |
| marun | jhesketh: I'm not expecting that job to run on 5,2, only on 6,2, and I'm confused as to why 5,2 is listed in that message. | 01:52 |
| clarkb | but ya everything but multihost should work (that and things like migration testing which come after multihost) | 01:52 |
| jogo | right, but we can see how specific patches work in multi-node (fixing unshelve) | 01:52 |
| jogo | clarkb: but I think you are right, lets disable the shelve test until its fixed and move on | 01:52 |
| clarkb | jogo: is shelve broken because the host moves but doesn't rearp? | 01:53 |
| marun | jeblair: a question regarding zuul job status messages? ^ | 01:53 |
| clarkb | jogo: or maybe the host moves and the floating ip stays on the old hypervisor | 01:53 |
| jogo | clarkb: not sure yet, it only happens on nova-net and not neutron | 01:53 |
| clarkb | jogo: let me try to reproduce on this setup. just shelve unshelve and ping/ssh until it breaks? | 01:53 |
| jogo | clarkb: yup, that should do it | 01:54 |
| clarkb | jogo: can do | 01:54 |
| jogo | clarkb: thanks | 01:54 |
| *** yamahata has quit IRC | 01:54 | |
| jogo | clarkb: on a unrelated note - if we pin stable/juno to the pip-freeze of dsvm-tempest grenade fails, if we pin stable/juno to the pip-freeze of grenade, dsvm-tempest fails | 01:54 |
| jogo | because the two pip-freezes are different | 01:55 |
| clarkb | jogo: wow | 01:55 |
| jogo | clarkb: isn't that fun :) | 01:55 |
| clarkb | jogo: ok reproduced. However its a little different than you describe. it unshelved onto the same host | 01:57 |
| *** amotoki has joined #openstack-infra | 01:59 | |
| *** mohitsharma has joined #openstack-infra | 01:59 | |
| clarkb | so now I am trying to see if the NAT rules are wrong (maybe it unshelved with a different kvm assigned ip?) | 01:59 |
| *** bradjones has quit IRC | 02:00 | |
| clarkb | jogo: ya ok, hitting the private ip works but not the floating ip however iptables shows nat rules that look correct to me | 02:01 |
| clarkb | time for tcpdump | 02:01 |
| *** bradjones has joined #openstack-infra | 02:01 | |
| *** ayoung has joined #openstack-infra | 02:01 | |
| *** amotoki has quit IRC | 02:03 | |
| *** packet has quit IRC | 02:07 | |
| *** shivharis has quit IRC | 02:08 | |
| clarkb | jogo: OHHH I see maybe | 02:09 |
| clarkb | jogo: the floating ip is being added to the other node (the one doing the pinging) but the floating ip is still on the other node where the vm is running | 02:10 |
| *** hdd has quit IRC | 02:10 | |
| jogo | so floating ip is on both nodes? | 02:10 |
| clarkb | jogo: ya that is what I was seeing | 02:11 |
| *** abhirc has quit IRC | 02:11 | |
| *** salv-orlando has quit IRC | 02:12 | |
| jogo | and where is the actual VM | 02:12 |
| clarkb | looks like the nat rule ends up on both nodes too | 02:13 |
| *** jerryz has joined #openstack-infra | 02:13 | |
| clarkb | but since we are multihost the routing needs to happen on the compute node with the VM I think | 02:13 |
| *** signed8bit_ZZZzz is now known as signed8bit | 02:13 | |
| clarkb | jogo: two node install A and B. VM is running on B. shelve unshelve a couple times, VM continues to run on B but A gets the floating ip and nat rules | 02:13 |
| clarkb | jogo: pings from A to floating ip fail. pings from B to floating ip succeed | 02:14 |
| jogo | clarkb: here is the bug I filed https://bugs.launchpad.net/nova/+bug/1404063 | 02:14 |
| *** fandi has joined #openstack-infra | 02:14 | |
| clarkb | jogo: I will update with what I have found | 02:14 |
| jogo | clarkb: thanks | 02:14 |
| marun | fungi: I'm having trouble interpreting a zuul build status, can you help? | 02:15 |
| jogo | so it sounds like we aren't cleaning something up correctly | 02:15 |
| marun | fungi: <Completed build, result: SUCCESS name: project-test-skip-if #21 changes: 5,2 6,2> | 02:15 |
| *** homeless1 has quit IRC | 02:15 | |
| jogo | clarkb: hmm https://bugs.launchpad.net/nova/+bug/1290635 | 02:16 |
| *** asettle has quit IRC | 02:16 | |
| marun | fungi: what does the inclusion of both 5,2 and 6,2 mean? In the configuration I'm working with, the 5,2 change should not have run the project-test-skip-if job | 02:16 |
| *** asettle has joined #openstack-infra | 02:16 | |
| jogo | clarkb: or this https://bugs.launchpad.net/nova/+bug/1350704 | 02:16 |
| clarkb | jogo: that second one appears to be the issue | 02:18 |
| jogo | clarkb: yeah and there is a patch up too :) | 02:18 |
| clarkb | jogo: let me get what I have into your first bug then you can link/merge/mark duplicate whatever is most appropriate | 02:18 |
| jogo | sounds good | 02:18 |
| jogo | want to try the fix as well? | 02:18 |
| clarkb | jogo: maybe? might be interesting to dig into nova but not sure where to start | 02:18 |
| jogo | clarkb: here is the fix https://review.openstack.org/#/c/110869/24 | 02:19 |
| *** _amrith_ is now known as amrith | 02:19 | |
| clarkb | jogo: your first bug is updated | 02:21 |
| jogo | clarkb: thanks | 02:22 |
| *** dmsimard_away is now known as dmsimard | 02:22 | |
| jogo | clarkb: ahh that does sound like a duplicate of https://bugs.launchpad.net/nova/+bug/1350704 which has a proposed fix: https://review.openstack.org/#/c/110869/24/nova/compute/manager.py,cm | 02:23 |
| *** dimtruck is now known as zz_dimtruck | 02:23 | |
| jhesketh | marun: sorry was out for lunch... the two changes are possibly because one is dependant on the other | 02:24 |
| marun | jhesketh: it's race-y though | 02:25 |
| marun | jhesketh: or do you mean there is a merge dependency? | 02:25 |
| marun | jhesketh: I can trigger the problem every 5 test runs or so | 02:25 |
| jhesketh | merge dependency... not sure what is race0y | 02:25 |
| marun | jhesketh: I'll upload the change and you can see what I mean. | 02:25 |
| jhesketh | marun: is this as a unit test? | 02:25 |
| jhesketh | okay :-) | 02:25 |
| marun | jhesketh: scenario test, as requested. | 02:26 |
| *** ddieterly has joined #openstack-infra | 02:28 | |
| *** otter768 has joined #openstack-infra | 02:29 | |
| clarkb | jogo: looks like Alex Xu is still working the fix? updated pretty recently | 02:29 |
| *** mohitsharma has quit IRC | 02:30 | |
| *** ddieterly has quit IRC | 02:32 | |
| *** ddieterly has joined #openstack-infra | 02:33 | |
| *** otter768 has quit IRC | 02:33 | |
| *** baoli has joined #openstack-infra | 02:34 | |
| *** dmsimard is now known as dmsimard_away | 02:34 | |
| *** achuprin_ has quit IRC | 02:35 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add networking-ofagent-ci group https://review.openstack.org/151512 | 02:35 |
| openstackgerrit | Maru Newby proposed openstack-infra/zuul: Add support for a skip-if filter on jobs https://review.openstack.org/146762 | 02:35 |
| marun | jhesketh: sorry for delay: https://review.openstack.org/#/c/146762/ | 02:36 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add networking-ofagent-ci group https://review.openstack.org/151512 | 02:36 |
| *** yamamot__ has joined #openstack-infra | 02:36 | |
| *** Kerralissa has quit IRC | 02:36 | |
| marun | jhesketh: line 1847 in https://review.openstack.org/#/c/146762/11/tests/test_scheduler.py | 02:36 |
| marun | jhesketh: I was seeing periodic test failures previous to implementing the workaround described there. | 02:37 |
| *** Rockyg has joined #openstack-infra | 02:38 | |
| *** baoli has quit IRC | 02:39 | |
| jogo | clarkb: I ran a recheck | 02:40 |
| *** mohitsharma has joined #openstack-infra | 02:40 | |
| *** melwitt has quit IRC | 02:41 | |
| *** amotoki has joined #openstack-infra | 02:42 | |
| *** amotoki has quit IRC | 02:42 | |
| jhesketh | marun: as in the failures happen in master? | 02:43 |
| marun | jhesketh: master in zuul? | 02:44 |
| marun | jhesketh: no, this is local to my test | 02:44 |
| marun | jhesketh: As per the comment, I need a way to detect whether the skip-configured job ran for a given change. | 02:44 |
| jhesketh | oh, I mis-read your explaination | 02:45 |
| marun | jhesketh: The only example of this that I could see was looking at the changes attached to the job status message | 02:45 |
| marun | jhesketh: job status -> build status (I guess?) | 02:45 |
| marun | jhesketh: and sometimes the build status references both a job that should have been skipped and one that should have ran it. | 02:46 |
| *** annegent_ has joined #openstack-infra | 02:46 | |
| marun | jhesketh: hence my question of merge dependency, since I am assuming that the inclusion of the preceding change id is because of something related to merge order rather than an indication that the job ran for both changes | 02:47 |
| *** amrith is now known as _amrith_ | 02:48 | |
| *** rwsu is now known as rwsu-afk | 02:48 | |
| jhesketh | marun: hmm, so basically they are entering the gate pipeline in an inconsistent order? | 02:50 |
| marun | jhesketh: I'm adding them in the same order every time. I can only assume that there is some kind of concurrent processing going on that introduces the variability. | 02:50 |
| marun | jhesketh: certainly the skip-if matching has no choice but to be deterministic. | 02:51 |
| *** annegent_ has quit IRC | 02:51 | |
| *** achuprin_ has joined #openstack-infra | 02:52 | |
| jhesketh | marun: I mean the scheduler is queuing them inconsistently ? | 02:53 |
| marun | jhesketh: Maybe? I'm pretty sure I know enough to answer that definitively. | 02:53 |
| marun | jhesketh: Do you see anything strange in the test? | 02:54 |
| marun | jhesketh: I cribbed from other similar tests. | 02:54 |
| jhesketh | nope, it looks good to me | 02:55 |
| marun | jhesketh: oops, I meant that I *don't* know enough to answer that definitively | 02:55 |
| jhesketh | I suspect the race is in the queue processing... if you put the queue in hold mode, queue them all and then release them all at once it might be more deterministic | 02:56 |
| jhesketh | look for a test where self.worker.hold_jobs_in_build is set | 02:57 |
| marun | jhesketh: but what does it mean when 2 changes are included for a built status message? | 02:57 |
| marun | jhesketh: I don't really care about the ordering, I just need to figure out which changes a job ran for. | 02:57 |
| marun | built -> build | 02:58 |
| *** harlowja_ is now known as harlowja_away | 02:58 | |
| jhesketh | marun: I think it's because it's a dependant pipeline. You're queuing 6 changes (ABCDEF) so for change C the tests are ran with master -> A -> B then C applied | 02:59 |
| jhesketh | it's built into the change so that cross project jobs can pull the correct order | 03:00 |
| marun | jhesketh: so does that imply that the only change that was actually tested was the last one in the list of changes on a build status? | 03:00 |
| *** bradjones has quit IRC | 03:00 | |
| marun | jhesketh: and that preceding changes are simply dependencies of the change the job ran against? | 03:00 |
| marun | jhesketh: that's what the code assumes right now, because that seemed the most likely conclusion | 03:01 |
| jhesketh | yes | 03:01 |
| jhesketh | although each other change will also have gone through... so E would have been tested with master -> ABCDE applied | 03:01 |
| *** bradjones has joined #openstack-infra | 03:02 | |
| marun | jhesketh: so why is the previous change included in some build statuses and not others? | 03:03 |
| openstackgerrit | Maru Newby proposed openstack-infra/zuul: Add support for a skip-if filter on jobs https://review.openstack.org/146762 | 03:04 |
| jhesketh | marun: it's possible that the build is finishing and being merged into master | 03:04 |
| jhesketh | so if A and B have merged, when testing E it'd look like master -> CDE | 03:04 |
| jhesketh | which is why if you put it in hold when queuing the changes it should be more consistent | 03:05 |
| marun | jhesketh: I'll take a look at how to do that | 03:05 |
| jhesketh | keep in mind I may be wrong ;p | 03:05 |
| *** unicell has quit IRC | 03:06 | |
| marun | jhesketh: worth a try, at least :) | 03:08 |
| marun | jhesketh: I'm not a huge fan of this kind of scenario testing, tbh | 03:08 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-ulimit: Add puppet module files for puppet-ulimit https://review.openstack.org/151514 | 03:08 |
| marun | jhesketh: too many moving pieces to learn and to go wrong, makes it hard to isolate the failure domain | 03:08 |
| marun | jhesketh: but I'll do what's asked :) | 03:09 |
| *** SumitNaiksatam has joined #openstack-infra | 03:09 | |
| *** SumitNaiksatam has quit IRC | 03:09 | |
| *** Rockyg has quit IRC | 03:09 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:10 | |
| jhesketh | marun: intresting, because I am a fan.. when zuul is used in the wild all these moving pieces can go wrong so it's good to test them | 03:11 |
| jhesketh | granted, it's a learning curve to write the tests, but I think it's worth it | 03:11 |
| marun | jhesketh: it's good to be able to do this kind of testing, don't get me wrong | 03:11 |
| marun | jhesketh: but I don't think it should be necessary to test more that the change matchers work in concert with everything else | 03:12 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-unbound: Add puppet module files for unbound https://review.openstack.org/151516 | 03:12 |
| *** hdd has joined #openstack-infra | 03:12 | |
| marun | jhesketh: it should be possible to isolate all specific testing of the change matchers, without involving other machinery | 03:12 |
| jhesketh | marun: I think this is how you uncover edge and race conditions to a certain extent though.. In an ideal world I would agree with you | 03:13 |
| marun | jhesketh: I had validated that job.changeMatches interacted correctly with the skip-if matcher, that the matcher could be constructed from configuration by the scheduler, and that the matchers worked. | 03:13 |
| marun | jhesketh: all of that in unit tests | 03:13 |
| marun | jhesketh: knowing that the existing tests are calling the job.changeMatches function, I know that everything below that would work properly. | 03:13 |
| marun | jhesketh: without that kind of separability for testing, the effort required for full coverage is exponential | 03:14 |
| jhesketh | well in this case you're doing both so it's even better ;-) | 03:14 |
| jhesketh | but I hear you, it's a burden | 03:14 |
| marun | jhesketh: I think the term 'test pyramid' accurately describes my philosophy of testing. | 03:15 |
| marun | jhesketh: validate everything at as low a level as feasible, because the cost is less. low -> unit, high-> integration, and everything in between. | 03:16 |
| * marun digresses | 03:16 | |
| jhesketh | I think your points are fair and I don't necessarily disagree | 03:17 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-user: Add puppet module files for puppet-user https://review.openstack.org/151517 | 03:17 |
| jhesketh | :-) | 03:17 |
| *** KanagarajM has joined #openstack-infra | 03:17 | |
| asselin_ | one more! | 03:18 |
| marun | jhesketh: :) | 03:18 |
| *** Sukhdev has quit IRC | 03:19 | |
| marun | jhesketh: your suggestion to use hold_jobs_in_build seems to be a good approach. | 03:19 |
| marun | jhesketh: I'm seeing consistent behavior at last. | 03:19 |
| *** xyang0 has joined #openstack-infra | 03:19 | |
| marun | jhesketh: ah crap, spoke too soon | 03:19 |
| marun | jhesketh: I guess the work-around stays for now | 03:20 |
| marun | jhesketh: maybe jeblair can figure it out. | 03:20 |
| clarkb | why not run tests in an independent pipeline? | 03:20 |
| jhesketh | :-( | 03:20 |
| *** otter768 has joined #openstack-infra | 03:20 | |
| marun | clarkb: https://review.openstack.org/#/c/146762/12/tests/test_scheduler.py | 03:21 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-unattended_upgrades: Add puppet module files for unattended_upgrades https://review.openstack.org/151519 | 03:22 |
| marun | clarkb: test_skip_if_jobs is just trying to verify that jobs are skipped as expected. | 03:22 |
| clarkb | ya instead of approving the changes just do patchset created events | 03:22 |
| *** annegent_ has joined #openstack-infra | 03:22 | |
| clarkb | then each change is tested independently | 03:22 |
| asselin_ | ^^ Last one! :) | 03:22 |
| marun | clarkb: do you have a pointer to an example? | 03:23 |
| * asselin_ going to eat | 03:23 | |
| marun | clarkb: oh wait | 03:23 |
| marun | clarkb: you mean run them in the check queue? | 03:23 |
| clarkb | yes | 03:23 |
| marun | clarkb: because I don't care about merging, just the job status | 03:23 |
| marun | clarkb: gotcha. serves me right for cribbing to hard | 03:23 |
| *** Ryan_Lane has quit IRC | 03:23 | |
| marun | clarkb: so what command triggers the check queue? | 03:24 |
| clarkb | its a patchset created event should be part of create change if you just drop the approval events | 03:24 |
| marun | clarkb: ah, ok | 03:25 |
| *** rkukura has quit IRC | 03:26 | |
| *** annegent_ has quit IRC | 03:27 | |
| *** sdake has joined #openstack-infra | 03:27 | |
| openstackgerrit | Maru Newby proposed openstack-infra/zuul: Add support for a skip-if filter on jobs https://review.openstack.org/146762 | 03:28 |
| marun | clarkb: that did it, thank you. | 03:28 |
| *** signed8bit has quit IRC | 03:28 | |
| *** bdpayne has joined #openstack-infra | 03:28 | |
| *** xyang0 has quit IRC | 03:29 | |
| *** adam_g_out is now known as adam_g | 03:29 | |
| *** bdpayne has quit IRC | 03:30 | |
| *** bradjones has quit IRC | 03:30 | |
| openstackgerrit | Maru Newby proposed openstack-infra/zuul: Add support for a skip-if filter on jobs https://review.openstack.org/146762 | 03:32 |
| *** bradjones has joined #openstack-infra | 03:36 | |
| openstackgerrit | Maru Newby proposed openstack-infra/zuul: Add support for a skip-if filter on jobs https://review.openstack.org/146762 | 03:40 |
| *** Sukhdev has joined #openstack-infra | 03:41 | |
| *** annegent_ has joined #openstack-infra | 03:46 | |
| *** mtreinish_ has joined #openstack-infra | 03:47 | |
| *** annegent_ has quit IRC | 03:51 | |
| *** dmsimard_away is now known as dmsimard | 03:53 | |
| *** dannywilson has joined #openstack-infra | 03:56 | |
| *** nelsnelson has quit IRC | 03:56 | |
| *** dmsimard is now known as dmsimard_away | 03:57 | |
| *** ChuckC_ has quit IRC | 03:58 | |
| *** ChuckC_ has joined #openstack-infra | 03:59 | |
| *** tiswanso has quit IRC | 04:00 | |
| *** dannywilson has quit IRC | 04:00 | |
| *** mohitsha_ has joined #openstack-infra | 04:02 | |
| openstackgerrit | Maru Newby proposed openstack-infra/zuul: Add support for a skip-if filter on jobs https://review.openstack.org/146762 | 04:03 |
| *** coolsvap|afk is now known as coolsvap | 04:03 | |
| *** sdake has quit IRC | 04:03 | |
| *** doug-fish has left #openstack-infra | 04:03 | |
| *** otter768 has quit IRC | 04:04 | |
| *** mtreinish has quit IRC | 04:04 | |
| *** mtreinish_ is now known as mtreinish | 04:04 | |
| *** mohitsharma has quit IRC | 04:05 | |
| *** hdd has quit IRC | 04:05 | |
| *** hdd has joined #openstack-infra | 04:06 | |
| marun | jhesketh: I've tried to implement Clark's suggestion but I appear to have muffed it: https://review.openstack.org/#/c/146762/ | 04:06 |
| marun | jhesketh: The jobs are no longer triggered. Am I doing anything obviously wrong in https://review.openstack.org/#/c/146762/16/tests/test_scheduler.py ? | 04:06 |
| *** rkukura has joined #openstack-infra | 04:06 | |
| marun | clarkb: ^^ | 04:08 |
| *** mohitsha_ has quit IRC | 04:08 | |
| jhesketh | marun: looks correct | 04:09 |
| jhesketh | just in the middle of a meeting or else I'd take a closer look | 04:09 |
| marun | jhesketh: no worries ;) | 04:09 |
| marun | :) | 04:09 |
| jhesketh | marun: it doesn't work locally? | 04:09 |
| marun | jhesketh: nope | 04:09 |
| *** spzala has quit IRC | 04:12 | |
| marun | jhesketh: this late in my workday it's likely pebkac and I should try tomorrow :) | 04:12 |
| jhesketh | heh, no worries | 04:13 |
| jhesketh | have a good one | 04:13 |
| *** woodster_ has quit IRC | 04:13 | |
| marun | jhesketh: you too! | 04:14 |
| *** nelsnelson has joined #openstack-infra | 04:15 | |
| *** amotoki has joined #openstack-infra | 04:16 | |
| *** mohitsharma has joined #openstack-infra | 04:19 | |
| marun | jhesketh: stubborn I am. Looks like the job is legitimately being skipped due to the change not correctly adding the files that would trigger a skip. | 04:28 |
| marun | jhesketh: or rather, the reverse. not adding files to the default set (commit message, readme) means the skip-if matcher always succeeds and the job never runs. | 04:30 |
| marun | clarkb: ^^ I've done as you suggested and tried to triggered job execution with a 'patch added' event, but now the changes don't seem to include the extra files I provide to addPatchset. | 04:32 |
| *** koolhead17 has joined #openstack-infra | 04:36 | |
| *** achanda has joined #openstack-infra | 04:36 | |
| *** ivar-laz_ has joined #openstack-infra | 04:39 | |
| *** achanda has quit IRC | 04:40 | |
| *** ivar-lazzaro has quit IRC | 04:43 | |
| *** marun has quit IRC | 04:44 | |
| *** ivar-laz_ has quit IRC | 04:44 | |
| *** annegent_ has joined #openstack-infra | 04:46 | |
| *** achanda has joined #openstack-infra | 04:49 | |
| *** nelsnelson has quit IRC | 04:50 | |
| *** annegent_ has quit IRC | 04:51 | |
| *** sdake has joined #openstack-infra | 04:51 | |
| *** unicell has joined #openstack-infra | 04:51 | |
| *** hdd has quit IRC | 04:54 | |
| *** pcrews has quit IRC | 04:56 | |
| *** mohitsharma has quit IRC | 05:03 | |
| *** zz_dimtruck is now known as dimtruck | 05:07 | |
| *** garyh has quit IRC | 05:09 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 05:11 | |
| *** fandi has quit IRC | 05:12 | |
| *** asettle has quit IRC | 05:17 | |
| *** achanda has quit IRC | 05:20 | |
| *** BharatK_ has joined #openstack-infra | 05:27 | |
| *** melwitt has joined #openstack-infra | 05:31 | |
| *** fandi has joined #openstack-infra | 05:31 | |
| *** achanda has joined #openstack-infra | 05:31 | |
| *** woodster_ has joined #openstack-infra | 05:32 | |
| *** garyh has joined #openstack-infra | 05:40 | |
| *** achanda has quit IRC | 05:41 | |
| *** Sukhdev has quit IRC | 05:41 | |
| *** afazekas_ has joined #openstack-infra | 05:43 | |
| *** jyuso1 has quit IRC | 05:43 | |
| *** vigneshvar has joined #openstack-infra | 05:45 | |
| *** koolhead_ has joined #openstack-infra | 05:45 | |
| *** annegent_ has joined #openstack-infra | 05:46 | |
| *** koolhead17 has quit IRC | 05:47 | |
| *** hdd has joined #openstack-infra | 05:48 | |
| *** mohitsharma has joined #openstack-infra | 05:49 | |
| *** ildikov has quit IRC | 05:49 | |
| *** annegent_ has quit IRC | 05:51 | |
| *** koolhead_ has quit IRC | 05:53 | |
| *** dimtruck is now known as zz_dimtruck | 05:53 | |
| *** achanda has joined #openstack-infra | 05:58 | |
| *** otter768 has joined #openstack-infra | 06:04 | |
| *** jyuso1 has joined #openstack-infra | 06:07 | |
| *** otter768 has quit IRC | 06:09 | |
| *** BharatK_ has quit IRC | 06:13 | |
| *** fandi has quit IRC | 06:18 | |
| *** BharatK has joined #openstack-infra | 06:23 | |
| *** achanda has quit IRC | 06:24 | |
| *** achanda has joined #openstack-infra | 06:25 | |
| *** harlowja_at_home has joined #openstack-infra | 06:27 | |
| *** EmilienM is now known as EmilienM|afk | 06:29 | |
| *** bhunter71 has quit IRC | 06:29 | |
| *** harlowja_at_home has quit IRC | 06:31 | |
| *** markmcclain has joined #openstack-infra | 06:33 | |
| *** fandi has joined #openstack-infra | 06:37 | |
| *** markmcclain has quit IRC | 06:40 | |
| *** markmcclain has joined #openstack-infra | 06:40 | |
| *** stevemar2 is now known as stevemar | 06:41 | |
| *** markmcclain has quit IRC | 06:44 | |
| *** fandi has quit IRC | 06:44 | |
| *** markmcclain has joined #openstack-infra | 06:44 | |
| *** Sukhdev has joined #openstack-infra | 06:45 | |
| *** annegent_ has joined #openstack-infra | 06:46 | |
| *** annegent_ has quit IRC | 06:51 | |
| *** achanda has quit IRC | 06:54 | |
| *** yolanda has joined #openstack-infra | 06:54 | |
| *** camunoz has quit IRC | 07:01 | |
| *** andreykurilin_ has joined #openstack-infra | 07:03 | |
| *** SumitNaiksatam has quit IRC | 07:03 | |
| *** ihrachyshka has joined #openstack-infra | 07:07 | |
| *** SumitNaiksatam has joined #openstack-infra | 07:08 | |
| *** jkraj has joined #openstack-infra | 07:08 | |
| *** fandi has joined #openstack-infra | 07:09 | |
| *** andreykurilin|ho has joined #openstack-infra | 07:11 | |
| *** andreykurilin_ has quit IRC | 07:12 | |
| *** mohitsharma has quit IRC | 07:13 | |
| *** afazekas_ has quit IRC | 07:16 | |
| *** mohitsharma has joined #openstack-infra | 07:19 | |
| *** mase_home has joined #openstack-infra | 07:20 | |
| *** ihrachyshka has quit IRC | 07:20 | |
| *** stevemar has quit IRC | 07:24 | |
| *** arxcruz has joined #openstack-infra | 07:24 | |
| *** jichen has joined #openstack-infra | 07:24 | |
| *** afazekas has quit IRC | 07:25 | |
| *** Sukhdev has quit IRC | 07:28 | |
| *** sdake has quit IRC | 07:29 | |
| *** koolhead17 has joined #openstack-infra | 07:37 | |
| *** avozza is now known as zz_avozza | 07:37 | |
| *** yamahata has joined #openstack-infra | 07:39 | |
| *** scheuran has joined #openstack-infra | 07:39 | |
| *** hdd has quit IRC | 07:45 | |
| *** annegent_ has joined #openstack-infra | 07:46 | |
| *** belmoreira has joined #openstack-infra | 07:46 | |
| *** annegent_ has quit IRC | 07:51 | |
| *** zz_avozza is now known as avozza | 07:52 | |
| *** woodster_ has quit IRC | 07:53 | |
| *** ildikov has joined #openstack-infra | 07:59 | |
| *** dvorak has quit IRC | 08:01 | |
| *** HeOS has quit IRC | 08:03 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: Add non-voting job for Nova V2.1 API with neutron https://review.openstack.org/151564 | 08:03 |
| openstackgerrit | Merged openstack-infra/puppet-vinz: Add missing LICENSE file https://review.openstack.org/151455 | 08:05 |
| *** otter768 has joined #openstack-infra | 08:05 | |
| *** doude has joined #openstack-infra | 08:06 | |
| *** mpaolino has joined #openstack-infra | 08:07 | |
| *** vigneshvar has quit IRC | 08:07 | |
| *** chlong has quit IRC | 08:08 | |
| *** dvorak has joined #openstack-infra | 08:09 | |
| *** otter768 has quit IRC | 08:11 | |
| *** unicell1 has joined #openstack-infra | 08:13 | |
| openstackgerrit | Merged openstack-infra/puppet-httpd: Add missing LICENSE file https://review.openstack.org/151432 | 08:13 |
| *** unicell has quit IRC | 08:14 | |
| *** Ala has joined #openstack-infra | 08:15 | |
| *** unicell has joined #openstack-infra | 08:23 | |
| *** unicell has quit IRC | 08:23 | |
| *** unicell has joined #openstack-infra | 08:23 | |
| *** unicell1 has quit IRC | 08:23 | |
| *** dtantsur|afk is now known as dtantsur | 08:24 | |
| *** yamahata has quit IRC | 08:24 | |
| *** melwitt1 has joined #openstack-infra | 08:27 | |
| *** melwitt has quit IRC | 08:27 | |
| *** MaxV has joined #openstack-infra | 08:28 | |
| *** jkraj has quit IRC | 08:31 | |
| *** dvorak has quit IRC | 08:32 | |
| *** jkraj has joined #openstack-infra | 08:33 | |
| *** avozza is now known as zz_avozza | 08:35 | |
| *** KanagarajM has quit IRC | 08:35 | |
| *** salv-orlando has joined #openstack-infra | 08:38 | |
| *** oomichi has quit IRC | 08:39 | |
| *** achuprin_ has quit IRC | 08:41 | |
| *** markus_z has joined #openstack-infra | 08:41 | |
| *** salv-orlando has quit IRC | 08:45 | |
| *** annegent_ has joined #openstack-infra | 08:46 | |
| *** cebruns has quit IRC | 08:47 | |
| *** wznoinsk_ has quit IRC | 08:47 | |
| *** yjiang5 has quit IRC | 08:47 | |
| *** wznoinsk has joined #openstack-infra | 08:47 | |
| *** yjiang5 has joined #openstack-infra | 08:48 | |
| *** cebruns has joined #openstack-infra | 08:48 | |
| *** dvorak has joined #openstack-infra | 08:50 | |
| *** annegent_ has quit IRC | 08:52 | |
| *** dtantsur is now known as dtantsur|brb | 08:52 | |
| *** achuprin_ has joined #openstack-infra | 08:54 | |
| *** dhritishikhar has joined #openstack-infra | 08:55 | |
| *** yamamot__ has quit IRC | 08:56 | |
| *** jcoufal has joined #openstack-infra | 08:57 | |
| *** tnovacik has joined #openstack-infra | 09:01 | |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** melwitt1 has quit IRC | 09:02 | |
| *** mase_home has quit IRC | 09:02 | |
| *** KanagarajM has joined #openstack-infra | 09:03 | |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: oslo.messaging: uses devstack-f21 instead f20 https://review.openstack.org/151322 | 09:06 |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: oslo.messaging: add zeromq funcionnal job https://review.openstack.org/151578 | 09:06 |
| *** vigneshvar has joined #openstack-infra | 09:10 | |
| *** Guest84671 has joined #openstack-infra | 09:13 | |
| *** afazekas has joined #openstack-infra | 09:17 | |
| *** derekh has joined #openstack-infra | 09:19 | |
| *** jlibosva has joined #openstack-infra | 09:22 | |
| *** yamamoto_ has joined #openstack-infra | 09:25 | |
| *** yamamoto_ has quit IRC | 09:25 | |
| *** jistr has joined #openstack-infra | 09:31 | |
| *** jp_at_hp has joined #openstack-infra | 09:32 | |
| *** pblaho has joined #openstack-infra | 09:35 | |
| *** HeOS has joined #openstack-infra | 09:36 | |
| *** yamamoto_ has joined #openstack-infra | 09:39 | |
| *** mrmartin has joined #openstack-infra | 09:39 | |
| *** yamamoto_ has quit IRC | 09:43 | |
| *** johnson_ has joined #openstack-infra | 09:45 | |
| *** ssam2 has joined #openstack-infra | 09:45 | |
| *** annegent_ has joined #openstack-infra | 09:46 | |
| *** annegent_ has quit IRC | 09:51 | |
| *** BharatK has quit IRC | 09:53 | |
| *** dhritishikhar has quit IRC | 09:53 | |
| *** jedimike has joined #openstack-infra | 09:54 | |
| *** hichihara has quit IRC | 09:57 | |
| *** johnson_ has quit IRC | 09:57 | |
| *** amotoki has quit IRC | 09:58 | |
| *** coolsvap is now known as coolsvap|afk | 09:58 | |
| *** mpaolino has quit IRC | 10:00 | |
| *** changbl has quit IRC | 10:01 | |
| *** sdake has joined #openstack-infra | 10:02 | |
| *** mrmartin has quit IRC | 10:04 | |
| *** fandi has quit IRC | 10:05 | |
| *** isviridov_away is now known as isviridov | 10:05 | |
| *** adalbas has joined #openstack-infra | 10:05 | |
| *** adalbas has quit IRC | 10:06 | |
| *** otter768 has joined #openstack-infra | 10:06 | |
| *** BharatK has joined #openstack-infra | 10:07 | |
| *** zhiwei has quit IRC | 10:08 | |
| *** e0ne has joined #openstack-infra | 10:10 | |
| *** dalgaaf has quit IRC | 10:10 | |
| *** dalgaaf has joined #openstack-infra | 10:10 | |
| *** otter768 has quit IRC | 10:11 | |
| *** teran has quit IRC | 10:11 | |
| *** mpaolino has joined #openstack-infra | 10:13 | |
| *** dhritishikhar has joined #openstack-infra | 10:19 | |
| *** pelix has joined #openstack-infra | 10:20 | |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: oslo.messaging: uses devstack-f21 instead f20 https://review.openstack.org/151322 | 10:22 |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: oslo.messaging: add zeromq funcionnal job https://review.openstack.org/151578 | 10:22 |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: oslo.messaging: enable functionnal jobs https://review.openstack.org/151593 | 10:22 |
| *** tsufiev_ is now known as tsufiev | 10:26 | |
| openstackgerrit | Petrut Lucian proposed openstack-dev/pbr: Fixes Windows generated scripts headers https://review.openstack.org/151595 | 10:29 |
| *** CIBuilds has quit IRC | 10:33 | |
| *** mohitsharma has quit IRC | 10:34 | |
| *** jamielennox is now known as jamielennox|away | 10:36 | |
| *** erlon has joined #openstack-infra | 10:36 | |
| *** erlon has quit IRC | 10:36 | |
| *** erlon has joined #openstack-infra | 10:37 | |
| openstackgerrit | Petrut Lucian proposed openstack-dev/pbr: Fixes Windows generated scripts headers https://review.openstack.org/151595 | 10:39 |
| *** pblaho has quit IRC | 10:42 | |
| *** chlong has joined #openstack-infra | 10:43 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 10:43 | |
| *** BharatK has quit IRC | 10:43 | |
| *** yamamoto_ has joined #openstack-infra | 10:44 | |
| *** jamielennox|away is now known as jamielennox | 10:44 | |
| *** teran has joined #openstack-infra | 10:46 | |
| *** annegent_ has joined #openstack-infra | 10:46 | |
| *** ldnunes has joined #openstack-infra | 10:47 | |
| *** vigneshvar has quit IRC | 10:49 | |
| *** teran has quit IRC | 10:50 | |
| *** yamamoto_ has quit IRC | 10:51 | |
| *** annegent_ has quit IRC | 10:51 | |
| *** BharatK has joined #openstack-infra | 10:55 | |
| *** sdake has quit IRC | 10:56 | |
| *** jamielennox is now known as jamielennox|away | 10:56 | |
| *** andreaf has joined #openstack-infra | 11:01 | |
| *** dhritishikhar has quit IRC | 11:02 | |
| openstackgerrit | Petrut Lucian proposed openstack-dev/pbr: Fixes Windows generated scripts headers https://review.openstack.org/151595 | 11:02 |
| *** HeOS has quit IRC | 11:04 | |
| *** rlandy has joined #openstack-infra | 11:05 | |
| *** HeOS has joined #openstack-infra | 11:07 | |
| *** BharatK has quit IRC | 11:13 | |
| *** rfolco has joined #openstack-infra | 11:14 | |
| *** sdake has joined #openstack-infra | 11:17 | |
| *** teran has joined #openstack-infra | 11:23 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/storyboard: Fix status code on delete https://review.openstack.org/151603 | 11:24 |
| *** dtantsur|brb is now known as dtantsur | 11:25 | |
| *** KanagarajM has quit IRC | 11:28 | |
| *** baoli has joined #openstack-infra | 11:28 | |
| *** Guest84671 has quit IRC | 11:29 | |
| *** sdake has quit IRC | 11:30 | |
| nibalizer | fungi: nice patchbomb | 11:32 |
| nibalizer | i hope that was scripted :P | 11:32 |
| *** adalbas has joined #openstack-infra | 11:33 | |
| *** cdent has joined #openstack-infra | 11:40 | |
| *** zz_avozza is now known as avozza | 11:41 | |
| *** ildikov has quit IRC | 11:42 | |
| *** yamamoto has joined #openstack-infra | 11:45 | |
| *** yamamoto has quit IRC | 11:45 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/infra-specs: StoryBoard integration tests specification https://review.openstack.org/150743 | 11:48 |
| *** yolanda_ has joined #openstack-infra | 11:49 | |
| yolanda | jeblair, krotscheck, amended spec for storyboard testing ^ | 11:50 |
| *** pblaho_ has joined #openstack-infra | 11:51 | |
| *** pblaho_ is now known as pblaho | 11:55 | |
| *** mpaolino has quit IRC | 11:57 | |
| *** aysyd has joined #openstack-infra | 11:57 | |
| *** e0ne is now known as e0ne_ | 11:58 | |
| *** andreykurilin|ho has quit IRC | 12:00 | |
| *** e0ne_ is now known as e0ne | 12:00 | |
| *** yamamoto has joined #openstack-infra | 12:04 | |
| *** pc_m has joined #openstack-infra | 12:05 | |
| *** dhritishikhar has joined #openstack-infra | 12:05 | |
| *** otter768 has joined #openstack-infra | 12:07 | |
| odyssey4me | Hi there - it would appear that zuul is stuck with one or more jobs which have succeeded, but they're not finishing as they usually do by reporting back to gerrit - and they just sit in the check queue. This is definitely happening for https://review.openstack.org/151390 | 12:08 |
| odyssey4me | in fact, then entire check queue is sitting... | 12:08 |
| odyssey4me | and the gate queue | 12:09 |
| odyssey4me | I think all the queues, actually. It would appear that perhaps zuul is dead. Long live zuul. | 12:09 |
| *** otter768 has quit IRC | 12:12 | |
| *** yolanda_ has quit IRC | 12:12 | |
| *** andreaf has quit IRC | 12:13 | |
| *** _amrith_ is now known as amrith | 12:13 | |
| *** andreaf has joined #openstack-infra | 12:14 | |
| *** HeOS has quit IRC | 12:20 | |
| *** dalgaaf has quit IRC | 12:20 | |
| *** HeOS has joined #openstack-infra | 12:21 | |
| *** baoli has quit IRC | 12:29 | |
| *** wuhg has quit IRC | 12:29 | |
| *** wuhg has joined #openstack-infra | 12:29 | |
| *** baoli has joined #openstack-infra | 12:29 | |
| *** rfolco has quit IRC | 12:29 | |
| *** jpich has quit IRC | 12:37 | |
| *** jpich has joined #openstack-infra | 12:38 | |
| *** woodster_ has joined #openstack-infra | 12:44 | |
| *** rfolco has joined #openstack-infra | 12:46 | |
| *** _nadya_ has joined #openstack-infra | 12:47 | |
| *** isviridov is now known as isviridov_na | 12:47 | |
| *** tnovacik has quit IRC | 12:48 | |
| *** liam_ has joined #openstack-infra | 12:50 | |
| *** mase_x200 has joined #openstack-infra | 12:50 | |
| *** liam_ is now known as Guest51703 | 12:50 | |
| *** Guest51703 has quit IRC | 12:56 | |
| *** berendt has joined #openstack-infra | 13:02 | |
| *** rfolco has quit IRC | 13:05 | |
| *** rfolco_ has joined #openstack-infra | 13:05 | |
| *** dkliban_afk is now known as dkliban | 13:13 | |
| BobBall | Why is the stackforge/xenapi-os-testing not merge changes? e.g. https://review.openstack.org/#/c/151602/ | 13:17 |
| BobBall | Has the project config needed changed? | 13:17 |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard: Remove duplicate register_options call https://review.openstack.org/151630 | 13:18 |
| *** bhunter71 has joined #openstack-infra | 13:22 | |
| andreaf | odyssey4me, yes indeed everything seems to be dead - no tests are started on new commit | 13:22 |
| odyssey4me | BobBall as andreaf has confirmed, zuul has stalled for all jobs | 13:23 |
| odyssey4me | everything in the queue has stayed exactly as it was for the last two hours or so | 13:23 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-user: Add puppet module files for puppet-user https://review.openstack.org/151517 | 13:26 |
| *** EmilienM|afk is now known as EmilienM | 13:28 | |
| *** radez_g0n3 is now known as radez | 13:28 | |
| *** mpaolino has joined #openstack-infra | 13:29 | |
| *** Guest84671 has joined #openstack-infra | 13:30 | |
| fungi | nibalizer: yes, scripted. i had held off until we had more of the pending boilerplate changes merged | 13:30 |
| fungi | looking into zuul status issues now | 13:31 |
| *** dprince has joined #openstack-infra | 13:32 | |
| *** mbacchi has joined #openstack-infra | 13:35 | |
| *** mbacchi has quit IRC | 13:35 | |
| fungi | looks like the scheduler is just queuing events and results, but not acting on them | 13:36 |
| *** markvoelker has joined #openstack-infra | 13:37 | |
| odyssey4me | fungi that queue must be looking quite long by now | 13:38 |
| *** tnurlygayanov has quit IRC | 13:41 | |
| *** changbl has joined #openstack-infra | 13:41 | |
| fungi | for some reason its signal handler didn't act on my attempt to generate a thread dump either | 13:41 |
| *** mbacchi has joined #openstack-infra | 13:41 | |
| *** doug-fish has joined #openstack-infra | 13:42 | |
| fungi | oh, wait it did | 13:42 |
| BobBall | thanks odyssey4me | 13:42 |
| fungi | i was looking in the wrong log | 13:42 |
| *** deva__ has joined #openstack-infra | 13:45 | |
| fungi | this traceback in the log may coincide with when things stopped moving http://paste.openstack.org/show/164159/ | 13:47 |
| fungi | trying to match it up to the graphs now | 13:47 |
| fungi | yeah, looks likely | 13:49 |
| fungi | seeing what the gerrit logs have to say about that... the change itself exists and is merged according to the webui | 13:50 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:50 | |
| *** jgallard_ has joined #openstack-infra | 13:51 | |
| *** markvoelker has quit IRC | 13:51 | |
| *** markvoelker has joined #openstack-infra | 13:52 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Delete created_at and updated_at fields from requests https://review.openstack.org/151641 | 13:53 |
| *** AJaeger has joined #openstack-infra | 13:56 | |
| *** HeOS has quit IRC | 13:56 | |
| *** markvoelker has quit IRC | 13:57 | |
| *** annegent_ has joined #openstack-infra | 13:57 | |
| fungi | nothing that looks at all related | 13:57 |
| *** scheuran has quit IRC | 14:00 | |
| fungi | the only thing anomalous i can find in gerrit or zuul's cacti graphs in that window is a spike in established tcp sessions for gerrit http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=30&rra_id=all | 14:01 |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Fix stories put method https://review.openstack.org/151643 | 14:02 |
| fungi | a few git uploads hanging around in the gerrit show-queue output for several hours, since that time | 14:02 |
| *** yamamoto has quit IRC | 14:03 | |
| *** tiswanso has joined #openstack-infra | 14:05 | |
| *** e0ne is now known as e0ne_ | 14:06 | |
| *** otter768 has joined #openstack-infra | 14:08 | |
| fungi | killed the stale git upload-pack threads in gerrit but it doesn't seem like that has started zuul moving | 14:08 |
| *** mattfarina has joined #openstack-infra | 14:10 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/storyboard: Pass the old entity values in the NotificationHook https://review.openstack.org/151645 | 14:12 |
| *** e0ne_ is now known as e0ne | 14:12 | |
| *** mriedem1 has joined #openstack-infra | 14:12 | |
| fungi | i wonder if zuul lost contact with its local gearman service | 14:13 |
| *** otter768 has quit IRC | 14:13 | |
| *** mriedem has quit IRC | 14:14 | |
| HenryG | sdague: Morning. This https://review.openstack.org/145793 could do with a +A to unblock a lot of dependent work. | 14:14 |
| *** spzala has joined #openstack-infra | 14:15 | |
| fungi | ugh. the gearman log doesn't seem to have timestamps | 14:15 |
| *** mfink_ has joined #openstack-infra | 14:16 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/storyboard: Pass the old entity values in the NotificationHook https://review.openstack.org/151645 | 14:17 |
| fungi | #status alert zuul isn't running jobs since ~10:30 utc, investigation underway | 14:17 |
| fungi | gah, and statusbot is missing? | 14:17 |
| *** dhritishikhar has quit IRC | 14:17 | |
| *** _nadya_ has quit IRC | 14:18 | |
| *** dustins has joined #openstack-infra | 14:18 | |
| BobBall | statusbot is always missing when there is something important to say :P | 14:19 |
| *** openstackstatus has joined #openstack-infra | 14:20 | |
| *** ChanServ sets mode: +v openstackstatus | 14:20 | |
| rcarrillocruz | hmm | 14:22 |
| rcarrillocruz | has zuul gone postal? | 14:22 |
| BobBall | yes | 14:22 |
| fungi | #status alert zuul isn't running jobs since ~10:30 utc, investigation underway | 14:22 |
| openstackstatus | fungi: sending alert | 14:22 |
| rcarrillocruz | k, thx fungi | 14:23 |
| annegent_ | happy attic day! | 14:24 |
| *** ddieterly has quit IRC | 14:24 | |
| -openstackstatus- NOTICE: zuul isn't running jobs since ~10:30 utc, investigation underway | 14:24 | |
| *** ChanServ changes topic to "zuul isn't running jobs since ~10:30 utc, investigation underway" | 14:24 | |
| fungi | i think the "exception moving gerrit event" is a red herring. i find plenty of them in zuul's log from previous days | 14:25 |
| openstackstatus | fungi: finished sending alert | 14:26 |
| *** markmcclain has quit IRC | 14:27 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Reject false creator_id from requests https://review.openstack.org/151650 | 14:28 |
| fungi | based on merger debug logs it looks like they were getting merge requests up until ~10:50 utc | 14:29 |
| fungi | which is very closely correlated to the spike in established tcp connections on gerrit | 14:30 |
| *** markmcclain has joined #openstack-infra | 14:31 | |
| fungi | right at that time fifieldt remarked on another channel about "gerrit downtime" | 14:31 |
| fungi | but he's not around now so i don't quite know what he meant by that | 14:32 |
| *** dmsimard_away is now known as dmsimard | 14:32 | |
| *** changbl has quit IRC | 14:33 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-etherpad_lite: Change /sbin/nologin to /usr/sbin/nologin for Ubuntu https://review.openstack.org/151651 | 14:33 |
| *** _nadya_ has joined #openstack-infra | 14:33 | |
| *** EmilienM is now known as EmilienM|afl | 14:35 | |
| *** EmilienM|afl is now known as EmilienM|afk | 14:35 | |
| fungi | jeblair: if you're waking up, zuul has gotten itself into a state where it's not processing its queue. it looks like it may have corresponded to some sort of network traffic anomaly for gerrit. it's not obvious to me from zuul's logs or thread dumps what it's waiting on and i don't want to restart it for fear we might lose a critical debugging opportunity | 14:36 |
| *** dims has joined #openstack-infra | 14:36 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-etherpad_lite: Enable websockets transport on Etherpad https://review.openstack.org/151652 | 14:36 |
| * fungi will brb | 14:36 | |
| fungi | made the mistake of looking at irc before i got my morning coffee | 14:36 |
| * rcarrillocruz redoing puppet changes that were by the time on system-config and now are unnmergable due to puppet modules split | 14:36 | |
| *** mriedem1 has quit IRC | 14:38 | |
| *** mrodden has quit IRC | 14:38 | |
| *** bknudson has joined #openstack-infra | 14:38 | |
| fungi | rcarrillocruz: thanks, it was on my agenda to -2/abandon those today anyway | 14:39 |
| *** wuhg has quit IRC | 14:39 | |
| rcarrillocruz | no worries, i can do, i guess you have a loooong list of changes to abandon | 14:39 |
| *** miqui_ has joined #openstack-infra | 14:39 | |
| *** mriedem has joined #openstack-infra | 14:40 | |
| *** mrodden has joined #openstack-infra | 14:41 | |
| *** mrmartin has joined #openstack-infra | 14:44 | |
| *** rlandy has quit IRC | 14:46 | |
| *** mase_x200 has quit IRC | 14:46 | |
| *** kgiusti has joined #openstack-infra | 14:47 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-etherpad_lite: Add connection-tuning file to new Apache 2.4 conf folder structure https://review.openstack.org/151656 | 14:48 |
| *** aysyd has quit IRC | 14:48 | |
| bknudson | is this supposed to work? https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack | 14:51 |
| bknudson | it's the link from https://launchpad.net/openstack | 14:51 |
| *** HeOS has joined #openstack-infra | 14:53 | |
| *** Guest84671 has quit IRC | 14:54 | |
| *** jichen_ has joined #openstack-infra | 14:56 | |
| *** signed8bit has joined #openstack-infra | 14:57 | |
| *** ddieterly has joined #openstack-infra | 14:57 | |
| *** sabeen has joined #openstack-infra | 14:57 | |
| *** bogdando has quit IRC | 14:59 | |
| *** zz_jgrimm is now known as jgrimm | 14:59 | |
| *** jichen has quit IRC | 15:00 | |
| *** jichen_ is now known as jichen | 15:00 | |
| *** armax has joined #openstack-infra | 15:00 | |
| fungi | bknudson: it's _supposed_ to work | 15:03 |
| fungi | but looks like the webserver is kaput on it | 15:03 |
| *** yamamoto has joined #openstack-infra | 15:03 | |
| fungi | bknudson: ohhhhh, http not https | 15:04 |
| fungi | sorry | 15:04 |
| fungi | caffeine is just starting to kick in finally | 15:04 |
| *** sdake has joined #openstack-infra | 15:04 | |
| * mordred is once again not here - but would like to annoucne that today is the last day of his two weeks of nearly constant meetings | 15:04 | |
| *** tonytan4ever has joined #openstack-infra | 15:04 | |
| *** amitgandhinz has joined #openstack-infra | 15:04 | |
| bknudson | can https://launchpad.net/openstack be updated? | 15:05 |
| fungi | mordred: too bad--we have a zuul outage i'm having little luck debugging. you get to miss the excitement | 15:05 |
| fungi | bknudson: yeah, i'm working on that right now | 15:05 |
| bknudson | or can https be enabled? | 15:05 |
| bknudson | fungi: thanks! | 15:05 |
| bknudson | someday that launchpad page config will be in gerrit. | 15:06 |
| fungi | bknudson: launchpad is updated now | 15:06 |
| *** mtanino has joined #openstack-infra | 15:06 | |
| bknudson | fungi: tried it and it works for me. Thanks | 15:06 |
| fungi | bknudson: someday that launchpad page config will be unnecessary because we won't be using launchpad | 15:06 |
| *** yamamoto has quit IRC | 15:08 | |
| *** ryanpetrello_ has joined #openstack-infra | 15:09 | |
| *** bogdando has joined #openstack-infra | 15:09 | |
| *** tmcpeak has joined #openstack-infra | 15:10 | |
| *** ryanpetrello has quit IRC | 15:11 | |
| *** ryanpetrello_ is now known as ryanpetrello | 15:11 | |
| fungi | i think i've about exhausted my options with zuul's situation and have collected any debugging info i know how, so i'm going to grab a snapshot of the current pipeline states and then restart it and try to figure out from the logs what else needs to get reenqueued which is hiding in the event queue currently | 15:13 |
| *** avozza is now known as zz_avozza | 15:14 | |
| *** prad has joined #openstack-infra | 15:18 | |
| fungi | it's restarted, reenqueuing previous changes now | 15:18 |
| fungi | gah, i didn't wait long enough for all the node types to register | 15:19 |
| *** ZZelle has quit IRC | 15:20 | |
| fungi | restarted again, and i'll toggle the gearman plugin on one of the jenkins masters once it's into the main scheduler loop | 15:20 |
| *** ZZelle has joined #openstack-infra | 15:20 | |
| tchaypo | is it possible to find how many bugs times a bug has been mentioned in a “recheck bug xxxx” message? | 15:22 |
| *** che-arne has joined #openstack-infra | 15:22 | |
| *** r-daneel has joined #openstack-infra | 15:22 | |
| *** markvoelker has joined #openstack-infra | 15:22 | |
| *** stevemar has joined #openstack-infra | 15:24 | |
| *** dhritishikhar has joined #openstack-infra | 15:24 | |
| *** markvoelker has quit IRC | 15:25 | |
| *** pradk has joined #openstack-infra | 15:25 | |
| *** markvoelker has joined #openstack-infra | 15:25 | |
| *** wenlock has joined #openstack-infra | 15:26 | |
| fungi | ohhhh! i think i know what may have precipitated this | 15:28 |
| *** zz_avozza is now known as avozza | 15:29 | |
| clarkb | tchaypo yes but we fpund the data to be mostly useless. which is why we dropped the bug requirement and focused on elastic recheck | 15:29 |
| fungi | rackspace was restarting trove database instances in dfw (and are still in progress doing so according to https://status.rackspace.com/ updates) | 15:29 |
| clarkb | fungi oh good I was going t have to start looking at logs | 15:29 |
| jeblair | fungi: morning | 15:29 |
| fungi | so that at least explains why gerrit's open tcp connections would have spiked up over about a 10-minute period | 15:30 |
| *** timcline has joined #openstack-infra | 15:30 | |
| jroll | interesting | 15:30 |
| fungi | i'm still trying to get bare-precise and py3k-precise workers registered in zuul's gearman before i reenqueue the lists i saved | 15:30 |
| *** zz_dimtruck is now known as dimtruck | 15:30 | |
| *** sdake has quit IRC | 15:31 | |
| clarkb | tchaypo I think sdague may have the script that combs comments for recheck bug #### if you want a way to do it. uses gerrit rest api iirc | 15:32 |
| jeblair | disabling the registration check is next on my list of zuul features to add | 15:32 |
| jeblair | you could probably use a gerrit search now, right? | 15:32 |
| jeblair | "comment:'recheck bug'" | 15:33 |
| *** packet has joined #openstack-infra | 15:33 | |
| fungi | unfortunately we have some stuck in a building state in nodepool for a similar timeframe so possible something went south with the (looks like rax-iad) nova api endoint | 15:33 |
| fungi | something tells me the rackspace "reboot all the things" scramble is not over and we'll probably be fighting fallout from it well into the day | 15:33 |
| jeblair | maybe we should just go ahead and move our maint window to now :) | 15:34 |
| fungi | can't hurt i guess | 15:34 |
| *** dansmith is now known as superdan | 15:34 | |
| fungi | i'm going to restart nodepool so i can clear these hung building nodes and try to get some workers registered | 15:35 |
| * clarkb notes he isnt quite awake yet but dont let that stop you | 15:35 | |
| jeblair | neither am i; if we wanted to move it, we'd probably still need a little while to get ready | 15:35 |
| tchaypo | thanks jeblair | 15:36 |
| *** achanda has joined #openstack-infra | 15:37 | |
| *** dims has quit IRC | 15:39 | |
| *** mrmartin has quit IRC | 15:39 | |
| *** roaet- has quit IRC | 15:43 | |
| *** achanda has quit IRC | 15:43 | |
| *** avozza is now known as zz_avozza | 15:44 | |
| *** jpich has quit IRC | 15:46 | |
| *** dims has joined #openstack-infra | 15:46 | |
| tchaypo | https://review.openstack.org/#/q/comment:%22recheck+bug%22,n,z isn’t finding me anything. | 15:46 |
| fungi | okay, bare-precise and py3k-precise are registered so reenqueuing saved changes now | 15:47 |
| fungi | and then i'll try to see what's in the event history in the zuul log and get those manually enqueued | 15:47 |
| *** salv-orlando has joined #openstack-infra | 15:47 | |
| *** carl_baldwin has joined #openstack-infra | 15:48 | |
| *** koolhead17 has quit IRC | 15:48 | |
| *** waynr has joined #openstack-infra | 15:49 | |
| fungi | jeblair: once you're more awake, there are a couple of thread dumps in my homedir on zuul (threaddump1 and threaddump2) taken a couple minutes apart while zuul was not processing its queues | 15:49 |
| *** rainya_ is now known as rainya | 15:50 | |
| *** dims has quit IRC | 15:50 | |
| jeblair | fungi: cool, will take a look | 15:51 |
| *** kragniz is now known as zingark | 15:53 | |
| jeblair | fungi: it was stuck trying to report back to gerrit | 15:54 |
| *** _nadya_ has quit IRC | 15:54 | |
| fungi | huh, so the paramiko thread? | 15:54 |
| jeblair | fungi: well, the reporting happens in the main thread | 15:54 |
| *** Ala has quit IRC | 15:54 | |
| fungi | oh, got it. paramiko is the stream events listener i guess | 15:54 |
| *** dangers_away is now known as dangers | 15:55 | |
| jeblair | fungi: Thread: 140344062621440 | 15:55 |
| *** _nadya_ has joined #openstack-infra | 15:55 | |
| jeblair | http://paste.openstack.org/show/164203/ | 15:55 |
| *** abhirc has joined #openstack-infra | 15:55 | |
| jeblair | so either gerrit went non-responsive, or the connection went bad | 15:56 |
| *** carl_baldwin has quit IRC | 15:56 | |
| *** timcline has quit IRC | 15:56 | |
| fungi | i _think_ gerrit's trove database was offline | 15:56 |
| *** carl_baldwin has joined #openstack-infra | 15:56 | |
| fungi | (being fanatically rebooted by rackspace) | 15:57 |
| *** timcline has joined #openstack-infra | 15:57 | |
| jeblair | i wonder what gerrit's behavior was in that case? did you restart gerrit? | 15:57 |
| fungi | nope | 15:57 |
| fungi | it continued on its merry way | 15:57 |
| jeblair | hrm | 15:57 |
| fungi | i need to dig deeper into the gerrit logs once i'm done getting the straggler changes reenqueued from the triggers in the zuul log | 15:57 |
| *** mpaolino has quit IRC | 15:58 | |
| clarkb | maybe a bug in the ssh command gerrit side? | 15:58 |
| jeblair | so figuring out how to time out an ssh command in zuul might be a good thing | 15:58 |
| *** jcoufal_ has joined #openstack-infra | 16:00 | |
| clarkb | zaro did you see replication in gerrit 2.9 requires an explicit flag to force push? we will nees that | 16:00 |
| jeblair | http://docs.paramiko.org/en/1.15/api/channel.html#paramiko.channel.Channel.settimeout | 16:02 |
| jeblair | we can probably work something out with that ^ | 16:02 |
| fungi | everything that got approved while zuul was offline is getting reenqueued into check now (and will propagate into the gate once it passes if it's ready) | 16:02 |
| *** ddieterly has quit IRC | 16:02 | |
| *** pcrews has joined #openstack-infra | 16:03 | |
| *** nelsnelson has joined #openstack-infra | 16:03 | |
| *** nelsnelson has quit IRC | 16:04 | |
| *** jcoufal has quit IRC | 16:04 | |
| superdan | do we need to recheck things after this morning's outage? (either now or at some point) | 16:04 |
| jeblair | https://storyboard.openstack.org/#!/story/2000141 | 16:04 |
| *** nelsnelson has joined #openstack-infra | 16:05 | |
| *** zz_avozza is now known as avozza | 16:05 | |
| *** EmilienM|afk is now known as EmilienM | 16:07 | |
| fungi | everything that got a patchset uploaded or was restored from an abandoned state while zuul was offline is being enqueued into the check pipeline now too | 16:07 |
| clarkb | superdan I think fungi has you covered | 16:07 |
| *** ddieterly has joined #openstack-infra | 16:07 | |
| superdan | rock on | 16:07 |
| fungi | superdan: i don't think so, except possibly for a few minutes between when zuul was restarted i've mined events out of the logs (since zuul was still listening to gerrit while it was stuck) | 16:07 |
| superdan | okay | 16:08 |
| fungi | superdan: in a few minutes everything will be done getting reenqueued and then you should check http://status.openstack.org/zuul/ for changes if in doubt as to whether or not zuul is planning to test them already | 16:08 |
| superdan | sure, thanks | 16:09 |
| fungi | you bet | 16:09 |
| fungi | and apologies for the disruption. clouds | 16:09 |
| *** mpaolino has joined #openstack-infra | 16:10 | |
| *** rfolco_ has quit IRC | 16:10 | |
| fungi | #status ok zuul is running again and changes have been reenqueud. seehttp://status.openstack.org/zuul/ before rechecking if in doubt | 16:11 |
| openstackstatus | fungi: sending ok | 16:11 |
| *** berendt has quit IRC | 16:11 | |
| *** ashleighfarnham has joined #openstack-infra | 16:11 | |
| *** belmoreira has quit IRC | 16:11 | |
| clarkb | last night I mostly figured out the multinode test failure. when you shelve then unshelve a node you can get floating ip and nat rules on both routers | 16:11 |
| *** salv-orlando has quit IRC | 16:12 | |
| clarkb | this creates brokeness with multi router nova net | 16:12 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/manual/ http://ci.openstack.org/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/" | 16:12 | |
| -openstackstatus- NOTICE: zuul is running again and changes have been reenqueud. seehttp://status.openstack.org/zuul/ before rechecking if in doubt | 16:12 | |
| clarkb | there is a proposed fix I need to test though so hopefully not an issue for long | 16:12 |
| mtreinish | clarkb: we can just add skip for the bug to tempest if you want to unblock faster | 16:14 |
| openstackstatus | fungi: finished sending ok | 16:15 |
| mtreinish | I don't think the loss of coverage on the shelve api for small time is really a big deal compared to getting multinode testing | 16:15 |
| clarkb | mtreinish if the nova fix works we can probably go with it | 16:15 |
| *** rfolco has joined #openstack-infra | 16:16 | |
| *** dannywilson has joined #openstack-infra | 16:16 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Reject false creator_id from requests https://review.openstack.org/151650 | 16:17 |
| *** sdake_ is now known as sdake | 16:18 | |
| fungi | okay, i'm going to catch up on e-mail and eat some breakfast now that there's a bit of a lull | 16:18 |
| *** david-lyle_afk is now known as david-lyle | 16:19 | |
| *** jgallard_ has quit IRC | 16:19 | |
| *** funzo has quit IRC | 16:21 | |
| *** jerryz has quit IRC | 16:22 | |
| *** funzo has joined #openstack-infra | 16:22 | |
| *** rfolco has quit IRC | 16:22 | |
| *** rfolco has joined #openstack-infra | 16:23 | |
| *** tonytan4ever has quit IRC | 16:25 | |
| *** apmelton_ is now known as apmelton | 16:25 | |
| *** tonytan4ever has joined #openstack-infra | 16:25 | |
| *** _nadya_ has joined #openstack-infra | 16:26 | |
| *** asselin_ has quit IRC | 16:26 | |
| *** carl_baldwin has quit IRC | 16:28 | |
| *** carl_baldwin has joined #openstack-infra | 16:29 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: DO NOT MERGE testing nova multinode fix https://review.openstack.org/151690 | 16:30 |
| clarkb | mtreinish: ^ let us see how that does | 16:30 |
| *** tonytan4ever has quit IRC | 16:30 | |
| *** avozza is now known as zz_avozza | 16:30 | |
| *** abhirc has quit IRC | 16:30 | |
| *** dims has joined #openstack-infra | 16:31 | |
| clarkb | and now to turn on subunit worker | 16:31 |
| *** crc32 has joined #openstack-infra | 16:31 | |
| *** dims has quit IRC | 16:31 | |
| *** dims has joined #openstack-infra | 16:32 | |
| *** zz_avozza is now known as avozza | 16:32 | |
| *** dims is now known as dimsum__ | 16:33 | |
| *** _nadya_ has quit IRC | 16:34 | |
| *** asselin_ has joined #openstack-infra | 16:35 | |
| *** asselin has joined #openstack-infra | 16:35 | |
| *** tonytan4ever has joined #openstack-infra | 16:37 | |
| *** andreaf has quit IRC | 16:37 | |
| *** andreaf has joined #openstack-infra | 16:38 | |
| *** craigbr has joined #openstack-infra | 16:42 | |
| *** jaypipes has joined #openstack-infra | 16:42 | |
| *** ashleighfarnham has quit IRC | 16:44 | |
| *** deva__ has quit IRC | 16:45 | |
| *** homeless has joined #openstack-infra | 16:45 | |
| *** _nadya_ has joined #openstack-infra | 16:46 | |
| *** cdent has quit IRC | 16:46 | |
| *** cdent has joined #openstack-infra | 16:47 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 16:47 |
| *** alexpilotti has joined #openstack-infra | 16:48 | |
| alexpilotti | mordred: hello | 16:48 |
| alexpilotti | mordred: we have a small blocking issues on pbr: https://review.openstack.org/#/c/151595/ | 16:48 |
| *** rwsu-afk is now known as rwsu | 16:49 | |
| alexpilotti | mordred: wonder if you could please take a quick look, it’s missing the second +2 blessing :-) | 16:49 |
| *** AJaeger has quit IRC | 16:49 | |
| *** jkraj has quit IRC | 16:49 | |
| *** vigneshvar has joined #openstack-infra | 16:49 | |
| clarkb | alexpilotti: by brackets do you mean quotes? I am trying to understand the code change given the commit message | 16:50 |
| *** salv-orlando has joined #openstack-infra | 16:51 | |
| alexpilotti | clarkb: quotes, possibly lpetruts wrote brackets instead of quotes :-) | 16:51 |
| *** marun has joined #openstack-infra | 16:51 | |
| *** e0ne is now known as e0ne_ | 16:51 | |
| *** mrmartin has joined #openstack-infra | 16:51 | |
| *** e0ne_ is now known as e0ne | 16:52 | |
| clarkb | alexpilotti: +A | 16:52 |
| alexpilotti | clarkb: tx a lot! | 16:52 |
| *** packet has quit IRC | 16:54 | |
| *** craigbr has quit IRC | 16:55 | |
| *** packet has joined #openstack-infra | 16:56 | |
| *** sabeen2 has joined #openstack-infra | 16:56 | |
| *** unicell has quit IRC | 16:56 | |
| *** woodster_ has quit IRC | 16:56 | |
| *** woodster_ has joined #openstack-infra | 16:56 | |
| *** ssam2 has quit IRC | 16:57 | |
| *** hj-hp has joined #openstack-infra | 16:58 | |
| *** che-arne has quit IRC | 16:58 | |
| *** sabeen has quit IRC | 16:58 | |
| *** ssam2 has joined #openstack-infra | 16:59 | |
| *** KanagarajM2 has joined #openstack-infra | 16:59 | |
| *** ChuckC has joined #openstack-infra | 17:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Add networking-cisco to stackforge https://review.openstack.org/145793 | 17:03 |
| *** annegent_ has quit IRC | 17:05 | |
| *** ChuckC_ has quit IRC | 17:05 | |
| clarkb | mtreinish: I ran into a thing yesterday when debugging the shelving test. The default security group after tempest ran had no rules in it. Does tempest remove all rules so that it can test specific things? | 17:05 |
| clarkb | or is the default security group completely locked down by default? | 17:05 |
| *** yjiang5 is now known as yginger5 | 17:06 | |
| *** denis_makogon has quit IRC | 17:06 | |
| *** ChuckC_ has joined #openstack-infra | 17:06 | |
| zaro | morning | 17:06 |
| zaro | clarkb: that's will look into that replication flag. | 17:07 |
| *** zingark is now known as krangiz | 17:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add test for a enqueuing complex dependencies https://review.openstack.org/151705 | 17:07 |
| *** krangiz is now known as kragniz | 17:07 | |
| zaro | that/thanks | 17:07 |
| clarkb | zaro: good morning. sounds good. I noticed it in a recent gerrit group thread | 17:07 |
| jeblair | clarkb, fungi, jhesketh: ^ okay that's the result of my tour down the rabbit hole from yesterday | 17:07 |
| jeblair | clarkb, fungi, jhesketh: the result is that basically, i think zuul is working correctly, and i made one tiny change that shouldn't really make a difference. | 17:08 |
| jeblair | but at least it's explained, documented, and tested now :) | 17:08 |
| *** craigbr has joined #openstack-infra | 17:08 | |
| jeblair | clarkb, fungi, jhesketh: and that tree ordering is actually a third option, different from the two yesterday (it's a true depth-first order) | 17:09 |
| *** ChuckC has quit IRC | 17:10 | |
| *** ChuckC has joined #openstack-infra | 17:10 | |
| *** jcoufal_ has quit IRC | 17:10 | |
| mtreinish | clarkb: it's that way by default. We create a new tenant for each test class and the default sg for a new tenant doesn't have any rules in it IIRC | 17:10 |
| clarkb | mtreinish: huh that is not how public clouds work :) | 17:11 |
| *** gyee has joined #openstack-infra | 17:11 | |
| clarkb | mtreinish: well shelve instance test failed with the supposed nova fix so we may want to skip it after all | 17:12 |
| clarkb | I am going to double check that the right versino of nova was installed | 17:12 |
| *** ChuckC_ has quit IRC | 17:14 | |
| clarkb | oh cool my checkout nova fix may be buggy let me fix that before we skip any tests | 17:14 |
| *** Sukhdev has joined #openstack-infra | 17:15 | |
| *** hdd has joined #openstack-infra | 17:15 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: DO NOT MERGE testing nova multinode fix https://review.openstack.org/151690 | 17:17 |
| mtreinish | clarkb: ok | 17:17 |
| jeblair | clarkb, fungi, mordred, jhesketh, SergeyLukjanov: http://lists.openstack.org/pipermail/openstack-dev/2015-January/055661.html | 17:17 |
| clarkb | the zuul mergers are not replicated to so cannot be treated as mirrors | 17:17 |
| *** dtantsur is now known as dtantsur|afk | 17:17 | |
| dougwig | hi infra, what's the right copyright header to put in source files? i've been using company name, which I assume then gets assigned via the CCLA, but have some review feedback to use openstack foundation. | 17:18 |
| *** spzala has quit IRC | 17:18 | |
| *** spzala_ has joined #openstack-infra | 17:18 | |
| jeblair | dougwig: if you do not work for the foundation, you should not indicate that your copyright is owned by it. the right answer is _probably_ your company (i can't say that for certain of course) | 17:18 |
| jeblair | dougwig: can you point me at the review? | 17:18 |
| *** spzala_ is now known as spzala | 17:18 | |
| dougwig | https://review.openstack.org/#/c/151375/ | 17:19 |
| clarkb | the CLA does not require copyright assignment so it basically boils down to who owns the work you did. Often your employer but not necessarily | 17:19 |
| clarkb | jeblair: ! | 17:19 |
| dougwig | clarkb: that's what i thought. | 17:20 |
| jeblair | i really love it when i get to write an email like that :) | 17:20 |
| *** alexpilotti has quit IRC | 17:20 | |
| * mtreinish is still happy to have a personal copyright in tempest from a week of unemployment | 17:20 | |
| *** salv-orlando has quit IRC | 17:21 | |
| jeblair | mtreinish: i don't work on apache licensed projects on my own time :) i spent my week of unemployment in the mountains | 17:21 |
| clarkb | jeblair: where you wrote GPL licensed smoke signal drivers? | 17:23 |
| *** afazekas has quit IRC | 17:23 | |
| mtreinish | jeblair: I didn't really have that luxury then. It was the week before I departed on my road trip to Atl so I had a bunch of summit prep stuff to do. I would have prefered not to be working that week | 17:24 |
| *** ddieterly has quit IRC | 17:25 | |
| *** craigbr has quit IRC | 17:26 | |
| openstackgerrit | Merged openstack-dev/hacking: Fix H238 to allow parent classes to be returned by a function https://review.openstack.org/148564 | 17:26 |
| mtreinish | clarkb: heh, well it would have to be gpl if it was for the linux kernel right? :) | 17:26 |
| *** ddieterly has joined #openstack-infra | 17:27 | |
| *** ddieterly has quit IRC | 17:27 | |
| clarkb | mtreinish: it could toggle a usb device from userspace I dunno | 17:27 |
| *** ddieterly has joined #openstack-infra | 17:27 | |
| *** craigbr has joined #openstack-infra | 17:28 | |
| mtreinish | clarkb: hahaha, fair enough | 17:28 |
| *** hj-hp has quit IRC | 17:28 | |
| *** ccrouch has joined #openstack-infra | 17:28 | |
| *** derekh has quit IRC | 17:29 | |
| *** hj-hp has joined #openstack-infra | 17:29 | |
| *** ddieterly has quit IRC | 17:30 | |
| *** mtanino has quit IRC | 17:31 | |
| *** patrickeast has joined #openstack-infra | 17:32 | |
| clarkb | oh hey the ceph job seems to be working now | 17:32 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Suspend Py3K jobs for oslo.messaging oslo.rootwrap https://review.openstack.org/151712 | 17:32 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Switch from Python 3.3 to 3.4 for Py3K jobs https://review.openstack.org/151713 | 17:32 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Stop building py3k-precise nodes https://review.openstack.org/151714 | 17:33 |
| ccrouch | clarkb: quick update on my "changing my openstack related email addresses" saga .. | 17:34 |
| ccrouch | I'm think I'm ready to flip them over from @redhat to @gmail | 17:34 |
| ccrouch | I can see my primary contact email address in Launchpad is @redhat, but my @gmail is confirmed there too | 17:34 |
| ccrouch | on review.openstack.org I can see @redhat is preferred email, but @gmail is also registered now | 17:34 |
| ccrouch | finally www.openstack.org has my primary as @redhat and secondary @gmail | 17:34 |
| ccrouch | clarkb: I think the next steps are... | 17:34 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Stop using py3k-precise nodes https://review.openstack.org/151715 | 17:35 |
| clarkb | ccrouch: in your foundation profile flip your primary to gmail then do the same for gerrit | 17:35 |
| ccrouch | switch @gmail to preferred email in review.openstack.org | 17:36 |
| ccrouch | then switch Launchpad so primary is @gmail, and I'll need to reset the password | 17:36 |
| ccrouch | then finally set www.openstack.org so primary is @gmail | 17:36 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/shade: Adds get_flavor() and get_network() functions https://review.openstack.org/150649 | 17:36 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/shade: Fixes broken image/object hashing https://review.openstack.org/150651 | 17:36 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/shade: Adds some more swift operations https://review.openstack.org/150650 | 17:36 |
| clarkb | ccrouch: ya that sounds about right | 17:36 |
| clarkb | fungi: ^ | 17:36 |
| *** melwitt has joined #openstack-infra | 17:36 | |
| clarkb | my only concern is when the launchpad change happens you may still get a new openid? | 17:36 |
| clarkb | but iirc fungi had a workaround for that so probably just a step missing if at all | 17:37 |
| ccrouch | I'm a little concerned on the openid side | 17:37 |
| ccrouch | in launchpad i have https://launchpad.net/~ccrouch | 17:37 |
| fungi | ccrouch: yeah, check the openid lp reports for you currently and make a note of it | 17:37 |
| *** unicell has joined #openstack-infra | 17:37 | |
| ccrouch | in review.openstack.org > Identities > @redhat has an entry | 17:37 |
| ccrouch | but @gmail is blank | 17:37 |
| fungi | ccrouch: stay logged into gerrit while you switch launchpad | 17:38 |
| fungi | ccrouch: then check the openid launchpad is reporting after the switch | 17:38 |
| clarkb | ccrouch: to check what launchpad reports view the source of your use page | 17:38 |
| fungi | ccrouch: if it's different, add it at https://review.openstack.org/#/settings/web-identities (maybe. for some reason i don't see the ability to add to that list though) | 17:38 |
| fungi | ccrouch: so it might end up with a new gerrit account the next time you log out of lp and back in anyway | 17:39 |
| fungi | ccrouch: if it does, we'll have to go poking in the gerrit database to disable one of the duplicate accounts, just give us a heads up | 17:39 |
| *** changbl has joined #openstack-infra | 17:39 | |
| *** salv-orlando has joined #openstack-infra | 17:40 | |
| mtreinish | clarkb: I don't think it will I switched mine from an ibm address to my domain a while ago and didn't have any issues | 17:40 |
| mtreinish | it did take me forever to figure out how to switch it in lp though | 17:40 |
| ccrouch | fungi: launchpad > OpenID login: and | 17:40 |
| ccrouch | review.openstack.org > Identities | 17:40 |
| ccrouch | are definitely showing different urls for the same @redhat email right now | 17:40 |
| *** dimsum__ has quit IRC | 17:40 | |
| fungi | yeah, i've never quite been able to pin down what triggers lp to create a new openid. definitely seemed to be tied to lp account modifications, just not sure what exactly | 17:40 |
| *** bdpayne has joined #openstack-infra | 17:40 | |
| fungi | ccrouch: does it show more than one there? | 17:41 |
| *** spzala has quit IRC | 17:41 | |
| zaro | zuul-dev.o.o will be up and down for testing | 17:41 |
| mtreinish | clarkb, fungi: qq can we change the permissions on the openstack-qa list to only allow the emails from the periodic jobs and reject all others? | 17:41 |
| jeblair | fungi, clarkb: i'm going to send an email about our 61 new repos; sound good? | 17:41 |
| fungi | jeblair: sounds great! | 17:41 |
| clarkb | jeblair: yup, I assume with some directions on making puppet changes too? | 17:41 |
| jeblair | (granted, probably only 57 of them are actually new, but who's counting) | 17:42 |
| *** hj-hp has quit IRC | 17:42 | |
| jeblair | clarkb: can you elaborate on that? | 17:42 |
| clarkb | mtreinish: I am no mailman expert but I believe you can give certain subscribers extra permissions | 17:42 |
| ccrouch | fungi: on review.openstack.org > Identities | 17:42 |
| ccrouch | there are two rows one for @redhat and one for @gmail | 17:42 |
| ccrouch | but only the @redhat has anything in the identity column | 17:42 |
| clarkb | jeblair: like "you will now need to make puppet changes to specific modules to modify their behavior" that seems to general | 17:42 |
| jeblair | clarkb: here's what i have: https://etherpad.openstack.org/p/ZRYtKd4OpG | 17:42 |
| clarkb | jeblair: just a heads up that system-config is no longer the place to update eg jenkins | 17:42 |
| *** dhritishikhar has quit IRC | 17:42 | |
| *** mpaolino has quit IRC | 17:43 | |
| jeblair | clarkb: okay, i grok. lemme add some stuff... | 17:43 |
| ccrouch | fungi: fyi url for @redhat email in review.openstack.org > Identities is https://login.launchpad.net/+id/n6xFnFx | 17:43 |
| fungi | jeblair: we're up to 109 infra projects, 255 active (not -attic) official openstack projects, 308 active stackforge projects and 569 total projects in gerrit | 17:44 |
| clarkb | speaking of attic. Do we need any prep changes for the 1900 renames? | 17:44 |
| jeblair | fungi: drat, i was really hoping we'd be a higher % of openstack by now; at least half. :) | 17:45 |
| ccrouch | fungi: again fwiw in LP the openid login is just https://launchpad.net/~ccrouch | 17:45 |
| clarkb | ccrouch: https://login.launchpad.net/+id/n6xFnFx is what I see for you in LP as well | 17:45 |
| clarkb | ccrouch: so those match up | 17:45 |
| fungi | jeblair: announcement lgtm | 17:46 |
| clarkb | ccrouch: if you view the source of your user page you will see them listed | 17:46 |
| *** ildikov has joined #openstack-infra | 17:46 | |
| ccrouch | ah ok | 17:46 |
| clarkb | ccrouch: so what happens is you are still ~ccrouch in lp but sometimeswhen you make changes to mail ou get a new +id/$hash string thing | 17:46 |
| ccrouch | clarkb: got it <link rel="openid.delegate" | 17:46 |
| clarkb | ccrouch: it doesn't remove the old string either you just end up with two of them, then order comes into play and it causes gerrit to get confused | 17:47 |
| ccrouch | so just to be crystal clear.. | 17:47 |
| *** armax has quit IRC | 17:47 | |
| ccrouch | i should be swapping emails in review.openstack.org *before* LP right? | 17:47 |
| ccrouch | and remaining logged into review.o.o while making the LP change | 17:48 |
| *** e0ne has quit IRC | 17:48 | |
| clarkb | ccrouch: yes | 17:48 |
| ccrouch | ack, thanks | 17:48 |
| * ccrouch dons his openid retardant suit | 17:48 | |
| *** melwitt_ has joined #openstack-infra | 17:49 | |
| *** harlowja_away is now known as harlowja_ | 17:50 | |
| clarkb | mtreinish: jogo http://logs.openstack.org/90/151690/2/experimental/check-tempest-dsvm-aiopcpu/95baf08/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-01-30_17_22_13_780 I got the shelve nova fix installed properly this time but the test still failed | 17:50 |
| jeblair | clarkb, fungi: puppet announcement email updated | 17:50 |
| *** yginger5 has quit IRC | 17:51 | |
| *** ashleighfarnham has joined #openstack-infra | 17:51 | |
| clarkb | jeblair: looks good, thank you | 17:51 |
| clarkb | mtreinish: jogo: so maybe a skip is best for now? can we skip based on multinode vs single node? eg could tempest ask nova for the number of computes and select that test only if one compute exists? makes that test seem somewhat artificial though | 17:52 |
| clarkb | "here is a feature for the cloud, it only works with a one node cloud" | 17:53 |
| jeblair | clarkb: ha! | 17:53 |
| fungi | ebadditions are great | 17:53 |
| fungi | er | 17:53 |
| jeblair | fungi: i hope they are not ebad | 17:53 |
| fungi | jeblair: additions are great | 17:53 |
| *** yjiang5 has joined #openstack-infra | 17:54 | |
| *** yjiang5 is now known as yginger5 | 17:54 | |
| fungi | for some reason my terminal ate the initial j in jeb and so my tab complete failed and i didn't notice | 17:54 |
| *** MaxV has quit IRC | 17:54 | |
| jeblair | fungi: "oh! The badditions you've made are great!" | 17:54 |
| fungi | _e_badditions | 17:54 |
| *** ddieterly has joined #openstack-infra | 17:54 | |
| *** bnemec is now known as beekneemech | 17:55 | |
| mtreinish | clarkb: well we can skip on the bug and a config flag | 17:55 |
| *** melwitt has quit IRC | 17:55 | |
| fungi | they're all terrigreat | 17:55 |
| *** marun has quit IRC | 17:55 | |
| mtreinish | but we don't allow skip decisions to be made by api responses | 17:55 |
| *** abhirc has joined #openstack-infra | 17:55 | |
| jeblair | fungi: sounds flawsome! | 17:55 |
| *** melwitt_ is now known as melwitt | 17:55 | |
| clarkb | mtreinish ok | 17:55 |
| fungi | craptacular | 17:55 |
| mtreinish | because the last time we did that we accidently ended up skipping all of the neutron testing and didn't even realize it | 17:56 |
| fungi | i'll whip up a change to the governance repo for the new e-badditions while i'm thinking about it | 17:56 |
| *** _nadya_ has quit IRC | 17:56 | |
| *** marun has joined #openstack-infra | 17:56 | |
| *** teran has quit IRC | 17:56 | |
| jeblair | fungi: already done | 17:56 |
| clarkb | mtreinish nice | 17:56 |
| jeblair | fungi: 151100 | 17:56 |
| zaro | zuul question: anybody know where the openstack_functions.py is located? that's the file that gets included in layout.yaml | 17:56 |
| fungi | jeblair: yep, i checked the proposed list just now when i started | 17:56 |
| fungi | zaro: project-config | 17:56 |
| fungi | zaro: it's possible the manifest for zuul-dev is lacking a file resource for that | 17:57 |
| fungi | zaro: alternatively, we could simply strip out the include directive from its layout.yaml | 17:57 |
| *** ashleighfarnham has quit IRC | 17:57 | |
| *** dhritishikhar has joined #openstack-infra | 17:57 | |
| jeblair | fungi, zaro: i think we changed it to be a directory, so the manifest shouldn't need changing | 17:57 |
| jeblair | (i mean, maybe double check that, but that's my recollection) | 17:58 |
| *** annegent_ has joined #openstack-infra | 17:58 | |
| *** ChuckC has quit IRC | 17:58 | |
| zaro | it's not on review-dev and i think that might be the problem. testing it now | 17:58 |
| *** jp_at_hp has quit IRC | 17:59 | |
| *** shivharis has joined #openstack-infra | 17:59 | |
| *** ashleighfarnham has joined #openstack-infra | 17:59 | |
| zaro | *review-dev/zuul-dev | 17:59 |
| *** _nadya_ has joined #openstack-infra | 18:00 | |
| *** ChuckC has joined #openstack-infra | 18:00 | |
| *** sarob has joined #openstack-infra | 18:00 | |
| fungi | has anybody started the maintenance plan for the project renames yet? if not i'll get the etherpad put together since we're t-minus one hour and counting | 18:00 |
| jeblair | fungi: nak | 18:01 |
| *** ChuckC_ has joined #openstack-infra | 18:01 | |
| *** crc32 has quit IRC | 18:02 | |
| *** spzala has joined #openstack-infra | 18:03 | |
| clarkb | I havent | 18:03 |
| *** salv-orlando has quit IRC | 18:04 | |
| *** harlowja_ has quit IRC | 18:04 | |
| *** salv-orlando has joined #openstack-infra | 18:05 | |
| *** julim has quit IRC | 18:05 | |
| *** ChuckC has quit IRC | 18:05 | |
| *** mestery has quit IRC | 18:05 | |
| *** julim has joined #openstack-infra | 18:06 | |
| *** mestery has joined #openstack-infra | 18:06 | |
| *** yamamoto has joined #openstack-infra | 18:06 | |
| *** charz has quit IRC | 18:07 | |
| *** harlowja has joined #openstack-infra | 18:07 | |
| *** dimsum__ has joined #openstack-infra | 18:07 | |
| *** mkoderer has quit IRC | 18:07 | |
| *** crc32 has joined #openstack-infra | 18:08 | |
| *** annegent_ has quit IRC | 18:08 | |
| *** begal-sop_ has joined #openstack-infra | 18:08 | |
| *** Guest74239 has quit IRC | 18:08 | |
| *** vladan has quit IRC | 18:08 | |
| *** loquacities has quit IRC | 18:08 | |
| *** morgabra has quit IRC | 18:08 | |
| *** ddieterly has quit IRC | 18:09 | |
| *** dkliban has quit IRC | 18:09 | |
| *** BobBall has quit IRC | 18:09 | |
| *** alaski has quit IRC | 18:09 | |
| *** alazarev has quit IRC | 18:09 | |
| *** yginger5 has quit IRC | 18:09 | |
| *** Vivek_ has joined #openstack-infra | 18:09 | |
| *** med_ has quit IRC | 18:10 | |
| *** radez is now known as radez_g0n3 | 18:10 | |
| *** BobBall has joined #openstack-infra | 18:10 | |
| *** BobBall is now known as Guest11352 | 18:10 | |
| *** yamamoto has quit IRC | 18:11 | |
| *** yjiang5 has joined #openstack-infra | 18:11 | |
| *** spzala has quit IRC | 18:12 | |
| *** begal-sop has quit IRC | 18:12 | |
| *** radez_g0n3 is now known as radez | 18:12 | |
| *** tonytan4ever has quit IRC | 18:13 | |
| *** alaski has joined #openstack-infra | 18:13 | |
| *** mkoderer has joined #openstack-infra | 18:13 | |
| *** alazarev has joined #openstack-infra | 18:13 | |
| ccrouch | fungi: clarkb: so the results for switching my primary email in review.o.o and LP were I think mixed :-/ | 18:13 |
| ccrouch | the good news: | 18:13 |
| ccrouch | 1) i was able to successfully switch both accounts to have @gmail as primary email | 18:13 |
| ccrouch | 2) I can then log into both accounts with the @gmail address | 18:13 |
| ccrouch | 3) the openid in launchpad remained the same (according to openid.delegate in the src for https://launchpad.net/~ccrouch, its still https://login.launchpad.net/+id/n6xFnFx) | 18:13 |
| ccrouch | now the bad news | 18:13 |
| ccrouch | a) when I'm logged into review.o.o I can no longer see any Starred Changes ( I could have sworn I had some) | 18:13 |
| ccrouch | I presume this is due to the fact that now when I go to https://review.openstack.org/#/settings/web-identities | 18:13 |
| ccrouch | I only see one row with my @gmail address but a new OpenId url: https://login.launchpad.net/+id/4wwPefA | 18:13 |
| *** morgabra has joined #openstack-infra | 18:13 | |
| *** vladan has joined #openstack-infra | 18:13 | |
| *** med_ has joined #openstack-infra | 18:13 | |
| *** charz has joined #openstack-infra | 18:14 | |
| *** med_ has quit IRC | 18:14 | |
| *** med_ has joined #openstack-infra | 18:14 | |
| *** yjiang5 has left #openstack-infra | 18:14 | |
| *** armax has joined #openstack-infra | 18:14 | |
| *** loquacities has joined #openstack-infra | 18:14 | |
| *** ssam2 has quit IRC | 18:15 | |
| clarkb | huh I would expect the starred changes table to use the internal account id as a foreign key not anything openid related | 18:16 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/storyboard: Pass the old entity values in the NotificationHook https://review.openstack.org/151645 | 18:16 |
| *** achanda has joined #openstack-infra | 18:16 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Only start 1 subunit worker per node https://review.openstack.org/151725 | 18:16 |
| *** markus_z has quit IRC | 18:17 | |
| fungi | clarkb: nope, if you log into gerrit via openid, gerrit seems to first consult its table of external identities looking for a match for the openid url. if it finds none, it creates a new account and associates that openid url with it | 18:18 |
| *** jlibosva has quit IRC | 18:18 | |
| *** dhritishikhar has quit IRC | 18:18 | |
| *** _nadya_ has quit IRC | 18:18 | |
| *** ChuckC_ has quit IRC | 18:18 | |
| ccrouch | clarkb: hmm I *thought* I had some watched changes. /me can't be sure | 18:18 |
| *** ChuckC has joined #openstack-infra | 18:18 | |
| clarkb | ccrouch: you have two accounts \o/ :) | 18:19 |
| ccrouch | lol | 18:19 |
| fungi | ccrouch: i'll deactivate the new account and associate that new openid url with the old account if so. what's your new account id number shown at https://review.openstack.org/#/settings/ ? | 18:19 |
| * ccrouch bets fungi has a simple and elegant fix to the above situation | 18:19 | |
| clarkb | account id 9352 is the old one and 14911 is the new one | 18:19 |
| fungi | k | 18:19 |
| ccrouch | thanks fungi clarkb | 18:20 |
| clarkb | fungi: is that all doable through the command line yet? or does it still require manual db edits? | 18:20 |
| clarkb | set-account should be able to set external ids but maybe only email addrs? | 18:20 |
| *** rcarrillocruz has quit IRC | 18:21 | |
| clarkb | ya just emails ... boo | 18:21 |
| fungi | yep, looks like http://paste.openstack.org/show/164265/ right now | 18:21 |
| fungi | i'll give it a shot via the api | 18:21 |
| *** dkliban has joined #openstack-infra | 18:21 | |
| clarkb | fungi: do we have an etherpad yet? I can etherpad while you fix account stuff | 18:22 |
| *** rcarrillocruz has joined #openstack-infra | 18:22 | |
| fungi | set-account --help doesn't mention anything for changing arbitrary external ids, just e-mail | 18:22 |
| clarkb | fungi: ya I think set account is limited in its abilities | 18:23 |
| clarkb | fungi: but set account 14911 inactive may be sufficient in this case? | 18:23 |
| *** craigbr has quit IRC | 18:23 | |
| *** tchaypo has quit IRC | 18:23 | |
| *** crc32 has quit IRC | 18:23 | |
| *** tchaypo has joined #openstack-infra | 18:24 | |
| fungi | ccrouch: log out of gerrit and back in again | 18:24 |
| fungi | clarkb: nah, because it'll just create yet another account with that new openid url if we only deactivate | 18:24 |
| clarkb | fungi: but ~ccrouch in LP only has the one openid | 18:25 |
| fungi | yeah, the new one, which doesn't match what gerrit had for the old one | 18:25 |
| *** markvoelker has quit IRC | 18:25 | |
| fungi | i did `update account_external_ids set account_id=9352 where account_id=14911;` | 18:25 |
| clarkb | I see | 18:25 |
| fungi | and then `gerrit set-account --inactive 14911` via the api | 18:25 |
| clarkb | a couple of the rename changes need rebases so I am doing that now | 18:26 |
| clarkb | I have not started on the etherpad | 18:26 |
| fungi | thanks. i'm nearly done wit the etherpad. should be ready for review shortly | 18:26 |
| *** e0ne has joined #openstack-infra | 18:26 | |
| *** yjiang5 has joined #openstack-infra | 18:26 | |
| *** yjiang5 is now known as yginger5 | 18:27 | |
| ccrouch | fungi: clarkb: \o/ old account number is back at review.o.o and I can see my Starred Changes :-) | 18:27 |
| ccrouch | thank you very much guys | 18:27 |
| *** ddieterly has joined #openstack-infra | 18:27 | |
| clarkb | hrm maybe they were just caught in the zuul merger blip. rechecking now | 18:27 |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:27 | |
| yginger5 | Hi, when grenade.sh upgrade the services, will it recreate the config files? I checked the code and seems they will not create the config file, am I right? | 18:27 |
| jeblair | clarkb, fungi: i set their topics to 'project-rename' | 18:28 |
| asselin_ | clarkb, fungi I think this patch has issues: https://review.openstack.org/#/c/147984/ | 18:28 |
| fungi | yginger5: it uses the old config files. the new services have to be capable of working with the previous configs | 18:28 |
| *** dimtruck is now known as zz_dimtruck | 18:28 | |
| fungi | jeblair: thanks | 18:28 |
| *** weshay has joined #openstack-infra | 18:29 | |
| asselin_ | still need to confirm | 18:29 |
| ccrouch | https://review.openstack.org/#/settings/web-identities is overflowing with identities :-), but everything looks good | 18:29 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Moves the <service>-api repos to the attic https://review.openstack.org/145289 | 18:29 |
| clarkb | fungi: jeblair ^ that one did have aconflict | 18:29 |
| *** dhritishikhar has joined #openstack-infra | 18:30 | |
| openstackgerrit | Alexander Tivelkov proposed openstack/requirements: Add semantic_version to global-requirements https://review.openstack.org/151466 | 18:30 |
| clarkb | and after rechecking the other one it claims to still conflict so rebase will be pushed | 18:30 |
| yginger5 | fungi: thanks for answer, so if a new config item added, the default value has to make sure it will not break anything, right? It really make sense. | 18:30 |
| *** Ryan_Lane has joined #openstack-infra | 18:30 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: move oslo.version to attic https://review.openstack.org/141812 | 18:31 |
| *** koolhead17 has joined #openstack-infra | 18:31 | |
| fungi | yginger5: correct--that's one of the things grenade is there to confirm. operators need to be able to upgrade services in-place with the config files from the previous release, then update their configs later (before the next release) | 18:31 |
| *** jpeeler1 has quit IRC | 18:32 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Just store preseed subunit stream https://review.openstack.org/151732 | 18:32 |
| *** shivharis has quit IRC | 18:33 | |
| *** ZZelle_ has joined #openstack-infra | 18:33 | |
| yginger5 | fungi: thanks for your answer. | 18:34 |
| *** ivar-lazzaro has joined #openstack-infra | 18:34 | |
| *** Ryan_Lane has quit IRC | 18:35 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:35 | |
| asselin_ | clarkb, fung, so reverting that patch resolves this issue: http://paste.openstack.org/show/164268/ I'll see if there's a fix | 18:35 |
| openstackgerrit | Maru Newby proposed openstack-infra/zuul: Add support for a skip-if filter on jobs https://review.openstack.org/146762 | 18:36 |
| clarkb | asselin_: oh hey at least the error is useful, what does your nodepool.yaml look like? | 18:37 |
| *** shivharis has joined #openstack-infra | 18:37 | |
| clarkb | asselin_: I did test that it builds both snapshot and dib images so I wonder if maybe the nodepool.yamls differ in ways that break it? | 18:37 |
| *** KanagarajM2 has quit IRC | 18:37 | |
| asselin_ | clarkb, it's possible. top half: http://paste.openstack.org/show/164269/ | 18:38 |
| clarkb | asselin_: ya your label name has to match the image name | 18:39 |
| clarkb | iirc that is necessary due to dib | 18:39 |
| clarkb | or rather how the dib machinery works | 18:39 |
| clarkb | dib itself knows nothing about labels in nodepool | 18:39 |
| *** annegent_ has joined #openstack-infra | 18:39 | |
| *** jpeeler has joined #openstack-infra | 18:39 | |
| *** jpeeler has quit IRC | 18:39 | |
| *** jpeeler has joined #openstack-infra | 18:39 | |
| clarkb | would need to spend time to see if we can remove that requirement | 18:40 |
| asselin_ | clarkb, ok I see. I was taking advantage of the ability for them to be different. | 18:40 |
| jeblair | status notice Gerrit and Zuul will be offline from 1900 to 1930 UTC for project renames | 18:41 |
| jeblair | clarkb, fungi: ^ lgty? | 18:41 |
| clarkb | jeblair: yup | 18:41 |
| jeblair | #status notice Gerrit and Zuul will be offline from 1900 to 1930 UTC for project renames | 18:41 |
| openstackstatus | jeblair: sending notice | 18:41 |
| clarkb | asselin_: from memory, the reason its that way is label providers provide labels they do not provide images | 18:41 |
| clarkb | asselin_: so the image key is redundant? its defnitely fixable it just wasn't the existing behavior | 18:42 |
| fungi | jeblair: lgtm | 18:42 |
| *** DuncanT has quit IRC | 18:42 | |
| shivharis | fungi: hi, how do i look up the group member list of "networking-brocade-core" and "networking-brocade-release" | 18:42 |
| *** DuncanT has joined #openstack-infra | 18:42 | |
| -openstackstatus- NOTICE: Gerrit and Zuul will be offline from 1900 to 1930 UTC for project renames | 18:42 | |
| *** pblaho has quit IRC | 18:43 | |
| *** jedimike has quit IRC | 18:43 | |
| asselin_ | clarkb, ok, I had a need to shorten the names of the nodes, but maybe there's another way. | 18:43 |
| openstackstatus | jeblair: finished sending notice | 18:43 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-etherpad_lite: Add connection-tuning file to new Apache 2.4 conf folder structure https://review.openstack.org/151656 | 18:44 |
| *** achanda has quit IRC | 18:44 | |
| *** achanda has joined #openstack-infra | 18:45 | |
| *** jaypipes has quit IRC | 18:45 | |
| fungi | jeblair: clarkb: i think https://etherpad.openstack.org/p/repo-renaming-2015-01-30 is ready to go | 18:47 |
| asselin_ | clarkb, fyi, docs would need to be updated: http://ci.openstack.org/nodepool/configuration.html#labels | 18:47 |
| *** baoli has quit IRC | 18:47 | |
| clarkb | asselin_: no I think we fix it to do the expected behavior | 18:48 |
| jeblair | fungi, clarkb: i'll do step2 | 18:48 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Fix invalid spacing in yaml file https://review.openstack.org/151735 | 18:48 |
| zaro | fungi ^ | 18:48 |
| fungi | shivharis: https://review.openstack.org/#/admin/groups/ (browsing people -> list groups in the top navigation bar of the webui gets you there) | 18:48 |
| *** craigbr has joined #openstack-infra | 18:49 | |
| shivharis | fungi: thanks | 18:49 |
| clarkb | asselin_: we will just need to untangle places where label.name is used instead of label.image and swap them out properly | 18:49 |
| *** jaypipes has joined #openstack-infra | 18:49 | |
| *** craigbr has quit IRC | 18:50 | |
| *** jaypipes is now known as leakypipes | 18:50 | |
| fungi | i've deleted the -2 or wip votes from https://review.openstack.org/#/q/status:open+topic:project-rename,n,z in preparation for the maintenance | 18:50 |
| *** mrmartin has quit IRC | 18:51 | |
| fungi | i'll go ahead and grab the gerrit cli steps | 18:51 |
| *** Sukhdev has quit IRC | 18:52 | |
| clarkb | asselin_: I can look into it mroe deeply once we rename things. This may actually be simpler than I thought | 18:52 |
| asselin_ | clarkb, ok. I can help test then :) | 18:53 |
| clarkb | asselin_: as we do seem to use label.image during disk image building so not sure why I thought that wasn't the case | 18:53 |
| *** sarob has quit IRC | 18:54 | |
| clarkb | I can do the github and mirror renames | 18:55 |
| *** elliottk has left #openstack-infra | 18:55 | |
| fungi | try your hand at the transfer ownership feature in github? | 18:55 |
| fungi | worked well for me last time around | 18:55 |
| clarkb | ya | 18:55 |
| *** baoli has joined #openstack-infra | 18:56 | |
| jeblair | fungi, clarkb: still waiting on puppet/ansible; will keep you updated | 18:56 |
| fungi | clarkb: might want to make sure to do that before manage-projects runs via puppet | 18:56 |
| fungi | otherwise we'll end up with new repos in the destination org | 18:56 |
| clarkb | fungi: yup | 18:56 |
| clarkb | asselin_: oh I remember, I think it is because image-update completely ignored the concept of a label which is where we map clouds to images | 18:58 |
| *** Ryan_Lane has joined #openstack-infra | 18:58 | |
| clarkb | asselin_: so we need to get the label and do that mapping properly but I seem to have done it wrongly | 18:58 |
| clarkb | asselin_: the old code just shotgunned images which wasn't quite right either. Maybe new change will be better than both | 18:58 |
| asselin_ | clarkb, +1 | 18:59 |
| *** mriedem has quit IRC | 18:59 | |
| clarkb | asselin_: like image-update would allow you to build an image on a cloud that it didn't have a label for | 18:59 |
| clarkb | asselin_: which should be an error | 18:59 |
| *** mtanino has joined #openstack-infra | 19:00 | |
| clarkb | because there is no way that nodepool would boot slaves off of that image so its just a waste of effort | 19:00 |
| jeblair | oh how interesting... | 19:00 |
| jeblair | a new ansible run has started even though the old one was still going | 19:01 |
| jeblair | i didn't think that was supposed to happen | 19:01 |
| clarkb | there should be a lock, however do we run two different ansible "jobs" now? | 19:01 |
| clarkb | maybe that is how it happened | 19:01 |
| fungi | huh. with flock around them? or were we just relying on puppet agent locks to prevent concurrent runs on a specific node? | 19:01 |
| asselin_ | clarkb, it seems you should be able to map a label to multiple images | 19:01 |
| clarkb | asselin_: possibly. that would be a new feature though | 19:02 |
| asselin_ | clarkb, and when bootstrapping, I would like to do a manual image-update, even though it's not on a label | 19:02 |
| clarkb | asselin_: like instead of a single image key allow for each provider in the provider list to specify a distinct image | 19:02 |
| asselin_ | clarkb, yes that would be new, but a bit implied by the way it's configured | 19:02 |
| clarkb | asselin_: so I really think that should be an error, because there is no way for nodepool to use that image if there is no label | 19:02 |
| clarkb | it will just make an image that is never booted off of | 19:03 |
| *** teran has joined #openstack-infra | 19:03 | |
| * jeblair is killing ansible processes | 19:03 | |
| fungi | with a vengeance? | 19:03 |
| clarkb | *shuts down puppetmaster | 19:04 |
| asselin_ | clarkb, yes, but the intent is to 'temporarity' not boot off of. | 19:04 |
| *** shardy is now known as shardy_afk | 19:04 | |
| clarkb | asselin_: in that case just add a label... | 19:04 |
| asselin_ | clarkb, ok I see, that can work | 19:04 |
| jeblair | clarkb, fungi: okay, ansible is stopped. i'm ready to proceed. shall i stop zuul now? | 19:05 |
| asselin_ | clarkb, that's actually a really good idea. it also solve the next issue I have: creating the image & testing it. thanks! | 19:05 |
| asselin_ | s/image/instance/ | 19:05 |
| jeblair | #status alert Gerrit and Zuul are offline until 1930 UTC for project renames | 19:06 |
| openstackstatus | jeblair: sending alert | 19:06 |
| *** adalbas has quit IRC | 19:06 | |
| fungi | yep | 19:06 |
| fungi | i'm ready to kill gerrit as soon as you're ready | 19:06 |
| jeblair | fungi: zuul is stopped | 19:06 |
| fungi | stopping gerrit now | 19:06 |
| ekarlso | what projects are renaming ? | 19:06 |
| jeblair | (i'm going to do nodepool cleanup now) | 19:06 |
| clarkb | fungi: jeblair: I am going to start renaming things on the mirrors | 19:06 |
| clarkb | then do github | 19:07 |
| zaro | fungi: so what should we do about the openstack_functions.py file? | 19:07 |
| fungi | ekarlso: http://lists.openstack.org/pipermail/openstack-dev/2015-January/055612.html | 19:07 |
| mrodden | ever thought about an ansible playbook to do the renaming? | 19:07 |
| zaro | is it even needed by zuul-dev? | 19:07 |
| zaro | should we keep it in the zuul-dev layout.yaml? | 19:07 |
| -openstackstatus- NOTICE: Gerrit and Zuul are offline until 1930 UTC for project renames | 19:07 | |
| *** baoli has quit IRC | 19:07 | |
| *** ChanServ changes topic to "Gerrit and Zuul are offline until 1930 UTC for project renames" | 19:07 | |
| clarkb | mrodden: yes, my only concern with it is I would still have to github because their api never works | 19:07 |
| fungi | gerrit is now offline. proceeding with database edits | 19:07 |
| mrodden | clarkb, yeah that would be an issue | 19:08 |
| clarkb | mrodden: but renaming on things we do control would be a simplish playbook if we can feed in a mapping | 19:08 |
| *** jistr has quit IRC | 19:08 | |
| clarkb | I just don't know how ansible expects that info | 19:08 |
| *** baoli has joined #openstack-infra | 19:08 | |
| * clarkb <- ansible noob | 19:08 | |
| *** pblaho has joined #openstack-infra | 19:08 | |
| mrodden | clarkb, yeah i just started with it, but they have like a ridiculous amount of ways to feed in variable data | 19:09 |
| *** pblaho has quit IRC | 19:09 | |
| fungi | gerrit database edits completed without error, proceeding with gerrit filesystem changes | 19:10 |
| openstackstatus | jeblair: finished sending alert | 19:10 |
| clarkb | now to do github this will be new and exciting | 19:10 |
| *** ashleighfarnham has quit IRC | 19:11 | |
| fungi | gerrit filesystem changes completed without incident, lucene search reindex running now | 19:12 |
| *** bknudson has quit IRC | 19:14 | |
| *** ZZelle has quit IRC | 19:14 | |
| *** ashleighfarnham has joined #openstack-infra | 19:15 | |
| *** Sukhdev has joined #openstack-infra | 19:15 | |
| *** salv-orlando has quit IRC | 19:16 | |
| fungi | zaro: we can either adjust the manifest for zuul-dev so that the file gets installed, or adjust the layout.yaml to not include it. depends on whether we feel it necessary to test functions in it (maybe keep it around so we can still set the offline node option for ephemeral nodes we might want to test on jenkins-dev?) | 19:16 |
| clarkb | ok github is done | 19:16 |
| *** salv-orlando has joined #openstack-infra | 19:16 | |
| *** mriedem has joined #openstack-infra | 19:16 | |
| fungi | lucene reindex has reached ~50% | 19:17 |
| *** tqtran_afk has joined #openstack-infra | 19:17 | |
| *** armax has quit IRC | 19:17 | |
| *** tqtran_afk is now known as tqtran | 19:18 | |
| *** ashleighfarnham has quit IRC | 19:18 | |
| *** mtanino_ has joined #openstack-infra | 19:19 | |
| zaro | fungi: ok. i'll make an attempt to adjust manifest to include it so it can be tested. | 19:19 |
| *** craigbr has joined #openstack-infra | 19:20 | |
| *** mtanino has quit IRC | 19:20 | |
| *** tqtran is now known as tqtran_afk | 19:20 | |
| fungi | lucene reindex just completed | 19:21 |
| *** marun has quit IRC | 19:21 | |
| fungi | everyone ready for me to start gerrit again? | 19:21 |
| jeblair | fungi: yep | 19:21 |
| fungi | clarkb? | 19:21 |
| zaro | rcarrillocruz: ping | 19:22 |
| clarkb | fungi: yes gitXX and github things have been renamed | 19:22 |
| *** annegent_ has quit IRC | 19:22 | |
| fungi | here goes | 19:22 |
| fungi | seems to be up and working | 19:22 |
| *** annegent_ has joined #openstack-infra | 19:22 | |
| jeblair | i'll start zuul now | 19:23 |
| jeblair | re-enqueueing | 19:24 |
| *** zul has quit IRC | 19:24 | |
| jeblair | done | 19:25 |
| *** carl_baldwin has quit IRC | 19:25 | |
| jeblair | should be okay to merge the project-config changes now | 19:25 |
| fungi | for some reason gerrit's not giving me the option to select verified votes on 141812 but does on the other two | 19:25 |
| fungi | gonna try flushing caches real quick | 19:25 |
| fungi | which of course makes gerrit very unresponsive for a minute :/ | 19:26 |
| clarkb | fungi: if you have previous votes it doesn't let you add categories | 19:26 |
| clarkb | fungi: so the fix is vote 0 | 19:26 |
| clarkb | then vote +2 all the things | 19:26 |
| clarkb | because gerrit | 19:26 |
| fungi | thanks clarkb--that did it | 19:27 |
| *** pelix has quit IRC | 19:27 | |
| fungi | okay, approving them now | 19:27 |
| *** zul has joined #openstack-infra | 19:27 | |
| jeblair | #status ok Gerrit is back online | 19:28 |
| openstackstatus | jeblair: sending ok | 19:28 |
| fungi | dawned on me i didn't need to set verified on these anyway | 19:28 |
| fungi | so just did normal +2 and approve | 19:28 |
| *** cdent has quit IRC | 19:29 | |
| zaro | can someone tell me where things like '$::project_config::gerrit_acls_dir' in system-config repo get defined? i thought it might be in project-config repo but can't find it there either. | 19:29 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/manual/ http://ci.openstack.org/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/" | 19:30 | |
| clarkb | zaro: its part of the project_config module which now lives at openstack-infra/puppet-project_config | 19:30 |
| -openstackstatus- NOTICE: Gerrit is back online | 19:30 | |
| zaro | clarkb: ahh, thanks! | 19:30 |
| *** marun has joined #openstack-infra | 19:30 | |
| *** vhoward has left #openstack-infra | 19:30 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 19:31 | |
| openstackstatus | jeblair: finished sending ok | 19:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Update gerrit rename instructions https://review.openstack.org/151742 | 19:32 |
| jeblair | wow, so many cloud errors | 19:37 |
| *** armax has joined #openstack-infra | 19:37 | |
| *** salv-orlando has quit IRC | 19:37 | |
| *** salv-orlando has joined #openstack-infra | 19:37 | |
| *** pradk has quit IRC | 19:38 | |
| fungi | cloud errors? | 19:39 |
| openstackgerrit | Merged openstack-infra/project-config: stackforge cookbook-ceph is being removed https://review.openstack.org/147250 | 19:40 |
| jeblair | fungi: nodepool -- rax-dfw nodes are going straight into error; novaclient 500s; timeouts waiting for servers in hpcloud to go active | 19:40 |
| jeblair | that's actually probably why our 'delete' ratio seems so high -- nodes going straing from build to delete :( | 19:40 |
| fungi | fun! | 19:41 |
| jeblair | clarkb, fungi: i cleaned up the alien list | 19:42 |
| fungi | awesome--i was just about to do that. now i can do something else ;) | 19:42 |
| clarkb | :/ | 19:44 |
| clarkb | jeblair: fungi: could we be leaking floating ips again? though timeouts probably aren't leaked floating IPs | 19:44 |
| *** spzala has joined #openstack-infra | 19:45 | |
| fungi | if it's hpcloud maybe | 19:45 |
| jroll | jeblair: what sort of errors on rax? | 19:45 |
| jeblair | jroll: nova error state | 19:45 |
| jroll | jeblair: any details, though? | 19:45 |
| *** vhoward has joined #openstack-infra | 19:45 | |
| jeblair | jroll: probably not at this point -- we should have nodepool query nova and log info there :/ | 19:46 |
| fungi | i'm going to start going through and marking system-config changes wip or abandoning depending on whether they're split between openstack_infra and other modules or don't modify openstack_infra and the remaining files in that repo at all | 19:46 |
| openstackgerrit | Merged openstack-infra/project-config: Moves the <service>-api repos to the attic https://review.openstack.org/145289 | 19:46 |
| openstackgerrit | Merged openstack-infra/project-config: move oslo.version to attic https://review.openstack.org/141812 | 19:46 |
| jroll | jeblair: ok, poking around to see if there's known things | 19:47 |
| jeblair | jroll: (because it immediately starts deleting servers in that case) | 19:47 |
| *** EmilienM is now known as EmilienM|afk | 19:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Better image checking in update_image command https://review.openstack.org/151749 | 19:48 |
| jeblair | fungi: all 3 changes are merged -- should i re-enable ansible cron? | 19:48 |
| *** dimsum__ has quit IRC | 19:48 | |
| clarkb | asselin_: ^ that is what I have whipped up after a few minutes of tackling the problem | 19:48 |
| fungi | jeblair: yep, go for it | 19:48 |
| jeblair | done | 19:48 |
| * asselin_ looking | 19:48 | |
| jeblair | i think that's everything then | 19:48 |
| clarkb | asselin_: I think it should fix the case you ran into and another case where the old code wasn't quite right | 19:48 |
| clarkb | jeblair: woot | 19:49 |
| fungi | jeblair: thanks--i'll check back in on the acls in a bit and make sure they're appropriately updated | 19:49 |
| clarkb | jeblair: do we need to debug the locking around ansible runs? | 19:49 |
| *** spzala has quit IRC | 19:49 | |
| jeblair | clarkb: looking at /opt/system-config/production/run_all.sh, i don't see any locking | 19:51 |
| jeblair | i have to run, back in a bit | 19:51 |
| *** hdd has quit IRC | 19:51 | |
| clarkb | I think the cron it self has the lock. I will check | 19:51 |
| asselin_ | clarkb, +1 | 19:52 |
| *** vigneshvar has quit IRC | 19:52 | |
| clarkb | jeblair: fungi hrm doesn't look like there is locking. I will write a patch for that | 19:53 |
| *** thedodd has joined #openstack-infra | 19:53 | |
| *** shivharis has quit IRC | 19:54 | |
| *** ddieterly has quit IRC | 19:54 | |
| fungi | k, awesome | 19:55 |
| clarkb | asselin_: so you ran it and it fixed your error? | 19:55 |
| asselin_ | clarkb, yes it did! | 19:55 |
| *** ddieterly has joined #openstack-infra | 19:55 | |
| asselin_ | clarkb, thank you :) | 19:56 |
| jroll | jeblair: still seeing rax issues? | 19:56 |
| asselin_ | clarkb, installed it here: http://paste.openstack.org/show/164309/ and ran the image-update as before as nodepool user | 19:58 |
| *** rfolco has quit IRC | 19:58 | |
| *** jedimike has joined #openstack-infra | 19:59 | |
| jroll | jeblair: at any rate, not seeing issues on our end | 20:01 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Use flock around puppet_run_all script https://review.openstack.org/151751 | 20:02 |
| clarkb | fungi: jeblair that should do it | 20:03 |
| clarkb | jroll: I can look in a secnd | 20:03 |
| asselin_ | clarkb, the error message could be improved however in the case that there is no label-image mapping: http://paste.openstack.org/show/164310/ | 20:03 |
| jroll | clarkb: ok, tenant id would be helpful too if you are seeing problems | 20:03 |
| clarkb | asselin_: hrm I thought I covered all errors with explicit errors | 20:04 |
| clarkb | asselin_: so that failure happened because no label has image dpcfc? | 20:04 |
| *** salv-orlando has quit IRC | 20:05 | |
| *** SumitNaiksatam has quit IRC | 20:05 | |
| clarkb | asselin_: line 185 of my change is what guards against that then line 225 should report that error | 20:05 |
| clarkb | asselin_: oh I see the bug derp | 20:05 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Better image checking in update_image command https://review.openstack.org/151749 | 20:06 |
| clarkb | asselin_: ^ I used the wrong variable name in one location | 20:06 |
| *** openstackgerrit has quit IRC | 20:06 | |
| *** openstackgerrit has joined #openstack-infra | 20:07 | |
| *** carl_baldwin has joined #openstack-infra | 20:11 | |
| *** spzala has joined #openstack-infra | 20:11 | |
| *** ldnunes has quit IRC | 20:15 | |
| *** jedimike has quit IRC | 20:17 | |
| *** e0ne has quit IRC | 20:20 | |
| pleia2 | thanks guys, I think now I should drink more beer at fosdem :) | 20:20 |
| clarkb | pleia2: yes! | 20:21 |
| *** teran_ has joined #openstack-infra | 20:21 | |
| *** teran has quit IRC | 20:23 | |
| *** teran has joined #openstack-infra | 20:23 | |
| asselin_ | clarkb, looking now | 20:25 |
| *** david-lyle has quit IRC | 20:25 | |
| *** teran_ has quit IRC | 20:25 | |
| *** shivharis has joined #openstack-infra | 20:25 | |
| *** ddieterl_ has joined #openstack-infra | 20:26 | |
| *** ddieterly has quit IRC | 20:26 | |
| *** markmcclain has quit IRC | 20:26 | |
| fungi | i checked the 8 renamed projects in gerrit and all of them have the new read-only acls applied correctly, so manage-projects seems to have been successful once puppet was restored to order | 20:26 |
| fungi | and i've skimmed all outstanding system-config changes, either abandoning (if they only applied to other puppet modules than openstack_project) or marked wip (if they applied to openstack_project and one or more other modules). i'll follow up to the mailing lists letting everyone know to check their abandoned changes | 20:28 |
| *** markmcclain has joined #openstack-infra | 20:28 | |
| *** harlowja is now known as harlowja_away | 20:29 | |
| *** tqtran_afk is now known as tqtran | 20:29 | |
| *** hdd has joined #openstack-infra | 20:30 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds a Sphinx theme for use for docs.openstack.org https://review.openstack.org/151766 | 20:32 |
| *** harlowja_away is now known as harlowja | 20:34 | |
| *** alexpilotti has joined #openstack-infra | 20:34 | |
| *** alexpilotti has quit IRC | 20:34 | |
| Mithrandir | pleia2: ooh, at fosdem? Slightly envious. | 20:36 |
| *** Ryan_Lane has quit IRC | 20:36 | |
| *** avozza is now known as zz_avozza | 20:36 | |
| fungi | looks like we have 6 more module-split topic reviews waiting for a second +2 and approval. the end is in sight! | 20:37 |
| *** crc32 has joined #openstack-infra | 20:38 | |
| *** dimsum__ has joined #openstack-infra | 20:41 | |
| *** carl_baldwin has quit IRC | 20:43 | |
| asselin_ | clarkb, -1, error message shows but is misleading. see my comment on the patch https://review.openstack.org/#/c/151749/ | 20:43 |
| *** pradk has joined #openstack-infra | 20:44 | |
| *** aysyd has joined #openstack-infra | 20:44 | |
| *** zz_dimtruck is now known as dimtruck | 20:46 | |
| *** markmcclain has quit IRC | 20:46 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds a Sphinx theme for use for docs.openstack.org https://review.openstack.org/151766 | 20:47 |
| *** bknudson has joined #openstack-infra | 20:47 | |
| *** koolhead17 has quit IRC | 20:48 | |
| *** jedimike has joined #openstack-infra | 20:48 | |
| fungi | clarkb: jhesketh: this change looks like it had a couple of log upload failures today https://review.openstack.org/151322 | 20:48 |
| *** aysyd has quit IRC | 20:49 | |
| *** markmcclain has joined #openstack-infra | 20:49 | |
| *** hdd has quit IRC | 20:51 | |
| *** radez is now known as radez_g0n3 | 20:51 | |
| *** _nadya_ has joined #openstack-infra | 20:51 | |
| clarkb | asselin_ thanks will fix | 20:52 |
| openstackgerrit | Merged openstack-infra/system-config: Fix module order https://review.openstack.org/151103 | 20:53 |
| clarkb | asselin_ not sure how it gets there though since the first loop should only add providers with labels that have that image | 20:53 |
| openstackgerrit | Merged openstack-infra/puppet-ulimit: Add puppet module files for puppet-ulimit https://review.openstack.org/151514 | 20:54 |
| asselin_ | clarkb, yea, I'm looking at that too...after re-reading your previous comment 12:05 | 20:54 |
| openstackgerrit | Merged openstack-infra/puppet-unbound: Add puppet module files for unbound https://review.openstack.org/151516 | 20:54 |
| openstackgerrit | Merged openstack-infra/puppet-user: Add puppet module files for puppet-user https://review.openstack.org/151517 | 20:54 |
| clarkb | oh i see because if you specify the specific provider then it goes there | 20:54 |
| openstackgerrit | Merged openstack-infra/puppet-unattended_upgrades: Add puppet module files for unattended_upgrades https://review.openstack.org/151519 | 20:54 |
| clarkb | I can fix that | 20:55 |
| *** hj-hp has joined #openstack-infra | 20:55 | |
| *** MaxV has joined #openstack-infra | 20:55 | |
| *** ashleighfarnham has joined #openstack-infra | 20:55 | |
| *** abhirc has quit IRC | 20:56 | |
| *** MaxV has quit IRC | 20:57 | |
| *** zz_avozza is now known as avozza | 20:57 | |
| *** craigbr has quit IRC | 20:58 | |
| *** shardy_afk has quit IRC | 20:59 | |
| *** e0ne has joined #openstack-infra | 21:00 | |
| *** hdd has joined #openstack-infra | 21:00 | |
| *** andreaf has quit IRC | 21:02 | |
| *** Ryan_Lane has joined #openstack-infra | 21:03 | |
| *** shivharis has quit IRC | 21:06 | |
| *** dhritishikhar has quit IRC | 21:06 | |
| *** abhirc has joined #openstack-infra | 21:08 | |
| *** david-lyle has joined #openstack-infra | 21:08 | |
| *** yamamoto has joined #openstack-infra | 21:09 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:13 | |
| *** yamamoto has quit IRC | 21:13 | |
| *** dimsum__ has quit IRC | 21:15 | |
| *** Sukhdev has quit IRC | 21:15 | |
| *** tiswanso has quit IRC | 21:17 | |
| *** shivharis has joined #openstack-infra | 21:17 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 21:18 | |
| *** dprince has quit IRC | 21:18 | |
| *** leakypipes has quit IRC | 21:20 | |
| *** mjturek has quit IRC | 21:21 | |
| *** _nadya_ has quit IRC | 21:21 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Better image checking in update_image command https://review.openstack.org/151749 | 21:25 |
| clarkb | asselin_: ^ how is that? | 21:25 |
| *** mrmartin has joined #openstack-infra | 21:25 | |
| * asselin_ looking | 21:26 | |
| *** erlon has quit IRC | 21:26 | |
| *** e0ne has quit IRC | 21:27 | |
| *** Duarte-DVR has joined #openstack-infra | 21:28 | |
| asselin_ | clarkb, looks good: Exception: No label with image dpc is provided by provider p219 | 21:28 |
| asselin_ | clarkb, will check the other cases now make sure they still work :) | 21:28 |
| *** annegent_ has quit IRC | 21:28 | |
| clarkb | asselin_: cool -1 away if necessary :) | 21:29 |
| *** unicell has quit IRC | 21:29 | |
| *** unicell has joined #openstack-infra | 21:30 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-etherpad_lite: Add missing LICENSE file https://review.openstack.org/151782 | 21:31 |
| fungi | that ^ one got missed | 21:31 |
| *** ddieterl_ has quit IRC | 21:31 | |
| fungi | i probably raced with a boilerplate approval on it and so didn't pick it up until my clean-up pass just now | 21:31 |
| clarkb | fungi: should we rm the py33 stuff now? | 21:32 |
| clarkb | fungi: eg have we reached enough consensus on the thread to do that? | 21:32 |
| fungi | clarkb: if you're ready | 21:32 |
| *** annegent_ has joined #openstack-infra | 21:32 | |
| fungi | clarkb: yeah, dhellmann confirmed | 21:32 |
| fungi | i'll send an announcement to the ml later this evening once it switches over | 21:33 |
| clarkb | fungi: did you see AJaeger's -1 at https://review.openstack.org/#/c/151713/ ? | 21:34 |
| clarkb | I approved the first patch in the chain though | 21:34 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-gerrit: Update Gerrit Puppet module for Trusty https://review.openstack.org/151783 | 21:34 |
| *** radez_g0n3 is now known as radez | 21:34 | |
| clarkb | and the -1 may explain why the test failed too | 21:34 |
| fungi | unfortunately the log is truncated | 21:35 |
| clarkb | fungi: ya but I am pretty sure that the thing ajaeger points out is why it failed | 21:35 |
| clarkb | I have just approved https://review.openstack.org/#/c/151475/ and can rebuild images to see if that makes jenkins happier once that merges | 21:36 |
| *** david-lyle_ has joined #openstack-infra | 21:37 | |
| *** david-lyle has quit IRC | 21:37 | |
| fungi | clarkb: i'm not sure what AJaeger suggests i'm missing in jenkins/jobs/python-jobs.yaml | 21:38 |
| *** annegen__ has joined #openstack-infra | 21:38 | |
| *** annegen__ has quit IRC | 21:39 | |
| *** achanda has quit IRC | 21:39 | |
| *** annegent_ has quit IRC | 21:39 | |
| fungi | though i do see that i missed deleting the template from the python-jobs group in the later wip change 151715 | 21:39 |
| *** mtanino_ has quit IRC | 21:39 | |
| *** achanda has joined #openstack-infra | 21:39 | |
| *** annegent_ has joined #openstack-infra | 21:39 | |
| clarkb | fungi: I think the jjb job group for python jobs must do 33 and not 34 | 21:40 |
| clarkb | so the layout check breaks when looking for 34 jos | 21:40 |
| fungi | clarkb: it has both in it already | 21:41 |
| clarkb | oh nevermind then | 21:41 |
| * clarkb runs the tox command locally to see the error | 21:41 | |
| fungi | i have it running right now | 21:41 |
| fungi | should know something momentarily | 21:41 |
| openstackgerrit | Merged openstack-infra/project-config: Suspend Py3K jobs for oslo.messaging oslo.rootwrap https://review.openstack.org/151712 | 21:42 |
| fungi | Job gate-ironic-discoverd-python34 not defined | 21:42 |
| fungi | there we go | 21:42 |
| fungi | fixing | 21:42 |
| *** amitgandhinz has quit IRC | 21:43 | |
| *** achanda has quit IRC | 21:43 | |
| clarkb | what is an ironic-discoverd? | 21:44 |
| fungi | a stackforge thing | 21:44 |
| *** mtanino has joined #openstack-infra | 21:44 | |
| clarkb | I can't help but think about ironic discoveries let us see what google has to say about that | 21:44 |
| *** mrmartin has quit IRC | 21:44 | |
| clarkb | wow ironic-discoverd comes up when I search ironic discovery gj | 21:45 |
| *** ashleighfarnham has quit IRC | 21:45 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Switch from Python 3.3 to 3.4 for Py3K jobs https://review.openstack.org/151713 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Stop using py3k-precise nodes https://review.openstack.org/151715 | 21:45 |
| fungi | anyway, fixed those couple changes ^ | 21:45 |
| openstackgerrit | Merged openstack-infra/project-config: Do not use cache while fetching console logs https://review.openstack.org/151475 | 21:46 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Add openstack_functions.py to zuul-dev.o.o https://review.openstack.org/151790 | 21:46 |
| *** achanda has joined #openstack-infra | 21:48 | |
| *** Kerralissa has joined #openstack-infra | 21:49 | |
| *** ddieterly has joined #openstack-infra | 21:49 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Declare a non-overlapping dynamic UID/GID range https://review.openstack.org/147643 | 21:49 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Run review-dev on Ubuntu Trusty https://review.openstack.org/147655 | 21:49 |
| clarkb | sweet rebuilding bare-trusty nodes now | 21:49 |
| *** achanda has quit IRC | 21:49 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Add openstack/oslo.versionedobjects https://review.openstack.org/151792 | 21:51 |
| *** david-lyle_ has quit IRC | 21:51 | |
| *** jgrimm is now known as zz_jgrimm | 21:52 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Run review-dev on Ubuntu Trusty https://review.openstack.org/147655 | 21:53 |
| *** amitgandhinz has joined #openstack-infra | 21:56 | |
| zaro | fungi: you know what happened here? https://review.openstack.org/#/c/151735/ | 21:58 |
| *** packet has quit IRC | 21:58 | |
| fungi | zaro: i'm wondering if we have a problem with the swift upload script on centos6 workers | 21:59 |
| fungi | clarkb: jhesketh: ^ | 21:59 |
| clarkb | mtreinish: still around? what do I need to do to skip the shelve test? | 21:59 |
| clarkb | mtreinish: I bet there are docs I should read huh? | 21:59 |
| mtreinish | clarkb: hmm, I'm not sure we have docs for that one | 22:00 |
| mtreinish | but basically just push a patch to tempest with a skip_because decorator on the test | 22:00 |
| clarkb | ok I can grep for examples | 22:00 |
| mtreinish | I can find an example one sec | 22:00 |
| *** achanda has joined #openstack-infra | 22:00 | |
| *** doug-fish has left #openstack-infra | 22:01 | |
| mtreinish | clarkb: oh look at that there is a doc on it :) http://docs.openstack.org/developer/tempest/HACKING.html#test-skips-because-of-known-bugs | 22:01 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-httpd: Rename apache2 to httpd globally https://review.openstack.org/151098 | 22:02 |
| *** craigbr has joined #openstack-infra | 22:02 | |
| nibalizer | afternoon y'all | 22:02 |
| nibalizer | awesome to see pleia2 getting nominated for core | 22:02 |
| *** ashleighfarnham has joined #openstack-infra | 22:03 | |
| *** SumitNaiksatam has joined #openstack-infra | 22:03 | |
| *** abhirc has quit IRC | 22:03 | |
| fungi | okay, heading out to grab some dinner, then i'll be back to send the couple of announcements i owe the ml | 22:03 |
| *** carl_baldwin has joined #openstack-infra | 22:04 | |
| *** craigbr has quit IRC | 22:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add a test for enqueuing complex dependencies https://review.openstack.org/151705 | 22:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Support cross-repo-dependencies in dependent pipelines https://review.openstack.org/144555 | 22:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Support cross-repo-dependencies in independent pipelines https://review.openstack.org/144556 | 22:05 |
| zaro | nibalizer: afternoon | 22:05 |
| *** craigbr has joined #openstack-infra | 22:05 | |
| zaro | nibalizer: hey how do you check that puppet is even running? is there a status command? | 22:06 |
| zaro | nibalizer: i meant the agent | 22:06 |
| jeblair | zaro: we don't run the agent, we run puppet apply from the puppetmaster | 22:07 |
| *** miqui_ is now known as miqui_away | 22:07 | |
| jeblair | zaro: you can check for an individual host on http://puppetboard.openstack.org/ | 22:07 |
| jeblair | zaro: review-dev, storyboard-dev, and zuul-dev are all having problems right now | 22:08 |
| jeblair | where problems == not able to run puppet | 22:08 |
| clarkb | fungi: we should ping ryanpetrello about running a pecan scaffolds 34 job | 22:08 |
| clarkb | ryanpetrello: ^ you have been pung | 22:08 |
| ryanpetrello | o/ | 22:09 |
| jeblair | that's some good pingin | 22:09 |
| clarkb | ryanpetrello: see https://review.openstack.org/#/c/151713/2/zuul/layout.yaml | 22:09 |
| *** abhirc has joined #openstack-infra | 22:09 | |
| clarkb | ryanpetrello: we are trying to move away from py33 since the test nodes for that are a bit crazy. But you get 34 now which is more awesomer :) | 22:09 |
| clarkb | ryanpetrello: just pointing out that we should switch your pecan job there to 34 | 22:09 |
| *** crc32 has quit IRC | 22:09 | |
| *** hj-hp has quit IRC | 22:10 | |
| ryanpetrello | fungi: you mind just changing that s/3/4 for me? | 22:10 |
| *** achanda has quit IRC | 22:10 | |
| ryanpetrello | there already exists a scaffolds-34 in pecan's tox | 22:11 |
| *** mattfarina has quit IRC | 22:11 | |
| jeblair | zaro: there were mismatched ssh keys because of host replacement. i have fixed it for review-dev and zuul-dev | 22:12 |
| jeblair | zaro: puppet should run on those hosts at 22:20 utc | 22:12 |
| *** achanda has joined #openstack-infra | 22:12 | |
| jeblair | zaro: or rather, between 22:20 and 22:30 | 22:12 |
| *** stevemar has quit IRC | 22:16 | |
| jogo | clarkb: thanks for https://review.openstack.org/#/c/151798/1 | 22:16 |
| clarkb | jogo: ya I tested the proposed nova fix and it idn't help so moving onward! | 22:17 |
| *** thedodd has quit IRC | 22:17 | |
| *** morganfainberg is now known as NotLurkingHere | 22:17 | |
| *** NotLurkingHere is now known as morganfainberg | 22:19 | |
| jogo | clarkb: yeah I saw | 22:19 |
| *** crc32 has joined #openstack-infra | 22:20 | |
| *** armax has quit IRC | 22:21 | |
| mtreinish | clarkb: I take it the centos 6 apply job is fubar? Or should I jsut recheck my patches | 22:23 |
| clarkb | we should have new bare-trusty images shortly across all providers | 22:23 |
| clarkb | mtreinish: in some cases yes because git | 22:23 |
| clarkb | mtreinish: did it timeout? | 22:23 |
| zaro | clarkb: no log file | 22:24 |
| mtreinish | clarkb: err yeah I can't check because I get a 404 on the log link | 22:24 |
| *** abhirc has quit IRC | 22:25 | |
| zaro | mtreinish: i've already recheck, same thing again. | 22:25 |
| clarkb | hrm you know hpcloud centos6 images may be old enough that they don't have the fix for the virtualenv stuff | 22:25 |
| clarkb | let me try rebuilding those. Likely an image was renamed | 22:25 |
| jeblair | mtreinish: link to change or 404 log link? | 22:26 |
| *** craigbr has quit IRC | 22:26 | |
| *** carl_baldwin has quit IRC | 22:26 | |
| mtreinish | jeblair: https://review.openstack.org/#/c/151725/ and https://review.openstack.org/#/c/151732/ | 22:26 |
| mtreinish | jeblair: the 404's are http://logs.openstack.org/32/151732/1/check/gate-infra-puppet-apply-centos6/6110001/ and http://logs.openstack.org/25/151725/1/check/gate-infra-puppet-apply-centos6/8290f49/ | 22:26 |
| *** dmsimard is now known as dmsimard_away | 22:27 | |
| *** ddieterly has quit IRC | 22:27 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Just store preseed subunit stream https://review.openstack.org/151732 | 22:27 |
| clarkb | ya I think that when we are under more node contention hpcloud gets used for those images and they are out of date | 22:28 |
| clarkb | and the reason they are out of date is epel | 22:28 |
| *** armax has joined #openstack-infra | 22:28 | |
| clarkb | ianw: are you weekending? | 22:28 |
| jeblair | https://jenkins06.openstack.org/job/gate-infra-puppet-apply-centos6/884/consoleFull | 22:29 |
| jeblair | mtreinish, clarkb: there's one of the failures | 22:29 |
| jeblair | 2015-01-30 20:28:05.706 | /tmp/hudson5916613440066750934.sh: line 2: /usr/zuul-swift-logs-env/bin/python: No such file or directory | 22:29 |
| clarkb | ya the venv is missing | 22:29 |
| clarkb | we need new images but Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again | 22:29 |
| jeblair | why would that only affect hpcloud? | 22:29 |
| *** Sukhdev has joined #openstack-infra | 22:29 | |
| jeblair | https://community.hpcloud.com/article/centos-63-instance-giving-cannot-retrieve-metalink-repository-epel-error | 22:30 |
| jeblair | i have not read that yet, but i'm kind of dreading what i'm about to read | 22:30 |
| mtreinish | jeblair: yeah, I just read it, you're not going to like it... | 22:31 |
| mtreinish | their fix to the problem is to run: sudo sed -i "s/mirrorlist=https/mirrorlist=http/" /etc/yum.repos.d/epel.repo && sudo yum update | 22:31 |
| *** annegent_ has quit IRC | 22:31 | |
| clarkb | uhm | 22:31 |
| Sukhdev | clarkb: can I ask you a quick question? | 22:32 |
| jeblair | where are these endpoints set? | 22:32 |
| *** dimsum__ has joined #openstack-infra | 22:32 | |
| clarkb | I guess no one is interested in fixing the base image. that is from november | 22:32 |
| jeblair | why is the solution not to remove the endpoints that use ssl3.0 (and then nuke them from orbit)? | 22:32 |
| jogo | mtreinish: wow is tempest's experimental list up to date | 22:32 |
| clarkb | Sukhdev: sure | 22:32 |
| jogo | mtreinish: are those all 'in progress' | 22:32 |
| *** carl_baldwin has joined #openstack-infra | 22:32 | |
| *** dimsum__ has quit IRC | 22:33 | |
| mtreinish | jogo: dunno, I haven't looked in a while, there's probably some pruning that could be done | 22:33 |
| *** dimsum__ has joined #openstack-infra | 22:33 | |
| Sukhdev | clarkb: How do I find out why has this not merged yet - https://review.openstack.org/#/c/148749/ | 22:33 |
| Sukhdev | I looked at zuul status dashboard - I can not see it there | 22:33 |
| jeblair | clarkb, mtreinish: "sudo yum upgrade ca-certificates --disablerepo=epel" from http://serverfault.com/questions/637549/epel-repo-for-centos-6-causing-error | 22:34 |
| jeblair | clarkb, mtreinish: so i'm guessing that what they actually need to do is to install a never ca-certs package on the base image? | 22:34 |
| mtreinish | jeblair: yeah that's what it sounds like | 22:34 |
| clarkb | Sukhdev: thats a good question. I suppose it is possible that change got caught up in our gerrit downtime today | 22:35 |
| clarkb | Sukhdev: you can safely recheck and it should go straight to the gate | 22:35 |
| Sukhdev | clarkb: "recheck no bug", right? | 22:35 |
| mtreinish | jogo: what were you looking at? | 22:35 |
| jogo | adding aiopcpu | 22:36 |
| clarkb | Sukhdev: just `recheck` will work | 22:36 |
| Sukhdev | clarkb: cool - thanks | 22:36 |
| clarkb | jeblair: hrm we have that in prepare_node.sh | 22:36 |
| clarkb | jeblair: I am looking at the log more closely to see if it needs to happen earlier | 22:37 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Run aiopcpu experimental job in a several places https://review.openstack.org/151806 | 22:37 |
| jogo | mtreinish: tempest has a very long list | 22:37 |
| mtreinish | jogo: yeah I just looked at the list, all of those are either in progress efforts or voting somewhere but not important enough to be voting on tempest | 22:37 |
| jogo | wow | 22:37 |
| Sukhdev | clarkb: I did the "recheck" it did not go to gate queue, instead it went to check queue - is this normal? | 22:38 |
| *** jedimike has quit IRC | 22:38 | |
| * jeblair quietly discards flaming commit he was authoring | 22:38 | |
| jogo | mtreinish: what about keystonev3 full? | 22:38 |
| mtreinish | jogo: the only one I'm not sure about is why there is a ceilo mongo job which is smoke and full we probably don't need one of those | 22:38 |
| mtreinish | jogo: that's tempest using v3 auth only | 22:39 |
| clarkb | jogo: oh lol | 22:39 |
| clarkb | er jeblair ^ | 22:39 |
| *** tmcpeak has quit IRC | 22:39 | |
| clarkb | jeblair: I will paste code and logs so we can see what happens | 22:39 |
| jeblair | clarkb: ++construcive approach | 22:39 |
| jogo | mtreinish: that isn't our default yet? | 22:39 |
| mtreinish | jogo: I'd have to check with andreaf_ on the status of that, my guess is it would probably work fine though | 22:40 |
| clarkb | jeblair: http://paste.openstack.org/show/164355/ the problem is the if yum repolist check fails due to the same error so then we skip the update ca-certs with the ignore epel option | 22:40 |
| *** leakypipes has joined #openstack-infra | 22:40 | |
| jogo | mtreinish: well added aiopcpu so if that lands I will run check experimental on clarkb's patch to turn off the bad test | 22:40 |
| clarkb | jeblair: I don't know enough about yum to know exactly what we want here, but what if we just remove that if else check and always update ca-certificates without epel then update with epel | 22:40 |
| mtreinish | jogo: the rest of your d-g stack landed? | 22:41 |
| clarkb | jeblair: that is my naive suggestions | 22:41 |
| *** kgiusti has quit IRC | 22:41 | |
| jogo | mtreinish: enough to get smoke working | 22:42 |
| jogo | next patch in series is turn on nova net multihost | 22:42 |
| jeblair | clarkb: i wonder if we can just check for a file's existence | 22:42 |
| jogo | but things should work without that too | 22:42 |
| jeblair | clarkb: i will spin up some hpcloud centos test nodes | 22:43 |
| clarkb | jeblair: sounds good | 22:43 |
| *** marun has quit IRC | 22:43 | |
| * jeblair learns nova boot | 22:44 | |
| *** armax has quit IRC | 22:44 | |
| *** mfink_ has quit IRC | 22:45 | |
| asselin_ | jeblair, nibalizer looks like module-split is basically done. last change that needs +A https://review.openstack.org/#/c/151782/ | 22:45 |
| *** achanda has quit IRC | 22:45 | |
| *** annegent_ has joined #openstack-infra | 22:45 | |
| clarkb | I was going to review that change but I already reviewed it | 22:45 |
| *** achanda has joined #openstack-infra | 22:46 | |
| asselin_ | jeblair, nibalizer I updated https://storyboard.openstack.org/#!/story/302, added a clean-up task to remove the extra pre-merged changes. | 22:46 |
| *** morganfainberg is now known as outforteaorcoffe | 22:47 | |
| jeblair | asselin_, sweston: i still think we should merge 137991 -- it's super cool, handy institutional knowledge we shouldn't forget | 22:48 |
| jeblair | sweston: do you have time to address the comments there or maybe replace it with the latest version of what you actually ran? | 22:49 |
| jeblair | asselin_: thanks! | 22:49 |
| asselin_ | jeblair, sweston did an awesome job. that saved a ton of time! | 22:49 |
| *** achanda has quit IRC | 22:50 | |
| sweston | jeblair: yes, I will address the comments now. | 22:50 |
| sweston | asselin_: thank you :-) | 22:50 |
| Sukhdev | clarkb: so, the recheck did not take https://review.openstack.org/#/c/148749/ to the gate - it is executing in the check queue - is there a way to get it into the gate - so that it can merge? | 22:52 |
| clarkb | Sukhdev: hrm, it should've gone to the gate unless it cannot merge for some reason. I may need to look closer | 22:53 |
| Sukhdev | clarkb: can you please? | 22:54 |
| *** baoli has quit IRC | 22:54 | |
| *** ChuckC has quit IRC | 22:54 | |
| *** pradk has quit IRC | 22:56 | |
| jeblair | clarkb: so we can either test for /etc/yum.repos.d/epel.repo | 22:57 |
| jeblair | clarkb: or run sudo yum update -y ca-certificates || sudo yum --disablerepo=epel update -y ca-certificates | 22:58 |
| clarkb | jeblair: I was thinking to double tap the yum update in order to get ca-certificates updated ptentially from epel after we do the first update? | 22:58 |
| clarkb | sudo yum --disablerepo=epel update -y ca-certificates ; sudo yum update -y ca-certificates | 22:59 |
| jeblair | clarkb: i don't think ca-certificates is expected to be in epel... | 22:59 |
| clarkb | jeblair: oh then why don't we just run it with the disablerepo always? | 22:59 |
| jeblair | clarkb: comment says "--disablerepo will fail" | 23:00 |
| jeblair | if we don't have epel configured | 23:00 |
| jeblair | yay, just corrupted my yum database; moving on to a new host :) | 23:00 |
| clarkb | ha | 23:01 |
| *** hdd has quit IRC | 23:01 | |
| jeblair | clarkb: i _think_ disablerepo=epel* might work | 23:02 |
| jeblair | (this is coming from finger memory from a past life; testing) | 23:02 |
| clarkb | if not your suggestion with the || sounds fine to me | 23:02 |
| *** amitgandhinz has quit IRC | 23:02 | |
| *** yamamoto has joined #openstack-infra | 23:03 | |
| *** harlowja is now known as harlowja_away | 23:05 | |
| clarkb | Sukhdev: I am not seeing anythin readily apparently in the zuul logs | 23:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Fix centos6 image updates on hpcloud https://review.openstack.org/151814 | 23:05 |
| jeblair | clarkb: i think epel* will work for us | 23:05 |
| jeblair | i will now detabify it. | 23:06 |
| Sukhdev | clarkb: interesting!! So, what would be the next step for me once it passes the check (it seems half way through) to get it to the gate? | 23:06 |
| clarkb | Sukhdev: well hopefully it goes to the gate, if not then need to figure out what is keeping it out | 23:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Fix centos6 image updates on hpcloud https://review.openstack.org/151814 | 23:07 |
| Sukhdev | clarkb: OK I will wait and see what happens next once it passes the check. Will ping you once the check is done | 23:07 |
| clarkb | oh hrm recheck isn't a thing on the gate | 23:08 |
| clarkb | jeblair: ^ when did that change? | 23:08 |
| clarkb | so that probably explains it. needs to go to check then gate | 23:08 |
| *** abhirc has joined #openstack-infra | 23:08 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Move missing periodic qa jobs to tempest experimental https://review.openstack.org/151815 | 23:09 |
| mtreinish | anteaya, clarkb, jeblair: ^^^ should be a quick review | 23:09 |
| jeblair | clarkb: my guess is that is since it's only useful if zuul misses an event (since all genuine failures should end up with negative votes and need to go to check for clean-check) | 23:09 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Clean-up deleted/refactored puppet module repo not https://review.openstack.org/151816 | 23:10 |
| *** ddieterly has joined #openstack-infra | 23:10 | |
| clarkb | jeblair: do you need to escape the * to avoid shell globbing? | 23:10 |
| *** leakypipes has quit IRC | 23:10 | |
| clarkb | I guess that is only a problem if something glob matches epel* in the current dir | 23:10 |
| jeblair | clarkb: or glob matches "--disablerepo=epel"? | 23:11 |
| *** arxcruz has quit IRC | 23:11 | |
| clarkb | right :) its probably fine as is | 23:11 |
| clarkb | +2 I can rebuild images as soon as that merges | 23:11 |
| *** andreykurilin_ has joined #openstack-infra | 23:12 | |
| *** ashleighfarnham has quit IRC | 23:12 | |
| *** dustins has quit IRC | 23:12 | |
| *** shivharis has quit IRC | 23:14 | |
| *** dimtruck is now known as zz_dimtruck | 23:15 | |
| *** wenlock has quit IRC | 23:17 | |
| clarkb | on the incomplete console log front: I can reproduce locally by fetching the same urls as the job | 23:17 |
| clarkb | and the --no-cache option doesn't seem to help | 23:17 |
| clarkb | also my browser exhibits the same behavior so I don't think wget and/or wget -c are at fault | 23:18 |
| jeblair | clarkb: oh interesting; how long after the job finished? | 23:18 |
| clarkb | however as soon as the job finishes I seem to get the complete log | 23:18 |
| jeblair | ah ok | 23:19 |
| clarkb | jeblair: so it seems to do that while the job is running but works after the job completes | 23:19 |
| jeblair | clarkb: are you seeing the whole log in the browser? | 23:19 |
| jeblair | clarkb: or does the browser look the same? | 23:19 |
| clarkb | jeblair: when the job was running browser was truncated, after job finished log was complete in browser | 23:20 |
| jeblair | clarkb: so it seems like buffering, yeah? | 23:20 |
| clarkb | ya | 23:20 |
| clarkb | maybe the slave process itself needs to flush | 23:20 |
| jeblair | clarkb: but we see it move one line at a time on slow jobs | 23:20 |
| clarkb | oh right so in the browser on the normal cosole page that worked | 23:20 |
| *** imcsk8 has quit IRC | 23:21 | |
| *** imcsk8 has joined #openstack-infra | 23:21 | |
| clarkb | so probably not the slave. sorry when I was talking about truncated things in the browser above I meanted looking at the urls that the swift uploader is grabbing | 23:21 |
| jeblair | ah | 23:21 |
| clarkb | https://jenkins01.openstack.org/job/gate-project-config-layout/420/console vs https://jenkins01.openstack.org/job/gate-project-config-layout/420/logText/progressiveHtml first one is complete second one was truncated until job end | 23:21 |
| *** ashleighfarnham has joined #openstack-infra | 23:23 | |
| jeblair | clarkb: jenkins actually using a POST with start=x parameter | 23:23 |
| *** timcline has quit IRC | 23:23 | |
| clarkb | jeblair: I am just going to approve https://review.openstack.org/#/c/151814/ now. I want those images rebuilt | 23:23 |
| *** timcline has joined #openstack-infra | 23:23 | |
| jeblair | clarkb: ++ | 23:23 |
| *** prad has quit IRC | 23:24 | |
| *** dkliban is now known as dkliban_afk | 23:26 | |
| *** yamamoto has quit IRC | 23:26 | |
| *** achanda has joined #openstack-infra | 23:26 | |
| clarkb | jeblair: wait the 420/console path is doing a POST to get the data back? | 23:26 |
| jeblair | clarkb: a POST to progressiveHtml | 23:27 |
| * clarkb turns on some chromium developer tools | 23:27 | |
| *** harlowja_away is now known as harlowja | 23:27 | |
| *** chuckC_ has joined #openstack-infra | 23:28 | |
| *** timcline has quit IRC | 23:28 | |
| *** sabeen2 has quit IRC | 23:29 | |
| *** harlowja has quit IRC | 23:29 | |
| *** ddieterly has quit IRC | 23:29 | |
| *** harlowja has joined #openstack-infra | 23:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix centos6 image updates on hpcloud https://review.openstack.org/151814 | 23:30 |
| clarkb | unfortunately that means we have to calculate our start: X offset manually since wget -c probably won't | 23:31 |
| clarkb | but it seems to work | 23:31 |
| jeblair | clarkb: i'm not understanding this yet | 23:31 |
| jeblair | clarkb: this isn't a problem in all cases -- do we know if there's a pattern? | 23:32 |
| jeblair | clarkb: i just watched https://jenkins04.openstack.org/job/gate-infra-puppet-apply-centos6/901/console and it worked normally | 23:32 |
| *** leakypipes has joined #openstack-infra | 23:32 | |
| clarkb | jeblair: I think it has to do with file length. the layout job console is much longer/larger than the other jobs | 23:33 |
| *** ddieterly has joined #openstack-infra | 23:33 | |
| clarkb | jeblair: so the other jobs get the entire log the first go around whereas the layout console is longer and needs to try again (assuming buffering is happening) | 23:33 |
| *** dangers is now known as dangers_away | 23:33 | |
| jeblair | clarkb: and you're thinking progressiveHTML has a limit to how much it will serve? | 23:34 |
| zaro | zuul question: noop jobs should get return success back to gerrit immediately correct? for some reason noop jobs are getting stuck in the zuul queue. http://zuul-dev.openstack.org/ | 23:34 |
| zaro | can anybody help? | 23:34 |
| clarkb | jeblair: ya | 23:34 |
| *** hdd has joined #openstack-infra | 23:36 | |
| *** Duarte-DVR has quit IRC | 23:36 | |
| jeblair | zaro: no merger running | 23:36 |
| jeblair | zaro: 2015-01-30 22:51:48,069 DEBUG zuul.MergeClient: Submitting job merger:merge with data {'items': [{'oldrev': None, 'newrev': None, 'refspec': u'refs/changes/72/5272/3', 'merge_mode': 2, 'number': u'5272', 'project': 'gtest-org/gtest', 'url': 'ssh://zuul-dev@review-dev.openstack.org:29418/gtest-org/gtest', 'branch': u'master', 'patchset': u'3', 'ref': 'Z5a2e180e51ae467faad89dcca5f8d13b'}]} | 23:36 |
| clarkb | I am rebuilding centos6 images now | 23:36 |
| jeblair | zaro: that's the last action from the log file | 23:36 |
| *** ZZelle_ has quit IRC | 23:37 | |
| zaro | jeblair: thanks | 23:37 |
| *** achanda has quit IRC | 23:39 | |
| fungi | clarkb: the scaffords33 job wasn't being run anyway, which is why i removed that section | 23:40 |
| clarkb | jeblair: I think we could probably switch over to `curl -XPOST --data="$(wc -c /tmp/console.html || echo 0)" $URLTOJENKINSPROGHTML` >> /tmp/console.html | 23:40 |
| fungi | ryanpetrello: ^ | 23:41 |
| fungi | er, scaffolds | 23:41 |
| fungi | oh, actually i do see it in the layout on master now | 23:42 |
| clarkb | though maybe stat would be faster than wc | 23:43 |
| fungi | ahh, i see my confusion. updating 151713 accordingly | 23:43 |
| jeblair | clarkb: sounds reasonable | 23:44 |
| *** achanda has joined #openstack-infra | 23:44 | |
| *** timcline has joined #openstack-infra | 23:44 | |
| fungi | ryanpetrello: i haven't tried that job on python 3.4 since it wasn't a standard py34 tox env, but if you suspect it works then fine by me | 23:44 |
| *** chuckC_ has quit IRC | 23:45 | |
| *** timcline_ has joined #openstack-infra | 23:46 | |
| asselin | jeblair, fungi: last one, really: https://review.openstack.org/151816 | 23:47 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Switch from Python 3.3 to 3.4 for Py3K jobs https://review.openstack.org/151713 | 23:47 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Stop using py3k-precise nodes https://review.openstack.org/151715 | 23:47 |
| *** hdd_ has joined #openstack-infra | 23:47 | |
| fungi | clarkb: ryanpetrello: ^ | 23:47 |
| *** michchap has joined #openstack-infra | 23:47 | |
| *** hdd has quit IRC | 23:48 | |
| jogo | clarkb: I just noticed tempest-full version of aiopcpu takes 20 minutes or so | 23:48 |
| jogo | while regular tempest-full takes 33 minutes | 23:48 |
| *** timcline has quit IRC | 23:49 | |
| jogo | hmm maybe not | 23:49 |
| *** michchap_ has quit IRC | 23:49 | |
| jogo | oh that was when devstack was broken | 23:49 |
| *** annegent_ has quit IRC | 23:50 | |
| *** ddieterly has quit IRC | 23:50 | |
| *** timcline_ has quit IRC | 23:51 | |
| clarkb | jeblair: centos6 iamge builds are progressing now. | 23:51 |
| fungi | clarkb: did you manato spot what was broken with swift log uploads for centos6? | 23:53 |
| *** shivharis has joined #openstack-infra | 23:53 | |
| *** bswartz has joined #openstack-infra | 23:53 | |
| * fungi skipped some of the copious scrollback which accumulated while he was dining | 23:53 | |
| clarkb | fungi: ya hpcloud images were ancient due to epel cert fun. jeblair whipped up a fix | 23:53 |
| *** timcline has joined #openstack-infra | 23:54 | |
| *** timcline has quit IRC | 23:58 | |
| *** changbl has quit IRC | 23:58 | |
| *** achanda has quit IRC | 23:59 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!