| *** dcramer_ has quit IRC | 00:00 | |
| *** senk has quit IRC | 00:00 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup builds for openstack Gerrit ver 2.8 https://review.openstack.org/60348 | 00:02 |
|---|---|---|
| jeblair | lifeless: writing some (moderate complexity) puppet changes for infra/config would also help increase your familiarity with the tree and our processes | 00:04 |
| *** rcleere has joined #openstack-infra | 00:04 | |
| lifeless | jeblair: ack; will keep an eye out for such | 00:05 |
| *** aardvark has quit IRC | 00:07 | |
| *** aardvark has joined #openstack-infra | 00:08 | |
| fungi | lifeless: just noticed you'd set the gantt{,client} addition to wip, but left a comment in there with regard to some minor missing parity with nova{,client} gating jobs | 00:09 |
| fungi | for once you're working on the subsequent non-wip pilish | 00:09 |
| fungi | polish | 00:09 |
| jeblair | asalkeld: the hook job does not upload its logs. :( | 00:10 |
| asalkeld | yeah for the moment I manually built the docs on readthedocs.org | 00:10 |
| lifeless | fungi: thats because the client isn't a restful API client for end users | 00:10 |
| fungi | oho! | 00:10 |
| lifeless | fungi: it's the RPC API definition, so skipping 3.3. and pypy is deliberate | 00:11 |
| fungi | right, it's backend/middleware huh? | 00:11 |
| lifeless | fungi: as today there is no usecase for standalone / CLI use | 00:11 |
| jeblair | asalkeld: so the other thing you can do is go spelunking in jenkins: | 00:11 |
| jeblair | asalkeld: https://jenkins02.openstack.org/job/hook-solum-rtfd/? | 00:11 |
| lifeless | it may grow that in future, and if so we'd want to add them in | 00:11 |
| jeblair | asalkeld: https://jenkins01.openstack.org/job/hook-solum-rtfd/? | 00:11 |
| clarkb | lifeless: ++ py3k is suppose to happen any month now :) | 00:11 |
| asalkeld | jeblair, looking | 00:11 |
| *** flaper87 is now known as flaper87|afk | 00:12 | |
| jeblair | asalkeld: looks like it clamed success; however, those jobs recently changed to use that new URL (using name instead of id); perhaps it's not quite right. | 00:12 |
| fungi | lifeless: though if it starts out working with py33 and pypy, you don't want that to regress as it diverges from novaclient right? | 00:12 |
| asalkeld | jeblair, yeah must be it | 00:13 |
| asalkeld | would be nice to have the output | 00:13 |
| lifeless | fungi: you make a valid point. But I'd rather have more work later than any issues that are not relevant today. | 00:13 |
| asalkeld | jeblair, https://jenkins01.openstack.org/job/hook-solum-rtfd/3/console | 00:13 |
| fungi | fair enough | 00:13 |
| lifeless | fungi: if this works on 3.3 but doesn't deprecate the nova scheduler, its a fail. | 00:13 |
| lifeless | fungi: the reverse is a success | 00:13 |
| fungi | sure | 00:13 |
| *** rcleere has quit IRC | 00:14 | |
| lifeless | fungi: oh, so this isn't going to diverge from nova client | 00:14 |
| lifeless | fungi: it's not starting from nova client | 00:14 |
| lifeless | fungi: it's starting from nova/scheduler/api.py + tests. | 00:14 |
| fungi | got it | 00:14 |
| jeblair | asalkeld: yeah, seems harmless. you want to patch that job definition to emit the output; and if you feel like it, upload logs too? | 00:14 |
| asalkeld | so that just triggers the build in readthedocs right? | 00:15 |
| asalkeld | so it might be ok | 00:15 |
| asalkeld | let me wait for the next one to see what happens | 00:15 |
| jeblair | asalkeld: yes; oh you think rtd might be behind? | 00:15 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Build mirror using proper requirements. https://review.openstack.org/60354 | 00:15 |
| asalkeld | well we have just added docs | 00:16 |
| asalkeld | and set it up right | 00:16 |
| asalkeld | so let me not waist your time | 00:16 |
| asalkeld | I'll see what happens next commit | 00:16 |
| clarkb | woot pbr change merged | 00:17 |
| jeblair | asalkeld: ok. also if rtd is fast, it's possible it's even starting a build before we've finished replicating (in which case the hook job could sleep) | 00:17 |
| asalkeld | ok | 00:17 |
| jeblair | fungi: OOOOH | 00:17 |
| jeblair | fungi, clarkb: i think the disk stuff happens _after_ the d-g rexec | 00:18 |
| jeblair | fungi, clarkb: nm, i'm still confused. | 00:18 |
| clarkb | jeblair: it was happening at a weird point in time at one time and we moved it | 00:18 |
| clarkb | because the mounts were overlaying all of the stuff in /opt | 00:18 |
| clarkb | but I think that is fixed now | 00:18 |
| fungi | jeblair: yeah, i'm still trying to match teh change up to the logs from the most recent run | 00:19 |
| jeblair | i was confused why bash-x wasn't working, but i think i understand | 00:19 |
| jeblair | i think it's because of the way it's initially invoked, i think i may need to set the option using a command if i want that. | 00:19 |
| fungi | right, or pass that as an option to the shell when invoking? | 00:20 |
| *** dims has quit IRC | 00:21 | |
| jeblair | fungi: oh, i think stdout goes to the setup log file, and stderr goes to the console log. that's somewhat confusing. | 00:23 |
| jeblair | clarkb: okay oh, er it's called both places | 00:23 |
| jeblair | clarkb: fix_disk_layout is called _before_ the rexec _and_ after. | 00:24 |
| fungi | i'm not seeing it repeat itself in the setup log | 00:24 |
| clarkb | jeblair: right, but it should only do stuff the first time | 00:24 |
| clarkb | because the check checks swap | 00:24 |
| *** adalbas has quit IRC | 00:24 | |
| jeblair | clarkb: yeah, but you can see how this is frustrating my attempts to test this change. :) | 00:24 |
| fungi | and we want to test having that happen after the reexec | 00:24 |
| fungi | harumph | 00:25 |
| klrmn | clarkb: i have a new answer for you as to why verbose…so that you know which tests *did* run when the number of tests comes back waaaaay less than you expected. (damnit, it ran 29 tests when i thought it was going to run over 100…how the hell do i debug *that* (with testr)?) | 00:25 |
| clarkb | oh right | 00:25 |
| clarkb | klrmn: all of the subunit is logged, you can check the logs directly or do testr last --subunit | 00:25 |
| jeblair | fungi: the first time the invocation is hidden (because of no -x), but the output appears in console log; the second time the invocation is visible in the setup log, but of course it's a no-op because there is now swap. | 00:25 |
| fungi | that's why it didn't look like it repeated the operation | 00:26 |
| jeblair | clarkb: do you recall why it was thought to be necessary to run it before the re-exec? | 00:26 |
| clarkb | jeblair: because we needed a place to put d-g itself | 00:26 |
| fungi | ramdiskramdiskramdisk ;) | 00:26 |
| clarkb | jeblair: this is a problem with /opt and not swap. | 00:26 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup builds for openstack Gerrit ver 2.8 https://review.openstack.org/60348 | 00:26 |
| jeblair | clarkb: it's in cwd. | 00:27 |
| *** esker has joined #openstack-infra | 00:27 | |
| jeblair | (that's why the job copies the script before invoking it, so that the copy in cwd can be safely updated and then exec'd (bash doesn't deal well with files changing under it)) | 00:27 |
| fungi | can we safely remove the earlier call? | 00:28 |
| jeblair | so as we keep pulling this thread -- i'm guessing that part of the re-org was to remove the update to cwd devstack-gate. | 00:28 |
| clarkb | jeblair: d-g doesn't do that though | 00:28 |
| clarkb | it runs it out of /opt | 00:28 |
| clarkb | the reexec | 00:29 |
| fungi | existing devstack-gate in cwd exec's proposed d-g in /opt | 00:30 |
| clarkb | fungi: right | 00:30 |
| jeblair | it didn't used to, but maybe that changed with the grenade refactor | 00:31 |
| *** Abhishek_ has quit IRC | 00:31 | |
| jeblair | let me see if i can fix this | 00:32 |
| *** dims has joined #openstack-infra | 00:33 | |
| *** jerryz has quit IRC | 00:34 | |
| klrmn | clarkb: stored file for the win. the environment that had testr installed was axed at the end of the jenkins job | 00:34 |
| *** Abhishek has joined #openstack-infra | 00:34 | |
| fungi | today's gerrit project count: 211 (stackforge hovering at 46% of the total) | 00:35 |
| clarkb | klrmn: http://logs.openstack.org/94/43194/27/check/gate-nova-python27/ae134d6/subunit_log.txt.gz definitely :) | 00:35 |
| *** adalbas has joined #openstack-infra | 00:37 | |
| *** harlowja has quit IRC | 00:40 | |
| *** harlowja has joined #openstack-infra | 00:42 | |
| *** esker has quit IRC | 00:43 | |
| *** esker has joined #openstack-infra | 00:43 | |
| clarkb | jeblair: reading the script, I have a hunch that if in reboot_devstack_gate after setup_project you copy from /opt to the workspace then reexec and do the the swap and /opt stuff after the reexec it iwll work | 00:47 |
| *** esker has quit IRC | 00:48 | |
| jeblair | clarkb: except that we're copying _to_ opt. | 00:48 |
| *** mrodden has quit IRC | 00:48 | |
| clarkb | setup_project dumps the d-g git repo in /opt but we only need a couple of files out of that repo right? | 00:49 |
| *** Abhishek has quit IRC | 00:49 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup builds for openstack Gerrit ver 2.8 https://review.openstack.org/60348 | 00:49 |
| clarkb | jeblair: or maybe put devstack-gate in $PROJECTS and setup_project devstack-gate twice | 00:50 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60391 | 00:51 |
| *** vipul has quit IRC | 00:51 | |
| jeblair | clarkb: or take it out of PROJECTS and only do it once ^ :) | 00:51 |
| *** jroll has joined #openstack-infra | 00:52 | |
| *** vipul has joined #openstack-infra | 00:52 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use the ephemeral disk on rackspace performance nodes https://review.openstack.org/60347 | 00:52 |
| *** jroll has left #openstack-infra | 00:52 | |
| *** ruhe has joined #openstack-infra | 00:53 | |
| *** ruhe has quit IRC | 00:54 | |
| *** jcooley_ has joined #openstack-infra | 00:56 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Ensure solum code is checked out https://review.openstack.org/60392 | 00:59 |
| clarkb | jeblair: I don't think devstack-checkout is sophisticated enough to make that work in JJB | 00:59 |
| clarkb | jeblair: I like the change though, so we should make devstack-checkout better | 01:00 |
| jeblair | clarkb: ? | 01:00 |
| clarkb | jeblair: devstack-checkout is the macro in JJB that makes it so that devstack-gate is in the workspace. It does not checkout the ZUUL_REF | 01:00 |
| clarkb | it resets to master | 01:00 |
| jeblair | clarkb: right we're only counting on devstack-checkout putting a copy of devstack in the workspace; i use setup_project to update to the ref before the re-exec | 01:01 |
| * clarkb rereads | 01:01 | |
| *** ^demon has joined #openstack-infra | 01:01 | |
| jeblair | russellb: which, interestingly enough, is i believe exactly why the change you just wrote won't work. that's like cosmically weird. :) | 01:02 |
| *** ^d has quit IRC | 01:03 | |
| clarkb | jeblair: I see it now. setup_project no longer clones and works on CWD | 01:03 |
| *** mestery has quit IRC | 01:04 | |
| jeblair | clarkb: yep, and clone_and_setup_project does the whole enchilada. | 01:04 |
| jeblair | (which is the normal case) | 01:04 |
| *** mestery has joined #openstack-infra | 01:05 | |
| clarkb | jeblair: something did break though (jenkins not happy with 60391) | 01:05 |
| *** mrodden has joined #openstack-infra | 01:05 | |
| *** Abhishek has joined #openstack-infra | 01:07 | |
| jeblair | clarkb: oh, it's not $WORKSPACE it's $WORKSPACE/devstack-gate | 01:07 |
| *** mrodden1 has joined #openstack-infra | 01:07 | |
| jeblair | that's actually what the original GATE_SCRIPT_DIR was for; i should probably just go back to that | 01:08 |
| *** mrodden has quit IRC | 01:08 | |
| clarkb | how did it get far enough to run setup_workspace? | 01:09 |
| jeblair | clarkb: remember that most of this change is not self-testing. :( | 01:10 |
| clarkb | ohhhhh | 01:11 |
| russellb | jeblair: heh ... i feel like i'm stabbing in the dark with this job | 01:11 |
| clarkb | right first chance to run $WORKSPACE/STUFF is after the reexec | 01:11 |
| clarkb | which is when it tries to run the gate_hook I think | 01:11 |
| *** Abhishek has quit IRC | 01:11 | |
| *** mrodden1 has quit IRC | 01:11 | |
| *** mrodden has joined #openstack-infra | 01:11 | |
| *** jergerber has quit IRC | 01:11 | |
| jeblair | russellb: at least you get test results for your change. :) i'm touching like the one part of devstack-gate that isn't self-testing. | 01:12 |
| jeblair | russellb: i left what i think should do the trick as a comment on your change | 01:12 |
| russellb | saw that thanks | 01:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60391 | 01:14 |
| *** adalbas has quit IRC | 01:14 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use the ephemeral disk on rackspace performance nodes https://review.openstack.org/60347 | 01:15 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Ensure solum code is checked out https://review.openstack.org/60392 | 01:16 |
| * russellb pokes it with a stick some more | 01:16 | |
| clarkb | jeblair: I am scratching my head at the pwd. Is that to get a fully rooted path? | 01:17 |
| jeblair | clarkb: yep. that's what it used to be, a long time ago. copied straight from devstack. | 01:17 |
| clarkb | ok not completely crazy over here, thnaks | 01:18 |
| *** annegentle has quit IRC | 01:18 | |
| jeblair | clarkb: so it'll get the full path of the directory the script is in, regardless of where you call it from | 01:18 |
| clarkb | yup | 01:20 |
| *** jerryz has joined #openstack-infra | 01:20 | |
| jeblair | russellb: lgtm; probably worth clarkb giving it a once over since his head is in this space right now too :) | 01:21 |
| russellb | thanks :) | 01:22 |
| clarkb | I will look in a minute | 01:22 |
| mgagne | clarkb: AFAIK, pwd returns the path of the folder you are in, not the folder in which the script is. | 01:22 |
| russellb | once this works, maybe i can turn this into a stackforge devstack-gate job template | 01:23 |
| mgagne | clarkb: http://paste.openstack.org/show/54565/ | 01:23 |
| clarkb | mgagne: right which is why it cd's in the dirname $0 first. I initially assumed dirname $0 was sufficient | 01:23 |
| jeblair | mgagne: DIR=$(cd $(dirname "$0") && pwd) | 01:23 |
| jeblair | russellb: ++ | 01:23 |
| mgagne | ^^' | 01:25 |
| *** annegentle has joined #openstack-infra | 01:25 | |
| clarkb | jeblair: do we need to reexec twice to have 60391 and 60347 get tested wit hthe new code as is? | 01:28 |
| openstackgerrit | A change was merged to openstack-infra/config: Ensure solum code is checked out https://review.openstack.org/60392 | 01:28 |
| clarkb | jeblair: brainstorming ways that would make this less painful | 01:29 |
| openstackgerrit | Sean Roberts proposed a change to openstack-infra/config: config changes to create the stackforge project milk https://review.openstack.org/60394 | 01:29 |
| *** bknudson has quit IRC | 01:30 | |
| *** melwitt has quit IRC | 01:30 | |
| *** melwitt has joined #openstack-infra | 01:30 | |
| jeblair | clarkb: i'm not sure we can ever make this better other than minimizing what happens before the re-exec, and it's getting pretty small now. | 01:30 |
| jeblair | clarkb: we might want to put devstack-gate back into PROJECTS so that the zuul_ref part of the checkout gets exercised after a re-exec (even though the code won't actually be used) | 01:31 |
| clarkb | a second rexec would ensure that changes that shrink the delta are actually tested | 01:32 |
| *** paul-- has joined #openstack-infra | 01:32 | |
| jeblair | clarkb: i'm not sure that's the case -- you can never test not doing something that the first run did before the first re-exec | 01:32 |
| jeblair | clarkb: (which is exactly what we want to do with the disk change) | 01:33 |
| *** ^demon has quit IRC | 01:33 | |
| clarkb | gah ya | 01:33 |
| clarkb | what if we gerrit git prepped d-g before d-g runs | 01:35 |
| jeblair | clarkb: i can't think of why that isn't exactly what we want to do. | 01:37 |
| *** sarob_ is now known as sarob | 01:37 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60391 | 01:38 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use the ephemeral disk on rackspace performance nodes https://review.openstack.org/60347 | 01:38 |
| jeblair | clarkb: oh, so one thing is ggp assumed a single workspace | 01:39 |
| clarkb | I think d-g should still setup everything else | 01:39 |
| jeblair | clarkb: yeah, that's not what i was saying though | 01:39 |
| *** markwash has quit IRC | 01:40 | |
| jeblair | clarkb: ggp assumes project-git-dir == $WORKSPACE; devstack-gate makes a subdir for the devstack-gate git repo (and another for logs) | 01:41 |
| jeblair | clarkb: so i think we'd need to accomodate that... | 01:41 |
| clarkb | good point | 01:41 |
| jeblair | clarkb: may be as simple as "mkdir devstack-gate; cd devstack-gate; ggp" | 01:42 |
| *** nosnos has joined #openstack-infra | 01:42 | |
| clarkb | wfm | 01:42 |
| jeblair | clarkb: i should work on dinner. i will WIP my 2 changes and work on this new plan tomorrow. thanks for the idea. :) | 01:43 |
| clarkb | I need to hunt down dinner too | 01:44 |
| clarkb | and np :) | 01:44 |
| klrmn | lifeless / clarkb: could you tell me where the subunit2XXX things live? | 01:47 |
| lifeless | klrmn: mostly in the subunit source tree? | 01:47 |
| reed | where is monty? | 01:49 |
| klrmn | lifeless: i've looked in every file hanging off of http://bazaar.launchpad.net/~subunit/subunit/trunk/files/235/python/subunit without finding 'subunit2pyunit' | 01:50 |
| *** bknudson has joined #openstack-infra | 01:50 | |
| klrmn | lifeless: found it | 01:53 |
| *** sjing has joined #openstack-infra | 01:54 | |
| *** marun has joined #openstack-infra | 01:59 | |
| *** jcooley_ has quit IRC | 02:03 | |
| *** jcooley_ has joined #openstack-infra | 02:04 | |
| *** julim has quit IRC | 02:06 | |
| *** jcooley_ has quit IRC | 02:08 | |
| mordred | morning jeblair clarkb | 02:11 |
| *** rongze has joined #openstack-infra | 02:12 | |
| * mordred is now in tel aviv - the scrollback makes me want to die - but I have a pretty wide open day tomorrow - where are we on the mirror bs - same place we were? I'd like to dive in and see if I can get it sorted | 02:12 | |
| fungi | mordred: clarkb worked this out for it https://review.openstack.org/60354 (now merged) | 02:14 |
| fungi | the d2to1 can presumably be ripped out once dev-requirements.txt demands a newer oslo.messaging prerelease tarball url, and once the projects using that update to match | 02:15 |
| *** changbl has joined #openstack-infra | 02:16 | |
| *** bingbu has joined #openstack-infra | 02:16 | |
| fungi | the alternative was to convince pypi-mirror how to also integrate tarball urls into the run for integration testing only, but we figured with impending wheels uploaded to pypi for prereleases that was time better spent on other broken things instead | 02:18 |
| *** jcooley_ has joined #openstack-infra | 02:19 | |
| *** guohliu has joined #openstack-infra | 02:22 | |
| *** sarob_ has joined #openstack-infra | 02:22 | |
| *** sarob has quit IRC | 02:25 | |
| *** sarob_ has quit IRC | 02:27 | |
| *** yamahata_ has joined #openstack-infra | 02:28 | |
| *** weshay has quit IRC | 02:28 | |
| mordred | fungi: awesome | 02:29 |
| mordred | so looking at that patch, that means we should be on more even footing | 02:29 |
| *** sarob has joined #openstack-infra | 02:29 | |
| mordred | fungi: we should be able to un-revert the patches we reverted now, yeah? since they weren't actually the problem? | 02:30 |
| *** mestery has quit IRC | 02:30 | |
| mordred | or - wait- I guess there was the other problem which was the capitalization one, yeah? | 02:30 |
| fungi | mordred: seems that way | 02:30 |
| mordred | btw - there is still no CentOS-6.5 dir on the mirror | 02:31 |
| fungi | capitalization was impacting us where? | 02:31 |
| mordred | the sphinx changes | 02:31 |
| fungi | mmmright | 02:31 |
| mordred | I mean, the things breaking when trying to install sphinx, and not finding it | 02:31 |
| mordred | fungi: wow. https://jenkins.openstack.org/job/periodic-mirror-python26/116/console | 02:32 |
| mordred | oh - build timed out | 02:32 |
| fungi | yep. consistently on just about every one | 02:33 |
| *** krotscheck has quit IRC | 02:33 | |
| *** mestery has joined #openstack-infra | 02:33 | |
| mordred | maybe we shold bump up build timeout' | 02:37 |
| mordred | ? | 02:37 |
| fungi | it's worth considering, though i'm wondering why it's started to take longer | 02:38 |
| *** adalbas has joined #openstack-infra | 02:39 | |
| fungi | maybe because we've just accumulated more projects and dependencies | 02:39 |
| mordred | well, it does double builds now | 02:40 |
| mordred | because it gets eveyrhting, AND then it builds wheels of everything | 02:40 |
| *** reed has quit IRC | 02:43 | |
| fungi | but wayyyyy more slowly than on 2.7 or 3.3 | 02:45 |
| mordred | fungi: coupla quick questoins- git*.o.o are the only things running on centos, yeah? | 02:46 |
| fungi | mordred: pbx (asterisk) | 02:46 |
| mordred | fungi: also - we have cinder volumes on static - anywhere else? | 02:47 |
| fungi | not yet, though rackspace performance nodes look like they may change that | 02:47 |
| mordred | how is that? | 02:48 |
| fungi | but again, implementation detail of the platform, not because we just want cinder on those | 02:48 |
| mordred | gotcha | 02:48 |
| *** asalkeld has left #openstack-infra | 02:48 | |
| fungi | smaller static disk on them, similar to hp | 02:48 |
| mordred | the static.o.o cinder raid array is up to a few TB now yeah? | 02:48 |
| fungi | mordred: i've got the second-to-last 0.5tb volume migrating onto a newer volume now. some time tomorrow we'll have 10tb on static.o.o | 02:49 |
| fungi | (pvmove just takes a really long time on an active vg) | 02:49 |
| mordred | wow | 02:50 |
| fungi | we'll already be using about half of it once i'm done sizing up | 02:50 |
| fungi | and i expect by spring we'll be near full with only a 6-month compressed build log history | 02:51 |
| fungi | so hopefully we either have a new storage solution working by then, or we ask for a quota bump | 02:51 |
| mordred | ++ | 02:52 |
| *** melwitt has quit IRC | 02:54 | |
| *** salv-orlando has quit IRC | 02:55 | |
| *** sdake_ has quit IRC | 02:57 | |
| fungi | jeblair's currently working through tweaking devstack-gate to make use of the ephemeral disk on rackspace performance, but he ran some initial baselines on it and it sounded nearly twice as fast as current rackspace nova nodes (can't remember if you were around when he was trying it earlier today) | 02:57 |
| *** sarob has quit IRC | 02:57 | |
| *** sarob has joined #openstack-infra | 02:57 | |
| *** thomasem has joined #openstack-infra | 02:59 | |
| fungi | 20:54z in the scrollback | 02:59 |
| *** thomasem has quit IRC | 02:59 | |
| clarkb | mordred: fungi: right I didn't fix the inability to install transitive deps of tarball deps and I didn't fix wheels | 03:01 |
| *** esker has joined #openstack-infra | 03:01 | |
| clarkb | but I made the integration test test the proper requirements ref and got the test to apss | 03:01 |
| clarkb | wheels make tarballs a non issue. So fixing wheels is probably the next step | 03:02 |
| *** sarob has quit IRC | 03:02 | |
| *** fifieldt_ has joined #openstack-infra | 03:03 | |
| *** vipul is now known as vipul-away | 03:05 | |
| *** senk has joined #openstack-infra | 03:08 | |
| *** vipul-away is now known as vipul | 03:11 | |
| *** dcramer_ has joined #openstack-infra | 03:11 | |
| *** pcrews_ has quit IRC | 03:12 | |
| *** senk has quit IRC | 03:13 | |
| *** sarob has joined #openstack-infra | 03:14 | |
| *** sdake_ has joined #openstack-infra | 03:16 | |
| *** fifieldt has quit IRC | 03:18 | |
| *** pcrews has quit IRC | 03:19 | |
| *** nati_ueno has quit IRC | 03:24 | |
| mordred | yah | 03:24 |
| mordred | fix wheels - finish wheel uploads, delete dev-requirements | 03:25 |
| clarkb | mordred: one thing that I was wondering about with requirements tests | 03:26 |
| lifeless | how do you change the topic of an existing patch? | 03:26 |
| *** mriedem has quit IRC | 03:26 | |
| mordred | lifeless: you upload a new patch - or you wait until we've updated to 2.8 | 03:26 |
| mordred | clarkb: yeah? | 03:26 |
| clarkb | mordred: is d-g applying the proposed requiremetn when running devstack? and if so are we letting the tempest tests run against pypi.o.o? | 03:26 |
| clarkb | mordred: the reason django 1.6 got through is those tests weren't tested against pypi proper | 03:27 |
| mordred | clarkb: d-g is applying the proposed requirements when running devstack | 03:29 |
| *** jcooley_ has quit IRC | 03:29 | |
| clarkb | mordred: ok, but then pypi.o.o is used? | 03:29 |
| mordred | clarkb: when the change is to requirements, the select-mirror script should abvoid pypi.o.o | 03:29 |
| mordred | because it's a change to requirements | 03:29 |
| mordred | there was a bug in the logic for that selection | 03:29 |
| mordred | which has been fixed | 03:29 |
| *** jcooley_ has joined #openstack-infra | 03:30 | |
| clarkb | did django 1.6 sneak through when it was broken? | 03:30 |
| mordred | yah | 03:30 |
| clarkb | actually it keys off of DEVSTACK_GATE_REQS_INTEGRATION, is that set in the job itself? | 03:31 |
| mordred | jeblair and I walked through the logic again a couple of days ago and I think we're satisfied that we've cloed the holes | 03:31 |
| mordred | it's set in some places - it was supposed to also skip the mirror if there was a change to openstack/requirements - but that part was broken | 03:31 |
| mordred | DEVSTACK_GATE_REQS_INTEGRATION is set on the requirements gate job | 03:31 |
| clarkb | right which isn't the problem | 03:32 |
| clarkb | the problem is that django 1.6 broke tempest | 03:32 |
| clarkb | so we need to make sure we don't use the mirror on those jobs either and I can't figure out how we are doing that today | 03:32 |
| mordred | if the change is to openstack/requirements | 03:33 |
| mordred | then the new select mirror logic | 03:33 |
| clarkb | is that used by d-g? | 03:33 |
| mordred | should skip using the mirror for ALL of the tempest tests - not just the requirements jobs | 03:33 |
| mordred | it is now | 03:33 |
| clarkb | where? I can't find it | 03:34 |
| *** jcooley_ has quit IRC | 03:34 | |
| mordred | heh | 03:35 |
| mordred | it's not merged | 03:35 |
| mordred | https://review.openstack.org/#/c/48064/ | 03:35 |
| mordred | we were working towards merging it when things fell apart | 03:35 |
| clarkb | ok I remember that change now | 03:36 |
| *** amotoki_ has quit IRC | 03:37 | |
| *** gyee has quit IRC | 03:37 | |
| *** wenlock has quit IRC | 03:37 | |
| clarkb | it is starred now, will try to keep an eye on it as it ties into this whole thing | 03:37 |
| mordred | actually - I think it's fair game - it's not tied to any of the wheel things | 03:38 |
| mordred | clarkb: we need to land https://review.openstack.org/#/c/58153/ | 03:38 |
| mordred | and then we can ask jeblair to remove his -2 | 03:39 |
| mordred | orthogonal issue | 03:39 |
| clarkb | looking | 03:39 |
| clarkb | doesn't that assume all slaves are built by nodepool? | 03:40 |
| clarkb | I think that will break on our regular slaves won't it? | 03:40 |
| mordred | nope - it just needs slaves to have jenkins_slave puppet module on them | 03:41 |
| *** julim has joined #openstack-infra | 03:41 | |
| *** nati_ueno has joined #openstack-infra | 03:41 | |
| mordred | so that they have /opt/requirements | 03:41 |
| mordred | or? | 03:41 |
| clarkb | oh that is a special case of the generic ndoepool stuff | 03:42 |
| mordred | it's just in the base puppet module | 03:42 |
| mordred | https://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/manifests/slave.pp#n310 | 03:43 |
| mordred | so all of our jenkins slaves should have the requirmeents repo now | 03:43 |
| *** sarob has quit IRC | 03:43 | |
| clarkb | yup, just noticed a small problem with that script. distro is used in the else but set in the if | 03:44 |
| clarkb | mordred: I can't babysit 58153 now. I can +2 it if you want to babysit | 03:44 |
| mordred | clarkb: it's 5:45AM where I am - I think me babysitting it is a bad idea | 03:45 |
| *** julim has quit IRC | 03:45 | |
| StevenK | Depends if you're yet to go to bed, or if you just woke | 03:46 |
| clarkb | mordred: np, I will put it in tomorrow | 03:46 |
| fungi | for me, it's both ;) | 03:46 |
| clarkb | also I see that the distro stuff was reverted, will just need to get it right second time around | 03:46 |
| StevenK | fungi: You just woke up at your desk? | 03:47 |
| fungi | StevenK: no, my desk just went to sleep | 03:47 |
| mordred | StevenK: both of those things are true | 03:47 |
| mordred | StevenK: except bed was two plane flights | 03:47 |
| mordred | fungi: haha | 03:48 |
| fungi | best^Wworst beds evar | 03:48 |
| StevenK | Haha | 03:48 |
| clarkb | mordred: can you respond to jeblair on the other change? | 03:48 |
| clarkb | mordred: so that I can convince him to remove the -2 tomorrow | 03:48 |
| StevenK | fungi: They're probably excellent at the pointy end of the plane, but who has the money for that | 03:48 |
| fungi | i saw the pointy end of a plane once. usually they make me avert my eyes as i pass by | 03:49 |
| *** dkranz has joined #openstack-infra | 03:49 | |
| clarkb | in other news it is really really cold in the northwest | 03:49 |
| clarkb | if all of seattle and portland disappears for a few days we probably froze | 03:49 |
| StevenK | I've had to deplane an A380 via first class, that was rather upsetting | 03:50 |
| mordred | clarkb: yup. once | 03:51 |
| mordred | https://review.openstack.org/58153 | 03:51 |
| mordred | is landed, we can recheck that one and verify that it does the right thing | 03:51 |
| *** nati_uen_ has joined #openstack-infra | 03:54 | |
| zaro | anybody have regular expression mojo? need some help. | 03:57 |
| *** nati_ueno has quit IRC | 03:57 | |
| clarkb | zaro: you will enjoy http://regexcrossword.com/ | 03:58 |
| fungi | zaro: i normally have regex mojo, but it's probably fairly tapped out at this hour. what's the puzzle? | 03:58 |
| *** guohliu has quit IRC | 03:59 | |
| zaro | fungi: probably not to hard for you then. | 03:59 |
| zaro | modules/jenkins/manifests/slave.pp | 04:00 |
| zaro | https://review.openstack.org/#/c/60348/6/modules/openstack_project/files/zuul/layout.yaml | 04:00 |
| zaro | ^ i want to express all branches except for < 2.4 | 04:00 |
| fungi | looking | 04:01 |
| clarkb | you want negative lookahead like on line 154 | 04:01 |
| fungi | yes, what clarkb said with his fancy words | 04:02 |
| clarkb | ^(?!openstack/2\.[0-4]).*$ | 04:02 |
| fungi | right. what you had instead would have matched all 2.x branches except 2.0-2.4 | 04:04 |
| zaro | hmm. i think i tried that on http://regexpal.com but didn't seem to match right. | 04:04 |
| fungi | though will there be an openstack/2.5.1 branch? how about an openstack/2.40 branch? | 04:06 |
| *** sarob has joined #openstack-infra | 04:06 | |
| clarkb | zaro: don't try it on some website. Instead fire up a python vm and try it there (same regex lib) | 04:06 |
| fungi | openstack/2.10 would be a more reasonable probability | 04:07 |
| clarkb | fungi: good point ^(?!openstack/2\.[0-4]\.).*$ | 04:07 |
| sarob | im looking for a quickie review | 04:07 |
| sarob | https://review.openstack.org/#/c/60394/ | 04:07 |
| zaro | ok. i guess that's my problem. i'll try the python vm. | 04:07 |
| zaro | thx again. | 04:08 |
| fungi | clarkb: though that one won't reject one without a trailing point version now | 04:08 |
| clarkb | fungi: gah | 04:08 |
| clarkb | fungi: good point ^(?!openstack/2\.[0-4]\.?).*$ | 04:08 |
| fungi | heh | 04:08 |
| clarkb | wait that doesn't work | 04:09 |
| fungi | nope | 04:09 |
| zaro | i think one of the problems iwth that is that it will match on something like openstack/2.25 ? | 04:09 |
| fungi | zaro: or even 2.10, which we'd hit much earlier | 04:09 |
| zaro | oh yeah. | 04:09 |
| clarkb | fungi: good point ^(?!openstack/2\.[0-4][^0-9]).*$ | 04:09 |
| clarkb | does [^0-9] match null string? | 04:09 |
| fungi | (since they're already almost to 2.9, 2.10 can't be too far away) | 04:09 |
| fungi | clarkb: no, not unless you add ? | 04:10 |
| zaro | isn't what we want just regex that says any value less than 4? | 04:10 |
| fungi | at which point you're back to the previous problem | 04:10 |
| fungi | zaro: regular expressions (not counting ios bgp prefix expression language) doesn't do integer comparison | 04:11 |
| clarkb | zaro: regexes don't really know less than | 04:11 |
| zaro | Argg! that's right, that was the real problem i was having. | 04:11 |
| fungi | yeah, it's character comparisons | 04:12 |
| fungi | you might just be better off blacklisting the branches you want to avoid | 04:12 |
| clarkb | yah I think that might be easiest to understand | 04:12 |
| fungi | though the potential problem is that they'll add some in between | 04:13 |
| fungi | but that can be covered with a pattern containing an explicit . | 04:13 |
| fungi | it'll be repetitive, but less voodoo | 04:13 |
| clarkb | right so you can use what I have above, then also explicitly list what we currently have | 04:14 |
| clarkb | er maybe that doesn't work | 04:15 |
| fungi | something like ^(?!openstack/2\.(0(|\.[0-9])|1(|\.[0-9])|2(|\.[0-9])|3(|\.[0-9])|4(|\.[0-9]))).*$ | 04:17 |
| fungi | er, without the .* | 04:17 |
| zaro | ahh fungi. you have the foo! | 04:17 |
| clarkb | or maybe shove the logic in the job | 04:17 |
| *** yamahata_ has quit IRC | 04:17 | |
| clarkb | zaro: does buck use pom.xml? | 04:17 |
| fungi | something like ^(?!openstack/2\.(0(|\.[0-9]*)|1(|\.[0-9]*)|2(|\.[0-9]*)|3(|\.[0-9]*)|4(|\.[0-9]*)))$ | 04:18 |
| zaro | i don't think it does. | 04:18 |
| clarkb | if not then if [ -f pom.xml ] ; then mvn ; else buck fi | 04:18 |
| zaro | it uses BUCK scripts | 04:18 |
| zaro | i think fungi already came up with a good regex. | 04:18 |
| clarkb | right, but do you want to remember what that does in 6 months? | 04:19 |
| fungi | it's definitely suffering from leaning toothpic syndrome | 04:19 |
| fungi | however, you might be overthinking this. we'll only run jobs when people propose or merge changes to these branches | 04:21 |
| clarkb | (I am not sure how ugle one job would be so the regex is definitely an option, just wanted to throw that alternative out there) | 04:21 |
| *** ^d has joined #openstack-infra | 04:21 | |
| fungi | at the moment, i can't see any reason for anyone to propose another change on anything prior to the latest 2.4.x we're using this moment | 04:21 |
| zaro | ok. need to take a break. kids need me to read to them. | 04:21 |
| fungi | night zaro! | 04:22 |
| *** nati_uen_ has quit IRC | 04:22 | |
| *** senk has joined #openstack-infra | 04:23 | |
| *** rcleere has joined #openstack-infra | 04:24 | |
| *** rcleere has quit IRC | 04:25 | |
| *** sdake_ has quit IRC | 04:25 | |
| *** senk has quit IRC | 04:28 | |
| *** senk has joined #openstack-infra | 04:28 | |
| openstackgerrit | Sean Roberts proposed a change to openstack-infra/config: config changes to create the stackforge project milk https://review.openstack.org/60394 | 04:42 |
| sarob | clarkb: more info in the commit and alpha order | 04:43 |
| clarkb | thanks | 04:44 |
| *** rongze has quit IRC | 04:50 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:54 | |
| *** rongze has joined #openstack-infra | 04:54 | |
| *** jcooley_ has joined #openstack-infra | 04:55 | |
| mordred | I agree with fungi | 04:59 |
| mordred | we might just be able to, you know switch to buck and say screw it | 04:59 |
| *** ArxCruz has quit IRC | 05:00 | |
| fungi | yeah, i similarly don't see us proposing new 2.4.x changes once we have working 2.8 or later in production | 05:01 |
| mordred | or recently | 05:01 |
| clarkb | mordred: did you get to watch the MNF game? | 05:02 |
| fungi | (until then we probably want to keep 2.4 buildable so we can rebase on security updates) | 05:02 |
| mordred | clarkb: yah. it wasn't as happy making forus | 05:04 |
| mordred | fungi: bah | 05:04 |
| mordred | fungi: you're making it sound like upstream is going to release security fixes for 2.4 | 05:04 |
| fungi | fie on security. fie i say | 05:04 |
| *** ljjjustin has joined #openstack-infra | 05:07 | |
| mordred | IT IS NOT AS WARM IN TEL AVIV RIGHT NOW AS I WANT IT TO BE | 05:07 |
| mordred | but my ocean-front room _is_ very lovely | 05:08 |
| StevenK | mordred: It's 24degC in Sydney, want that? | 05:08 |
| mordred | StevenK: YES | 05:08 |
| *** paul-- has quit IRC | 05:08 | |
| mordred | StevenK: also, your beaches are very nice | 05:08 |
| StevenK | :-) | 05:08 |
| mordred | although I was just a few days ago at the outer banks ... | 05:11 |
| mordred | fungi: your beaches are also very nice | 05:11 |
| mordred | clarkb: yours are not | 05:11 |
| clarkb | mordred: they are a different kind of nice. good for a fire and chowder | 05:12 |
| clarkb | also EFREEZING right now | 05:12 |
| fungi | i intend to move to my beaches next fall | 05:12 |
| mordred | fungi: nice | 05:12 |
| mordred | clarkb: I'd complain about the EFREEZING here, but somehow I think I'd lose | 05:12 |
| fungi | looking for good waterfront houses facing the sound | 05:13 |
| * StevenK sends mordred a box of Sydney's warmth | 05:13 | |
| mordred | fungi: good luck with that | 05:13 |
| clarkb | mordred: it is just below 0C | 05:13 |
| clarkb | going to drop over night. On saturday we won't get above 0C | 05:13 |
| mordred | clarkb: it's 15C here | 05:13 |
| pleia2 | it's been getting into the 30s F here at night, so much panic ;) | 05:13 |
| pleia2 | lots of instagramming about frost on cars | 05:14 |
| mordred | but what with the wind off the mediterranean and all - it's chilly | 05:14 |
| StevenK | mordred: Just to make you sob, here is the forecast for the new four days: 27degC ; 30degC ; 33degC ; 34degC | 05:14 |
| StevenK | s/new/next/ | 05:14 |
| * mordred shoots StevenK in the head | 05:14 | |
| StevenK | Ow | 05:14 |
| mordred | that's ok - I'll be in Perth ina month | 05:15 |
| pleia2 | StevenK: that's actually kind of unpleasant | 05:15 |
| *** elasticio has joined #openstack-infra | 05:15 | |
| clarkb | we should have a infra sprint in micronesia | 05:15 |
| mordred | clarkb: I would attend that | 05:15 |
| clarkb | then we can all complain about how humid it is | 05:15 |
| pleia2 | clarkb: ++ | 05:15 |
| SergeyLukjanov | heh, -2C atm and -12C should be next week :) | 05:15 |
| StevenK | I'm amused that tripit is already telling me the weather in Perth | 05:15 |
| fungi | there are some good available options in the kill devil hills area, affordable too since that area's been pretty depressed tourism-wise lately | 05:15 |
| StevenK | pleia2: All of them? 27 is wonderful, 30 is okayish, 33 and 34 I can deal with if the humidity is low | 05:16 |
| pleia2 | StevenK: 27 is on the edge of being ok ;) | 05:16 |
| *** esker has quit IRC | 05:16 | |
| StevenK | Hobbsee, meanwhile is happy with anything up to 36 | 05:17 |
| pleia2 | I grew up in walk home from school uphill both ways in snow land | 05:17 |
| StevenK | Walk home from school both ways? :-P | 05:18 |
| fungi | clarkb: count me in for micronesian get-togethers, i love islands | 05:18 |
| fungi | the more remote, the better | 05:18 |
| pleia2 | StevenK: haha, doh, it's been a long week :) | 05:18 |
| clarkb | I love this 25C at midnight with 90% humidity | 05:18 |
| StevenK | I'd love a micronesian get-together, it's closer than the US | 05:18 |
| pleia2 | yes, equally painful for everyone | 05:19 |
| StevenK | Micronesian is only seven hours or something, it's not really long enough to annoy me | 05:19 |
| clarkb | StevenK: you'd probably fly to guam then into micronesia proper | 05:20 |
| pleia2 | ok, time to find some dinner | 05:20 |
| StevenK | Hm, I'm wrong. Guam is 11h30 | 05:20 |
| StevenK | Slightly better than LA, but not by much. | 05:20 |
| *** sarob has quit IRC | 05:21 | |
| clarkb | I like the idea in theory but we would all go crazy at the lack of Internets | 05:21 |
| *** sarob has joined #openstack-infra | 05:21 | |
| *** jcooley_ has quit IRC | 05:22 | |
| StevenK | clarkb: Just bring a fiber vampire tap? I'm sure a submarine cable is close enough ... | 05:22 |
| *** jcooley_ has joined #openstack-infra | 05:22 | |
| clarkb | they have fiber now, but most of the BW apparently belongs to the US military and the RTT is still bad and ugh | 05:23 |
| clarkb | (I was there earlier this year and I didn't haev a good time) | 05:23 |
| clarkb | well I enjoyed my trip. Did not enjoy the internets | 05:23 |
| StevenK | What sort of RTT? | 05:23 |
| * clarkb checks | 05:23 | |
| clarkb | wow they have it down to 161ms now | 05:24 |
| clarkb | it was closer to 300 then | 05:24 |
| clarkb | www.fm is the thing to traceroute. most of that for me currently is seattle to tokyo | 05:24 |
| StevenK | 300 is roughly my RTT to the UK | 05:25 |
| clarkb | when I lived there the connection to the outside world was a satellite. 300ms was one hop into space | 05:25 |
| clarkb | so this is definitely better | 05:25 |
| *** ^d has quit IRC | 05:26 | |
| *** sarob has quit IRC | 05:26 | |
| *** jcooley_ has quit IRC | 05:27 | |
| *** sdake_ has joined #openstack-infra | 05:31 | |
| *** sdake_ has quit IRC | 05:31 | |
| *** sdake_ has joined #openstack-infra | 05:31 | |
| *** chandankumar has joined #openstack-infra | 05:34 | |
| *** nicedice has quit IRC | 05:35 | |
| *** fallenpegasus has joined #openstack-infra | 05:39 | |
| *** DennyZhang has joined #openstack-infra | 05:47 | |
| *** vipul has quit IRC | 05:52 | |
| *** vipul has joined #openstack-infra | 05:53 | |
| *** elasticio has quit IRC | 05:55 | |
| *** fallenpegasus has quit IRC | 06:00 | |
| *** markwash has joined #openstack-infra | 06:02 | |
| *** DennyZhang has quit IRC | 06:03 | |
| *** senk has quit IRC | 06:06 | |
| *** nati_ueno has joined #openstack-infra | 06:08 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 06:11 | |
| *** blamar has quit IRC | 06:12 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 06:22 | |
| *** SergeyLukjanov has quit IRC | 06:24 | |
| *** markwash has quit IRC | 06:25 | |
| *** markwash has joined #openstack-infra | 06:26 | |
| *** vkozhukalov has joined #openstack-infra | 06:26 | |
| *** denis_makogon has joined #openstack-infra | 06:27 | |
| *** amotoki has quit IRC | 06:28 | |
| *** nosnos_ has joined #openstack-infra | 06:30 | |
| *** nosnos has quit IRC | 06:33 | |
| *** pcrews has joined #openstack-infra | 06:38 | |
| *** pblaho has joined #openstack-infra | 06:48 | |
| *** rainya has quit IRC | 06:49 | |
| *** dkliban_ has quit IRC | 06:50 | |
| *** vkozhukalov has quit IRC | 06:51 | |
| *** pcrews has quit IRC | 06:52 | |
| *** boris-42 has joined #openstack-infra | 06:53 | |
| *** senk has joined #openstack-infra | 06:59 | |
| *** denis_makogon has quit IRC | 07:14 | |
| *** andreaf has quit IRC | 07:18 | |
| *** andreaf has joined #openstack-infra | 07:19 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:29 | |
| *** senk has quit IRC | 07:32 | |
| *** WarrenUsui has joined #openstack-infra | 07:45 | |
| *** aardvark has quit IRC | 07:48 | |
| *** afazekas_ has joined #openstack-infra | 07:49 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added basic popup messages https://review.openstack.org/59706 | 07:50 |
| *** nati_uen_ has joined #openstack-infra | 07:54 | |
| *** jcoufal has joined #openstack-infra | 07:54 | |
| *** flaper87|afk is now known as flaper87 | 07:56 | |
| *** senk has joined #openstack-infra | 07:57 | |
| *** nati_ueno has quit IRC | 07:58 | |
| *** markwash has quit IRC | 08:00 | |
| *** senk1 has joined #openstack-infra | 08:00 | |
| *** senk has quit IRC | 08:02 | |
| *** osanchez has joined #openstack-infra | 08:07 | |
| *** markwash has joined #openstack-infra | 08:12 | |
| *** nati_uen_ has quit IRC | 08:20 | |
| *** markwash has quit IRC | 08:21 | |
| anteaya | a question based on a conversation in -neutron about cpu usage for a devstack test instance | 08:22 |
| anteaya | is cpu usage pegged at 100% all the time? | 08:23 |
| *** senk has joined #openstack-infra | 08:23 | |
| *** nprivalova has joined #openstack-infra | 08:23 | |
| *** senk1 has quit IRC | 08:23 | |
| *** senk has quit IRC | 08:24 | |
| *** locke1051 has quit IRC | 08:25 | |
| *** mrodden has quit IRC | 08:25 | |
| *** beagles has quit IRC | 08:26 | |
| *** bnemec has quit IRC | 08:27 | |
| *** SergeyLukjanov_ has joined #openstack-infra | 08:28 | |
| *** pete5_ has joined #openstack-infra | 08:28 | |
| *** davidlenwell_ has joined #openstack-infra | 08:29 | |
| *** bknudson has quit IRC | 08:29 | |
| *** vkozhukalov has joined #openstack-infra | 08:29 | |
| *** bpokorny has quit IRC | 08:29 | |
| *** beagles has joined #openstack-infra | 08:29 | |
| *** EntropyWorks_ has joined #openstack-infra | 08:30 | |
| openstackgerrit | Max Rydahl Andersen proposed a change to openstack-infra/jenkins-job-builder: add private-repository to maven-target https://review.openstack.org/60161 | 08:30 |
| *** plomakin has joined #openstack-infra | 08:30 | |
| *** hub_cap_ has joined #openstack-infra | 08:31 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack-dsvm https://review.openstack.org/58702 | 08:33 |
| *** esmute_ has joined #openstack-infra | 08:33 | |
| *** mtreinish_ has joined #openstack-infra | 08:34 | |
| *** esmute has quit IRC | 08:34 | |
| *** pete5 has quit IRC | 08:34 | |
| *** lifeless has quit IRC | 08:34 | |
| *** EntropyWorks has quit IRC | 08:34 | |
| *** plomakin_ has quit IRC | 08:34 | |
| *** SergeyLukjanov has quit IRC | 08:34 | |
| *** hub_cap has quit IRC | 08:34 | |
| *** mtreinish has quit IRC | 08:34 | |
| *** davidlenwell has quit IRC | 08:34 | |
| *** mtreinish_ is now known as mtreinish | 08:35 | |
| *** pete5_ is now known as pete5 | 08:35 | |
| *** rongze_ has joined #openstack-infra | 08:35 | |
| *** EntropyWorks_ is now known as EntropyWorks | 08:35 | |
| *** esmute_ is now known as esmute | 08:35 | |
| *** lifeless has joined #openstack-infra | 08:36 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add manila repos to PROJECTS in d-g job https://review.openstack.org/60442 | 08:36 |
| *** fifieldt_ has quit IRC | 08:36 | |
| *** rongze has quit IRC | 08:37 | |
| *** ljjjustin has quit IRC | 08:37 | |
| *** ljjjustin has joined #openstack-infra | 08:39 | |
| *** shardy_afk is now known as shardy | 08:44 | |
| *** sjing has quit IRC | 08:44 | |
| *** yolanda has joined #openstack-infra | 08:52 | |
| *** jpich has joined #openstack-infra | 09:11 | |
| *** yassine has joined #openstack-infra | 09:13 | |
| *** ljjjustin has quit IRC | 09:27 | |
| *** nosnos_ has quit IRC | 09:29 | |
| *** nosnos has joined #openstack-infra | 09:29 | |
| *** boris-42 has quit IRC | 09:29 | |
| *** bingbu has quit IRC | 09:37 | |
| *** sdake_ has quit IRC | 09:43 | |
| *** HenryG has quit IRC | 09:47 | |
| *** HenryG has joined #openstack-infra | 09:47 | |
| *** klrmn has quit IRC | 09:51 | |
| *** hdd has quit IRC | 09:51 | |
| *** echohead has quit IRC | 09:51 | |
| *** echohead has joined #openstack-infra | 09:51 | |
| *** cody-somerville has quit IRC | 09:51 | |
| *** klrmn1 has joined #openstack-infra | 09:51 | |
| *** hdd has joined #openstack-infra | 09:51 | |
| *** klrmn1 is now known as klrmn | 09:51 | |
| *** cody-somerville has joined #openstack-infra | 09:51 | |
| *** senk has joined #openstack-infra | 09:54 | |
| *** senk has quit IRC | 09:59 | |
| *** Ryan_Lane has joined #openstack-infra | 09:59 | |
| *** senk has joined #openstack-infra | 09:59 | |
| *** hashar has joined #openstack-infra | 10:00 | |
| *** fbo_away is now known as fbo | 10:09 | |
| *** DinaBelova has joined #openstack-infra | 10:14 | |
| *** nprivalova has quit IRC | 10:21 | |
| *** Ryan_Lane has quit IRC | 10:21 | |
| *** DinaBelova has quit IRC | 10:25 | |
| *** senk has quit IRC | 10:26 | |
| *** salv-orlando has joined #openstack-infra | 10:27 | |
| *** SergeyLukjanov_ has quit IRC | 10:32 | |
| *** johnthetubaguy has joined #openstack-infra | 10:37 | |
| *** che-arne has joined #openstack-infra | 10:44 | |
| *** nosnos_ has joined #openstack-infra | 10:44 | |
| *** nprivalova has joined #openstack-infra | 10:46 | |
| *** nosnos has quit IRC | 10:48 | |
| openstackgerrit | Roman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic https://review.openstack.org/53917 | 10:55 |
| *** senk has joined #openstack-infra | 11:02 | |
| *** heyongli has joined #openstack-infra | 11:08 | |
| *** senk has quit IRC | 11:09 | |
| *** senk has joined #openstack-infra | 11:10 | |
| *** senk has quit IRC | 11:15 | |
| *** rongze_ has quit IRC | 11:17 | |
| *** rfolco has joined #openstack-infra | 11:19 | |
| *** mkerrin has quit IRC | 11:20 | |
| *** mkerrin has joined #openstack-infra | 11:21 | |
| *** yamahata_ has joined #openstack-infra | 11:21 | |
| *** wenlock has joined #openstack-infra | 11:26 | |
| *** rongze has joined #openstack-infra | 11:28 | |
| *** ArxCruz has joined #openstack-infra | 11:29 | |
| *** markmc has joined #openstack-infra | 11:32 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Neutron support in Grenade jobs https://review.openstack.org/59039 | 11:36 |
| *** fallenpegasus has joined #openstack-infra | 11:37 | |
| EmilienM | fungi: i hope i did well ^ | 11:37 |
| *** yamahata_ has quit IRC | 11:48 | |
| *** fallenpegasus has quit IRC | 11:49 | |
| *** adalbas has quit IRC | 11:51 | |
| *** senk has joined #openstack-infra | 12:07 | |
| *** bpokorny has joined #openstack-infra | 12:07 | |
| *** locke105 has joined #openstack-infra | 12:08 | |
| *** senk1 has joined #openstack-infra | 12:09 | |
| *** senk has quit IRC | 12:11 | |
| *** bknudson has joined #openstack-infra | 12:12 | |
| *** nati_ueno has joined #openstack-infra | 12:14 | |
| *** senk1 has quit IRC | 12:16 | |
| *** boris-42 has joined #openstack-infra | 12:16 | |
| *** pblaho has quit IRC | 12:18 | |
| *** pblaho has joined #openstack-infra | 12:18 | |
| HenryG | Noticed yesterday after updating devstack that git.openstack.org is *much* slower than github.com for cloning repos | 12:22 |
| sdague | HenryG: can you build some timing numbers on that? (also location from where you are cloning would be useful) | 12:30 |
| sdague | if we can quantify it better we can figure out options | 12:30 |
| *** mrodden has joined #openstack-infra | 12:31 | |
| BobBall | Anyone seen beam.smp using >2G ram? It's only started to happen recently and I'm trying to work out what's changed to cause it | 12:32 |
| HenryG | sdague: what timing are you looking for? time to clone, say, nova? | 12:34 |
| sdague | HenryG: sure, honestly it would be useful to get it across the collection of repos that get cloned | 12:35 |
| HenryG | sdague: is that list of repos easily obtainable? | 12:37 |
| sdague | HenryG: it's in the devstack source | 12:37 |
| * HenryG looks ... | 12:37 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/devstack-gate: Adding an option to use qpid instead of rabbit or zeromq https://review.openstack.org/55829 | 12:40 |
| *** vkozhukalov has quit IRC | 12:41 | |
| *** vkozhukalov has joined #openstack-infra | 12:45 | |
| *** boris-42 has quit IRC | 12:47 | |
| *** adalbas has joined #openstack-infra | 12:48 | |
| *** nati_ueno has quit IRC | 12:49 | |
| *** dcramer_ has quit IRC | 12:51 | |
| *** mriedem has joined #openstack-infra | 12:54 | |
| HenryG | sdague: looks like git.openstack.org is faster than github today. The slowness must have been temporary | 12:56 |
| *** boris-42 has joined #openstack-infra | 12:56 | |
| sdague | HenryG: cool | 12:57 |
| sdague | zuul is pretty quiet at the moment as well. It would be useful to figure out if it turns out that we are overloading the system during heavy zuul times | 12:58 |
| sdague | so if you had something you could rerun periodically to see if it changes, that would be good | 12:58 |
| *** yamahata_ has joined #openstack-infra | 12:59 | |
| HenryG | I have a couple of servers for devstack. I could set up something periodic, but no promises on when I'd get that done. | 13:01 |
| *** DinaBelova has joined #openstack-infra | 13:04 | |
| *** salv-orlando has quit IRC | 13:06 | |
| sdague | yeh, even something manual so if you see it return, we can narrow it down | 13:07 |
| *** salv-orlando has joined #openstack-infra | 13:07 | |
| *** weshay has joined #openstack-infra | 13:09 | |
| HenryG | might also be a good idea to periodically log the cpu/disk/net load on the server and graph it together with the zuul graphs? | 13:09 |
| *** boris-42 has quit IRC | 13:09 | |
| *** beagles has quit IRC | 13:11 | |
| *** nosnos_ has quit IRC | 13:11 | |
| *** senk has joined #openstack-infra | 13:16 | |
| *** senk has quit IRC | 13:16 | |
| *** senk has joined #openstack-infra | 13:17 | |
| *** senk has quit IRC | 13:21 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Run tests and add the ability to release for tooz https://review.openstack.org/60504 | 13:22 |
| *** loq_mac has joined #openstack-infra | 13:23 | |
| *** loq_mac has quit IRC | 13:25 | |
| *** beagles has joined #openstack-infra | 13:26 | |
| *** bpokorny has quit IRC | 13:27 | |
| *** prad has joined #openstack-infra | 13:29 | |
| *** heyongli has quit IRC | 13:40 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add e-r query for bug 1257644 https://review.openstack.org/60508 | 13:41 |
| uvirtbot | Launchpad bug 1257644 in nova "gate-tempest-dsvm-postgres-full fails - unable to schedule instance" [Critical,Triaged] https://launchpad.net/bugs/1257644 | 13:41 |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Install ZooKeeper on slaves https://review.openstack.org/60509 | 13:41 |
| *** mindjive1 is now known as mindjiver | 13:42 | |
| *** bpokorny has joined #openstack-infra | 13:45 | |
| *** tma996 has joined #openstack-infra | 13:50 | |
| *** dprince has joined #openstack-infra | 13:52 | |
| *** beagles is now known as beagles_brb | 14:06 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:07 | |
| *** senk has joined #openstack-infra | 14:07 | |
| *** esker has joined #openstack-infra | 14:10 | |
| openstackgerrit | Darragh O'Reilly proposed a change to openstack-infra/os-loganalyze: Provide a script that starts a simple WSGI server https://review.openstack.org/60325 | 14:11 |
| *** mfer has joined #openstack-infra | 14:12 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Stories and Tasks search https://review.openstack.org/60515 | 14:12 |
| *** dkliban_ has joined #openstack-infra | 14:12 | |
| *** dcramer_ has joined #openstack-infra | 14:14 | |
| *** senk has quit IRC | 14:16 | |
| *** senk has joined #openstack-infra | 14:16 | |
| *** thomasem has joined #openstack-infra | 14:17 | |
| *** weshay has quit IRC | 14:18 | |
| jog0 | notmyname: yes, that is on my list of todo's biggest blocker is don't know how to get a single datapoint in graphite yet | 14:18 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add e-r query for bug 1257644 https://review.openstack.org/60508 | 14:18 |
| uvirtbot | Launchpad bug 1257644 in nova "gate-tempest-dsvm-postgres-full fails - unable to schedule instance" [Critical,Triaged] https://launchpad.net/bugs/1257644 | 14:19 |
| jog0 | clarkb: how hard would it be to get warn level logs in elasticsearch | 14:19 |
| jog0 | clarkb: by that I mean they would really really really help | 14:19 |
| jog0 | https://review.openstack.org/#/c/60508/ | 14:19 |
| *** DinaBelova has quit IRC | 14:23 | |
| *** DinaBelova has joined #openstack-infra | 14:26 | |
| sdague | so we think all the rax nodes are performance nodes now? | 14:27 |
| jog0 | fingured out how to get the raw data to get failure rate from graphite | 14:28 |
| jog0 | http://paste.openstack.org/show/54592/ | 14:28 |
| sdague | jog0: we have warn in es | 14:28 |
| sdague | we don't have debug | 14:28 |
| jog0 | sdague: I couldn't find it | 14:28 |
| jog0 | ohh its warning | 14:29 |
| jog0 | lol | 14:29 |
| jog0 | sdague: I won't be able to get a patch out for showing the failure rate using graphite today | 14:31 |
| sdague | http://ubuntuone.com/2ZhczqLoUjkfs6CiLofc6N | 14:31 |
| jog0 | so if you want to turn that paste into something that runs on statu.o.o/e-r | 14:31 |
| sdague | ok, I'll see if I can get to it | 14:31 |
| jog0 | thanks | 14:32 |
| sdague | i've been pushing off some paper work till today, so I think I have to tackle that first | 14:32 |
| *** rnirmal has joined #openstack-infra | 14:32 | |
| jog0 | the graphite graph from the e-r repo version has only gate in it | 14:32 |
| jog0 | but the logic is the same | 14:32 |
| sdague | anyway, the above graph seems to show the rax cloud is now slightly slower than hp cloud, with massively more variance in performance | 14:32 |
| jog0 | that variance is scary | 14:33 |
| sdague | that is successful runs only out of the check queue | 14:33 |
| sdague | of tempest full | 14:33 |
| sdague | https://jenkins01.openstack.org/job/check-tempest-dsvm-full/buildTimeTrend - throw out all fails | 14:34 |
| jog0 | wow 30-60 minutes big variance | 14:35 |
| sdague | actually hp az2 is pretty crazy as well | 14:35 |
| jog0 | yeah I wonder if thats the new one | 14:35 |
| jog0 | sdague: anyway have to go AFK (its almost 5here) | 14:36 |
| jog0 | o/ | 14:36 |
| sdague | enjoy | 14:37 |
| *** russellb is now known as rustlebee | 14:38 | |
| *** dcramer_ has quit IRC | 14:42 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Stories and Tasks search https://review.openstack.org/60515 | 14:43 |
| *** bnemec has joined #openstack-infra | 14:45 | |
| *** pabelanger_ has joined #openstack-infra | 14:46 | |
| *** pabelanger_ has quit IRC | 14:46 | |
| *** pabelanger_ has joined #openstack-infra | 14:46 | |
| *** DinaBelova has quit IRC | 14:46 | |
| *** pabelanger has quit IRC | 14:46 | |
| *** pabelanger_ is now known as pabelanger | 14:46 | |
| *** pabelanger_ has joined #openstack-infra | 14:47 | |
| *** DinaBelova has joined #openstack-infra | 14:48 | |
| *** pabelanger has quit IRC | 14:48 | |
| *** pabelanger has joined #openstack-infra | 14:48 | |
| *** mkerrin has quit IRC | 14:49 | |
| *** loq_mac has joined #openstack-infra | 14:50 | |
| fungi | HenryG: we also have graphs of resource utilization for the servers in the git farm... http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=43 http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=47 http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=46 | 14:50 |
| fungi | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=45 http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=44 | 14:50 |
| fungi | the first url is the haproxy load balancer, and the other four are the git servers themselves | 14:51 |
| fungi | so if you have specific times you notice it's slow, there's a good chance we can figure out what the bottleneck is (so long as it's in our systems) | 14:51 |
| *** loq_mac has quit IRC | 14:52 | |
| *** senk has quit IRC | 14:52 | |
| *** jergerber has joined #openstack-infra | 14:52 | |
| HenryG | Thanks, I'll check these when/if we notice slowness again. | 14:53 |
| *** mestery_ has joined #openstack-infra | 14:53 | |
| *** chandankumar has quit IRC | 14:53 | |
| *** boris-42 has joined #openstack-infra | 14:54 | |
| fungi | i look forward to the day when barbican devs stop referring to commits in code review as "pull requests" and git repositories as "github" | 14:56 |
| *** mestery has quit IRC | 14:57 | |
| *** jcoufal has quit IRC | 14:57 | |
| *** mestery has joined #openstack-infra | 14:57 | |
| *** weshay has joined #openstack-infra | 14:58 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Stories and Tasks search https://review.openstack.org/60515 | 14:58 |
| *** mestery_ has quit IRC | 14:58 | |
| *** rongze has quit IRC | 15:00 | |
| *** mestery has quit IRC | 15:05 | |
| *** ruhe has joined #openstack-infra | 15:06 | |
| *** dansmith is now known as damnsmith | 15:07 | |
| *** vkozhukalov has quit IRC | 15:12 | |
| *** mestery has joined #openstack-infra | 15:13 | |
| *** rongze has joined #openstack-infra | 15:16 | |
| *** pblaho has quit IRC | 15:18 | |
| *** gfidente has joined #openstack-infra | 15:20 | |
| *** ^d has joined #openstack-infra | 15:24 | |
| *** bnemec is now known as beekneemech | 15:25 | |
| *** boris-42 has quit IRC | 15:25 | |
| *** gfidente is now known as giulivo | 15:26 | |
| *** markwash has joined #openstack-infra | 15:26 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Stories and Tasks search https://review.openstack.org/60515 | 15:26 |
| mtreinish | fungi: found an interesting failure: http://logs.openstack.org/79/60379/1/check/gate-tempest-dsvm-neutron-large-ops/70d71d4/console.html#_2013-12-05_23_28_56_056 | 15:27 |
| mtreinish | it looks like subunit hit an exception when importing a test file. My guess is that a dep didn't get correctly installed | 15:28 |
| pleia2 | flaper87: when you have a moment, can you see https://review.openstack.org/#/c/60240/ to get channel permissions sorted for us? | 15:29 |
| pleia2 | flaper87: mmagr tried, but doesn't have enough access in the channel | 15:29 |
| fungi | mtreinish: that looks sorta like one clarkb and lifeless were talking about earlier in the week | 15:29 |
| *** boris-42 has joined #openstack-infra | 15:30 | |
| *** boris-42 has quit IRC | 15:30 | |
| fungi | mtreinish: i'm wondering whether that's discover catching an exception when trying to import some module | 15:30 |
| fungi | used to trip up nose anyway | 15:30 |
| mtreinish | fungi: that's what the error has been in my experience (I've hit it locally when writing tests) | 15:30 |
| flaper87 | pleia2: done! | 15:31 |
| * flaper87 completely forgot he was the owner of that channel | 15:31 | |
| flaper87 | :D | 15:31 |
| pleia2 | flaper87: hm, still now showing up in the access list /msg chanserv access #packstack-dev list | 15:31 |
| pleia2 | flaper87: s/now/not | 15:32 |
| mtreinish | fungi: I opened a bug about it not catching errors on import before (we ended up skipping broken tests in tempest because subunit didn't error out and just changed the test name) | 15:32 |
| mtreinish | fungi: https://bazaar.launchpad.net/~subunit/subunit/trunk/revision/231 | 15:32 |
| fungi | oh, yeah i remember that now | 15:33 |
| pleia2 | flaper87: you want this command (chanserv should confirm addition): /msg chanserv access #packstack-dev add openstackinfra +AFRfiorstv | 15:33 |
| flaper87 | pleia2: yeah, that's the one I'm running, but it just said I'm not authorized | 15:33 |
| pleia2 | hrm | 15:34 |
| mtreinish | fungi: I'm just wondering what could have caused the import error there, because it passed all the other jenkins jobs (just not the large ops ones) | 15:34 |
| pleia2 | flaper87: you are the channel founder, and you're identified with nickserv, very odd | 15:34 |
| pleia2 | flaper87: sure you're using #packstack-dev and not #channel as in my (poor) example? | 15:35 |
| flaper87 | pleia2: done, there was a typo in the channel name | 15:35 |
| pleia2 | hurrah! | 15:35 |
| pleia2 | flaper87: looks good, thank you :) | 15:35 |
| flaper87 | pleia2: yeah, It just looks like I don't know how to write 'dev' | 15:35 |
| pleia2 | hehe | 15:35 |
| sdague | fungi: so do you know for sure that we're on rax performance nodes on all the rax stuff? | 15:35 |
| *** datsun180b has joined #openstack-infra | 15:36 | |
| fungi | sdague: unless the ephemeral disk modifications for devstack-gate landed yet, then we're not using them no | 15:37 |
| * fungi checks | 15:37 | |
| sdague | fungi: ok | 15:37 |
| fungi | sdague: https://review.openstack.org/#/c/60347/ | 15:38 |
| fungi | pretty sure without that we can't use them, because the root filesystem is too small | 15:38 |
| fungi | similar to hpcloud | 15:38 |
| *** vkozhukalov has joined #openstack-infra | 15:40 | |
| fungi | sdague: but jeblair ran some baselines by hand and the numbers look favorable. the same or slightly faster than hpcloud nodes (from the old region not the newer region), and well faster than our current rackspace slaves | 15:40 |
| sdague | cool | 15:41 |
| *** fbo has quit IRC | 15:42 | |
| *** fbo has joined #openstack-infra | 15:42 | |
| jeblair | oh, no, we're using performance for all rax nodes | 15:45 |
| jeblair | we're just not using them in gate yet | 15:45 |
| jeblair | sdague, fungi: ^ | 15:45 |
| mordred | oh neat | 15:45 |
| jeblair | mordred, fungi, zaro: i don't want to be backed into a corner with gerrit -- we need to still be able to build 2.4. new jobs or jobs that check for pom.xml are fine either way. | 15:45 |
| jeblair | mordred, fungi, zaro: i don't think we need to handle new <2.8 branches though, i don't expect to add any of those. | 15:45 |
| fungi | jeblair: oh! excellent | 15:46 |
| mordred | jeblair: ++ on gerrit - sorry, it was late, I was being snarky about security from upstream, not actually suggesting we lose the ability to build 2.4 | 15:46 |
| mordred | :) | 15:46 |
| jeblair | mordred: ok. :) we _have_ patched 2.4 with security updates twice since we thought they 'stopped supporting it'. one was actually a 2.4 patch, the other was a backport. | 15:47 |
| fungi | jeblair: right, i agree 2.4.x equal to or later than 2.4.4 needs to be buildable as well as 2.8 and later | 15:47 |
| jeblair | fungi: yep, that's the main thing. | 15:47 |
| *** SushilKM has joined #openstack-infra | 15:48 | |
| *** herndon has joined #openstack-infra | 15:49 | |
| sdague | jeblair: ok | 15:49 |
| *** beagles_brb is now known as beagles | 15:49 | |
| sdague | so the perf nodes are better, but still pretty variant | 15:49 |
| mikal | What instance size are you using? | 15:50 |
| mikal | Is there some data I can point people at? | 15:50 |
| jeblair | mikal: 8g | 15:50 |
| *** SushilKM has quit IRC | 15:50 | |
| jeblair | mikal, sdague: i'm dusting off an old graph, give me a few mins | 15:50 |
| mikal | jeblair: sure, NP | 15:50 |
| sdague | http://ubuntuone.com/2ZhczqLoUjkfs6CiLofc6N | 15:50 |
| sdague | that I scraped the morning | 15:51 |
| mikal | jeblair: I'm finishing up exposing logs for TH test runs now, because I care about your opinion | 15:51 |
| sdague | thats tempest full runs from the check queue, which succeeded | 15:51 |
| *** sandy__ has quit IRC | 15:51 | |
| jeblair | mikal: you bring tears to my eyes | 15:51 |
| mikal | LOL | 15:52 |
| *** oubiwann_ has joined #openstack-infra | 15:56 | |
| *** dims has quit IRC | 15:58 | |
| jeblair | mikal, sdague: well, crap. i lost the runtime stats when moving to nodepool. i'll work on putting them back. sorry. | 15:59 |
| mordred | moar graphs | 16:00 |
| *** oubiwann_ has quit IRC | 16:02 | |
| *** adalbas has quit IRC | 16:03 | |
| *** dims has joined #openstack-infra | 16:03 | |
| hashar | we had nice graphs related to Nelson Mandella deaths yesterday | 16:03 |
| *** andreaf has quit IRC | 16:04 | |
| *** adalbas has joined #openstack-infra | 16:05 | |
| hashar | http://imgur.com/NtJXGvg http req/s there is a nice spike at 22:00 UTC | 16:05 |
| *** andreaf has joined #openstack-infra | 16:07 | |
| *** jcooley_ has joined #openstack-infra | 16:10 | |
| *** adalbas has quit IRC | 16:11 | |
| *** mrodden has quit IRC | 16:11 | |
| openstackgerrit | Denis M. proposed a change to openstack/requirements: Update python-troveclient version https://review.openstack.org/60553 | 16:11 |
| *** pcrews has joined #openstack-infra | 16:17 | |
| *** sdake has quit IRC | 16:24 | |
| *** sdake has joined #openstack-infra | 16:24 | |
| *** hub_cap_ is now known as hub_cap | 16:25 | |
| *** adalbas has joined #openstack-infra | 16:25 | |
| *** Abhishek has joined #openstack-infra | 16:26 | |
| *** sdake has quit IRC | 16:27 | |
| *** sdake has joined #openstack-infra | 16:27 | |
| *** sdake has joined #openstack-infra | 16:27 | |
| *** mrodden has joined #openstack-infra | 16:29 | |
| *** tma996 has quit IRC | 16:30 | |
| *** fallenpegasus has joined #openstack-infra | 16:31 | |
| *** weshay has quit IRC | 16:32 | |
| mriedem | anyone know which version of libvirt we're running in the gate? | 16:34 |
| mriedem | >= 1.0.1? | 16:34 |
| openstackgerrit | Denis M. proposed a change to openstack/requirements: Update python-troveclient version https://review.openstack.org/60553 | 16:35 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add job runtime stats https://review.openstack.org/60557 | 16:36 |
| jeblair | mordred, sdague, mikal: ^ that should give us back the runtime stats | 16:36 |
| mordred | jeblair: I am in support of our new runtime stats overlords | 16:36 |
| sdague | mriedem: it's a 0.9x I think | 16:37 |
| sdague | it will be in the console logs somewhere | 16:37 |
| *** ^d has quit IRC | 16:37 | |
| mordred | jeblair: your patch does 4 things. can you split it in to four patches? (TOTALLY KIDDING) | 16:37 |
| fungi | "your patch has four lines--can you split it into four patches?" | 16:38 |
| mriedem | sdague: yup Setting up libvirt0 (0.9.8-2ubuntu17.16) ... | 16:39 |
| jeblair | hehe | 16:41 |
| *** mgagne has quit IRC | 16:41 | |
| *** yassine has quit IRC | 16:42 | |
| *** markmcclain has joined #openstack-infra | 16:42 | |
| *** mgagne has joined #openstack-infra | 16:43 | |
| *** vkozhukalov has quit IRC | 16:43 | |
| *** Abhishek has quit IRC | 16:46 | |
| *** fallenpegasus has quit IRC | 16:49 | |
| jeblair | mordred: so you're ready to land https://review.openstack.org/#/c/58153/ ? | 16:50 |
| jeblair | mordred: i see clarkb wanted to babysit it, but i think i can get that started | 16:51 |
| *** niska has quit IRC | 16:52 | |
| *** beekneemech has quit IRC | 16:52 | |
| *** SergeyLukjanov has quit IRC | 16:53 | |
| *** guitarzan has quit IRC | 16:53 | |
| *** bnemec has joined #openstack-infra | 16:53 | |
| *** dkliban_ has quit IRC | 16:53 | |
| *** iccha has quit IRC | 16:54 | |
| *** dstanek has quit IRC | 16:54 | |
| *** odyi has quit IRC | 16:54 | |
| *** dkliban_ has joined #openstack-infra | 16:54 | |
| *** iccha has joined #openstack-infra | 16:54 | |
| *** niska has joined #openstack-infra | 16:54 | |
| *** guitarzan has joined #openstack-infra | 16:54 | |
| mordred | jeblair: yes. I see no reason to not move forward with that | 16:55 |
| *** odyi has joined #openstack-infra | 16:55 | |
| mordred | I should send a message to the list about the behavior change | 16:55 |
| openstackgerrit | A change was merged to openstack-infra/config: Use requirements project list for mirror selection https://review.openstack.org/58153 | 16:55 |
| jeblair | mordred: excellent. :) | 16:55 |
| *** DinaBelova has quit IRC | 16:56 | |
| *** sdake has quit IRC | 16:57 | |
| *** sdake has joined #openstack-infra | 16:58 | |
| mordred | woot | 16:58 |
| mordred | so - once that rolls out in the next puppet update, we can probably re-check the other one | 16:59 |
| jeblair | *nod* | 16:59 |
| mordred | jeblair: I'm mildly worried that we have the implementation of this system spread over 4 repos | 16:59 |
| *** sdake has quit IRC | 17:00 | |
| mordred | but I have been unable to conceive of appropriate changes to fix it | 17:00 |
| *** sdake has joined #openstack-infra | 17:00 | |
| *** sdake has quit IRC | 17:00 | |
| *** sdake has joined #openstack-infra | 17:00 | |
| *** talluri has joined #openstack-infra | 17:00 | |
| *** pete5 has quit IRC | 17:00 | |
| *** nati_ueno has joined #openstack-infra | 17:01 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Neutron support in Grenade jobs https://review.openstack.org/59039 | 17:01 |
| *** openstackstatus has quit IRC | 17:03 | |
| *** openstackstatus has joined #openstack-infra | 17:04 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Bump the min ram for HP 1.1 https://review.openstack.org/60561 | 17:04 |
| mordred | jeblair: there's a change to try bumping the ram to see if it helps | 17:04 |
| jeblair | #status log i'm running statusbot in screen to try to catch why it dies after a while. | 17:05 |
| mordred | jeblair: do you think we should run for a while with your new nodepool stats on 8k first to collect good numbers there | 17:05 |
| jeblair | fungi, mordred, clarkb: ^ see status note | 17:05 |
| mordred | jeblair: nod | 17:05 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add job runtime stats https://review.openstack.org/60557 | 17:06 |
| *** rossella_s has joined #openstack-infra | 17:06 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: implementation of FacetSet for client side nested facets https://review.openstack.org/59822 | 17:06 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Remove unecessary debug message https://review.openstack.org/60355 | 17:06 |
| jeblair | mordred: not sure; i'm happy to, but i wasn't around when it was being tested so i'm not familiar with the numbers we do have | 17:06 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: port check_success to facets https://review.openstack.org/59832 | 17:06 |
| *** markwash has quit IRC | 17:07 | |
| mordred | jeblair: the numbers we have are clark saying that the numbers are simliar to the rax nova non-perf nodes | 17:07 |
| mordred | although, in some cases, things ran as much as twice as slow and hit timeouts | 17:07 |
| *** zehicle_at_dell has joined #openstack-infra | 17:07 | |
| fungi | jeblair: now that i've seen it, i can't un-see it. you started it under a shell in screen so that we'll be able to reattach and see the traceback? (obviously if you spawn it directly from screen, screen will terminate when the daemon does) | 17:08 |
| jeblair | fungi: yes | 17:08 |
| *** nicedice has joined #openstack-infra | 17:08 | |
| EmilienM | fungi: sorry, you were very clear. I'm fixing the job. | 17:09 |
| zaro | good morning | 17:09 |
| fungi | mordred: i think in a lot of cases, "slightly slower than rackspace non-performance" is enough to tip a few of those jobs over into timeouts | 17:09 |
| jeblair | mordred: a spot check of a slave indicates select-mirror has been updated | 17:10 |
| *** weshay has joined #openstack-infra | 17:10 | |
| zaro | fungi: will you have time to start creating accounts today? or should we wait until next week? | 17:10 |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Neutron support in Grenade jobs https://review.openstack.org/59039 | 17:10 |
| fungi | EmilienM: no worries. it's a complex system, takes some getting used to | 17:10 |
| *** sandywalsh has joined #openstack-infra | 17:10 | |
| *** jcooley_ has quit IRC | 17:10 | |
| jeblair | mordred: https://jenkins01.openstack.org/job/gate-neutron-python26/4017/console used the new code | 17:10 |
| fungi | zaro: yep, i need about 30 minutes to wrap up the last of the cinder/lvm work on static, and then i'm working on accounts | 17:11 |
| *** jpich has quit IRC | 17:11 | |
| mordred | jeblair: neat. it seems to have worked | 17:12 |
| EmilienM | fungi: thank you for your patience | 17:12 |
| fungi | EmilienM: you're very welcome | 17:12 |
| *** denis_makogon has joined #openstack-infra | 17:13 | |
| jeblair | mordred: i agree with that assesment | 17:13 |
| denis_makogon | goodday, could this review be reviewed https://review.openstack.org/#/c/60553/ ? | 17:13 |
| *** nati_ueno has quit IRC | 17:13 | |
| jeblair | mordred: i rechecked https://review.openstack.org/#/c/48064/ | 17:13 |
| *** johnthetubaguy has quit IRC | 17:14 | |
| *** nprivalova has quit IRC | 17:15 | |
| *** FallenPegasus has joined #openstack-infra | 17:20 | |
| *** FallenPegasus has quit IRC | 17:20 | |
| *** FallenPegasus has joined #openstack-infra | 17:21 | |
| *** rcleere has joined #openstack-infra | 17:24 | |
| *** SushilKM has joined #openstack-infra | 17:30 | |
| *** marun has quit IRC | 17:30 | |
| *** sarob has joined #openstack-infra | 17:30 | |
| *** ryanpetrello has joined #openstack-infra | 17:31 | |
| *** pabelanger has quit IRC | 17:33 | |
| *** ^d has joined #openstack-infra | 17:36 | |
| *** FallenPegasus has quit IRC | 17:37 | |
| *** loq_mac has joined #openstack-infra | 17:39 | |
| mordred | jeblair: this: https://review.openstack.org/#/c/58145/ | 17:40 |
| mordred | jeblair: is WIP - I think I'm just goingto wait until gerrit 2.8 to see if it's fixed there | 17:41 |
| lifeless | mtreinish: remember https://bugs.launchpad.net/subunit/+bug/1245672 ? :) | 17:41 |
| uvirtbot | Launchpad bug 1245672 in testtools "testtools.run --list-tests exits 0 when discover fails to import" [Critical,Fix released] | 17:41 |
| jeblair | mordred: ok | 17:42 |
| lifeless | mtreinish: the UI is a huge fail right now, I need to circle back on that | 17:42 |
| *** markmcclain has quit IRC | 17:42 | |
| lifeless | mtreinish: but right at the end: | 17:42 |
| lifeless | import errors\x1etempest.tests.test_rest_client | 17:42 |
| lifeless | tempest.tests.test_rest_client failed to import | 17:42 |
| lifeless | so running the full suite would have failed | 17:43 |
| clarkb | morning | 17:43 |
| *** loquacities has joined #openstack-infra | 17:43 | |
| clarkb | also ++ to merging the mirror selection change. I didn't want to exclude anyone else from babysitting :) | 17:43 |
| *** markmcclain has joined #openstack-infra | 17:43 | |
| jeblair | clarkb: good morning | 17:43 |
| lifeless | mtreinish: and now we fail at listing time, because as you pointed out, a filtered test suite might not pick up the lazy report of a failing import leading to silently not running expected tests | 17:44 |
| jeblair | clarkb: after further thought, i think i want to proceed with https://review.openstack.org/#/c/60391/ and not the ggp approach ... | 17:44 |
| clarkb | jeblair: ok | 17:44 |
| *** loq_mac has quit IRC | 17:44 | |
| lifeless | fungi: clarkb: ^ See my reply to mtreinish :) | 17:44 |
| jeblair | clarkb: i don't think ggp is going to easily deal with updating the devstack-gate checkout for projects other than devstack-gate (that the current project is the only project is another assumption it makes) | 17:45 |
| jeblair | clarkb: so making it handle checking out the zuul ref for devstack-gate master branch for a change to nova stable/havana would be adding significant complexity to it. | 17:45 |
| *** ruhe has quit IRC | 17:45 | |
| SushilKM | wanted to know on trove gate which version of python 2.6 we are using like 2.6.? | 17:46 |
| clarkb | SushilKM: Python 2.6.6 | 17:46 |
| SushilKM | thanks clarkb | 17:46 |
| *** markmcclain has quit IRC | 17:46 | |
| clarkb | jeblair: good point | 17:46 |
| *** hashar has quit IRC | 17:47 | |
| clarkb | jeblair: I hadn't considered non master checkouts and how that works with d-g | 17:47 |
| fungi | lifeless: okay, makes sense. thanks! | 17:48 |
| fungi | SushilKM: basically, the latest python 2.6 provided in the current release of centos 6 | 17:48 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Add an alert file publisher https://review.openstack.org/60568 | 17:48 |
| SushilKM | Fine and thanks @fungi | 17:50 |
| clarkb | mordred: jeblair: the slowdown for tempest smoke and swift functional tests wasn't much worse than rax, but tempest full was pretty bad | 17:50 |
| notmyname | ? | 17:50 |
| *** nati_ueno has joined #openstack-infra | 17:51 | |
| mordred | notmyname: $ | 17:51 |
| jeblair | clarkb: mordred wants to experiment with a larger flavor. meanwhile i just wrote a patch to add runtime stats to nodepool... | 17:51 |
| jeblair | clarkb: do you have good enough numbers that we can make mordred's change and have a meaningful comparison? | 17:51 |
| mordred | notmyname: oh - swift ping. nah, wasn't actually swift releated | 17:51 |
| *** krotscheck has joined #openstack-infra | 17:51 | |
| *** reed has joined #openstack-infra | 17:51 | |
| jeblair | clarkb: or should we run with the nodepool change on 8g for a while to get those numbers? | 17:51 |
| clarkb | jeblair: probably not, I was just glancing at https://jenkins01.openstack.org/job/check-tempest-dsvm-full/buildTimeTrend on the various jenkins servers and since we don't have many slaves we don't get great numbers there | 17:52 |
| *** esker has quit IRC | 17:52 | |
| notmyname | mordred: thanks. just checking :-) | 17:52 |
| *** loquacities has quit IRC | 17:53 | |
| clarkb | jeblair: but looking at that for the full tempest runs they seem to consistently timeout | 17:53 |
| clarkb | jeblair: so we can at least check if they stop timing out | 17:53 |
| mtreinish | lifeless: the thing is running the full suite passed. It' only fails on 2 of the jobs: https://review.openstack.org/#/c/60379/ | 17:53 |
| clarkb | jeblair: zaro: btw gerrit 2.9 release is aiming for february 1st | 17:54 |
| *** flaper87 is now known as flaper87|afk | 17:54 | |
| *** markmc has quit IRC | 17:55 | |
| lifeless | mtreinish: that means something in those two jobs is preventing the import of that module | 17:55 |
| *** gyee has joined #openstack-infra | 17:56 | |
| mtreinish | lifeless: yeah, that's what I was starting to dig into. But nothing was standing out as obvious causes from a quick glance at d-g. | 17:56 |
| *** nikhil__ is now known as nikhil_ | 17:58 | |
| *** nikhil_ is now known as Guest32160 | 17:59 | |
| *** rnirmal has quit IRC | 18:05 | |
| *** Guest32160 is now known as nikhil__ | 18:05 | |
| *** SushilKM has quit IRC | 18:05 | |
| *** dripton has quit IRC | 18:05 | |
| *** boris-42 has joined #openstack-infra | 18:06 | |
| jeblair | clarkb, fungi, mordred: could you look at https://review.openstack.org/#/c/60391/ ? it's going to spend most of the day trying to get through the gate. | 18:06 |
| clarkb | sure | 18:06 |
| *** alcabrera has joined #openstack-infra | 18:07 | |
| lifeless | mtreinish: are there any dep differences between the jobs? | 18:07 |
| fungi | hah, i already had that one open in a tab, staring at it while waiting for this ext2resize to complete | 18:07 |
| lifeless | mtreinish: e.g. what does enabling large ops do to the python librariess | 18:08 |
| lifeless | mtreinish: or to your new test file | 18:08 |
| *** dripton has joined #openstack-infra | 18:09 | |
| *** dripton is now known as dripton_gym | 18:09 | |
| *** esker has joined #openstack-infra | 18:10 | |
| mtreinish | lifeless: it shouldn't do anything it's a just a different tox job with a different regex filter: http://git.openstack.org/cgit/openstack/tempest/tree/tox.ini#n50 | 18:11 |
| clarkb | jeblair: has zuul been restarted with the split pipeline fix? | 18:11 |
| mtreinish | and on the devstack gate side it just sets some different config options: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n197 | 18:11 |
| jeblair | clarkb: yes | 18:11 |
| lifeless | mtreinish: does the choice of virt driver impact what deps are installed? | 18:13 |
| jeblair | clarkb: i want to add a stanza inside of the meetbot vhost. did we ever figure out a pattern for that? | 18:13 |
| jeblair | clarkb: (a non-meetbot related stanza) | 18:13 |
| lifeless | mtreinish: though the file that fails looks super ordinary | 18:13 |
| mtreinish | lifeless: I don't know that's a good question. I feel like it shouldn't | 18:13 |
| lifeless | no obvious exotic dependencies | 18:13 |
| mtreinish | sdague: ^^^ any idea | 18:14 |
| fungi | clarkb: eys | 18:14 |
| fungi | clarkb: or yes | 18:14 |
| mtreinish | lifeless: yeah it's pretty straightforward | 18:14 |
| fungi | oh, nm, you got an answer | 18:14 |
| clarkb | jeblair: I don't think we did | 18:16 |
| clarkb | jeblair: re https://review.openstack.org/#/c/60391/3/devstack-vm-gate-wrap.sh the only thing I have a question about is whether or not we need the logs dir before reexecing | 18:17 |
| clarkb | since we do log the reexec process don't we? | 18:17 |
| sdague | lifeless: virt driver selection does change dependencies | 18:17 |
| *** che-arne has quit IRC | 18:17 | |
| *** rossella_s has quit IRC | 18:17 | |
| lifeless | mtreinish: | 18:17 |
| fungi | clarkb: that all gets captured to stdout (console log) right? | 18:18 |
| clarkb | fungi: some of it does, some of it goes in the setup-* log files. After reading further it looks like the log into file stuff happens after reexec | 18:19 |
| clarkb | fungi: I think you are correct | 18:19 |
| *** markwash has joined #openstack-infra | 18:19 | |
| fungi | that's what i meant, everything prior to any reexec hits the console log, not a redirect | 18:19 |
| clarkb | yup, I am approving now | 18:19 |
| *** SergeyLukjanov has joined #openstack-infra | 18:20 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 18:21 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 18:22 | |
| openstackgerrit | A change was merged to openstack-infra/gearman-plugin: fix function registration. https://review.openstack.org/58140 | 18:24 |
| jeblair | clarkb, fungi: agreed (re: log to file stuff) | 18:28 |
| clarkb | http://logs.openstack.org/64/48064/7/check/check-tempest-dsvm-full/3e46bc9/logs/devstack-gate-setup-host.txt.gz select mirror was called but appears to be running the old version. We need to rebuild images | 18:29 |
| clarkb | I am going to start nodepool image builds if no one has beaten me to that | 18:30 |
| *** rnirmal has joined #openstack-infra | 18:31 | |
| clarkb | hrm we have a lot of images (more than the typical 1 or 2 per region/az) | 18:31 |
| jeblair | clarkb: i built images a few times yesterday | 18:32 |
| *** afazekas_ has quit IRC | 18:32 | |
| clarkb | jeblair: safe to ignore then? | 18:32 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/nodepool: Fix README file https://review.openstack.org/60579 | 18:33 |
| jeblair | clarkb: for now; i'll look for errors | 18:33 |
| *** ryanpetrello has quit IRC | 18:35 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/gearman-plugin: [PLEASE IGNORE] Launchpad sync test https://review.openstack.org/60580 | 18:37 |
| SergeyLukjanov | hey guys | 18:38 |
| SergeyLukjanov | my last CR shows that everything ok with LP sync | 18:38 |
| SergeyLukjanov | using launchpad option in projects.yamk | 18:38 |
| SergeyLukjanov | there is now gearman plugin in jeepyb's mapping - https://github.com/openstack-infra/jeepyb/blob/master/jeepyb/projects.py#L134 | 18:38 |
| SergeyLukjanov | but it was auto commented to https://bugs.launchpad.net/openstack-ci/+bug/1258609 | 18:39 |
| uvirtbot | Launchpad bug 1258609 in openstack-ci "Bug for testing Launchpad sync" [Undecided,In progress] | 18:39 |
| clarkb | jeblair: mordred: fungi: btw nodepool failed to spin up a slave in dfw for image building we are at quota there. Is that quota shared with iad and ord? doesn't seem to be because images are building in those regions just fine | 18:39 |
| clarkb | nodepool claims to know of only one host in dfw | 18:39 |
| clarkb | querying nova now | 18:39 |
| jeblair | clarkb: dfw holds the long running slaves | 18:39 |
| clarkb | oh right | 18:40 |
| jeblair | clarkb: (nodepool alien-list will show you the diff) | 18:40 |
| clarkb | there they are. Should we request a quota bump or ignore that for now? | 18:41 |
| HenryG | sdague, fungi: hi. I think git.openstack.org is starting to show some signs of sluggishness | 18:41 |
| jeblair | clarkb: ignore for now; though i thought there should be enough for an image build | 18:41 |
| clarkb | HenryG: can you be more specific? | 18:41 |
| clarkb | HenryG: is there a particular repo and protocol that is slow? | 18:41 |
| sarob | Can I get this https://review.openstack.org/#/c/60394/ merged? | 18:43 |
| sarob | i need to start promoting the stackforge project for new devs | 18:43 |
| HenryG | I have been spinning up devstack several times today with RECLONE=yes. It has been fast for the past ~4 hours, now it is getting slower. The "git fetch origin" operations are taking longer. | 18:43 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Serve the statusbot alert file from eavesdrop https://review.openstack.org/60582 | 18:44 |
| HenryG | Not as bad as it was yesterday though | 18:45 |
| clarkb | HenryG: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=2 you can see git farm numbers there | 18:47 |
| clarkb | I wonder if this happens over time after the cron that packs refs runs | 18:48 |
| clarkb | then it gets faster again | 18:48 |
| clarkb | could also be a network thing as load in rackspace and our ci infra ramps up | 18:48 |
| fungi | maybe we could run that under nice? | 18:48 |
| HenryG | yes, I am poking around in those graphs | 18:48 |
| *** osanchez has quit IRC | 18:48 | |
| clarkb | fungi: no I don't think the process is causing the problem. The addition of unpacked refs is problematic | 18:49 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Serve the statusbot alert file from eavesdrop https://review.openstack.org/60582 | 18:49 |
| fungi | though if it's i/o starvation then yeah, nicing won't help | 18:49 |
| clarkb | fungi: we might want to run it more often | 18:49 |
| clarkb | fungi: we run that because that version of git sucks when there are a lot of refs loose | 18:49 |
| fungi | oh, that. possible | 18:49 |
| HenryG | Seat of the pants observation is that the peaks in "TCP Open" connections correlates with when I saw slowness. | 18:49 |
| jeblair | clarkb: it takes a long time to add enough refs to make that kind of difference. | 18:49 |
| fungi | though that could be a symptom rather than a cause | 18:49 |
| fungi | more open sockets because slower response means clients waiting longer on them | 18:50 |
| *** vkozhukalov has joined #openstack-infra | 18:50 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=862&rra_id=all it does push a non trivial amount of traffic especially considering what rackspace gives us for network bits | 18:50 |
| *** reed has quit IRC | 18:51 | |
| *** yolanda has quit IRC | 18:53 | |
| *** mrodden has quit IRC | 18:54 | |
| clarkb | fungi: any chance we can delete the precise-dev and centos6-dev slaves? are they still in use? | 18:55 |
| clarkb | fungi: if not deleting them will free up 16GB of DFW quota | 18:55 |
| fungi | clarkb: they are slaves of jenkins-dev | 18:55 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add e-r query for bug 1257644 https://review.openstack.org/60508 | 18:56 |
| uvirtbot | Launchpad bug 1257644 in nova "gate-tempest-dsvm-postgres-full fails - unable to schedule instance" [Critical,Triaged] https://launchpad.net/bugs/1257644 | 18:56 |
| clarkb | fungi: gah that won't work then | 18:56 |
| * clarkb goes back to ignoring dfw quota | 18:56 | |
| fungi | i've actually been trying to add a precisepy3k-dev slave to test the revised puppet pip providers | 18:56 |
| clarkb | maybe we can put the dev slaves in another region? | 18:56 |
| fungi | reasonable | 18:56 |
| *** reed has joined #openstack-infra | 18:57 | |
| *** dstanek has joined #openstack-infra | 18:58 | |
| *** mrodden has joined #openstack-infra | 18:58 | |
| *** pabelanger has joined #openstack-infra | 18:59 | |
| *** DinaBelova has joined #openstack-infra | 19:01 | |
| jeblair | clarkb: do you have any image builds happening now? | 19:03 |
| clarkb | jeblair: yes I have builds going for everywhere but dfq | 19:03 |
| clarkb | *dfw | 19:03 |
| jeblair | ok | 19:03 |
| *** loq_mac has joined #openstack-infra | 19:05 | |
| *** dripton has joined #openstack-infra | 19:05 | |
| denis_makogon | guys, could someone please review this patch (in blocks, somehow, development of trove related features, like dashboard and heat resource) https://review.openstack.org/#/c/60553/ | 19:06 |
| *** dripton_gym has quit IRC | 19:06 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Restructure periodic cleanup https://review.openstack.org/60585 | 19:06 |
| clarkb | denis_makogon: is that the absolute minimum version that can work? | 19:07 |
| jeblair | clarkb: i do think there is still a problem with the periodic cleanup ^ that change should help | 19:07 |
| clarkb | denis_makogon: I ask because that change shouldn't hold up anything the new version will be available to you | 19:07 |
| clarkb | jeblair: thanks | 19:07 |
| *** westmaas is now known as westmau5 | 19:07 | |
| clarkb | denis_makogon: http://pypi.openstack.org/openstack/python-troveclient/ | 19:07 |
| denis_makogon | clarkb, there's big difference between 1.0.0 and 1.0.3 | 19:08 |
| *** sarob has quit IRC | 19:08 | |
| denis_makogon | clarkb, improvements requires 1.0.3 | 19:08 |
| clarkb | denis_makogon: sure, but 1.0.3 is available now, you don't need that change to merge to get work done (it isn't a blocker) | 19:08 |
| *** sarob has joined #openstack-infra | 19:08 | |
| *** sarob has quit IRC | 19:09 | |
| denis_makogon | clarkb, is it possible to make 1.0.3 minimal version ? | 19:09 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Serve the statusbot alert file from eavesdrop https://review.openstack.org/60582 | 19:09 |
| *** sarob_ has joined #openstack-infra | 19:09 | |
| clarkb | denis_makogon: it definitely is. I just want to understand why the change is necessary and how urgent it is | 19:10 |
| *** adalbas has quit IRC | 19:10 | |
| clarkb | denis_makogon: infra doesn't like approving requirements changes unless it is actually urgent like breaking the gate. We prefer the requiremetns core team review those changes | 19:10 |
| jeblair | clarkb, mordred, fungi, pabelanger_: any reason not to cut zuul 2.0 now? | 19:10 |
| clarkb | jeblair: you fixed the major bug that I wanted to see get fixed | 19:11 |
| clarkb | no opposition from me | 19:11 |
| denis_makogon | clarkb, heat team blocks my patch until v1.0.3 would land | 19:11 |
| *** loq_mac has quit IRC | 19:11 | |
| denis_makogon | clarkb, as minimal requirements | 19:11 |
| jeblair | (or, i gues 2.0.0 to be consistent) | 19:11 |
| clarkb | new image is in place on hpcloud az1-3 | 19:13 |
| *** talluri has quit IRC | 19:15 | |
| fungi | zuul 2.0.0 wfm | 19:15 |
| jeblair | pushed | 19:15 |
| clarkb | denis_makogon: I would try to get the non infra members of requirements core to review the change, we do try to be hands off as much as possible. Being hands off gives the python and distro folks a chance to review | 19:16 |
| denis_makogon | clarkb, thanks a lot =) | 19:17 |
| *** sarob_ has quit IRC | 19:18 | |
| *** sarob has joined #openstack-infra | 19:18 | |
| fungi | denis_makogon: the infra cores are core on requirements mainly so that we can get emergency changes through quickly to un-break gating and make decisions relevant to support and infrastructure, but otherwise don't really want to be in the position of deciding what versions of libraries various projects are going to use | 19:19 |
| markwash | SpamapS: good stuff on ML, thanks | 19:19 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Remove launchpad project mappings for -api repos https://review.openstack.org/60593 | 19:20 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Remove rest openstack/* repos mappings https://review.openstack.org/60594 | 19:20 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Remove launchpad mappings for cookbook-* repos https://review.openstack.org/60595 | 19:20 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Remove launchpad mapping for the rest projects https://review.openstack.org/60596 | 19:20 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Remove openstack-infra projects mappings to LP https://review.openstack.org/60332 | 19:20 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Set launchpad project mapping for all -api repos https://review.openstack.org/60597 | 19:20 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Set launchpad mappings for rest openstack/* repos https://review.openstack.org/60598 | 19:20 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Set launchpad mappings for all cookbook-* repos https://review.openstack.org/60599 | 19:20 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Set launchpad mappings for the rest projects https://review.openstack.org/60600 | 19:20 |
| *** loq_mac has joined #openstack-infra | 19:20 | |
| SergeyLukjanov | ooops, sorry for spam | 19:20 |
| DinaBelova | wow :) | 19:20 |
| denis_makogon | fungi, problem it that older client (v1.0.0) doesn't allow to use all features of upstream trove, but 1.0.3 does | 19:20 |
| SergeyLukjanov | it's about movin git2lp mappings | 19:20 |
| ogelbukh | SergeyLukjanov: squash it ) | 19:20 |
| SergeyLukjanov | I've split them to make review easier | 19:20 |
| SergeyLukjanov | ogelbukh, I separate them by project 'groups', it makes them more reviewable | 19:21 |
| SergeyLukjanov | ogelbukh, and I've separated them after looking on one big commit ;) | 19:22 |
| SpamapS | markwash: ty.. sometimes feel like I'm the crazy guy yelling at the world on the street corner. ;) | 19:22 |
| *** loq_mac has quit IRC | 19:22 | |
| markwash | I assume that happens to all of us :-) | 19:22 |
| *** sarob has quit IRC | 19:23 | |
| *** pete5 has joined #openstack-infra | 19:23 | |
| *** mrodden has quit IRC | 19:25 | |
| clarkb | sdague: can you tl;dr the e-r status page stuff for me so that I can review and approve | 19:26 |
| clarkb | sdague: is the only needed change https://review.openstack.org/#/c/60318/ ? | 19:27 |
| sdague | clarkb: I think that's it | 19:27 |
| sdague | we've landed all the stuff in er | 19:27 |
| clarkb | thanks reviewing that now so it can get in before end of day | 19:27 |
| sdague | the basic model is as follows... | 19:27 |
| sdague | we pbr install the static pages into $prefix/share/elastic-recheck | 19:28 |
| sdague | then assume the host env will provide that as /elastic-recheck in apache | 19:28 |
| sdague | and the state dir with data as /elastic-recheck/data in apache | 19:28 |
| *** mrodden has joined #openstack-infra | 19:29 | |
| sdague | so hopefully that set of config changes does all that | 19:29 |
| sdague | and if not, I screwed up again :) | 19:29 |
| jeblair | 2013-12-06 19:29:07.669 | setUpClass (tempest.api.compute.v3.admin.test_simple_tenant_usage.TenantUsagesV3TestXML) ... FAIL | 19:30 |
| jeblair | what does that mean? | 19:30 |
| jeblair | the test setup class failed, before running any tests? | 19:30 |
| clarkb | correct | 19:30 |
| jeblair | classy | 19:31 |
| *** sandywalsh has quit IRC | 19:31 | |
| mordred | SpamapS: you said something good? | 19:32 |
| mordred | ah setUpClass | 19:32 |
| mordred | one of these days we'll get rid of that | 19:32 |
| clarkb | sdague: I just noop ran it on logstash.o.o and it fails in bot.pp because it can't find the various requires from init.pp. I would change the requires in that file to be require => Class['elastic_recheck'] | 19:33 |
| clarkb | sdague: I am leaving a comment now | 19:33 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond https://review.openstack.org/60348 | 19:33 |
| sdague | clarkb: cool | 19:33 |
| clarkb | sdague: one other thing I want to change since we need to make a new patchset | 19:33 |
| *** gyee has quit IRC | 19:33 | |
| ArxCruz | jeblair: fungi how welcome a patch to devstack-gate.sh run a generic script after and before run tempest? | 19:33 |
| sdague | sure, I'm trying to move the histogram stuff from graph.py back into results.py in facets, so I'll look to do a rev in 30 minutes or so | 19:34 |
| ArxCruz | jeblair: fungi for example, we need to set some configs in our power machine, before ran devstack, and after ran devstack, in order to tempest run fine | 19:34 |
| fungi | ArxCruz: you mean additional hook points in the script, like the pre, setup and post ones? | 19:35 |
| ArxCruz | fungi: yes | 19:35 |
| ArxCruz | fungi: we need to do some black magic in our vm to run tempest | 19:35 |
| fungi | zaro: btw, sorry for the delay on the sonatype jira account... they're throwing java exceptions when i try to add one, so i'm playing around with it a little | 19:35 |
| ArxCruz | it would be cool if we can point ian script to run as a envar | 19:35 |
| ArxCruz | like | 19:35 |
| ArxCruz | DEVSTACK_RUN_SCRIPT_AFTER=my_black_magic.sh | 19:36 |
| clarkb | sdague: commented | 19:36 |
| jeblair | ArxCruz: why doesn't the pre_test_hook work for that? | 19:36 |
| ArxCruz | then in devstack-vm-gate.sh check if have this envar, if so, run after | 19:36 |
| ArxCruz | jeblair: hummm didn't knew about that | 19:37 |
| clarkb | all regions but dfw have new images | 19:37 |
| ArxCruz | jeblair: any documentation how to use ? | 19:37 |
| clarkb | I will recheck mordred's change in a bit after hosts have a chance to cycle | 19:37 |
| jeblair | ArxCruz: there should be some example in the infra/config repo jjb definitions | 19:37 |
| *** oubiwann_ has joined #openstack-infra | 19:38 | |
| ArxCruz | jeblair: cool! you guys save me a lot of time :D | 19:38 |
| clarkb | ArxCruz: the solum, swift functional, and pbr integration tests all use the hooks | 19:38 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Serve the statusbot alert file from eavesdrop https://review.openstack.org/60582 | 19:38 |
| mordred | clarkb: which change? | 19:39 |
| clarkb | mordred: the d-g change to use select-mirror | 19:39 |
| clarkb | mordred: needed an image update to get the newer version of select-mirror in place | 19:39 |
| mordred | oh. duh | 19:39 |
| mtreinish | sdague: on http://logs.openstack.org/79/60379/1/check/gate-tempest-dsvm-large-ops/070b9cf/console.html I recreated the env locally and it's failing on importing mock when testr runs. | 19:41 |
| mtreinish | it looks like mock isn't installed in /usr/local/lib/python2.7/dist-packages and tox isn't putting it in it's venv | 19:42 |
| *** oubiwann_ has quit IRC | 19:42 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond https://review.openstack.org/60348 | 19:42 |
| sdague | mtreinish: dvsm shouldn't be using a venv | 19:43 |
| sdague | why does large ops need mock? | 19:44 |
| clarkb | tempest uses a venv in the gate | 19:44 |
| mordred | sdague: no? I thought we used it anyway? | 19:44 |
| mtreinish | sdague: large-ops doesn't, but the unit tests do | 19:44 |
| mtreinish | and subunit discover imports the test files | 19:44 |
| mordred | haha | 19:45 |
| mordred | that's teh funny | 19:45 |
| sdague | clarkb: I thought we bypass that? | 19:45 |
| clarkb | sdague: no, we enable site packages so it is still a venv | 19:45 |
| sdague | or is this one of those things where you can never fully bypass it in tox | 19:45 |
| mtreinish | I just can't figure out why large-ops doesn't work but tempest-full (or the other tempest jobs) have no issue | 19:46 |
| clarkb | right no fully bypassing. we approximate with site packages but the python is running in a virtualenv | 19:46 |
| sdague | man, I sort of what run_tests.sh back it's so silly to have a venv there | 19:46 |
| mtreinish | mock is in the tempest test-requirements file so shouldn't it be installed? | 19:47 |
| *** sandywalsh has joined #openstack-infra | 19:48 | |
| mtreinish | mordred: heh, yeah I know. It almost makes me want to make tempest-unit-tests be in a separate repo. :) | 19:49 |
| *** loq_mac has joined #openstack-infra | 19:49 | |
| clarkb | mtreinish: test-requirements will be installed by pbr I think | 19:49 |
| mordred | clarkb: no | 19:49 |
| mordred | we only install requirements by default | 19:49 |
| ArxCruz | clarkb: thanks, found here, I will check if this will work to me :) | 19:50 |
| clarkb | mtreinish: in that case only the testenv:venv and testenv:pep8 install test requreiemtns according to the tox.ini I have (whcih is probably old) | 19:50 |
| *** MarkAtwood has joined #openstack-infra | 19:50 | |
| mtreinish | clarkb: what about testenv:py27? | 19:51 |
| mtreinish | clarkb: also then how do the other tempest jobs pass? | 19:51 |
| *** mwagner_lap has joined #openstack-infra | 19:52 | |
| clarkb | mtreinish: I am updating my repo to make sure this isn't super old | 19:52 |
| clarkb | yup just venv and pep8, notice the deps = lines | 19:52 |
| clarkb | mtreinish: tox will install tempest to the virtuelnvs with pbr, installing the requirements | 19:53 |
| clarkb | maybe that is sufficient to run the tests? | 19:53 |
| *** loq_mac has quit IRC | 19:53 | |
| mtreinish | clarkb: requirements is sufficient to run the regular tempest tests. But, the testr list command will import the unit tests which is where the mock dep is coming from. | 19:55 |
| mtreinish | but it only fails the large-ops jobs | 19:55 |
| mtreinish | clarkb: ok maybe we should add that the deps lines to testenv just as a quick workaround | 19:55 |
| *** sandywalsh has quit IRC | 19:55 | |
| clarkb | do we need to change the test discovery dirs? | 19:56 |
| clarkb | mtreinish or that | 19:56 |
| mtreinish | clarkb: the problem with changing the discover dir is that we want it to run on the unit test dir for the unit test jobs | 19:57 |
| fungi | zaro: they do try hard to make this process complicated... what should our groupId be? https://docs.sonatype.org/display/Repository/Choosing+your+Coordinates | 19:58 |
| *** loq_mac has joined #openstack-infra | 19:58 | |
| *** loq_mac has quit IRC | 19:58 | |
| mtreinish | clarkb: unless there is an easy way to swap testr.confs or switch the test_command line based on the tox job | 19:59 |
| *** alcabrera has quit IRC | 19:59 | |
| *** yamahata_ has quit IRC | 19:59 | |
| *** vipul is now known as vipul-away | 19:59 | |
| *** vipul-away is now known as vipul | 19:59 | |
| notmyname | whatever bug this is https://jenkins01.openstack.org/job/gate-tempest-dsvm-full/739/console is about to throw away a successful gate pass for 6 jobs | 19:59 |
| mordred | lovely | 20:00 |
| *** elasticio has joined #openstack-infra | 20:01 | |
| *** elasticio is now known as Guest82733 | 20:01 | |
| mordred | notmyname: I dunno. I tnk that might be a real bug in your release notes | 20:01 |
| *** loq_mac has joined #openstack-infra | 20:01 | |
| *** loq_mac has quit IRC | 20:02 | |
| notmyname | mordred: it's even more frustrating when you see that it had already passed all the checks on a previous run (maybe just checks, but it seemed to run all the gate jobs) | 20:02 |
| *** rfolco has quit IRC | 20:03 | |
| *** loq_mac has joined #openstack-infra | 20:04 | |
| mtreinish | clarkb: ok, the other tempest jobs also fail, just report success: http://logs.openstack.org/79/60379/1/check/check-tempest-dsvm-full/0558615/console.html#_2013-12-06_15_47_32_311 | 20:04 |
| mtreinish | looks like I've found 2 bugs... | 20:04 |
| notmyname | mordred: looks like this bug? https://bugs.launchpad.net/tempest/+bug/1242898 | 20:04 |
| mordred | notmyname: I believe someone was looking at tha earlier | 20:04 |
| uvirtbot | Launchpad bug 1242898 in tempest "tearDownClass (tempest.api.compute.servers.test_server_rescue.ServerRescueTestXML): tearDownClass does not call the super's tearDownClass " [Undecided,New] | 20:04 |
| *** datsun_F40PH has joined #openstack-infra | 20:06 | |
| mtreinish | notmyname: "the tearDownClass does not call the super's tearDownClass" is a secondary effect caused by an exception being raised elsewhere | 20:06 |
| *** vkozhukalov has quit IRC | 20:06 | |
| *** loq_mac has quit IRC | 20:06 | |
| jeblair | mordred, clarkb: is there any reason i should not be extremely upset about the thing mtreinish just pointed out where it's reporting success because it can't count the tests? | 20:06 |
| mtreinish | notmyname: if you look at the test results a bunch of volumes tests failed | 20:07 |
| mtreinish | so my guess is that the cinder logs will show the real bug you're hitting | 20:07 |
| fungi | zaro: i'm filling out the jira ticket, so any of the following details would be helpful. we can etherpad it for simplicity... https://etherpad.openstack.org/p/0ldKRW4BiT | 20:07 |
| reed | does anybody have a gpg key for Richard Fontana? | 20:08 |
| mordred | jeblair: I believe that is a bug that's got a fix | 20:08 |
| reed | since I changed my gpg setup according to fungi, I can't search anything | 20:08 |
| jeblair | mordred: i feel like we have fixed that bug many, many times. | 20:08 |
| * reed has fungi as the new scapegoat | 20:08 | |
| *** datsun180b has quit IRC | 20:08 | |
| *** datsun_F40PH is now known as datsun180b | 20:08 | |
| fungi | reed: gpg2 --search-keys fontana | 20:09 |
| *** sandywalsh has joined #openstack-infra | 20:09 | |
| fungi | reed: i see 0x66CB352B | 20:09 |
| jeblair | mordred: the first time we 'fixed' it was when we moved from nose to testr | 20:09 |
| reed | fungi, I'm still getting gpgkeys: HTTP search error 77: :( | 20:09 |
| fungi | reed: also 0x35D5D78B but it looks like he may have retired that one | 20:09 |
| reed | damn it, I broke my gpg2 | 20:10 |
| mtreinish | jeblair: I know I broke and fixed it once in the past 6 months with a testr wrapper script that I screwed in tempest :) | 20:10 |
| mordred | jeblair: https://bugs.launchpad.net/subunit/+bug/1245672 | 20:10 |
| uvirtbot | Launchpad bug 1245672 in testtools "testtools.run --list-tests exits 0 when discover fails to import" [Critical,Fix released] | 20:10 |
| mtreinish | mordred: that's the bug that's causing the list command to fail | 20:10 |
| *** ArxCruz has quit IRC | 20:10 | |
| mtreinish | but testr is reports success with running 0 tests | 20:10 |
| mtreinish | mordred: err, the fix for that bug is causing the list command to fail | 20:11 |
| *** SergeyLukjanov_ has joined #openstack-infra | 20:11 | |
| mordred | gotcha | 20:11 |
| fungi | reed: my ~/.gnupg/gpg.conf looks like http://paste.openstack.org/show/54620 (maybe you have something odd set in yours?) | 20:12 |
| mordred | oh - right - because when list-tests doesn't list any tests, then testr 'successfully' runs all 0 of them, right? | 20:12 |
| *** SergeyLukjanov has quit IRC | 20:13 | |
| *** mriedem has quit IRC | 20:13 | |
| SpamapS | lifeless: ^^ note that testr exitting 0 with 0 run tests is _still_ confusing .. correct or not. :-P | 20:14 |
| mtreinish | mordred: except not always because some of the jobs fail | 20:14 |
| *** hashar has joined #openstack-infra | 20:14 | |
| *** mrodden1 has joined #openstack-infra | 20:14 | |
| notmyname | mtreinish: I'm not really sure what to look for. where are you seeing cinder or volume errors? the full logs are hard to look through, and searching for "error" isn't too helpful | 20:14 |
| fungi | reed: when you get that error 77, does it follow with a line about CAfile: /path/to/CA/sks-keyservers.netCA.pem | 20:14 |
| reed | fungi, yeah, the path to the pem file needs to be specified | 20:15 |
| mordred | SpamapS, lifeless: yes. if you think it's not confusing, you've overthought the abstractions | 20:15 |
| mtreinish | notmyname: for example: http://logs.openstack.org/28/60428/3/gate/gate-tempest-dsvm-full/c541ad0/console.html#_2013-12-06_19_48_16_315 | 20:15 |
| *** rongze has quit IRC | 20:15 | |
| reed | I wonder why it worked last time I used it | 20:15 |
| fungi | reed: seems to be a common error if you configured gpg to look in the wrong place for the hkps ca cert | 20:15 |
| fungi | (or didn't have the path entry for it at all in your config) | 20:15 |
| reed | I am sure I haven't changed the path... or I may have invoked gpg2 from .gnupg | 20:15 |
| reed | fungi, problem solved, you're now officially a "scapegoat with a solution" | 20:16 |
| *** mrodden has quit IRC | 20:16 | |
| mtreinish | notmyname: the job timedout so most of the service logs aren't there but something screwed up with cinder which is making the jobs super slow (causing the jenkins timeout) and causing the volume tests to fail. | 20:16 |
| fungi | reed: glad i've been upgraded slightly | 20:16 |
| jeblair | clarkb, fungi, mordred: the poosibly breaking devstack-gate change lands in at least 27 mins, but it's time for me to find lunch | 20:16 |
| notmyname | mtreinish: am I just walking several step behind you in finding the error? do you already have a LP bug number? | 20:17 |
| jeblair | clarkb, fungi, mordred: if you see it land while i'm gone, keep an eye on things and force a revert if it breaks. | 20:17 |
| fungi | jeblair: i'll grip my seat and look out for it | 20:17 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: move the graph generating code over to facets https://review.openstack.org/60609 | 20:17 |
| sdague | jeblair: so your eyes on that one would be good | 20:17 |
| mtreinish | notmyname: I don't have a bug number. I was just pointing out that cinder was the culprit behind the failure. I don't think you'll be able to identify the true failure because you're missing the service logs | 20:17 |
| sdague | I believe it's completely equivalent | 20:18 |
| sdague | well, with the caveat of graph start time | 20:18 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Bump up results per page in Kibana https://review.openstack.org/60610 | 20:18 |
| notmyname | mtreinish: jeblair: so does that mean we have a "reverify no bug"? has that been disabled yet? | 20:18 |
| *** oubiwann_ has joined #openstack-infra | 20:18 | |
| mordred | jeblair: ++ | 20:19 |
| *** pafuent has joined #openstack-infra | 20:19 | |
| pafuent | Hi | 20:19 |
| mtreinish | mordred, jeblair: I figured out the success bug for those jobs. It's my wrapper script again :( | 20:20 |
| *** mrodden1 has quit IRC | 20:20 | |
| mtreinish | http://git.openstack.org/cgit/openstack/tempest/tree/tools/pretty_tox.sh#n4 subunit2pyunit returns 0 | 20:20 |
| pafuent | There is a REST API for gerrit? | 20:20 |
| sdague | pafuent: not in the version we have deployed | 20:22 |
| mordred | pafuent: not in the version we're running - we're working on upgrading to 2.8 which does have one | 20:22 |
| *** melwitt has joined #openstack-infra | 20:23 | |
| *** harlowja has quit IRC | 20:23 | |
| clarkb | back from lunch | 20:23 |
| *** harlowja has joined #openstack-infra | 20:23 | |
| clarkb | jeblair: take it up with lifeless it has been argued | 20:24 |
| pafuent | sdague: mordred: Ohhhh, thanks for your reply | 20:24 |
| clarkb | jeblair: see review on https://review.openstack.org/#/c/60557/ | 20:25 |
| zaro | fungi: clouddocs is already on central: https://oss.sonatype.org/index.html#nexus-search;quick~clouddocs | 20:26 |
| fungi | zaro: so this is about setting up a repository specifically for clouddocs then, not more generally setting up access to create repositories? | 20:27 |
| zaro | fungi: dcramer has an account to upload to there. we just need access to be able to upload to same location. | 20:27 |
| fungi | zaro: so, if it's the sonatype jira account, that exists and is called openstackinfra | 20:27 |
| fungi | zaro: i take it i just needed to stop after step $2 in the instructions you linked? | 20:28 |
| * zaro looks at instructions again. | 20:29 | |
| zaro | fungi: yep, just stop after step #2 and then we need to test with openstackinfra account. | 20:30 |
| zaro | fungi: not sure how to test it though. | 20:31 |
| clarkb | sdague: still planning on updating the puppet change? | 20:31 |
| fungi | zaro: though there's this at the end of step #3... "How does a new user ask for publish rights to existing repositories? Follow step 2, sign up and then leave a comment on the JIRA ticket with the username. We will assign related roles to him and update the ticket." | 20:31 |
| fungi | zaro: but maybe dcramer can simply grant access for openstackinfra instead | 20:32 |
| sdague | clarkb: yes, I'm just fighting with python | 20:32 |
| sdague | python setup.py develop now provides an error on elastic recheck | 20:32 |
| sdague | locally | 20:32 |
| sdague | I'm so annoyed | 20:32 |
| fungi | "fighting with pythons, tonight on the discovery channel" | 20:32 |
| sdague | AttributeError: Distribution instance has no attribute 'entry_points' | 20:33 |
| sdague | so many back trace before that | 20:33 |
| jeblair | clarkb: can you explain: 20:27 < clarkb> jeblair: take it up with lifeless it has been argued | 20:33 |
| zaro | fungi: yeah, not sure if that's possible. let me ask him if he knows the process. | 20:33 |
| fungi | sdague: is the entry_points section in your setup.cfg malformed, maybe? | 20:33 |
| sdague | it wasn't | 20:34 |
| clarkb | jeblair: re testr returning success if no tests run | 20:34 |
| sdague | and I didn't change it. I think i'm just in a crazy setuptools wedge | 20:34 |
| *** bpokorny has quit IRC | 20:34 | |
| jeblair | clarkb: i literally do not understand the words you are saying to me when you say "take it up with lifeless it has been argued". | 20:34 |
| jeblair | clarkb: i can not parse that sentence. | 20:35 |
| fungi | clarkb: jeblair: the counterargument was that if you specify a test name pattern which matches no discovered tests, then it's valid that you didn't expect any tests to run | 20:35 |
| clarkb | jeblair: many people have argued that that is a bug. Lifeless seems to disagree. I think he does agree that discover failing forsome reason should result in failure, but discovering 0 tests successfully is not a failure | 20:35 |
| jeblair | clarkb, fungi: sure. i realize how fruitless it is to get into an argument with lifeless about this, so i have no intention in doing so. | 20:36 |
| jeblair | clarkb: however, i am concerned because i thought we had a workaround. | 20:36 |
| fungi | i think the workaround is that we normally don't ask testr to run subsets of tests in our jobs | 20:37 |
| sdague | yeh, it was just pip installed vs. distro | 20:37 |
| sdague | oh python packaging, why do you need to be so broke... :{ | 20:37 |
| clarkb | there is a thing that checks the number tests discovered against the number of tests run, that cannot catch the 0 discovered 0 run scenario | 20:37 |
| jeblair | clarkb: why not? | 20:38 |
| clarkb | becuase it relies on discover to do it | 20:38 |
| clarkb | if dicsover discovers 0 tests then you run 0 tests and check passes | 20:38 |
| sdague | honestly, testr should be handling all this internally | 20:38 |
| sdague | I'm annoyed that we keep breaking ourselves working around things testr punted on | 20:39 |
| clarkb | gah gate reset behind head :( | 20:39 |
| jeblair | clarkb: why don't we do a string comparison on the output? | 20:40 |
| clarkb | jeblair: that is what we do | 20:40 |
| clarkb | but 0 == 0 | 20:40 |
| fungi | also, testr does die properly (if somewhat opaquely) if discover exits nonzero | 20:40 |
| *** reed has quit IRC | 20:40 | |
| jeblair | clarkb: awesome. we know it's 0. 0 is always wrong for us. | 20:40 |
| clarkb | we would need a different way of detecting this. Perhaps run discover directly and check the return code | 20:40 |
| clarkb | jeblair: I see, so we would need a second check | 20:40 |
| sdague | honestly, I don't understand how a test runner would ever believe 0 was a right answer :) | 20:41 |
| sdague | if I say run some tests, 0 is a wrong answer, because I fat fingered something | 20:41 |
| fungi | sdague: for pattern matches, it's a potential use case to say "descend into these projects and run tests matching this if they exist" | 20:41 |
| *** hogepodge has joined #openstack-infra | 20:42 | |
| jeblair | clarkb: where is the run=discover check performed? | 20:42 |
| clarkb | jeblair: I can't find it anymore. I wonder if it got lost in the switch to run-unittests.sh too | 20:42 |
| sdague | fungi: that's an extradordinary case, that should require another flag to let that pass | 20:42 |
| clarkb | jeblair: it used to be in the old run-something.sh script that ran unittests. is not in run-unittets.sh | 20:42 |
| fungi | sdague: i don't disagree with you there | 20:42 |
| mtreinish | sdague, clarkb: the failure this time is different though. It's the tox_pretty.sh wrapper script, testr returns the right code, but it's getting eaten by subunit2pyunit (which get's the piped output of testr) | 20:43 |
| mtreinish | I suck at bash so I'm trying to figure out how to get around it | 20:43 |
| fungi | mtreinish: -o pipefail? | 20:43 |
| clarkb | mtreinish: so foo | bar, foo returns 1 bar return 0? | 20:43 |
| clarkb | if so what fungi said | 20:43 |
| jeblair | mtreinish: what's a tox-pretty? | 20:43 |
| * fungi uses words | 20:43 | |
| sdague | mtreinish: right, but that pipe is again a work around for missing testr ui | 20:43 |
| clarkb | mtreinish: lifeless recommends the end of the pipe deal with failures properly | 20:44 |
| clarkb | I thought subunit2pyunit did that | 20:44 |
| mtreinish | jeblair: git.openstack.org/cgit/openstack/tempest/tree/tools/pretty_tox.sh | 20:44 |
| mtreinish | jeblair: or serially: http://git.openstack.org/cgit/openstack/tempest/tree/tools/pretty_tox_serial.sh | 20:44 |
| mtreinish | it was something sdague required because he was scared by 30min with no output | 20:45 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: sort the graph list by number of fails https://review.openstack.org/60613 | 20:46 |
| sdague | clarkb: ok, back to the puppet thing | 20:46 |
| devananda | mordred: shall i take a guess at what those missing jobs are and add them? | 20:46 |
| *** yolanda has joined #openstack-infra | 20:47 | |
| mordred | devananda: look at python-novaclient | 20:47 |
| mordred | devananda: you'll see things in the post and release pipelines | 20:47 |
| *** Guest82733 has quit IRC | 20:48 | |
| *** Guest82733 has joined #openstack-infra | 20:49 | |
| jeblair | fungi, clarkb, mordred: https://review.openstack.org/#/c/40006/2 | 20:50 |
| jeblair | fungi, clarkb, mordred: you removed the test count check. | 20:50 |
| zaro | fungi: sent an email to dcramer about access to clouddocs. | 20:50 |
| sdague | clarkb: so https://review.openstack.org/#/c/60318/6/modules/openstack_project/manifests/static.pp | 20:51 |
| clarkb | jeblair: I remember now. The reason that was done is that discover finds some number of tests, but the actualy number of tests can vary wildy because you can build tests at runtime with stuff like testscenarios | 20:51 |
| zaro | fungi: doing this one today as well? https://review.openstack.org/#/c/57333 | 20:51 |
| sdague | I should also change the require on 318 right? | 20:51 |
| clarkb | jeblair: however that shouldn't affect a comparison against 0, I think we can have the check if that is all it does | 20:51 |
| jeblair | clarkb: that would be so lovely. | 20:51 |
| *** nerdinthewest has joined #openstack-infra | 20:52 | |
| clarkb | sdague: yup good catch | 20:52 |
| fungi | zaro: yep | 20:52 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/config: Add release jobs for python-ironicclient https://review.openstack.org/60614 | 20:52 |
| nerdinthewest | lifeless: I can't remember if it was you, but someone was talking to Jared Smith from Bluehost about getting some hardware donated | 20:52 |
| nerdinthewest | I'm sure it was one of you guys in here anyway, sound familiar to anyone? | 20:52 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: move to elastic-recheck web console in share https://review.openstack.org/60318 | 20:52 |
| *** denis_makogon_ has joined #openstack-infra | 20:52 | |
| sdague | clarkb: ok there it is | 20:52 |
| *** yolanda has quit IRC | 20:52 | |
| clarkb | jeblair: I will whip that up right now | 20:53 |
| devananda | mordred: ^^. also, should I do the same for ironic? | 20:53 |
| *** oubiwann_ has quit IRC | 20:53 | |
| mordred | the other breakage in this case is that list-tests would also list 0 tests - so the check would check 0 against 0 | 20:53 |
| mordred | clarkb: make sure to account for that ^^ | 20:53 |
| mordred | devananda: nope. we don't release the servers that way yet | 20:54 |
| devananda | k | 20:54 |
| clarkb | mordred: right, I am just going to compare the reported number that ran against 0 rather than the number found | 20:54 |
| clarkb | mordred: because I think in either case jeblair wants a test failure | 20:54 |
| mordred | clarkb: works for me | 20:54 |
| jeblair | nerdinthewest: probably lifeless, mordred may be able to help you | 20:54 |
| mordred | lifeless: ^^ you know - a real fix would be neat | 20:54 |
| *** esker has quit IRC | 20:54 | |
| fungi | zaro: which kind of credential do we want there? the one we use for regular nodepool->jenkins looks like a uuid... the options in the drop-down are certificate, ssh username/key or username/password | 20:55 |
| *** rossella_s has joined #openstack-infra | 20:55 | |
| mordred | nerdinthewest: almost certainly lifeless - but I'll be happy to try to help put you in touch with people | 20:55 |
| *** esker has joined #openstack-infra | 20:55 | |
| nerdinthewest | yeah, let me know what you guys need and I'll try and get it available | 20:55 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond https://review.openstack.org/60348 | 20:55 |
| nerdinthewest | not just hardware specs, but bandwidth, access, any datacenter hands, etc | 20:56 |
| *** DinaBelova has quit IRC | 20:56 | |
| zaro | fungi: i didn't set that up, so i'm not sure. but i believe it's the uuid, same as the one for regular nodepool. | 20:57 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: If zero tests run fail the job. https://review.openstack.org/60617 | 20:57 |
| clarkb | jeblair: something like ^ | 20:57 |
| *** denis_makogon has quit IRC | 20:58 | |
| *** denis_makogon_ is now known as denis_makogon | 20:58 | |
| *** dmakogon_ has joined #openstack-infra | 20:58 | |
| fungi | zaro: yeah, that's what i was asking... wondering which of those options it is, or whether we simply entered a uuid as a password. maybe i can tell by comparing it in the webui. checkiong | 20:58 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/config: Add release jobs for python-ironicclient https://review.openstack.org/60614 | 20:59 |
| clarkb | mordred: jeblair: http://logs.openstack.org/64/48064/7/check/check-tempest-dsvm-neutron/7adfbec/logs/devstack-gate-setup-host.txt.gz successful mirror selection with the d-g select-mirror change | 20:59 |
| zaro | fungi: jeblair might know. i think he set that up. | 20:59 |
| mordred | clarkb: WOOT | 20:59 |
| *** esker has quit IRC | 21:00 | |
| mordred | clarkb: ^^ deva's change above - he's trying to cut a release, if you get a sec | 21:00 |
| *** pafuent has left #openstack-infra | 21:01 | |
| jeblair | fungi: what's the question? | 21:01 |
| *** rossella_s has quit IRC | 21:02 | |
| devananda | mordred: as far as tagging an I-1 release of Ironic, any suggestions there? | 21:02 |
| fungi | jeblair: the nodepool credential for authenticating to jenkins... in hiera it looks like a uuid, just wondering whether that was intentional or just random choice | 21:02 |
| jeblair | fungi: it's an identifier for a jenkins credential record, which is itself an ssh public key | 21:03 |
| fungi | jeblair: getting ready to generate the one on jenkins-dev for the nodepool dev instance zaro's adding | 21:03 |
| fungi | jeblair: perfect--that's what i needed to know. thanks! | 21:03 |
| jeblair | fungi: if it's not in the ui, you should be able to see it in one of the ~jenkins/*.xml files | 21:03 |
| devananda | mordred: or an I-2, if its too late for I-1 now | 21:04 |
| fungi | jeblair: even better. thanks | 21:04 |
| *** oubiwann_ has joined #openstack-infra | 21:04 | |
| mordred | devananda: I _think_ that the pattern is 2014.1.b1 | 21:05 |
| mordred | devananda: process is same - git tag -s 2014.1.b1 ; git push gerrit 2014.1.b1 | 21:05 |
| clarkb | jeblair: fungi: mordred: I think we should merge https://review.openstack.org/#/c/60561/ at the very least we can see if the full tempest runs stop timing out which will tell us if it is an improvement. I don't want ot stick with 8GB because they just don't work | 21:06 |
| jeblair | clarkb: wfm, aprvd | 21:06 |
| devananda | mordred: any jobs i need to add to ironic's pipe? or the tag is just there for others to reference? | 21:07 |
| mtreinish | clarkb: subunit2pyunit already does that you're right(otherwise all the gate jobs would never fail) It's just that this is failing in the list command which I think is what was tripping things up. (it's probably a subunit2pyunit bug) | 21:07 |
| mordred | devananda: you're set already | 21:08 |
| devananda | coolness | 21:08 |
| *** Guest82733 has quit IRC | 21:09 | |
| clarkb | fungi: did you want to leave a comment about gerrit familiarity and so on at https://review.openstack.org/#/c/60132/ ? | 21:09 |
| jeblair | clarkb: regarding node run times -- are we really interested in run times for failed jobs? | 21:09 |
| clarkb | jeblair: I think so, particularly since in the case of hpcloud region-b one of the tests is failing consistently at 90 minutes | 21:10 |
| jeblair | clarkb: it's certainly theoretically interesting, but there are going to be _a lot_ of extra whisper files. | 21:10 |
| jeblair | clarkb: ok, but what graph would contain that data? | 21:11 |
| jeblair | clarkb: i mean, i'm struggling to figure when we're going to look at a graph of failed run times. | 21:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Bump the min ram for HP 1.1 https://review.openstack.org/60561 | 21:11 |
| clarkb | nodepool.job.tempest.jobname.region.FAILURE | 21:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Add release jobs for python-ironicclient https://review.openstack.org/60614 | 21:11 |
| clarkb | jeblair: I would be ok with successful jobs for now, then if we decide we need the other data down the road we can add that then? | 21:12 |
| jeblair | clarkb: (if we're trying to answer the question "should we increase the timeout", i imagine we would want to look at successful runs, not failures) | 21:12 |
| clarkb | jeblair: unless there are no successful runs | 21:12 |
| clarkb | the point about the whisper file explosion is valid, I didn't really consider that | 21:12 |
| devananda | mordred: ! [remote rejected] 2014.1.b1 -> 2014.1.b1 (can not create new references) | 21:13 |
| clarkb | happy to +2 the existing change for now and see what the data gives us | 21:13 |
| jeblair | clarkb: ok, let's do this then: i'll patch that change to include the job result in the key, but still only do success | 21:13 |
| jeblair | that way we can add more later if we want | 21:13 |
| mordred | devananda: you do not have permissions! I believe we may have you set already to be under ttx control | 21:13 |
| clarkb | jeblair: sounds good, thanks | 21:13 |
| mordred | devananda: go ahead and delete the local tag | 21:13 |
| mordred | devananda: and send ttx a note asking him to tag whatever git sha you want tagged as 2014.1.b1 for you | 21:14 |
| devananda | aahhhh | 21:14 |
| devananda | hehe | 21:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Restructure periodic cleanup https://review.openstack.org/60585 | 21:14 |
| *** rossella_s has joined #openstack-infra | 21:15 | |
| jeblair | clarkb: actually, i think i would probably put it on the end, so i think we can leave the patch the way it is. | 21:19 |
| *** oubiwann_ has quit IRC | 21:20 | |
| clarkb | jeblair: ok I will lift my -1 (I gave the other nodepool change a +2_ | 21:20 |
| devananda | clarkb: i see add-ironicclient-release-pipe has merged. how long before I can push a tag and have the magic happen? | 21:20 |
| clarkb | devananda: I would wait for a few minutes after 2130 | 21:21 |
| devananda | clarkb: awesome, thanks! | 21:21 |
| clarkb | basically give puppet about half an hour | 21:21 |
| jeblair | the devstack-gate possibly-breaking change is merging | 21:24 |
| devananda | also, for reference, who can push release tags to client projects (besides the release team)? is it just the PTL, or also -core? | 21:24 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60391 | 21:24 |
| jeblair | devananda: generally PTLs and their close delegates | 21:25 |
| jeblair | devananda: we like to keep the group small because the process is manual and has little error checking | 21:25 |
| devananda | jeblair: fair 'nuf. how is the auth handled? | 21:25 |
| devananda | like, how do I add a delegate? | 21:25 |
| jeblair | devananda: we'd create an ironic-ptl group, and you can add people to it | 21:26 |
| jeblair | in gerrit | 21:26 |
| mtreinish | clarkb: ok, no I'm confused: https://review.openstack.org/#/c/60611/1 it only worked on the postgres job... | 21:26 |
| *** esker has joined #openstack-infra | 21:26 | |
| clarkb | jeblair: https://review.openstack.org/#/c/60568/ has a comment | 21:27 |
| devananda | jeblair: gotcha. there is an ironic-milestone group. taht's separate, yes? | 21:27 |
| jeblair | clarkb: thx | 21:28 |
| jeblair | devananda: yes, that's people who can merge changes to milestone branches (presumably) | 21:28 |
| clarkb | mtreinish: are the test-requirements coplete? | 21:28 |
| clarkb | mtreinish: is there a dependency that is missing? | 21:29 |
| *** resker has joined #openstack-infra | 21:29 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Refactor reconfiguration into a management event https://review.openstack.org/60623 | 21:29 |
| devananda | jeblair: ahh. ty for the explanation. an ironic-ptl group probably isn't necessary now, but in keeping consistent with other projects (looks like most have this), one would be great | 21:29 |
| *** resker has quit IRC | 21:30 | |
| mtreinish | clarkb: I don't think so, but I'm not sure. (I'll double check) But, that patch only adds the test-requirements to the job, which pass normally, so it shouldn't be missing anything it needs to run. | 21:31 |
| *** esker has quit IRC | 21:31 | |
| *** loq_mac has joined #openstack-infra | 21:32 | |
| *** loq_mac has quit IRC | 21:32 | |
| clarkb | jeblair: mordred https://review.openstack.org/#/c/48064/ looks ready. I didn't +A to give jeblair a chance to rereview and to avoid more potential d-g weirdness while the latest d-g change settles in | 21:32 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Revert "Don't fix disks before rexecing" https://review.openstack.org/60624 | 21:33 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Revert "Don't fix disks before rexecing" https://review.openstack.org/60624 | 21:33 |
| clarkb | I guess that tells us why there is a lot of red in the zuul queue | 21:33 |
| jeblair | well that didn't work. | 21:33 |
| *** rossella_s has quit IRC | 21:33 | |
| mordred | poop | 21:33 |
| jeblair | hrm. i have an idea. | 21:35 |
| clarkb | sdague: https://review.openstack.org/#/c/60318/ lgtm. let us see if we can get another core to approve it | 21:37 |
| clarkb | I ran it noop on logstash.o.o and it runs | 21:38 |
| *** senk has joined #openstack-infra | 21:38 | |
| sdague | yeh, probably jeblair | 21:38 |
| *** reed has joined #openstack-infra | 21:38 | |
| jeblair | i'll look real quick | 21:38 |
| sdague | as he did the prior +2 | 21:38 |
| mordred | "Red Hat Software Collections (which includes a Red Hat supported Python 3.3 release) hit general availability in September, so even RHEL users have a fully supported migration option these days." | 21:38 |
| mordred | (looking at my psf mail) | 21:38 |
| clarkb | mordred: right, but red hat doesn't want people using that stuff for openstack :) | 21:39 |
| jeblair | clarkb, sdague: lgtm | 21:39 |
| sdague | mordred: my understanding is software collections is an upcharge | 21:39 |
| mordred | clarkb: I have days where I care less what distros want people doing | 21:39 |
| sdague | you need another license for it | 21:39 |
| clarkb | mtreinish: I am really stumped about that | 21:39 |
| mordred | clarkb: and days where I want the distros to care more about what we want them to do | 21:39 |
| clarkb | mtreinish: I am looking at tempest code now o_O | 21:39 |
| sdague | because I was excited that it meant we could dump 2.6 | 21:40 |
| sdague | but we can't | 21:40 |
| mordred | sdague: well | 21:40 |
| mordred | we CAN | 21:40 |
| sdague | sure | 21:40 |
| mordred | we've chosen not to | 21:40 |
| sdague | fair | 21:40 |
| mordred | mainly because we like russel and markmc | 21:40 |
| rustlebee | <3 | 21:41 |
| *** jkyle has quit IRC | 21:41 | |
| mordred | rustlebee: damn you and your being a good human | 21:41 |
| jeblair | it's one thing to hold on to 2.6 because the distros lack the technical ability to support something else; it's another thing to hold onto it because it doesn't fit their favored pricing structure. :) | 21:41 |
| rustlebee | i do try | 21:41 |
| rustlebee | in all seriousness ... if we as a project have significant amounts to gain from dropping it, then I don't think I would fight that | 21:41 |
| rustlebee | if it's dropping it just because we can, then sure, i'd rather not | 21:42 |
| *** senk has quit IRC | 21:42 | |
| rustlebee | if it's blocking some python 3 work for example, that makes sense | 21:42 |
| rustlebee | and we would just have to deal with it | 21:42 |
| jeblair | rustlebee: that is so sensible. :) | 21:42 |
| rustlebee | a heads up would be good of course, like, early in the release cycle | 21:43 |
| clarkb | mtreinish: http://logs.openstack.org/11/60611/1/check/check-tempest-dsvm-full/ea7ab24/logs/tempest.txt.gz is my bread crumb | 21:43 |
| *** syerrapragada1 has quit IRC | 21:43 | |
| clarkb | mtreinish: leading me to http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/base.py#n34 | 21:44 |
| *** syerrapragada has joined #openstack-infra | 21:44 | |
| clarkb | mtreinish: which will be run during discover | 21:44 |
| clarkb | which makes me want to cry | 21:44 |
| clarkb | executing code during import is bad bad bad | 21:44 |
| rustlebee | "don't think I would fight that" -- in fact, i would encourage it | 21:44 |
| *** jkyle has joined #openstack-infra | 21:45 | |
| fungi | needing working python2.7 on rhel so you can install and run openstack might be incentive enough for them to start offering it without an upcharge | 21:45 |
| clarkb | mtreinish: so, best I can figure is something is affecting that import path, perhaps import ordering? | 21:45 |
| clarkb | 2.7 will be availabe in rhel 7 base image right? | 21:45 |
| *** loq_mac has joined #openstack-infra | 21:46 | |
| rustlebee | rhel 7 yes | 21:46 |
| *** loq_mac has quit IRC | 21:46 | |
| *** senk has joined #openstack-infra | 21:47 | |
| openstackgerrit | A change was merged to openstack-infra/config: move to elastic-recheck web console in share https://review.openstack.org/60318 | 21:49 |
| sdague | clarkb: so how long until that applies? | 21:51 |
| clarkb | puppet on master updates on the hour, then puppet runs every ten minutes so ~20 minutes max | 21:51 |
| *** dprince has quit IRC | 21:52 | |
| clarkb | did we lose mtreinish for the weekend? | 21:52 |
| annegentle | reed: around? interesting community request for ya | 21:52 |
| reed | annegentle, i am | 21:52 |
| sdague | clarkb: I don't think yet | 21:52 |
| sdague | I bet he's hanging in the hallway there, it tends to get social about this hour | 21:53 |
| clarkb | mtreinish: if you get this, my hunch is that something in that code path is being executed during import and either import ordering is breaking us or the InvalidConfiguration exception is being raised or something I haven't considered yet :) | 21:54 |
| openstackgerrit | A change was merged to openstack-infra/config: new maven plugin jobs to build and deploy maven plugins to maven nexus https://review.openstack.org/46099 | 21:54 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add check-devstack-gate-tempest-dsvm-full jobs https://review.openstack.org/60627 | 21:55 |
| jeblair | clarkb: ^ check that out -- uses your idea from yesterday. | 21:56 |
| clarkb | looking | 21:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Add an alert file publisher https://review.openstack.org/60568 | 21:56 |
| *** pabelanger has quit IRC | 21:57 | |
| jeblair | oops, found an error | 21:57 |
| clarkb | jeblair: what about running that against branches | 21:57 |
| jeblair | clarkb: sorry? | 21:58 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add check-devstack-gate-tempest-dsvm-full jobs https://review.openstack.org/60627 | 21:58 |
| clarkb | jeblair: the job templates can't be made to run against icehouse | 21:58 |
| jeblair | clarkb: oh, right, those should be jobs, not templates | 21:59 |
| clarkb | jeblair: I think they should be templates | 21:59 |
| mordred | clarkb, jeblair: I saw you guys talking about that, and I mostly follow, but my brain has too much jetlag to critique | 21:59 |
| jeblair | clarkb: why? | 21:59 |
| clarkb | unless we never plan to run them against icehouse | 21:59 |
| jeblair | clarkb: i never plan to run them against icehouse. :) | 21:59 |
| salv-orlando | Hi - stupid question on devstack. Is it correct that devstack removes untagged changes in git trees when you unstack/restack? | 21:59 |
| clarkb | jeblair: ok jobs are fine then | 21:59 |
| salv-orlando | s/untagged/unstaged | 21:59 |
| jeblair | clarkb: there is a teensy hole in our testing in that the pre-rexec code is never tested until it lands; i'm trying to at least close that in the check queue | 22:00 |
| clarkb | jeblair: yup. Wasnt sure if we wanted to extend that to new releases as they happen | 22:00 |
| jeblair | clarkb: i think doing this on master is sufficient because there is no per-branch logic that happens pre-reexec | 22:00 |
| clarkb | jeblair: good point. Will there be a patchset 3? | 22:01 |
| jeblair | clarkb: yes, i need to change them to jobs | 22:01 |
| jeblair | clarkb: actually -- maybe not; that way node at least is still templated. | 22:02 |
| *** oubiwann_ has joined #openstack-infra | 22:02 | |
| clarkb | jeblair: if you want them to still be templates I think they need to go in a different job group as the parameters accepted is different | 22:03 |
| *** jcooley_ has joined #openstack-infra | 22:03 | |
| fungi | salv-orlando: i think it's supposed to not do that if you have uncommitted code in the git repositories, but where that falls apart is if you commit changes locally and then stack (haven't tried myself, but that's what i heard) | 22:03 |
| clarkb | fungi: salv-orlando: I just looked and there is logic to undo requirements changes using git reset --hard | 22:03 |
| clarkb | and that is enabled by default so yes looks like devstack will do that | 22:04 |
| salv-orlando | fungi & clarkb: thanks. it really seems a hard reset from git log. I'll try to keep this in mind for the future. | 22:04 |
| mordred | salv-orlando: we'd certaily like to make that better ... | 22:05 |
| mtreinish | clarkb: sorry I stepped out of my office | 22:05 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add check-devstack-gate-tempest-dsvm-full jobs https://review.openstack.org/60627 | 22:05 |
| openstackgerrit | Pranesh Pandurangan proposed a change to openstack-infra/config: Added new project, entropy https://review.openstack.org/60132 | 22:06 |
| devananda | clarkb: https://jenkins01.openstack.org/job/python-ironicclient-tarball/1/ upload lost ? | 22:06 |
| devananda | i'm sure i did something wrong .... | 22:06 |
| salv-orlando | mordred: I was assuming it was working in this way for a reason; perhaps a way to handle that better would be to stash rather than resetting | 22:06 |
| *** mfer has quit IRC | 22:06 | |
| mordred | salv-orlando: hrm. | 22:06 |
| *** gyee has joined #openstack-infra | 22:07 | |
| mordred | salv-orlando: the only time devstack run reset --hard | 22:07 |
| mordred | salv-orlando: is if you've set UNDO_REQUIREMENTS | 22:07 |
| mtreinish | clarkb: it doesn't take much any exception raised during import will cause the failure. Which is why I wrote: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/tests/test_list_tests.py | 22:07 |
| mordred | salv-orlando: ah - that defaults to true | 22:07 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: sort the graph list by number of fails https://review.openstack.org/60613 | 22:08 |
| salv-orlando | mordred: thanks for finding that out! Is there a reason for defaulting to true? | 22:08 |
| mordred | salv-orlando: I think the logic here is worth rethinking - it might be complex... | 22:08 |
| clarkb | mtreinish: I would try a change that moves that code path from import time to run time | 22:08 |
| clarkb | devananda: gah, puppet/JJB may not be applying jobs to jenkins.o.o properly | 22:09 |
| gyee | mordred, is there a wiki on how to run the grenade tests on my local box? I am trying to chase down the failure in https://review.openstack.org/#/c/56333/ | 22:09 |
| devananda | clarkb: that sounds like not-something-i-did-wrong? | 22:09 |
| sdague | salv-orlando: because otherwise it will leave your trees with changes to the requirements files | 22:10 |
| mtreinish | clarkb: actually look what I found: https://jenkins02.openstack.org/job/gate-tempest-python27/1616/console (from the job fixing the wrapper script) it looks like 2 of the prexisting unit tests are broken... | 22:10 |
| sdague | which then make it harder to do clean commits | 22:10 |
| salv-orlando | sdague: correct, I remember that other issue with the three being dirty everytime | 22:10 |
| clarkb | devananda: right not something you did wrong | 22:10 |
| sdague | there were a series of patches around this, I was never really happy with any of the options | 22:10 |
| sdague | but it's where we landed | 22:11 |
| salv-orlando | perhaps we can then just add a note in README.md to warn dumb developers like me | 22:11 |
| clarkb | salv-orlando: sdague: really you shouldn't dev direclty in devstack anyways right? | 22:11 |
| mordred | clarkb: no | 22:11 |
| sdague | clarkb: that's what I tell people | 22:11 |
| mordred | clarkb: many many people do direct dev in devstack | 22:11 |
| sdague | but they don't believe me :) | 22:11 |
| clarkb | you should set it up to clone from your homedir or whever | 22:11 |
| clarkb | mordred: right but permissions and all kinds of ugly | 22:11 |
| sdague | so I gave up that one | 22:11 |
| clarkb | don't do it please | 22:11 |
| mordred | losing battle | 22:11 |
| mordred | it's going to happen | 22:11 |
| salv-orlando | yeah, make sense. I'm always cutting corners. | 22:11 |
| jeblair | clarkb: it's pretty much what it was written for. | 22:11 |
| mordred | it is the easiest way to set up an environment in which you can iteratively develop | 22:12 |
| mordred | thus "dev"stack | 22:12 |
| sdague | salv-orlando: that being said, if you don't develop on master in your tree | 22:12 |
| sdague | then I think yuo end up safer | 22:12 |
| salv-orlando | sdague: that's a good advice. I'll do that. | 22:12 |
| salv-orlando | thanks guys a lot for your help | 22:13 |
| sdague | honestly, you shouldn't need to stack / unstack either | 22:13 |
| sdague | just ^C the process you want to restart | 22:13 |
| sdague | and restart it in it's screen session | 22:13 |
| sdague | that's actually why we run in screen | 22:13 |
| sdague | to enable that | 22:13 |
| clarkb | mordred: jeblair: except that many people run it as root and it makes all sorts of nasty assumptions not just about cleaning repos | 22:13 |
| salv-orlando | sdague: I'm trying to make the parallel jobs work on neutron and as the tests fail leave stuff behind | 22:13 |
| clarkb | it is really good to get a dev cloud up, not really good for managing your dev repos | 22:14 |
| sdague | salv-orlando: ah fair | 22:14 |
| sdague | salv-orlando: so the best way to really do something where you are going to need to do that much teardown | 22:14 |
| clarkb | though I think devstack won't let you run it as root anymore so ++ to that | 22:14 |
| *** reed has quit IRC | 22:14 | |
| sdague | is to have a local clone, then just set the localrc in yuor devstack to make that other local clone your clone from tree | 22:14 |
| jeblair | clarkb: that's not relevant really -- it requires and does things as root anywy. | 22:14 |
| jeblair | clarkb: and it's always been intended to be run on a dedicated vm. | 22:15 |
| clarkb | devananda: https://jenkins.openstack.org/job/python-ironicclient-pypi-upload/ does exist, maybe we tried to run the upload job too early? | 22:15 |
| salv-orlando | sdague: that makes sense as well. | 22:15 |
| clarkb | jeblair: ^ any ideas on why the ironicclient job was lost? | 22:15 |
| devananda | clarkb: i waited until after 22:00 | 22:15 |
| *** reed has joined #openstack-infra | 22:15 | |
| jeblair | 2013-12-06 22:04:43,099 ERROR zuul.Gearman: Job <gear.Job 0x7f8029e74b10 handle: None name: build:python-ironicclient-pypi-upload unique: 12ab4922cb89473085a838fb6ceadbe2> is not registered with Gearman | 22:16 |
| clarkb | jeblair: it does those things cleanly as root. If you run it directly as root won't it make all of the repos owned by root. Which means pushing and pull ing code as root which is scary | 22:16 |
| fungi | clarkb: keep in mind that the jjb run is far from instantaneous, so you have puppetmaster update + puppet agent update + jjb run time | 22:16 |
| clarkb | fungi: right which is why I gave it ~two puppet run iterations :) | 22:17 |
| sdague | jeblair / clarkb: https://review.openstack.org/#/c/60609/ that's the refactor of the graph script to take the histograms into core code | 22:17 |
| fungi | but it does look like jenkins.o.o doesn't think it was ever asked to run that job | 22:17 |
| sdague | which even works :) | 22:17 |
| sdague | and the patch on top, sorts the graphs by fails, which also works (have it up locally) | 22:17 |
| * fungi needs to disappear to hang holiday lights while there's still some daylight, and grab dinner | 22:17 | |
| clarkb | you have light after 5pm? | 22:18 |
| clarkb | what is this magic | 22:18 |
| sdague | fungi: where is there daylight? | 22:18 |
| sdague | it's been dark here for at least 30 minutes | 22:18 |
| clarkb | sdague: gah, I think the puppet stuff isn't working quite right on static.o.o looking into it | 22:18 |
| sdague | clarkb: ok | 22:18 |
| sdague | brb | 22:21 |
| *** harlowja has quit IRC | 22:21 | |
| devananda | clarkb: shall i try again in a bit? | 22:22 |
| *** jcooley_ has quit IRC | 22:24 | |
| *** melwitt has quit IRC | 22:24 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Fix puppet subscription for elastic-recheck. https://review.openstack.org/60628 | 22:25 |
| clarkb | devananda: no | 22:25 |
| *** dims has quit IRC | 22:25 | |
| clarkb | devananda: the problem with tags is that you should treat them as immutable, one of us infra cores will need to try retriggeringthe job | 22:25 |
| clarkb | sdague: ^ | 22:25 |
| clarkb | devananda: I guess that isn't really a problem, it is a desired trait, we just have to retrigger it off to the side | 22:26 |
| clarkb | jeblair: fungi 60628 makes puppet agent --test --noop not fail on static.o0.o | 22:27 |
| *** fifieldt has joined #openstack-infra | 22:27 | |
| clarkb | devananda: I will give that a shot shortly | 22:27 |
| clarkb | I get to learn the new zuul cli command | 22:27 |
| jeblair | clarkb: it doesn't work on post jobs | 22:28 |
| jeblair | clarkb: you still need to use the old new cli command, in tools. :) | 22:28 |
| sdague | clarkb: nice | 22:29 |
| *** melwitt has joined #openstack-infra | 22:30 | |
| sdague | jeblair: is there an easy way to force the y axis on graphite? | 22:30 |
| sdague | I'd like the failure rates to no auto change based on max val, as it gets jarring when the rates are low | 22:31 |
| jeblair | sdague: yes. in the composer: graph options -> y axis | 22:31 |
| clarkb | jeblair: this is the release pipeline but I assume same deal | 22:32 |
| jeblair | sdague: i think the resulting params are yMax=... yMin=... | 22:32 |
| jeblair | clarkb: yes | 22:32 |
| jeblair | clarkb: you want tools/trigger-job.py | 22:32 |
| devananda | clarkb: i see. thanks! | 22:33 |
| clarkb | jeblair: yup thanks | 22:33 |
| sdague | jeblair: yep yMax | 22:34 |
| clarkb | hmm script is hanging, I think the problem may be that the job didn't register itself with gearman after being added | 22:34 |
| clarkb | jeblair: zaro: is that something you want to look more closely at? | 22:35 |
| *** weshay has quit IRC | 22:35 | |
| jeblair | clarkb: 1s | 22:36 |
| *** fifieldt has quit IRC | 22:36 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: set y axis on failure rate graph https://review.openstack.org/60630 | 22:36 |
| sdague | ok, I'm going to run away for a bit to grab some groceries for dinner. May or may not be back on later. | 22:36 |
| clarkb | sdague: I am going to approve the fix to the puppet stuff in a minute, would be great if you could check that it didn't all fall over later (but is friday so np if you can't) | 22:37 |
| sdague | yeh, so the thing that occured to me | 22:37 |
| sdague | in local testing... we don't actually install to index.html now | 22:38 |
| clarkb | does that mean my fix will let puppet run and break us again? | 22:38 |
| sdague | the rest works, I guess the only question is fix that on the er side, or on the infra side | 22:38 |
| sdague | well, you'll need another url | 22:39 |
| sdague | I could move it to index and be done with it on the er side | 22:39 |
| sdague | why don't I just do that | 22:39 |
| clarkb | sdague: ok | 22:39 |
| *** denis_makogon is now known as bot_bot | 22:40 | |
| *** dims has joined #openstack-infra | 22:40 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: move the main page to index.html https://review.openstack.org/60631 | 22:40 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: move the graph generating code over to facets https://review.openstack.org/60609 | 22:40 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: set y axis on failure rate graph https://review.openstack.org/60630 | 22:40 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: sort the graph list by number of fails https://review.openstack.org/60613 | 22:40 |
| sdague | so I stuck that under the other changes | 22:40 |
| sdague | so it's an easy +A | 22:41 |
| jeblair | clarkb: yes, it appears that it's not registered. i'd disconnect and reconnect the pypi slave. | 22:41 |
| clarkb | jeblair: I want to say this has happened before and toggling the project on and off worked? | 22:41 |
| clarkb | zaro: ^ | 22:41 |
| jeblair | clarkb: that should work too. | 22:41 |
| clarkb | woot job is building now | 22:42 |
| sdague | clarkb: ok, I'm self approving the file move to index.html | 22:43 |
| *** SergeyLukjanov has joined #openstack-infra | 22:43 | |
| clarkb | devananda: https://pypi.python.org/pypi/python-ironicclient/0.0.1 | 22:43 |
| clarkb | sdague: wfm | 22:43 |
| sdague | so puppet should work after it | 22:43 |
| clarkb | I just approved the puppet change | 22:43 |
| * zaro adds investigation of gearman bug on td list | 22:44 | |
| sdague | ok, store time. I'll check in once I get the quiche put together and in the oven | 22:44 |
| devananda | clarkb: sweet! ty | 22:44 |
| *** SergeyLukjanov_ has quit IRC | 22:44 | |
| *** bot_bot has quit IRC | 22:45 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: move the main page to index.html https://review.openstack.org/60631 | 22:45 |
| clarkb | devananda: that won't trigger the mirror jobs, but they will run a little later today anyways | 22:45 |
| *** datsun180b has quit IRC | 22:48 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix puppet subscription for elastic-recheck. https://review.openstack.org/60628 | 22:48 |
| *** hashar has quit IRC | 22:48 | |
| *** che-arne has joined #openstack-infra | 22:50 | |
| *** cody-somerville has quit IRC | 22:51 | |
| *** sdake_ has joined #openstack-infra | 22:52 | |
| *** fifieldt has joined #openstack-infra | 22:52 | |
| SpamapS | pcrews: You're the random sql query generator guy, right? | 22:52 |
| SpamapS | or was it shrews | 22:52 |
| SpamapS | I can never remember | 22:52 |
| pcrews | SpamapS: I used to use it back in the day ;) What's up? | 22:53 |
| *** cody-somerville has joined #openstack-infra | 22:53 | |
| *** cody-somerville has quit IRC | 22:53 | |
| *** cody-somerville has joined #openstack-infra | 22:53 | |
| SpamapS | pcrews: I'm just wondering if we should do something similar for Heat and generate random templates. | 22:53 |
| clarkb | SpamapS: tempest wants to do similar for tempest negative testing | 22:53 |
| SpamapS | yeah | 22:53 |
| SpamapS | I recall krow having a library that injects random failure and data into syscalls that he uses on gearmand too. | 22:54 |
| clarkb | it LD_PRELOADS disaster | 22:54 |
| *** bnemec has quit IRC | 22:57 | |
| *** bnemec has joined #openstack-infra | 22:57 | |
| *** boris-42 has quit IRC | 22:59 | |
| *** prad has quit IRC | 23:00 | |
| *** bnemec has quit IRC | 23:03 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: make maven and freestyle job creation consistent https://review.openstack.org/60633 | 23:04 |
| *** thomasem has quit IRC | 23:05 | |
| clarkb | jeblair: https://review.openstack.org/#/c/60627/ lgtm, I will let you approve when you are ready to tackle the reexec problem again | 23:05 |
| *** reed has quit IRC | 23:06 | |
| *** reed_ has joined #openstack-infra | 23:06 | |
| notmyname | looks likethe whole gate is about to be flushed again? https://jenkins02.openstack.org/job/gate-tempest-dsvm-full/775/ | 23:06 |
| jeblair | clarkb: aprvd; it shouldn't affect anything | 23:06 |
| *** bnemec has joined #openstack-infra | 23:06 | |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/config: Mark puppet 3.0 unit tests as non-voting https://review.openstack.org/60635 | 23:06 |
| *** nerdinthewest is now known as geekinutah | 23:07 | |
| clarkb | notmyname: looks likeit | 23:07 |
| mgagne | I have to go but if someone could take a look at this review, it will be much appreciated. it's blocking our gates. | 23:07 |
| jeblair | notmyname: looks like another timeout; did you file a bug for the previous one? | 23:07 |
| openstackgerrit | A change was merged to openstack-infra/config: Add check-devstack-gate-tempest-dsvm-full jobs https://review.openstack.org/60627 | 23:08 |
| notmyname | jeblair: no. all I got was "probably/maybe a cinder thing, but there were no logs" | 23:08 |
| clarkb | mgagne: looking | 23:08 |
| mgagne | clarkb: tyvm | 23:08 |
| jeblair | notmyname: we should file a bug; it's clearly a problem that needs to be tracked | 23:08 |
| notmyname | jeblair: indeed | 23:08 |
| *** giulivo has quit IRC | 23:09 | |
| geekinutah | lifeless: ping | 23:09 |
| *** mgagne has quit IRC | 23:09 | |
| *** bnemec has quit IRC | 23:09 | |
| *** bnemec has joined #openstack-infra | 23:10 | |
| jeblair | notmyname: i think even one that just said "this might be a cinder bug, here's the output, but there are no logs" would be useful. | 23:11 |
| clarkb | elastic-recheck status page is broken, looking into it | 23:11 |
| *** bnemec has quit IRC | 23:11 | |
| notmyname | jeblair: all I got was from mtreinish. I can file one now. against cinder then? | 23:11 |
| jeblair | notmyname: yeah. maybe also target it to tempest since it's timing out. | 23:12 |
| notmyname | jeblair: well, I'm not really sure. there are FAILs all over the place https://jenkins02.openstack.org/job/gate-tempest-dsvm-full/775/console | 23:14 |
| notmyname | nova, cinder, and (maybe) neutron | 23:14 |
| *** bnemec has joined #openstack-infra | 23:15 | |
| *** rnirmal has quit IRC | 23:18 | |
| reed_ | jeblair, I keep forgetting if you can modify the alias mail list of openstack.org | 23:20 |
| reed_ | jeblair, can you? | 23:20 |
| jeblair | reed_: yes | 23:20 |
| reed_ | jeblair, if you have time now, can you check who receives logo@openstack.org? | 23:20 |
| jeblair | reed_: claire, lauren, margie, mark, todd | 23:22 |
| reed_ | great | 23:22 |
| reed_ | not | 23:22 |
| reed_ | I'll email you, thanks | 23:22 |
| clarkb | jeblair: AuthType not set! is what apache is complaining about | 23:24 |
| jeblair | clarkb: ? | 23:24 |
| clarkb | jeblair: the reason elastic-recheck is broken. | 23:24 |
| clarkb | I think sdague's directory directives don't like me | 23:25 |
| *** bnemec has quit IRC | 23:25 | |
| *** afazekas has quit IRC | 23:26 | |
| *** bnemec has joined #openstack-infra | 23:27 | |
| clarkb | jeblair: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/status.vhost.erb#n44 should that be made to look more like the directory on line 34? | 23:27 |
| clarkb | I think the problem may be that we require all granted, need mod_authz which isn't there | 23:28 |
| clarkb | instead we should just let the bits flow | 23:28 |
| jeblair | clarkb: yeah, i had rather assumed that was copied from the other file... | 23:28 |
| jeblair | those line's don't make any sense. you are correct they should look like the block at #34 | 23:29 |
| jeblair | clarkb: any need to follow symlinks? | 23:29 |
| clarkb | jeblair: I don't think so | 23:30 |
| clarkb | I am whipping up a change now | 23:30 |
| jeblair | clarkb: yeah, i don't see any. | 23:30 |
| notmyname | jeblair: https://bugs.launchpad.net/tempest/+bug/1258682 | 23:33 |
| uvirtbot | Launchpad bug 1258682 in tempest "timeout causing gate-tempest-dsvm-full to fail" [Undecided,New] | 23:33 |
| *** rnirmal has joined #openstack-infra | 23:33 | |
| jeblair | notmyname: thanks | 23:33 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Fix elastic-recheck vhost settings. https://review.openstack.org/60636 | 23:34 |
| clarkb | jeblair: how does that look? | 23:35 |
| jeblair | clarkb: lgtm | 23:36 |
| *** randallburt has joined #openstack-infra | 23:38 | |
| *** jergerber has quit IRC | 23:39 | |
| *** senk has quit IRC | 23:39 | |
| randallburt | quick question/confirmation for the infra gurus: I need to add a job to heat; all I need to do is add my "gate-heat-tox-contrib" job to modules/openstack_project/files/zuul/layout.yaml, yes? | 23:42 |
| clarkb | randallburt: you need to make sure that there is a jenkins job defined for that job name in modules/openstack_project/files/jenkins_job_builder/config as well | 23:43 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack/requirements: Add python-ironicclient to global requirements https://review.openstack.org/60638 | 23:44 |
| randallburt | clarkb: cool, thanks! | 23:44 |
| *** reed_ has quit IRC | 23:44 | |
| randallburt | IIRC, its one of the jobs templates for python, but I'll double check | 23:44 |
| fungi | holy scrollback batman | 23:45 |
| *** gyee has quit IRC | 23:47 | |
| clarkb | fungi: just for you | 23:47 |
| sdague | clarkb: oh... right, apache 2.2 | 23:47 |
| clarkb | sdague: ubuntu apache 2.2 even fancier | 23:48 |
| sdague | yeh | 23:48 |
| sdague | well I've updated all my systems to 13.10 | 23:48 |
| fungi | d yeah, the daylight was mostly wishful thinking on my part | 23:48 |
| sdague | so I took all the pain of 2.4 | 23:48 |
| clarkb | http://status.openstack.org/elastic-recheck/ is semi up now | 23:48 |
| clarkb | I applied the change by hand | 23:48 |
| clarkb | I don't see any graphs | 23:48 |
| *** rcleere has quit IRC | 23:49 | |
| clarkb | sdague: File does not exist: /usr/local/share/elastic-recheck/data | 23:49 |
| sdague | clarkb: did cron run yet? | 23:49 |
| clarkb | sdague: yes, but graph.json is in /var/lib/elastic-recheck | 23:49 |
| clarkb | not /usr/local/share/elastic-recheck/data unless I am misunderstanding how this works | 23:50 |
| sdague | ok, do we have to restack those in the other direction? | 23:50 |
| *** bnemec has quit IRC | 23:50 | |
| clarkb | sdague: I don't understand what restack those in the other direction means | 23:50 |
| sdague | hold on, let me experiment here | 23:50 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix elastic-recheck vhost settings. https://review.openstack.org/60636 | 23:50 |
| sdague | https://review.openstack.org/#/c/60636/1/modules/openstack_project/templates/status.vhost.erb | 23:50 |
| sdague | nested aliases | 23:50 |
| sdague | in apache | 23:50 |
| *** bnemec has joined #openstack-infra | 23:50 | |
| clarkb | oh right | 23:51 |
| sdague | yeh, the aliases have to go narrow to wide | 23:51 |
| sdague | so the more specific one first, sorry about that | 23:51 |
| fungi | all these cool new changes to review, but they're already merged... just as well | 23:51 |
| sdague | I'll push the puppet bit | 23:52 |
| *** bnemec has quit IRC | 23:52 | |
| clarkb | sdague: hard refresh, I made the change by hand, it works | 23:52 |
| sdague | woot, thanks | 23:53 |
| *** SergeyLukjanov has quit IRC | 23:53 | |
| clarkb | puppet is current not running on status.o.o | 23:53 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: fix alias order for elastic-recheck https://review.openstack.org/60640 | 23:56 |
| openstackgerrit | Max Rydahl Andersen proposed a change to openstack-infra/jenkins-job-builder: add private-repository to maven-target https://review.openstack.org/60161 | 23:56 |
| fungi | ooh, i got caught up in time to review an apache config correction | 23:56 |
| sdague | clarkb: sorry for causing the scramble. | 23:56 |
| clarkb | no problem, I want this in now so that you can keep hacking on making the whole thing better :) | 23:57 |
| sdague | heh | 23:57 |
| fungi | make it ALL better | 23:57 |
| sdague | well, most of that will wait until next week. As we have friends coming in for the weekend. So no computers this weekend. But I'll start diving again on monday. | 23:59 |
| clarkb | I may end up working simply because going outside is too cold | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!