| *** markvoelker has quit IRC | 00:02 | |
| *** changbl has joined #openstack-infra | 00:03 | |
| *** sdake_ has joined #openstack-infra | 00:04 | |
| *** sdake has quit IRC | 00:07 | |
| *** ruagair has quit IRC | 00:10 | |
| *** ruagair has joined #openstack-infra | 00:11 | |
| *** sdake has joined #openstack-infra | 00:20 | |
| *** sdake_ has quit IRC | 00:23 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Use ostestr for running unit tests https://review.openstack.org/213565 | 00:23 |
|---|---|---|
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add more db api tests https://review.openstack.org/213566 | 00:24 |
| *** salv-orlando has joined #openstack-infra | 00:25 | |
| *** hdd has joined #openstack-infra | 00:25 | |
| *** achanda has quit IRC | 00:26 | |
| *** salv-orlando has quit IRC | 00:29 | |
| *** mriedem has joined #openstack-infra | 00:30 | |
| *** hdd has quit IRC | 00:30 | |
| *** mriedem has quit IRC | 00:30 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Migration to using requests https://review.openstack.org/205637 | 00:33 |
| *** olaph has joined #openstack-infra | 00:38 | |
| tchaypo | I think I finally understand why shade is called shade | 00:43 |
| *** baoli has joined #openstack-infra | 00:45 | |
| StevenK | clarkb: I am an admin | 00:48 |
| StevenK | clarkb: As is camunoz | 00:48 |
| *** baoli has quit IRC | 00:50 | |
| clarkb | cool just finishing dinner now | 00:50 |
| *** asettle has joined #openstack-infra | 00:54 | |
| *** sflanigan has joined #openstack-infra | 00:55 | |
| sflanigan | G'Day StevenK, clarkb, I'm standing in for camunoz. Let me know if you need a hand with the Zanata upgrade. | 00:57 |
| StevenK | sflanigan: Hi! clarkb is just finishing off dinner and we'll approve the branch | 00:58 |
| *** mestery has joined #openstack-infra | 00:59 | |
| clarkb | hello | 01:01 |
| clarkb | yup going to approve now then trigger puppet real quick, everyone ready? | 01:01 |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Test to try capture occasional failures https://review.openstack.org/211776 | 01:02 |
| clarkb | StevenK: this is the only one right https://review.openstack.org/#/c/202899/ ? | 01:02 |
| StevenK | clarkb: Correct. | 01:03 |
| clarkb | approved now | 01:04 |
| clarkb | then once that merges I will manually get it to run quicker than the normal puppet cycle | 01:05 |
| *** camunoz has quit IRC | 01:05 | |
| *** camunoz has joined #openstack-infra | 01:05 | |
| openstackgerrit | Merged openstack-infra/puppet-zanata: Move to Zanata 3.7.1 https://review.openstack.org/202899 | 01:09 |
| clarkb | ok now to get that to run on the server | 01:09 |
| *** boris-42 has quit IRC | 01:10 | |
| clarkb | puppet is running now | 01:11 |
| *** vmbrasseur_GONE has quit IRC | 01:13 | |
| clarkb | its nice and slow | 01:15 |
| clarkb | I am guessing thats a war downloda happening | 01:15 |
| StevenK | Yeah | 01:16 |
| StevenK | I've noticed a few timeouts over various puppet runs, yay sourceforge | 01:17 |
| clarkb | ok can rerun if that happens | 01:17 |
| clarkb | ok its done I don't see any error | 01:17 |
| clarkb | StevenK: but I am guessing I have to restart services | 01:18 |
| clarkb | hrm is our wildfly version new enough? I guess we will find out | 01:18 |
| clarkb | its up running 3.7.1 now | 01:20 |
| StevenK | translate-dev.openstack.org is showing the wildfly AS page. IIRC, it takes ~40 seconds or so for Zanata to start | 01:20 |
| clarkb | StevenK: sflanigan at this point I defer to you | 01:20 |
| clarkb | StevenK: I see it saying 3.7.1 now | 01:20 |
| clarkb | try a refresh? | 01:20 |
| *** vmbrasseur has joined #openstack-infra | 01:21 | |
| clarkb | I can log in and do things, I do think our version of wildfly may be too old but we can fix that later as things appear to be working | 01:22 |
| StevenK | clarkb: I can login fine, the project list looks correct, and I've had a poke around ceilometer since it was on the first page | 01:22 |
| sflanigan | Which version of WF are you using? | 01:22 |
| StevenK | 8.1.0 currently | 01:22 |
| clarkb | StevenK: you gonna do the reindex that is required according to the release notes? | 01:22 |
| sflanigan | We have been testing on 9, but 8.1 is probably okay, apart from the logout bug. | 01:23 |
| *** zhenguo has joined #openstack-infra | 01:23 | |
| sflanigan | actually, it's more of an intermittent 500? error, but especially on logout | 01:23 |
| StevenK | clarkb: I was just about to ask about that, which reindex? | 01:23 |
| clarkb | sflanigan: ok, we can probably avoid dealing iwth that until a little later unless StevenK wants to try upgrading that now too | 01:23 |
| sflanigan | sure | 01:23 |
| StevenK | I didn't see much point jumping to 8.2, personally | 01:24 |
| sflanigan | oh, and apart from security... | 01:24 |
| clarkb | StevenK: http://docs.zanata.org/en/latest/release-notes/#370 under deployment there is a reindex listed | 01:24 |
| sflanigan | wildfly 8.1 doesn't have all the security fixes in wf 9 | 01:24 |
| clarkb | sflanigan: ah ok | 01:24 |
| StevenK | :-/ | 01:24 |
| StevenK | sflanigan: When do you think the team will have finished testing under Wildfly 9? | 01:25 |
| sflanigan | of course, eap 6.4.1 is better still in that respect, but that's another tale. :-) | 01:25 |
| sflanigan | we did the release testing for zanata 3.7 on wildfly 9 | 01:25 |
| StevenK | Ah, right, excellent. | 01:26 |
| *** sputnik13 has quit IRC | 01:26 | |
| StevenK | I will investigate upgrading wildfly to 9 later today | 01:26 |
| clarkb | StevenK: awesome, do ping me here when you have something to report back and I will follow up in my morning | 01:27 |
| *** yamahata has joined #openstack-infra | 01:27 | |
| *** baoli has joined #openstack-infra | 01:27 | |
| clarkb | I will hang around now for reindexing and any othre testing stuff you want to do now | 01:27 |
| *** FallenPegasus has joined #openstack-infra | 01:27 | |
| StevenK | clarkb: Reindex done | 01:28 |
| *** zz_dimtruck is now known as dimtruck | 01:28 | |
| StevenK | sflanigan: That was painless, thanks for the assist :-) | 01:29 |
| sflanigan | no worries | 01:29 |
| *** baoli has quit IRC | 01:31 | |
| clarkb | ok, anything else? or was that it :) | 01:32 |
| *** FallenPegasus has quit IRC | 01:32 | |
| StevenK | clarkb: I'm waiting for my proposal slave for some more testing, but I think we're good | 01:33 |
| clarkb | cool putting laptop away and will keep an eye on irc from my phone, thanks@! | 01:33 |
| StevenK | Heh heh | 01:35 |
| StevenK | Nice, create-zanata-xml will die horribly | 01:35 |
| *** mestery has quit IRC | 01:36 | |
| *** mestery has joined #openstack-infra | 01:36 | |
| *** salv-orlando has joined #openstack-infra | 01:37 | |
| *** mestery has quit IRC | 01:37 | |
| *** boris-42 has joined #openstack-infra | 01:38 | |
| *** salv-orlando has quit IRC | 01:42 | |
| *** salv-orlando has joined #openstack-infra | 01:45 | |
| *** yamahata has quit IRC | 01:47 | |
| *** salv-orlando has quit IRC | 01:49 | |
| *** larainema has joined #openstack-infra | 01:50 | |
| *** unicell1 has quit IRC | 01:52 | |
| *** unicell has joined #openstack-infra | 01:52 | |
| *** zhenguo has quit IRC | 01:53 | |
| *** tiswanso has joined #openstack-infra | 01:55 | |
| *** asettle has quit IRC | 01:55 | |
| *** tiswanso has quit IRC | 01:57 | |
| *** baoli has joined #openstack-infra | 01:57 | |
| *** tiswanso has joined #openstack-infra | 01:58 | |
| *** markvoelker has joined #openstack-infra | 01:58 | |
| *** asettle has joined #openstack-infra | 02:01 | |
| *** baoli has quit IRC | 02:02 | |
| *** markvoelker has quit IRC | 02:03 | |
| *** camunoz has quit IRC | 02:03 | |
| *** asettle has quit IRC | 02:05 | |
| *** camunoz has joined #openstack-infra | 02:05 | |
| *** asettle has joined #openstack-infra | 02:05 | |
| *** asettle is now known as asettle-afk | 02:06 | |
| *** asettle-afk is now known as asettle | 02:07 | |
| anteaya | nice work all | 02:11 |
| *** unicell1 has joined #openstack-infra | 02:12 | |
| anteaya | and hey Canada is having a pycon! https://2015.pycon.ca/ | 02:13 |
| *** unicell has quit IRC | 02:13 | |
| anteaya | who knew? | 02:13 |
| * anteaya didn't know | 02:13 | |
| anteaya | I just checked my email and there was a post about it | 02:13 |
| anteaya | so submissions closed friday but I am going to submit something anyway | 02:13 |
| anteaya | because hey, canada | 02:13 |
| openstackgerrit | Baohua Yang proposed openstack-infra/project-config: Add the necessary configs for new easyovs project https://review.openstack.org/212396 | 02:16 |
| *** sputnik13 has joined #openstack-infra | 02:21 | |
| *** asettle is now known as asettle-afk | 02:23 | |
| anteaya | and the talk submission form is still accepting submissions | 02:23 |
| *** yamamoto has joined #openstack-infra | 02:27 | |
| *** markvoelker has joined #openstack-infra | 02:31 | |
| *** aukhan has joined #openstack-infra | 02:41 | |
| *** enikanorov2_ has quit IRC | 02:41 | |
| *** jeadr has quit IRC | 02:42 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Revert "zuul/layout: extend puppet-openstack-integration testing" https://review.openstack.org/213582 | 02:43 |
| *** jeadr has joined #openstack-infra | 02:43 | |
| *** sputnik13 has quit IRC | 02:44 | |
| *** sputnik13 has joined #openstack-infra | 02:44 | |
| *** laughterwym_ has joined #openstack-infra | 02:50 | |
| *** laughterwym has quit IRC | 02:53 | |
| *** sabeen has joined #openstack-infra | 02:54 | |
| *** laughterwym_ has quit IRC | 02:55 | |
| *** laughterwym has joined #openstack-infra | 02:55 | |
| *** gildub has joined #openstack-infra | 02:58 | |
| *** sputnik13 has quit IRC | 03:04 | |
| *** sputnik13 has joined #openstack-infra | 03:07 | |
| *** sputnik13 has quit IRC | 03:10 | |
| *** asettle-afk is now known as asettle | 03:10 | |
| *** sputnik13 has joined #openstack-infra | 03:13 | |
| *** sabeen has quit IRC | 03:14 | |
| *** sabeen has joined #openstack-infra | 03:15 | |
| *** FallenPegasus has joined #openstack-infra | 03:16 | |
| *** jeadr has quit IRC | 03:16 | |
| *** jeadr has joined #openstack-infra | 03:17 | |
| *** tiswanso has quit IRC | 03:20 | |
| *** FallenPegasus has quit IRC | 03:21 | |
| *** camunoz has quit IRC | 03:29 | |
| *** ruagair has quit IRC | 03:30 | |
| *** ruagair has joined #openstack-infra | 03:30 | |
| *** flepied has joined #openstack-infra | 03:33 | |
| *** flepied1 has quit IRC | 03:34 | |
| *** sabeen has quit IRC | 03:44 | |
| *** camunoz has joined #openstack-infra | 03:44 | |
| *** jeadr has quit IRC | 03:47 | |
| *** jeadr has joined #openstack-infra | 03:48 | |
| *** ayoung has quit IRC | 03:52 | |
| *** dimtruck is now known as zz_dimtruck | 03:54 | |
| *** salv-orlando has joined #openstack-infra | 03:54 | |
| *** fedexo has joined #openstack-infra | 03:56 | |
| *** jeadr has quit IRC | 03:58 | |
| *** jeadr has joined #openstack-infra | 03:59 | |
| openstackgerrit | Merged openstack-infra/system-config: Add Mistral channel to eavesdrop https://review.openstack.org/212550 | 04:03 |
| *** chen12 has joined #openstack-infra | 04:04 | |
| *** changbl has quit IRC | 04:04 | |
| *** camunoz has quit IRC | 04:06 | |
| *** sergmelikyan has joined #openstack-infra | 04:06 | |
| *** salv-orlando has quit IRC | 04:06 | |
| *** camunoz has joined #openstack-infra | 04:06 | |
| *** sergmelikyan has quit IRC | 04:10 | |
| *** ybathia has joined #openstack-infra | 04:15 | |
| *** openstack has joined #openstack-infra | 04:20 | |
| openstackgerrit | Asha proposed openstack/requirements: Version bump for hacking and coverage https://review.openstack.org/213255 | 04:21 |
| *** ybathia_ has joined #openstack-infra | 04:22 | |
| *** ybathia has quit IRC | 04:23 | |
| *** ybathia_ is now known as ybathia | 04:23 | |
| *** yamamoto has quit IRC | 04:23 | |
| *** markvoelker has quit IRC | 04:31 | |
| *** sputnik13 has quit IRC | 04:31 | |
| *** jeadr has quit IRC | 04:31 | |
| *** jeadr has joined #openstack-infra | 04:32 | |
| *** BharatK has joined #openstack-infra | 04:43 | |
| *** geoffarnoldX is now known as geoffarnold | 04:49 | |
| *** geoffarnold is now known as geoffarnoldX | 04:50 | |
| *** abregman has joined #openstack-infra | 04:51 | |
| *** abregman is now known as abregman|afk | 04:51 | |
| *** yamamoto has joined #openstack-infra | 04:51 | |
| *** jeadr has quit IRC | 04:53 | |
| *** jeadr has joined #openstack-infra | 04:54 | |
| *** bpokorny has quit IRC | 04:56 | |
| *** deepakcs has joined #openstack-infra | 04:57 | |
| *** sputnik13 has joined #openstack-infra | 04:57 | |
| *** jerryz has joined #openstack-infra | 05:05 | |
| *** FallenPegasus has joined #openstack-infra | 05:05 | |
| *** fedexo has quit IRC | 05:05 | |
| *** FallenPegasus has quit IRC | 05:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "zuul/layout: extend puppet-openstack-integration testing" https://review.openstack.org/213582 | 05:11 |
| *** geoffarnoldX is now known as geoffarnold | 05:15 | |
| *** FallenPegasus has joined #openstack-infra | 05:20 | |
| *** ttx has joined #openstack-infra | 05:22 | |
| *** flepied has quit IRC | 05:24 | |
| *** flepied1 has joined #openstack-infra | 05:24 | |
| *** asettle has quit IRC | 05:25 | |
| *** flepied1 has quit IRC | 05:26 | |
| *** asettle has joined #openstack-infra | 05:26 | |
| *** asettle has quit IRC | 05:26 | |
| *** Sree has joined #openstack-infra | 05:26 | |
| *** asettle has joined #openstack-infra | 05:27 | |
| *** flepied has joined #openstack-infra | 05:27 | |
| *** geoffarnold is now known as geoffarnoldX | 05:27 | |
| *** mrunge has joined #openstack-infra | 05:28 | |
| *** achanda has joined #openstack-infra | 05:30 | |
| *** flepied has quit IRC | 05:31 | |
| *** markvoelker has joined #openstack-infra | 05:32 | |
| *** coolsvap|away is now known as coolsvap | 05:32 | |
| *** geoffarnoldX has quit IRC | 05:32 | |
| *** nadya has joined #openstack-infra | 05:33 | |
| *** markvoelker has quit IRC | 05:36 | |
| *** enikanorov2 has joined #openstack-infra | 05:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental scenario dsvm jobs for Sahara https://review.openstack.org/213541 | 05:38 |
| *** hdd has joined #openstack-infra | 05:38 | |
| *** enikanorov2 has quit IRC | 05:43 | |
| *** salv-orlando has joined #openstack-infra | 05:44 | |
| *** FallenPegasus has quit IRC | 05:45 | |
| *** peristeri has joined #openstack-infra | 05:48 | |
| *** alexpilotti has quit IRC | 05:50 | |
| *** salv-orlando has quit IRC | 05:51 | |
| *** nadya has quit IRC | 05:52 | |
| *** coolsvap is now known as coolsvap|away | 05:55 | |
| *** ybathia has quit IRC | 05:56 | |
| *** yonglihe has joined #openstack-infra | 06:01 | |
| ianw | dstufft: so i believe https://github.com/pypa/pip/issues/2940 (pip installing binaries into /usr/lib instead of /usr/lib64) is causing f22 breakages ... it's subtle because the packaged library remains, so you only notice down the track at runtime when you finally start making calls that don't exist | 06:03 |
| yonglihe | just tell this team, Intel PCI/NGFW/PTAS CI is shunting down for maintenance due to kernel bug, will back for today or tomorrow. | 06:04 |
| ianw | dstufft: if you can take a look, awesome. i can dive a bit deeper later | 06:04 |
| ianw | jamielennox lifeless : cc ^ | 06:04 |
| *** afazekas has quit IRC | 06:10 | |
| *** abregman|afk is now known as abregman | 06:13 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/213606 | 06:14 |
| *** yamamoto has quit IRC | 06:16 | |
| *** Sukhdev has joined #openstack-infra | 06:17 | |
| *** yamamoto has joined #openstack-infra | 06:17 | |
| *** vlaza has joined #openstack-infra | 06:18 | |
| *** claudiub has joined #openstack-infra | 06:21 | |
| *** salv-orlando has joined #openstack-infra | 06:23 | |
| *** hdd has quit IRC | 06:24 | |
| *** scheuran has joined #openstack-infra | 06:26 | |
| *** scheuran has quit IRC | 06:27 | |
| *** flepied has joined #openstack-infra | 06:27 | |
| *** laughterwym has quit IRC | 06:29 | |
| *** akscram has quit IRC | 06:32 | |
| *** akscram has joined #openstack-infra | 06:33 | |
| *** yamamoto has quit IRC | 06:34 | |
| *** yamamoto has joined #openstack-infra | 06:34 | |
| openstackgerrit | Doron Chen proposed openstack-infra/project-config: adding python27 test to the storlets project https://review.openstack.org/213613 | 06:36 |
| *** ifarkas has joined #openstack-infra | 06:40 | |
| *** sdake_ has joined #openstack-infra | 06:44 | |
| *** scheuran has joined #openstack-infra | 06:45 | |
| *** Sukhdev has quit IRC | 06:47 | |
| *** asettle has quit IRC | 06:47 | |
| *** sdake has quit IRC | 06:47 | |
| *** vinay_yadhav has joined #openstack-infra | 06:48 | |
| *** zhenguo has joined #openstack-infra | 06:49 | |
| *** zeih has joined #openstack-infra | 06:52 | |
| *** peristeri has quit IRC | 06:53 | |
| *** AJaeger has joined #openstack-infra | 06:54 | |
| *** jeadr has quit IRC | 06:54 | |
| *** scheuran has quit IRC | 06:54 | |
| *** sileht has quit IRC | 06:55 | |
| *** jeadr has joined #openstack-infra | 06:55 | |
| *** sileht has joined #openstack-infra | 06:56 | |
| AJaeger | gerrit seems to be really slow right now. Is that just me? | 06:56 |
| StevenK | Nope, it isn't just you. | 06:56 |
| AJaeger | Hi StevenK ! | 06:56 |
| StevenK | AJaeger: Hai! | 06:57 |
| AJaeger | jhesketh: are you around and could check why gerrit is soo slow, please? | 06:57 |
| jhesketh | AJaeger: actually already looking | 06:57 |
| *** scheuran has joined #openstack-infra | 06:57 | |
| jhesketh | CPU and load are huge, but there is nothing obvious in the logs | 06:57 |
| jhesketh | or strace etc | 06:57 |
| jhesketh | there seems to be 8 or so threads that have hung around, but I'm not sure what they are doing yet | 06:58 |
| jhesketh | I'm very hesitent to do a restart of gerrit without knowing the cause | 06:59 |
| AJaeger | thanks, jhesketh ! | 06:59 |
| * AJaeger gives up to interact with gerrit for now | 07:00 | |
| *** afazekas_ has joined #openstack-infra | 07:01 | |
| *** Sree has quit IRC | 07:01 | |
| AJaeger | jhesketh: I understand your hesitance, hope you find the problem | 07:01 |
| jhesketh | don't thank me yet... I'm out of ideas | 07:01 |
| *** Sree has joined #openstack-infra | 07:02 | |
| jhesketh | a slow gerrit is (probably) better than no gerrit, so I may wait for another root to appear | 07:02 |
| jhesketh | jeblair, clarkb, fungi, mordred, pleia2, SergeyLukjanov: ^ | 07:02 |
| AJaeger | jhesketh: in that case: Could you send an IRC notification and an email out, please? | 07:03 |
| jhesketh | AJaeger: good idea! | 07:03 |
| AJaeger | jhesketh: FYI, fungi won't be here probably for another 5 hours - and he's the first root to show up (unless SergeyLukjanov does) | 07:04 |
| jhesketh | #status notice Gerrit is currently under very high load and may be unresponsive. infra are looking into the issue. | 07:05 |
| openstackstatus | jhesketh: sending notice | 07:05 |
| jhesketh | AJaeger: yeah, I realise it'll be a while :-( | 07:05 |
| *** Sree has quit IRC | 07:06 | |
| -openstackstatus- NOTICE: Gerrit is currently under very high load and may be unresponsive. infra are looking into the issue. | 07:06 | |
| *** camunoz has quit IRC | 07:06 | |
| openstackstatus | jhesketh: finished sending notice | 07:07 |
| AJaeger | jhesketh: the load is low, it's Monday and vacation time - so might work to wait ;) | 07:07 |
| *** sdake_ has quit IRC | 07:08 | |
| jhesketh | that's what I'm hoping | 07:08 |
| jhesketh | maybe people will write more patches this way ;-) | 07:08 |
| AJaeger | more *quality* patches that are *locally* tested and pass directly ;) | 07:09 |
| *** jtomasek has joined #openstack-infra | 07:09 | |
| nibalizer | the show-queue only says 7 jobs | 07:09 |
| nibalizer | but one has been around for more than an hour | 07:09 |
| nibalizer | 693d5b85 06:50:51.204 github/stackforge/fuel-library | 07:09 |
| nibalizer | Idunno what that job means though | 07:10 |
| nibalizer | but it also seems to have cleared those | 07:10 |
| nibalizer | oh not more than an hour, just several minutes | 07:11 |
| AJaeger | I don't see that job in status.o.o/zuul | 07:11 |
| *** sshnaidm has joined #openstack-infra | 07:11 | |
| nibalizer | I'm looking at 'gerrit show-queue' (which is available form the gerrit ssh cli) | 07:11 |
| nibalizer | at any rate, it has completed | 07:11 |
| *** markus_z has joined #openstack-infra | 07:14 | |
| jhesketh | gerrit is "working" (for certain values of working), it's just very slow | 07:14 |
| jhesketh | I can't find anything that's particularly broken, but it's more likely I've just missed it | 07:14 |
| *** pblaho has joined #openstack-infra | 07:15 | |
| *** vinay_yadhav has quit IRC | 07:16 | |
| *** k4n0 has joined #openstack-infra | 07:17 | |
| *** Hal has joined #openstack-infra | 07:17 | |
| *** peristeri has joined #openstack-infra | 07:17 | |
| *** Hal is now known as Guest38687 | 07:18 | |
| *** zhenguo has quit IRC | 07:18 | |
| *** camunoz has joined #openstack-infra | 07:19 | |
| *** peristeri has quit IRC | 07:21 | |
| *** peristeri has joined #openstack-infra | 07:21 | |
| *** Guest38687 has quit IRC | 07:23 | |
| *** ildikov has quit IRC | 07:23 | |
| *** peristeri has quit IRC | 07:31 | |
| lifeless | ianw: AIUI we install cffi then install python-cffi after | 07:31 |
| lifeless | ianw: I may be remembering wrongly | 07:31 |
| *** pblaho has quit IRC | 07:32 | |
| *** pblaho has joined #openstack-infra | 07:32 | |
| *** markvoelker has joined #openstack-infra | 07:33 | |
| *** FallenPegasus has joined #openstack-infra | 07:33 | |
| *** enikanorov2 has joined #openstack-infra | 07:34 | |
| yolanda | mm, i'll leave reviews for the afternoon | 07:34 |
| *** peristeri has joined #openstack-infra | 07:34 | |
| *** sthillma has joined #openstack-infra | 07:35 | |
| *** dizquierdo has joined #openstack-infra | 07:35 | |
| *** devvesa has joined #openstack-infra | 07:35 | |
| *** Sree has joined #openstack-infra | 07:36 | |
| *** sthillma_ has joined #openstack-infra | 07:36 | |
| nibalizer | jhesketh: http://paste.openstack.org/show/418861/ I tend to be suspicious of the nova operations since its the biggest repo | 07:37 |
| *** markvoelker has quit IRC | 07:37 | |
| *** FallenPegasus has quit IRC | 07:38 | |
| *** matbu has joined #openstack-infra | 07:38 | |
| *** zhenguo has joined #openstack-infra | 07:39 | |
| *** fhubik has joined #openstack-infra | 07:39 | |
| *** sthillma has quit IRC | 07:39 | |
| *** sthillma_ is now known as sthillma | 07:39 | |
| *** barthalion has joined #openstack-infra | 07:41 | |
| *** camunoz has quit IRC | 07:41 | |
| *** sthillma_ has joined #openstack-infra | 07:41 | |
| *** arxcruz has joined #openstack-infra | 07:42 | |
| barthalion | hi folks, who should I poke about adding me as a core for newly created repositories on stackforge? these three to be more specific: https://review.openstack.org/#/q/owner:%22Bart%25C5%2582omiej+Piotrowski%22+status:merged+project:openstack-infra/project-config,n,z | 07:42 |
| deepakcs | barthalion: at this TZ it should be jhesketh | 07:44 |
| AJaeger | nibalizer: but those disappear again, so they do finish - don't they? | 07:44 |
| *** sthillma has quit IRC | 07:44 | |
| *** sthillma_ is now known as sthillma | 07:44 | |
| barthalion | deepakcs: thanks | 07:45 |
| jhesketh | barthalion: will look soon, currently working on gerrit stuff | 07:45 |
| barthalion | jhesketh: cool, thank you :) | 07:45 |
| *** jlanoux has joined #openstack-infra | 07:46 | |
| jhesketh | hmm, getting a lot of these http://paste.openstack.org/show/418862/ in the gerrit log... not sure why I didn't notice them earlier | 07:47 |
| nibalizer | AJaeger: I'm not really sure how it works, but at a high level ibm-zkvm-ci and hyper-v-ci have been trying to clone nova for almost an hour (since I've been watching) | 07:47 |
| nibalizer | is that a symptom or a cause I do not know | 07:47 |
| *** fhubik is now known as fhubik_brb | 07:49 | |
| AJaeger | nibalizer: clone? Does git-upload-pack mean they are cloning? | 07:49 |
| nibalizer | I think so | 07:49 |
| jhesketh | nibalizer: Are you cloning from git.o.o? | 07:54 |
| jhesketh | unfortunately I have to run to an appointment so I'll be back later | 07:54 |
| *** gildub has quit IRC | 07:55 | |
| *** nadya has joined #openstack-infra | 07:56 | |
| *** dizquierdo has quit IRC | 07:57 | |
| *** matbu has quit IRC | 07:57 | |
| *** dizquierdo has joined #openstack-infra | 07:57 | |
| *** matbu has joined #openstack-infra | 08:02 | |
| *** Yanyanhu has joined #openstack-infra | 08:04 | |
| *** zhenguo has quit IRC | 08:04 | |
| *** matbu has quit IRC | 08:07 | |
| *** sthillma has quit IRC | 08:07 | |
| *** sshnaidm has quit IRC | 08:08 | |
| *** fhubik_brb is now known as fhubik | 08:08 | |
| *** ildikov has joined #openstack-infra | 08:08 | |
| *** shardy has joined #openstack-infra | 08:09 | |
| *** achanda has quit IRC | 08:10 | |
| *** afazekas_ is now known as afazkas | 08:12 | |
| nibalizer | I'm off as well, sorry | 08:12 |
| afazkas | review.openstack.org seams DOWN, are there any known issue or is it a planned downtime ? | 08:12 |
| *** devvesa has quit IRC | 08:14 | |
| *** fhubik is now known as fhubik_brb | 08:14 | |
| *** derekh has joined #openstack-infra | 08:14 | |
| nibalizer | afazkas: it is a known issue | 08:15 |
| nibalizer | it should be functional just very very slow | 08:16 |
| nibalizer | but yes I can't load the web interface at this point :/ | 08:16 |
| *** fhubik_brb is now known as fhubik | 08:16 | |
| *** ihrachyshka has joined #openstack-infra | 08:17 | |
| *** fhubik is now known as fhubik_brb | 08:18 | |
| jd__ | yeah too slow to be working | 08:18 |
| *** achanda has joined #openstack-infra | 08:20 | |
| *** jistr has joined #openstack-infra | 08:21 | |
| *** fhubik_brb is now known as fhubik | 08:21 | |
| openstackgerrit | Simon Pasquier proposed openstack-infra/project-config: Enable merge commits for fuel-plugin-influxdb-grafana-release group https://review.openstack.org/213627 | 08:22 |
| *** fhubik is now known as fhubik_brb | 08:24 | |
| *** sshnaidm has joined #openstack-infra | 08:24 | |
| ianw | lifeless: yeah, it's not that ... the pypi version gets built into a wheel and installed, but it's picking up the .so from the rpm version because of https://github.com/pypa/pip/issues/2940 | 08:25 |
| *** matbu has joined #openstack-infra | 08:28 | |
| *** fbo has joined #openstack-infra | 08:29 | |
| *** zhenguo has joined #openstack-infra | 08:29 | |
| *** notnownikki has joined #openstack-infra | 08:30 | |
| notnownikki | jhesketh, hi, gerrit seems *really* slow at the moment, is there an issue? | 08:32 |
| *** shardy_ has joined #openstack-infra | 08:33 | |
| lifeless | ianw: ah | 08:34 |
| *** shardy has quit IRC | 08:34 | |
| lifeless | notnownikki: yes; see -dev list | 08:34 |
| *** achanda has quit IRC | 08:35 | |
| *** dguitarbite has joined #openstack-infra | 08:35 | |
| *** matbu has quit IRC | 08:35 | |
| *** fhubik_brb is now known as fhubik | 08:37 | |
| *** jyuso1 has quit IRC | 08:37 | |
| *** matbu has joined #openstack-infra | 08:37 | |
| *** fhubik is now known as fhubik_brb | 08:38 | |
| *** liam__ has joined #openstack-infra | 08:38 | |
| *** dizquierdo has quit IRC | 08:38 | |
| *** shardy_ has quit IRC | 08:38 | |
| *** liam__ has quit IRC | 08:38 | |
| *** dizquierdo has joined #openstack-infra | 08:39 | |
| *** shardy has joined #openstack-infra | 08:39 | |
| openstackgerrit | Daniel Danner proposed openstack-infra/jenkins-job-builder: Add support for GroovyAxis https://review.openstack.org/213630 | 08:39 |
| notnownikki | notnownikki, i'd offer to help (lots of root experience on our downstream CI!) but I guess there's not much I can do here :( | 08:40 |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/infra-manual: developer's guide: Add bug handling description https://review.openstack.org/192232 | 08:40 |
| notnownikki | grr, i meant lifeless ^^ not myself haha | 08:43 |
| notnownikki | i need coffee... | 08:43 |
| *** jeadr has quit IRC | 08:43 | |
| *** prmtl_ has joined #openstack-infra | 08:43 | |
| *** jeadr has joined #openstack-infra | 08:44 | |
| yolanda | at this point , what we do is to check which task is starving gerrit,try to kill it. If not possible, just restart gerrit | 08:45 |
| *** lucasagomes has joined #openstack-infra | 08:47 | |
| *** dingyichen has quit IRC | 08:47 | |
| lucasagomes | hi all, gerrit is really really slow today. There's any maintenance going on? | 08:47 |
| yolanda | lucasagomes, seems to be a known issue | 08:48 |
| AJaeger | lucasagomes: see http://lists.openstack.org/pipermail/openstack-dev/2015-August/072185.html | 08:48 |
| notnownikki | yolanda, lifeless, I noticed it took a *long* time negotiating a secure connection, I'd also check for low entropy just in case it's blocked waiting for /dev/random to deliver things | 08:48 |
| lucasagomes | yolanda, AJaeger thanks guys. Sorry I just noticed it and thought about reporting here | 08:48 |
| *** ttx has quit IRC | 08:48 | |
| *** ttx has joined #openstack-infra | 08:48 | |
| *** dingyichen has joined #openstack-infra | 08:48 | |
| *** dingyichen has quit IRC | 08:52 | |
| *** robcresswell has joined #openstack-infra | 08:53 | |
| *** marekd_404 has quit IRC | 08:54 | |
| *** skolekonov has joined #openstack-infra | 08:55 | |
| *** aarefiev has quit IRC | 08:56 | |
| lifeless | notnownikki: I don't know where the investigation is up to tbh sorry | 08:56 |
| notnownikki | lifeless, i guess we don't have many roots in EMEA timezone? | 08:56 |
| notnownikki | i mean I've had git mirrors go down and not be able to get changes through check, and there's been no support in this timezone to even investigate | 08:57 |
| AJaeger | notnownikki: read the backscroll on eavesdrop ;) | 08:58 |
| *** fhubik_brb is now known as fhubik | 08:58 | |
| *** aarefiev has joined #openstack-infra | 08:58 | |
| AJaeger | notnownikki: jhesketh is around but currently in a meeting... | 08:58 |
| StevenK | AJaeger: Which isn't EMEA. :-) | 08:58 |
| *** rraja has joined #openstack-infra | 08:58 | |
| AJaeger | StevenK: Correct | 08:59 |
| notnownikki | so who are our EMEA roots? | 08:59 |
| *** aarefiev has quit IRC | 08:59 | |
| yolanda | SergeyLukjanov | 08:59 |
| AJaeger | notnownikki: SergeyLukjanov but he's travelling also often to the US ;/ | 08:59 |
| AJaeger | And has now other priorities AFAIU | 08:59 |
| *** aarefiev has joined #openstack-infra | 08:59 | |
| notnownikki | may I suggest this is another reason we find it difficult to contribute in this timezone? | 09:00 |
| yolanda | there is some hour gaps between jhesketh goes off and fungi starts | 09:00 |
| AJaeger | notnownikki: I completly agree but we need volunteers that have the right expertise for this. | 09:01 |
| yolanda | i volunteer | 09:01 |
| AJaeger | notnownikki: It's not that bad in general. We had a few hickups over the last few weeks but in general it's fine. | 09:01 |
| yolanda | and i've been like a year and a half managing downstream CI, so i have the expertise | 09:02 |
| notnownikki | AJaeger, i've been root on HP's internal CI for over a year | 09:02 |
| AJaeger | yolanda: Great, please talk with jeblair as PTL about this. | 09:02 |
| yolanda | i did it on the past, i'll try to move it again | 09:02 |
| yolanda | myself or anyone from Gozer team will be happy to be | 09:02 |
| AJaeger | notnownikki: same comment as to yolanda - please discuss with jeblair | 09:03 |
| yolanda | notnownikki, do you want that we add the topic to the Gozer agenda for tomorrow? | 09:04 |
| yolanda | are you able to attend? | 09:04 |
| notnownikki | yolanda, no, i have an appointment in the evening. *another* reason EMEA contributions get overlooked... | 09:05 |
| yolanda | i can be there and represent our concerns | 09:05 |
| yolanda | or we can try to talk informally today | 09:05 |
| notnownikki | yolanda, and if it's so difficult to even get our reviews looked at, I don't see what hope we have of getting root to support the EMEA timezone | 09:05 |
| yolanda | well, worth trying | 09:05 |
| *** pelix has joined #openstack-infra | 09:06 | |
| yolanda | i guess we've had here in Gozer the same or more problems that upstream, so we know how to deal with them | 09:06 |
| notnownikki | more problems I'd say | 09:06 |
| yolanda | more pressure to fix at the same moment for sure | 09:06 |
| *** nadya has quit IRC | 09:06 | |
| *** samuelBartel has quit IRC | 09:06 | |
| notnownikki | yeah | 09:06 |
| notnownikki | AJaeger, if Gozer has, for example, a problem with the pypi mirrors, we have customers on our backs asking for fixes within *minutes* :) So we're used to diagnosing and fixing problems under pressure, with all the openstack CI systems | 09:07 |
| AJaeger | Great to hear, notnownikki! | 09:08 |
| *** jschwarz has joined #openstack-infra | 09:08 | |
| jschwarz | Hi guys.. any reason why review.openstack.org is so slow today? :< | 09:09 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Add check-requirements job to networking-ofagent project https://review.openstack.org/213639 | 09:09 |
| AJaeger | jschwarz: known issue, under investigation. | 09:09 |
| jschwarz | yey, thanks AJaeger :) | 09:09 |
| *** nadya has joined #openstack-infra | 09:10 | |
| *** AJaeger has quit IRC | 09:10 | |
| *** FallenPegasus has joined #openstack-infra | 09:14 | |
| *** FallenPegasus has quit IRC | 09:19 | |
| *** sshnaidm has quit IRC | 09:20 | |
| *** dizquierdo has quit IRC | 09:23 | |
| *** abregman is now known as abregman|afk | 09:25 | |
| *** scheuran has quit IRC | 09:29 | |
| *** dizquierdo has joined #openstack-infra | 09:31 | |
| *** samuelBartel has joined #openstack-infra | 09:31 | |
| *** sshnaidm has joined #openstack-infra | 09:32 | |
| *** markvoelker has joined #openstack-infra | 09:33 | |
| *** abregman|afk is now known as abregman | 09:35 | |
| *** samuelBartel has quit IRC | 09:38 | |
| *** markvoelker has quit IRC | 09:38 | |
| * jhesketh returns | 09:44 | |
| *** devvesa has joined #openstack-infra | 09:44 | |
| jhesketh | it doesn't look like anything has changed | 09:46 |
| ttx | jhesketh: I confirm. | 09:48 |
| *** Guest47951 is now known as d0ugal | 09:49 | |
| *** d0ugal has quit IRC | 09:49 | |
| *** d0ugal has joined #openstack-infra | 09:49 | |
| jhesketh | seems worse actually | 09:51 |
| jhesketh | possibly due to more people being awake | 09:51 |
| *** vinay_yadhav has joined #openstack-infra | 09:54 | |
| jhesketh | looks like we might be hitting https://code.google.com/p/gerrit/wiki/Scaling#Jetty | 09:57 |
| *** jlanoux_ has joined #openstack-infra | 09:57 | |
| *** Sree__ has joined #openstack-infra | 09:59 | |
| *** jlanoux has quit IRC | 09:59 | |
| lexloofah | wait gerrit runs on jetty? | 09:59 |
| *** lexloofah is now known as lxsli | 10:00 | |
| lxsli | I'm kinda impressed it manages as well as it does | 10:00 |
| *** Sree has quit IRC | 10:03 | |
| notnownikki | jhesketh, we've seen that problem too | 10:04 |
| jhesketh | notnownikki: from my earlier paste? | 10:04 |
| notnownikki | jhesketh, i didn't see your paste, but we've seen the 100% cpu and "failed to dispatch" error relating to jetty | 10:04 |
| jhesketh | notnownikki: right, sounds the same... did you find a solution? | 10:05 |
| notnownikki | jhesketh, the only workaround we could find was the one mentioned there, http.maxqueued = 0, although our gerrit is much lower volume so I wouldn't be comfortable saying to use that here without significant stress testing beforehand | 10:06 |
| jhesketh | hmm okay | 10:07 |
| notnownikki | temporary solution was to restart gerrit | 10:08 |
| jhesketh | yeah, I'm leaning towards doing that | 10:08 |
| jhesketh | the question is, if it doesn't come back up, are we worse off | 10:09 |
| jhesketh | ie waiting for another root to help fix the slowness or another root to help fix everything | 10:09 |
| notnownikki | so what's the situation, are you seeing the jetty errors in logs and getting 100% cpu usage from gerrit? | 10:10 |
| jhesketh | notnownikki: yep | 10:11 |
| notnownikki | jhesketh, yeah, there's not much you can do without restarting at that point, that message comes from the JettyHttpServerTransportHandler and it's jetty getting clogged up, not being able to dispatch new rest requests through the HttpServerAdapter | 10:14 |
| jhesketh | figured as much | 10:14 |
| jhesketh | I'm just worried I won't be able to do much if it doesn't come back up cleanly | 10:15 |
| notnownikki | we never had a problem with that, the only problems we saw with it was zuul taking some time to get reconnected and processing events again, but that problem was a while ago with an old zuul version | 10:16 |
| jhesketh | notnownikki: okay, thanks | 10:16 |
| jhesketh | ttx: what's your opinion. Should I restart gerrit with the possibility it won't come up cleanly, or shall we put up with the slowness until somebody more skilled than myself turns up | 10:16 |
| *** maxandersen has joined #openstack-infra | 10:17 | |
| *** maxandersen has joined #openstack-infra | 10:17 | |
| *** dims has joined #openstack-infra | 10:17 | |
| *** fhubik has quit IRC | 10:18 | |
| jhesketh | hmm, it's only gotten worse, to the point of unusable now | 10:18 |
| jhesketh | I'll do a restart | 10:18 |
| notnownikki | yeah it gets to the point where you can't make any new connections | 10:18 |
| *** fhubik has joined #openstack-infra | 10:18 | |
| jhesketh | #status alert review.openstack.org (aka gerrit) is going down for an emergency restart | 10:19 |
| openstackstatus | jhesketh: sending alert | 10:19 |
| jhesketh | okay, backup taken, going to restart | 10:20 |
| -openstackstatus- NOTICE: review.openstack.org (aka gerrit) is going down for an emergency restart | 10:21 | |
| *** ChanServ changes topic to "review.openstack.org (aka gerrit) is going down for an emergency restart" | 10:21 | |
| jhesketh | restart complete | 10:22 |
| notnownikki | ah, that's better :) | 10:22 |
| jhesketh | seems to have come back up, thankfully | 10:22 |
| notnownikki | best check zuul is connected properly | 10:22 |
| jhesketh | yep, just looking now | 10:22 |
| notnownikki | :) | 10:23 |
| openstackstatus | jhesketh: finished sending alert | 10:23 |
| yolanda | woot, much better | 10:24 |
| dims | definitely better jhesketh. thanks! | 10:24 |
| jhesketh | looks like zuul is catching up | 10:24 |
| notnownikki | cool :) | 10:24 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-project_config: Allow to define project_config branch from parameter https://review.openstack.org/211584 | 10:25 |
| jhesketh | thanks for your help notnownikki :-) | 10:25 |
| notnownikki | jhesketh, no problem :) me, yolanda, and rcarrillocruz run the HP internal CI in this timezone, so if you ever need to ping someone, we're happy to help :) | 10:26 |
| *** jlanoux has joined #openstack-infra | 10:26 | |
| jhesketh | notnownikki: awesome, good to know, thanks ::-) | 10:27 |
| jhesketh | * :-) | 10:28 |
| *** Sree__ has quit IRC | 10:28 | |
| *** baoli has joined #openstack-infra | 10:28 | |
| *** Sree has joined #openstack-infra | 10:28 | |
| *** jlanoux_ has quit IRC | 10:29 | |
| yolanda | jhesketh, thx for your reviews on nodepool, really good ones. I agree with you on removing the targets keyword, but i'd like to expose to more people as it's a change in config | 10:30 |
| *** AJaeger has joined #openstack-infra | 10:30 | |
| jhesketh | yolanda: no trouble | 10:30 |
| *** dims has quit IRC | 10:30 | |
| jhesketh | yolanda: I suspect if we do remove the jenkins key we'd keep a depreciated support for it | 10:30 |
| jhesketh | which wouldn't actually be that hard | 10:31 |
| *** dims has joined #openstack-infra | 10:31 | |
| jhesketh | yolanda: your change, from memory, already requires a config change since deployers will have to move their jenkins credentials | 10:31 |
| yolanda | yes, it's a very disruptive chane | 10:31 |
| jhesketh | yolanda: maybe we want to load the whole jenkins dict in with a depreciated warning for a while and drop it later | 10:31 |
| yolanda | having a dict for a single keyword makes no sense, so i mostly agree | 10:32 |
| *** Sree has quit IRC | 10:33 | |
| *** baoli has quit IRC | 10:33 | |
| yolanda | even we could just drop it without deprecation, using the opportunity that we are introduing that disruptive change, we agreed with jeblair that we'll move it forward with that secure.conf, without caring about backwards compatibility | 10:33 |
| yolanda | i'll raise the topic with some infra people this afternoon, and i'll amend the change if we agree | 10:33 |
| *** Yanyanhu has quit IRC | 10:34 | |
| jhesketh | yeah if we're breaking compat anyway, I think we should make this change while we're there | 10:34 |
| jhesketh | otherwise we'll have to make it another time and that'll be lame | 10:34 |
| jhesketh | yolanda: I'd also like to see nodepool not require jenkins (that's how we run it for turbo-hipster), but that's a follow up change I'll look at making | 10:35 |
| yolanda | jhesketh, do you know if that test-job is even actively used? haven't seen any usage on upstream | 10:35 |
| jhesketh | yolanda: no idea... It's a new feature so chances are no | 10:35 |
| yolanda | jhesketh , well, jenkins should be removed soon as part of zuul v3 | 10:36 |
| *** dims_ has joined #openstack-infra | 10:36 | |
| yolanda | jhesketh, you run a patched version of nodepool to use turbo-hipster instead of jenkins? | 10:36 |
| jhesketh | yolanda: right | 10:36 |
| jhesketh | (on both parts) | 10:36 |
| *** yamamoto has quit IRC | 10:38 | |
| yolanda | ah, about the comment you made about the name of logfile, you just mean mention the name of the nodepool log? | 10:38 |
| *** dims has quit IRC | 10:38 | |
| *** marekd has joined #openstack-infra | 10:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove extra pyhon-jobs from networking-ofagent https://review.openstack.org/213656 | 10:40 |
| *** marekd is now known as marekd_404 | 10:40 | |
| *** mwagner_lap has quit IRC | 10:42 | |
| *** erlon has joined #openstack-infra | 10:43 | |
| jhesketh | #status ok Gerrit restart has resolved the issue and systems are back up and functioning | 10:45 |
| openstackstatus | jhesketh: sending ok | 10:45 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 10:48 | |
| -openstackstatus- NOTICE: Gerrit restart has resolved the issue and systems are back up and functioning | 10:48 | |
| openstackstatus | jhesketh: finished sending ok | 10:50 |
| *** dims_ has quit IRC | 10:50 | |
| *** dims has joined #openstack-infra | 10:51 | |
| SergeyLukjanov | yolanda, AJaeger, jhesketh, hey folks, sorry, I'm later today | 10:51 |
| jhesketh | SergeyLukjanov: all good, just wanted a second person before restarting gerrit | 10:52 |
| jhesketh | done anyway and surprisingly painless | 10:52 |
| *** mase_home has joined #openstack-infra | 10:52 | |
| SergeyLukjanov | jhesketh, AFAIR gerrit restart was always good ;) | 10:52 |
| SergeyLukjanov | jhesketh, what was the issue in a few words? | 10:52 |
| yolanda | yes, gerrit start is normally safe, less disruptive than others | 10:53 |
| *** deepakcs has quit IRC | 10:53 | |
| *** fhubik is now known as fhubik_brb | 10:53 | |
| jhesketh | SergeyLukjanov: basically paste.openstack.org/show/418862/ in gerrits error log | 10:54 |
| *** pc_m has joined #openstack-infra | 10:54 | |
| jhesketh | using lots of CPU and under high load (see cacti) | 10:54 |
| jhesketh | which appears to be a somewhat known issue, mentioned https://code.google.com/p/gerrit/wiki/Scaling#Jetty | 10:54 |
| barthalion | jhesketh: humbly reminding about myself, if you have spare moment now | 10:56 |
| jhesketh | barthalion: sure, got those links again please | 10:59 |
| SergeyLukjanov | jhesketh, ok, I've never seen this error in gerrit | 10:59 |
| *** dims has quit IRC | 10:59 | |
| barthalion | jhesketh: https://review.openstack.org/#/q/owner:%22Bart%25C5%2582omiej+Piotrowski%22+status:merged+project:openstack-infra/project-config,n,z | 10:59 |
| *** dims has joined #openstack-infra | 11:00 | |
| jhesketh | barthalion: should be all done now | 11:02 |
| *** FallenPegasus has joined #openstack-infra | 11:03 | |
| *** leopoldj has joined #openstack-infra | 11:04 | |
| *** marcusvrn has joined #openstack-infra | 11:05 | |
| *** dims has quit IRC | 11:06 | |
| *** FallenPegasus has quit IRC | 11:07 | |
| *** dims has joined #openstack-infra | 11:08 | |
| *** boris-42 has quit IRC | 11:10 | |
| *** rkukura has quit IRC | 11:10 | |
| *** aukhan has quit IRC | 11:11 | |
| *** dims_ has joined #openstack-infra | 11:12 | |
| *** dims has quit IRC | 11:12 | |
| *** larainema has quit IRC | 11:13 | |
| *** rkukura has joined #openstack-infra | 11:13 | |
| *** vinay_yadhav has quit IRC | 11:13 | |
| openstackgerrit | Nicko Glayre proposed openstack-infra/jenkins-job-builder: Stash publisher's credentials configuration https://review.openstack.org/212004 | 11:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add flavor access API https://review.openstack.org/211710 | 11:17 |
| *** dims_ has quit IRC | 11:17 | |
| Shrews | mordred: ^^^^ s/tenant/project/ | 11:18 |
| *** mase_home has quit IRC | 11:19 | |
| openstackgerrit | fumihiko kakuma proposed openstack/requirements: Add networking-ofagent to projects.txt https://review.openstack.org/210772 | 11:20 |
| *** dims has joined #openstack-infra | 11:22 | |
| *** frank_zhou has joined #openstack-infra | 11:23 | |
| openstackgerrit | fumihiko kakuma proposed openstack/requirements: Add networking-ofagent to projects.txt https://review.openstack.org/210772 | 11:24 |
| openstackgerrit | Daniel Danner proposed openstack-infra/jenkins-job-builder: Add support for GroovyAxis https://review.openstack.org/213630 | 11:25 |
| *** kzaitsev1ws has joined #openstack-infra | 11:26 | |
| *** kzaitsev1ws has quit IRC | 11:26 | |
| *** dims has quit IRC | 11:27 | |
| *** lucasagomes is now known as lucas-hungry | 11:28 | |
| *** dims has joined #openstack-infra | 11:29 | |
| *** vponomaryov has joined #openstack-infra | 11:32 | |
| *** zeih has quit IRC | 11:34 | |
| *** markvoelker has joined #openstack-infra | 11:34 | |
| *** dims_ has joined #openstack-infra | 11:34 | |
| *** dims has quit IRC | 11:35 | |
| *** thorst has joined #openstack-infra | 11:36 | |
| *** jeadr has quit IRC | 11:36 | |
| *** jeadr has joined #openstack-infra | 11:37 | |
| *** bdemers has quit IRC | 11:37 | |
| dstufft | ianw: I'm back now, was sleeping at a hotel | 11:38 |
| *** markvoelker has quit IRC | 11:39 | |
| *** dims_ has quit IRC | 11:39 | |
| *** kzaitsev_ws has quit IRC | 11:40 | |
| *** dims has joined #openstack-infra | 11:40 | |
| *** kzaitsev_ws has joined #openstack-infra | 11:40 | |
| *** shardy_ has joined #openstack-infra | 11:43 | |
| *** yamamoto has joined #openstack-infra | 11:43 | |
| *** leopoldj has quit IRC | 11:44 | |
| *** shardy has quit IRC | 11:45 | |
| *** ldnunes has joined #openstack-infra | 11:45 | |
| *** BharatK has quit IRC | 11:45 | |
| *** woodster_ has joined #openstack-infra | 11:46 | |
| *** frank_zhou has quit IRC | 11:46 | |
| *** fhubik_brb is now known as fhubik | 11:48 | |
| *** shardy_ has quit IRC | 11:49 | |
| *** dims has quit IRC | 11:49 | |
| *** shardy has joined #openstack-infra | 11:49 | |
| *** jeadr has quit IRC | 11:52 | |
| *** jeadr has joined #openstack-infra | 11:53 | |
| *** dims has joined #openstack-infra | 11:53 | |
| *** notnownikki has quit IRC | 11:55 | |
| *** funzo has joined #openstack-infra | 11:55 | |
| *** dims_ has joined #openstack-infra | 11:56 | |
| *** dims has quit IRC | 11:58 | |
| *** chlong has quit IRC | 11:59 | |
| *** warpc__ has joined #openstack-infra | 12:00 | |
| *** funzo has quit IRC | 12:00 | |
| *** otter768 has quit IRC | 12:00 | |
| openstackgerrit | Simon Pasquier proposed openstack-infra/project-config: Enable merge commits for fuel-plugin-influxdb-grafana-release group https://review.openstack.org/213627 | 12:03 |
| *** prmtl_ has left #openstack-infra | 12:03 | |
| *** dims_ has quit IRC | 12:04 | |
| *** dims has joined #openstack-infra | 12:04 | |
| openstackgerrit | Merged openstack/requirements: README.rst: generate-constraints requires -r option https://review.openstack.org/213091 | 12:05 |
| *** yamamoto has quit IRC | 12:06 | |
| *** dizquierdo has quit IRC | 12:07 | |
| openstackgerrit | Nicko Glayre proposed openstack-infra/jenkins-job-builder: Customized pollscm trigger https://review.openstack.org/211993 | 12:07 |
| *** dims_ has joined #openstack-infra | 12:08 | |
| *** dims has quit IRC | 12:10 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Fix the example configuration for Vagrant deployment https://review.openstack.org/213687 | 12:11 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: jenkins/copy_puppet_logs.sh: create apache logs dir before copy https://review.openstack.org/213688 | 12:12 |
| *** dims has joined #openstack-infra | 12:12 | |
| *** dims_ has quit IRC | 12:14 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: jenkins/copy_puppet_logs.sh: create apache logs dir before copy https://review.openstack.org/213688 | 12:16 |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Separate tests in separate files and classes https://review.openstack.org/213521 | 12:17 |
| *** dims_ has joined #openstack-infra | 12:18 | |
| *** dims has quit IRC | 12:18 | |
| *** k4n0 has quit IRC | 12:18 | |
| *** dkranz has joined #openstack-infra | 12:19 | |
| *** dims has joined #openstack-infra | 12:19 | |
| *** k4n0 has joined #openstack-infra | 12:21 | |
| *** aysyd has joined #openstack-infra | 12:21 | |
| *** digvijay has joined #openstack-infra | 12:22 | |
| *** dims_ has quit IRC | 12:23 | |
| *** k4n0 has quit IRC | 12:23 | |
| *** Sree has joined #openstack-infra | 12:25 | |
| *** dims_ has joined #openstack-infra | 12:26 | |
| *** dims has quit IRC | 12:26 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Separate tests in separate files and classes https://review.openstack.org/213521 | 12:28 |
| *** shardy_ has joined #openstack-infra | 12:28 | |
| *** shardy has quit IRC | 12:29 | |
| *** dims has joined #openstack-infra | 12:30 | |
| *** lucas-hungry is now known as lucasagomes | 12:31 | |
| *** dims_ has quit IRC | 12:31 | |
| *** shardy_ has quit IRC | 12:33 | |
| *** shardy has joined #openstack-infra | 12:34 | |
| *** dims has quit IRC | 12:35 | |
| *** dims has joined #openstack-infra | 12:35 | |
| openstackgerrit | Asha proposed openstack/requirements: Version bump for hacking https://review.openstack.org/213255 | 12:35 |
| *** dims_ has joined #openstack-infra | 12:38 | |
| *** dims has quit IRC | 12:40 | |
| *** cfarquhar has quit IRC | 12:42 | |
| *** baoli has joined #openstack-infra | 12:44 | |
| *** ekarlso has quit IRC | 12:44 | |
| *** ekarlso has joined #openstack-infra | 12:44 | |
| *** kdas_ has joined #openstack-infra | 12:45 | |
| *** dims has joined #openstack-infra | 12:45 | |
| *** dims_ has quit IRC | 12:45 | |
| *** kdas_ has quit IRC | 12:46 | |
| *** kdas_ has joined #openstack-infra | 12:47 | |
| *** matbu has quit IRC | 12:47 | |
| *** baoli has quit IRC | 12:48 | |
| EmilienM | hello infra, I have an outstanding review request: https://review.openstack.org/213688 - thanks | 12:49 |
| *** matbu has joined #openstack-infra | 12:51 | |
| *** FallenPegasus has joined #openstack-infra | 12:52 | |
| openstackgerrit | fumihiko kakuma proposed openstack/requirements: Add networking-ofagent to projects.txt https://review.openstack.org/210773 | 12:53 |
| *** jswarren_ has joined #openstack-infra | 12:53 | |
| *** jswarren_ is now known as jswarren | 12:54 | |
| *** tlian has joined #openstack-infra | 12:54 | |
| *** mjblack has joined #openstack-infra | 12:54 | |
| openstackgerrit | fumihiko kakuma proposed openstack/requirements: Add networking-ofagent to projects.txt https://review.openstack.org/210773 | 12:54 |
| *** tlian has quit IRC | 12:54 | |
| *** jswarren has quit IRC | 12:54 | |
| *** dprince has joined #openstack-infra | 12:54 | |
| *** dims has quit IRC | 12:55 | |
| *** jswarren has joined #openstack-infra | 12:55 | |
| *** FallenPegasus has quit IRC | 12:56 | |
| *** dims has joined #openstack-infra | 12:58 | |
| *** eharney has joined #openstack-infra | 12:58 | |
| *** digvijay has quit IRC | 12:59 | |
| *** scheuran has joined #openstack-infra | 12:59 | |
| *** kdas_ is now known as kushal | 12:59 | |
| *** kushal has quit IRC | 12:59 | |
| *** kushal has joined #openstack-infra | 12:59 | |
| *** scheuran has quit IRC | 12:59 | |
| *** sdake has joined #openstack-infra | 13:00 | |
| *** tlian has joined #openstack-infra | 13:00 | |
| *** geoffarnold has joined #openstack-infra | 13:01 | |
| *** rfolco has joined #openstack-infra | 13:02 | |
| *** xyang1 has joined #openstack-infra | 13:05 | |
| *** matbu has quit IRC | 13:05 | |
| *** lifeless has quit IRC | 13:05 | |
| *** samuelBartel has joined #openstack-infra | 13:05 | |
| *** thiagop has joined #openstack-infra | 13:06 | |
| *** thiagop has left #openstack-infra | 13:06 | |
| *** dims has quit IRC | 13:06 | |
| *** yamamoto has joined #openstack-infra | 13:07 | |
| *** rlandy has joined #openstack-infra | 13:08 | |
| *** HeOS_ has quit IRC | 13:08 | |
| *** dims has joined #openstack-infra | 13:09 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add tables for Worklists and Boards https://review.openstack.org/213701 | 13:09 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add an API endpoint for Worklists https://review.openstack.org/213702 | 13:09 |
| *** zeih has joined #openstack-infra | 13:11 | |
| *** yamamoto has quit IRC | 13:11 | |
| *** mrmartin has joined #openstack-infra | 13:12 | |
| *** yamamoto has joined #openstack-infra | 13:12 | |
| *** tiswanso has joined #openstack-infra | 13:13 | |
| *** larainema has joined #openstack-infra | 13:13 | |
| *** dims has quit IRC | 13:14 | |
| *** HeOS_ has joined #openstack-infra | 13:14 | |
| *** dims has joined #openstack-infra | 13:14 | |
| *** sdake_ has joined #openstack-infra | 13:15 | |
| *** amitgandhinz has joined #openstack-infra | 13:18 | |
| *** mriedem has joined #openstack-infra | 13:18 | |
| *** dkranz has quit IRC | 13:18 | |
| *** sdake has quit IRC | 13:19 | |
| *** sdake_ has quit IRC | 13:19 | |
| *** zeih has quit IRC | 13:20 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/infra-specs: StoryBoard: Worklists and Boards https://review.openstack.org/202989 | 13:20 |
| *** dims has quit IRC | 13:21 | |
| *** geoffarnold has quit IRC | 13:21 | |
| *** matbu has joined #openstack-infra | 13:22 | |
| SotK | That spec has two +1s and has been waiting around for a few weeks. Is there any chance other folks could look at it soon? | 13:23 |
| SotK | If not is there any chance it could be merged soon? :) | 13:23 |
| *** bdemers has joined #openstack-infra | 13:24 | |
| openstackgerrit | Merged openstack-infra/storyboard: Fix the example configuration for Vagrant deployment https://review.openstack.org/213687 | 13:24 |
| *** leakypipes is now known as jaypipes | 13:25 | |
| *** alivigni has joined #openstack-infra | 13:25 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Separate tests in separate files and classes https://review.openstack.org/213521 | 13:25 |
| *** che-arne has joined #openstack-infra | 13:25 | |
| lxsli | SotK: we're waaay past spec freeze for Liberty | 13:26 |
| *** BharatK has joined #openstack-infra | 13:26 | |
| *** funzo has joined #openstack-infra | 13:26 | |
| clarkb | lxsli we dont strictly follow release cycle things in infra | 13:27 |
| lxsli | oh sorry, thought I was in Nova room >.< | 13:27 |
| *** dims has joined #openstack-infra | 13:27 | |
| *** larainema has quit IRC | 13:27 | |
| *** lifeless has joined #openstack-infra | 13:28 | |
| fungi | jhesketh: yeah, restarting gerrit to speed it back up is acceptable. getting to the bottom of why it keeps bogging down would be even better, but to my knowledge none of us has managed to get very far with that yet | 13:28 |
| jhesketh | fungi: so it appears to be due to jetty | 13:30 |
| jhesketh | but I don't know anything about that | 13:30 |
| barthalion | jhesketh: great, thank you | 13:30 |
| jhesketh | the suggestions seem to be to use an alternative or tweak the maxqueued (https://code.google.com/p/gerrit/wiki/Scaling#Jetty) | 13:30 |
| *** julim has joined #openstack-infra | 13:31 | |
| *** larainema has joined #openstack-infra | 13:31 | |
| *** funzo has quit IRC | 13:31 | |
| fungi | jhesketh: yeah, the few times i've been able to nail down a cause, it's that web crawlers were following links from our (recently added) html irc logs, dragging down both review.o.o and paste.o.o | 13:31 |
| *** dkranz has joined #openstack-infra | 13:32 | |
| clarkb | we could also give gerrit more memory to increase headroo. | 13:32 |
| *** pc_m has quit IRC | 13:32 | |
| jhesketh | clarkb: memory usuage seemed fine (not that that means anything) | 13:32 |
| jhesketh | it wsa just the CPU at max and the load around 10 | 13:32 |
| fungi | given that we have irc bots which announce urls to most changes, some in multiple channels, it results in a lot of queryspam to gerrit | 13:32 |
| fungi | jhesketh: yep, that's what i've seen in the past too | 13:33 |
| jhesketh | we don't have a robots file? | 13:33 |
| jhesketh | or do you mean due to bad bots | 13:33 |
| fungi | jhesketh: sometimes if you catch it when it's really bad, it can spike up around 30 | 13:33 |
| clarkb | usually when e have trouble is memory pressure | 13:33 |
| clarkb | its capped at 8GB so cacti doesnt show it but what happens is you spebd a large amount of time garbage collecting | 13:34 |
| jhesketh | right so maybe we should increase the memory anyway | 13:34 |
| clarkb | that GC overhead seems to lead to the cpu sluggiahness | 13:34 |
| jhesketh | clarkb: capped? it looks fine to me http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 13:34 |
| clarkb | yes gerrit us told to use 8gb and wont use more | 13:35 |
| *** fhubik is now known as fhubik_brb | 13:35 | |
| *** markvoelker has joined #openstack-infra | 13:35 | |
| jhesketh | oh right, gerrit is capped | 13:35 |
| clarkb | so cacti is mostly useless for this as you just see it using the 8GB | 13:35 |
| jhesketh | (sorry, thought you meant cacti didn't graph more than 8gb) | 13:35 |
| *** pc_m has joined #openstack-infra | 13:35 | |
| jhesketh | well upping gerrit certainly couldn't hurt | 13:36 |
| *** dims has quit IRC | 13:36 | |
| *** krtaylor has quit IRC | 13:36 | |
| *** pc_m has quit IRC | 13:37 | |
| fungi | i wonder if that cap is a relic of when we were running on a smaller flavor | 13:37 |
| *** kgiusti has joined #openstack-infra | 13:37 | |
| *** sergmelikyan has joined #openstack-infra | 13:38 | |
| clarkb | see https://review.openstack.org/monitoring?part=graph&graph=usedMemory and https://review.openstack.org/monitoring?part=graph&graph=gc | 13:38 |
| *** dizquierdo has joined #openstack-infra | 13:38 | |
| jhesketh | that monitoring page is new to me | 13:39 |
| *** markvoelker has quit IRC | 13:39 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Separate tests in separate files and classes https://review.openstack.org/213521 | 13:41 |
| *** FallenPegasus has joined #openstack-infra | 13:41 | |
| ekarlso | what's the diff if you guys know of doing patcher = mock.patch("foo"); self.foo = patcher.start() vs the decorator or with pattern in tests ? | 13:41 |
| *** dims has joined #openstack-infra | 13:42 | |
| *** changbl has joined #openstack-infra | 13:43 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add an API endpoint for Worklists https://review.openstack.org/213702 | 13:45 |
| *** baoli has joined #openstack-infra | 13:45 | |
| fungi | jhesketh: yeah, i always forget it's there. also you only have the "monitoring" button and access to that interface if you're a gerrit admin | 13:45 |
| *** FallenPegasus has quit IRC | 13:45 | |
| *** mwagner_lap has joined #openstack-infra | 13:46 | |
| *** fhubik_brb is now known as fhubik | 13:46 | |
| *** pabelanger_ is now known as pabelanger | 13:47 | |
| jhesketh | fungi: ah cool :-) | 13:47 |
| EmilienM | fungi or clarkb: please, can you look https://review.openstack.org/#/c/213688/ - it will unblock 2 modules's gating - thanks | 13:47 |
| *** mjblack has quit IRC | 13:48 | |
| *** dims has quit IRC | 13:48 | |
| *** pc_m has joined #openstack-infra | 13:48 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add paging controls to the result set pager https://review.openstack.org/198768 | 13:48 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Implement offset/limit pagination of lists https://review.openstack.org/198769 | 13:48 |
| *** sdake has joined #openstack-infra | 13:49 | |
| *** sdake has quit IRC | 13:49 | |
| fungi | EmilienM: that commit message is confusing compared to the diff | 13:49 |
| *** krtaylor has joined #openstack-infra | 13:49 | |
| *** sdake has joined #openstack-infra | 13:49 | |
| fungi | EmilienM: it talks about creating the apache log destination but instead creates somewhere to copy apache configs | 13:50 |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Create experimental jobs for VPN DevStack plugin https://review.openstack.org/209887 | 13:50 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Stop using fixed-width containers https://review.openstack.org/201124 | 13:50 |
| *** baoli has quit IRC | 13:50 | |
| EmilienM | fungi: ok, let me modify it | 13:50 |
| fungi | EmilienM: if it's just the commit message that needs updating, then do that real quick and i'll approve it | 13:50 |
| EmilienM | fungi: ok, I'm changing the text | 13:50 |
| *** serg_melikyan has joined #openstack-infra | 13:51 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: jenkins/copy_puppet_logs.sh: prepare destination apache logs https://review.openstack.org/213688 | 13:52 |
| EmilienM | fungi: do you think it's better? ^ | 13:52 |
| *** xyang1 has quit IRC | 13:52 | |
| *** sergmelikyan has quit IRC | 13:52 | |
| *** xyang1 has joined #openstack-infra | 13:52 | |
| fungi | EmilienM: i'm still confused. it says "we need to create the destination repository that will host apache logs" but your change is creating a directory to host apache configuration. logs seem to be handled by another part of the script entirely | 13:54 |
| *** dims has joined #openstack-infra | 13:54 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: jenkins/copy_puppet_logs.sh: prepare destination apache logs https://review.openstack.org/213688 | 13:54 |
| *** matbu has quit IRC | 13:55 | |
| EmilienM | fungi: err | 13:55 |
| fungi | EmilienM: also if you're updating the commit message, please try to keep the text wrapped at <80 characters (preferably <70) so it's easier to look at in git log output | 13:56 |
| EmilienM | fungi: I'm updating | 13:56 |
| jeblair | we just had a M4.2 earthquake on the hayward fault | 13:56 |
| *** bswartz has quit IRC | 13:56 | |
| fungi | the gerrit webui unfortunately does a terrible job of making it look like it's wrapping text fir you when it's not | 13:56 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch nova temprevert to a cherrypick https://review.openstack.org/213714 | 13:57 |
| fungi | jeblair: woah! | 13:57 |
| *** matbu has joined #openstack-infra | 13:57 | |
| fungi | jeblair: any problems? | 13:57 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: jenkins/copy_puppet_logs.sh: prepare apache_config dir https://review.openstack.org/213688 | 13:57 |
| *** fhubik is now known as fhubik_brb | 13:57 | |
| EmilienM | fungi: here we go | 13:57 |
| jeblair | fungi: no, wasn't that bad here | 13:58 |
| * jhesketh is off for the night | 13:58 | |
| *** zul has joined #openstack-infra | 13:58 | |
| jhesketh | seeya! | 13:58 |
| *** pradk has joined #openstack-infra | 13:59 | |
| *** moravec has joined #openstack-infra | 13:59 | |
| fungi | g'night jhesketh, thanks for all the help while i was sleeping ;) | 13:59 |
| * jhesketh tries to help :-) | 13:59 | |
| EmilienM | fungi: I'm off today, I'll catch-up tomorrow, thanks for the fast review | 13:59 |
| *** spzala has joined #openstack-infra | 13:59 | |
| *** dims has quit IRC | 13:59 | |
| jhesketh | fungi: also you should be able to sleep without worrying! | 14:00 |
| *** boris-42 has joined #openstack-infra | 14:00 | |
| pabelanger | EmilienM: fungi: Why not use --parents flag? That way cp doesn't care if the directory exists, it will use the full source on the root DIRECTORY name. This way, you don't need to build a custom folder structure for each cp command | 14:00 |
| *** otter768 has joined #openstack-infra | 14:01 | |
| *** tvanderwiel has joined #openstack-infra | 14:01 | |
| *** tvanderwiel has left #openstack-infra | 14:01 | |
| *** tongli has joined #openstack-infra | 14:01 | |
| fungi | pabelanger: i assumed it was so that the configs would wind up together in a flat directory structure | 14:02 |
| *** bexelbie_ has joined #openstack-infra | 14:02 | |
| *** bexelbie_ is now known as bexelbie | 14:03 | |
| *** chlong has joined #openstack-infra | 14:03 | |
| *** fhubik_brb is now known as fhubik | 14:04 | |
| *** dims has joined #openstack-infra | 14:04 | |
| *** otter768 has quit IRC | 14:06 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Adds Cloudbees folder plugin support https://review.openstack.org/180185 | 14:06 |
| pabelanger | fungi: Ya, I tend to keep the structure out of habit | 14:06 |
| *** yamamoto has quit IRC | 14:09 | |
| *** zz_dimtruck is now known as dimtruck | 14:09 | |
| *** doug-fish has joined #openstack-infra | 14:09 | |
| jeblair | fungi: oh, there was damage! my kermit the frog pez dispensor fell off of a high shelf, and broke! | 14:10 |
| *** sdake has quit IRC | 14:10 | |
| *** dims has quit IRC | 14:11 | |
| fungi | jeblair: you should probably better secure your pez dispensers | 14:11 |
| *** tonytan4ever has joined #openstack-infra | 14:14 | |
| *** w_ has joined #openstack-infra | 14:14 | |
| openstackgerrit | Kaitlin Farr proposed openstack/requirements: Add Castellan to global requirements https://review.openstack.org/184874 | 14:15 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:15 | |
| *** dims has joined #openstack-infra | 14:16 | |
| *** olaph has quit IRC | 14:16 | |
| *** spredzy is now known as spredzy|afk | 14:16 | |
| EmilienM | pabelanger: we can change that in a next patch | 14:16 |
| *** sdake has joined #openstack-infra | 14:17 | |
| *** flepied has quit IRC | 14:17 | |
| ekarlso | 7buf5 | 14:17 |
| openstackgerrit | Merged openstack-infra/project-config: jenkins/copy_puppet_logs.sh: prepare apache_config dir https://review.openstack.org/213688 | 14:18 |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Migration to using requests https://review.openstack.org/205637 | 14:18 |
| *** olaph has joined #openstack-infra | 14:20 | |
| yolanda | hi jeblair, so i got a good comment from jhesketh in the nodepool secure.conf change https://review.openstack.org/#/c/189762/ | 14:21 |
| yolanda | he proposes to drop the jenkins dictionary and flatten the test-job setting | 14:21 |
| *** bswartz has joined #openstack-infra | 14:21 | |
| *** notnownikki has joined #openstack-infra | 14:21 | |
| *** w_ has quit IRC | 14:21 | |
| *** dims has quit IRC | 14:22 | |
| yolanda | i'd like to check with you before amending that, initially i support it | 14:22 |
| *** serg_melikyan has quit IRC | 14:22 | |
| *** fhubik is now known as fhubik_brb | 14:24 | |
| openstackgerrit | Kaitlin Farr proposed openstack/requirements: Add Castellan to global requirements https://review.openstack.org/184874 | 14:24 |
| *** zeih has joined #openstack-infra | 14:25 | |
| *** doug-fish has quit IRC | 14:26 | |
| *** Sree has quit IRC | 14:27 | |
| *** funzo has joined #openstack-infra | 14:27 | |
| *** austin81 has joined #openstack-infra | 14:27 | |
| *** doug-fish has joined #openstack-infra | 14:30 | |
| *** sergmelikyan has joined #openstack-infra | 14:30 | |
| *** flepied has joined #openstack-infra | 14:30 | |
| *** vinay_yadhav has joined #openstack-infra | 14:31 | |
| *** afazkas has quit IRC | 14:31 | |
| *** funzo has quit IRC | 14:32 | |
| *** zeih has quit IRC | 14:32 | |
| *** olaph has quit IRC | 14:33 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Set default credentials in the tools/ scripts to work with CI https://review.openstack.org/213732 | 14:33 |
| *** arxcruz has quit IRC | 14:34 | |
| *** fhubik_brb is now known as fhubik | 14:38 | |
| *** tsekiyama has joined #openstack-infra | 14:38 | |
| openstackgerrit | Jiri Stransky proposed openstack/diskimage-builder: Fedora: install older corosync https://review.openstack.org/213736 | 14:38 |
| *** yamamoto has joined #openstack-infra | 14:39 | |
| *** peristeri has quit IRC | 14:40 | |
| *** vlaza has quit IRC | 14:40 | |
| *** mtanino has joined #openstack-infra | 14:42 | |
| *** jeadr has quit IRC | 14:43 | |
| *** jeadr has joined #openstack-infra | 14:44 | |
| *** fhubik is now known as fhubik_brb | 14:44 | |
| *** ayoung has joined #openstack-infra | 14:44 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Fedora: install older corosync https://review.openstack.org/213737 | 14:45 |
| *** doug-fis_ has joined #openstack-infra | 14:45 | |
| *** flepied has quit IRC | 14:45 | |
| *** alexpilotti_ has joined #openstack-infra | 14:47 | |
| *** annegentle has joined #openstack-infra | 14:48 | |
| *** doug-fish has quit IRC | 14:48 | |
| *** dims has joined #openstack-infra | 14:48 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image builds using gearman https://review.openstack.org/195175 | 14:49 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image uploads using gearman https://review.openstack.org/197722 | 14:49 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image delete using gearman https://review.openstack.org/198114 | 14:49 |
| *** markvoelker has joined #openstack-infra | 14:51 | |
| *** yamamoto has quit IRC | 14:54 | |
| *** AJaeger has quit IRC | 14:54 | |
| *** bpokorny has joined #openstack-infra | 14:54 | |
| *** changbl has quit IRC | 14:55 | |
| *** jeadr has quit IRC | 14:55 | |
| *** markvoelker has quit IRC | 14:55 | |
| *** gordc has joined #openstack-infra | 14:56 | |
| *** yolanda has quit IRC | 14:56 | |
| *** jeadr has joined #openstack-infra | 14:56 | |
| *** ashleighfarnham has quit IRC | 14:56 | |
| *** sergmelikyan has quit IRC | 14:57 | |
| *** bpokorny has quit IRC | 14:59 | |
| *** shardy_ has joined #openstack-infra | 14:59 | |
| *** sergmelikyan has joined #openstack-infra | 14:59 | |
| *** achanda has joined #openstack-infra | 14:59 | |
| *** doug-fis_ has quit IRC | 14:59 | |
| *** smccully has quit IRC | 14:59 | |
| openstackgerrit | greghaynes proposed openstack-infra/glean: Create systemwide lock for glean https://review.openstack.org/212177 | 15:00 |
| *** changbl has joined #openstack-infra | 15:00 | |
| *** doug-fish has joined #openstack-infra | 15:00 | |
| *** shardy has quit IRC | 15:01 | |
| *** smccully has joined #openstack-infra | 15:02 | |
| *** vinay_yadhav has quit IRC | 15:02 | |
| *** doug-fish has quit IRC | 15:03 | |
| *** pcrews has quit IRC | 15:03 | |
| *** ildikov has quit IRC | 15:03 | |
| *** sergmelikyan has quit IRC | 15:03 | |
| *** funzo has joined #openstack-infra | 15:04 | |
| *** krtaylor has quit IRC | 15:04 | |
| *** shardy_ has quit IRC | 15:04 | |
| *** achanda has quit IRC | 15:04 | |
| *** shardy has joined #openstack-infra | 15:05 | |
| *** asselin has joined #openstack-infra | 15:06 | |
| *** morgan_503 is now known as morgan_2549 | 15:06 | |
| *** mmedvede has quit IRC | 15:07 | |
| *** esker has joined #openstack-infra | 15:08 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/project-config: Submit ironic-python-agent releases to pypi https://review.openstack.org/213749 | 15:09 |
| *** moravec has joined #openstack-infra | 15:09 | |
| *** mmedvede has joined #openstack-infra | 15:10 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: create growroot element https://review.openstack.org/206636 | 15:11 |
| *** yamahata has joined #openstack-infra | 15:12 | |
| *** sputnik13 has quit IRC | 15:12 | |
| *** ZZelle-off has joined #openstack-infra | 15:12 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Fix the test_backend.sh scripts to work usefully CI https://review.openstack.org/213732 | 15:12 |
| *** yamamoto has joined #openstack-infra | 15:13 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Run geard for nodepool tests https://review.openstack.org/194928 | 15:13 |
| *** yolanda has joined #openstack-infra | 15:13 | |
| *** BharatK has quit IRC | 15:13 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image builds using gearman https://review.openstack.org/195175 | 15:14 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image uploads using gearman https://review.openstack.org/197722 | 15:14 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image delete using gearman https://review.openstack.org/198114 | 15:14 |
| *** markvoelker has joined #openstack-infra | 15:15 | |
| *** krtaylor has joined #openstack-infra | 15:16 | |
| *** sergmelikyan has joined #openstack-infra | 15:16 | |
| *** bexelbie has quit IRC | 15:17 | |
| *** bpokorny has joined #openstack-infra | 15:18 | |
| *** zul has quit IRC | 15:18 | |
| *** chlong has quit IRC | 15:18 | |
| *** dprince has quit IRC | 15:20 | |
| *** armax has joined #openstack-infra | 15:20 | |
| pelix | what's the best way to add information to the console on a test that failed to raise an exception? | 15:24 |
| pelix | looking for suggestions on how I should pass back some information to help debugging a test failure when using https://review.openstack.org/211776 | 15:25 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Stop using fixed-width containers https://review.openstack.org/201124 | 15:26 |
| *** e0ne has joined #openstack-infra | 15:26 | |
| *** doug-fish has joined #openstack-infra | 15:27 | |
| jeblair | pelix: i usually just log, that way it shows up in order with the rest of the output and nicely timestamped | 15:28 |
| *** mpopow has joined #openstack-infra | 15:28 | |
| *** sshnaidm has quit IRC | 15:28 | |
| jeblair | pelix: (if you print to stdout, it gets stored in a different subunit stream so you loose context and ordering) | 15:29 |
| pelix | yeap, that's that what I'm trying to avoid | 15:29 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: WIP: Fix the test_backend.sh scripts to work usefully CI https://review.openstack.org/213732 | 15:29 |
| pelix | the logging tends to get output when it gets called instead of along with the failed test information | 15:29 |
| *** spzala has quit IRC | 15:30 | |
| jeblair | pelix: that sounds strange -- usually we output all of the logging when there is a failed test; do you have a link to a failed run? | 15:31 |
| pelix | I guess I could try using testtools.addDetail | 15:31 |
| *** chlong has joined #openstack-infra | 15:32 | |
| pelix | http://logs.openstack.org/76/211776/3/check/gate-python-jenkins-python34/163daea/console.html | 15:32 |
| *** BharatK has joined #openstack-infra | 15:32 | |
| pelix | fine for a single failure and single log message, but I've found that approach to be difficult to deal with if there are multiple of both | 15:32 |
| *** dizquierdo has quit IRC | 15:32 | |
| pelix | I miss having a clear association | 15:33 |
| *** mpopow has quit IRC | 15:33 | |
| jeblair | yolanda: i don't think jhesketh's change is necessary or related to your current effort | 15:34 |
| yolanda | well, but we are doing a massive refactor and disruptive change here | 15:34 |
| yolanda | so we could use that oportunity | 15:34 |
| yolanda | and it's a bit related because we are moving everything from targets, leaving only a single element... | 15:34 |
| *** alexpilotti_ is now known as alexpilotti | 15:35 | |
| jeblair | pelix: yeah, there's no log output there. that seems wrong. | 15:35 |
| *** dannywilson has joined #openstack-infra | 15:35 | |
| pelix | log line for the exception is http://logs.openstack.org/76/211776/3/check/gate-python-jenkins-python34/163daea/console.html#_2015-08-17_15_01_29_860 | 15:36 |
| jeblair | yolanda: 'jenkins' was in there to support future non-jenkins targets without needing a refactor. | 15:36 |
| pelix | that's what is outputted by the test that fails to raise the exception | 15:36 |
| jeblair | yolanda: there is no need or purpose to make that change. let's just leave it alone and get on with other work | 15:37 |
| yolanda | ok | 15:37 |
| *** _ody has quit IRC | 15:37 | |
| yolanda | i'll amend a pair of nits there, and answer jhesketh about his comment | 15:37 |
| *** chlong has quit IRC | 15:38 | |
| jeblair | yolanda: also, the change you are making is fairly straightforward and has a simple migration path. the suggested change breaks forward compatability and so makes things much harder. | 15:39 |
| clarkb | tests should use the logging fixture from fixtures to associate logs properly | 15:39 |
| *** Sukhdev has joined #openstack-infra | 15:40 | |
| jeblair | pelix: ah that's what's missing ^ | 15:40 |
| *** yfried|afk has quit IRC | 15:40 | |
| pelix | ah, that's useful | 15:40 |
| *** chlong has joined #openstack-infra | 15:40 | |
| *** _ody has joined #openstack-infra | 15:40 | |
| yolanda | jeblair, honestly that single label pending from target looked strange to me, i even missed that on previous changes. But if that's left there with some future intention, it's ok | 15:41 |
| pelix | thanks, figured it'd be something straight forward, just need to know about it ;) | 15:41 |
| *** geoffarnold has joined #openstack-infra | 15:42 | |
| *** geoffarnold is now known as geoffarnoldX | 15:42 | |
| *** hdd has joined #openstack-infra | 15:43 | |
| *** fhubik_brb is now known as fhubik | 15:43 | |
| *** nadya has quit IRC | 15:43 | |
| jeblair | yolanda: well, we probably won't get around to using it before zuulv3, but it was intentionally there in the first place. | 15:44 |
| clarkb | its used in the new dsvm test | 15:44 |
| *** sergmelikyan has quit IRC | 15:44 | |
| clarkb | we have a jenkinsless target | 15:44 |
| jeblair | clarkb: oh neat :) | 15:44 |
| jeblair | it fulfilled its destiny | 15:45 |
| jeblair | yolanda: ^ | 15:45 |
| yolanda | ah ok, then i cannot drop it, for sure | 15:45 |
| *** flepied has joined #openstack-infra | 15:46 | |
| *** hemnafk is now known as hemna | 15:46 | |
| *** dannywilson has quit IRC | 15:46 | |
| *** dguerri` is now known as dguerri | 15:46 | |
| *** sergmelikyan has joined #openstack-infra | 15:47 | |
| *** geoffarnoldX is now known as geoffarnold | 15:47 | |
| clarkb | otherwise we would have to run a jenkins or fake it | 15:49 |
| clarkb | this way we just get a pool assigned to a dummy target works well for testing | 15:49 |
| *** jeadr has quit IRC | 15:49 | |
| *** mriedem is now known as mriedem_away | 15:50 | |
| *** mriedem_away has quit IRC | 15:50 | |
| *** jeadr has joined #openstack-infra | 15:50 | |
| yolanda | i wasn't aware of that use case | 15:50 |
| *** mestery has joined #openstack-infra | 15:53 | |
| *** ifarkas has quit IRC | 15:56 | |
| *** gyee has joined #openstack-infra | 15:56 | |
| zaro | morning | 15:57 |
| *** angdraug has joined #openstack-infra | 15:59 | |
| *** ihrachyshka has quit IRC | 15:59 | |
| *** jerryz has quit IRC | 16:00 | |
| *** hemna has quit IRC | 16:01 | |
| *** jlanoux has quit IRC | 16:01 | |
| *** otter768 has joined #openstack-infra | 16:02 | |
| *** mrunge has quit IRC | 16:02 | |
| *** dprince has joined #openstack-infra | 16:03 | |
| *** akraminsky has joined #openstack-infra | 16:03 | |
| *** lucasagomes is now known as lucas-brb | 16:04 | |
| *** dosaboy_ has joined #openstack-infra | 16:04 | |
| *** dosaboy_ has quit IRC | 16:04 | |
| *** dosaboy has quit IRC | 16:04 | |
| *** armax has quit IRC | 16:05 | |
| *** abregman has quit IRC | 16:05 | |
| *** otter768 has quit IRC | 16:07 | |
| *** warpc__ has quit IRC | 16:08 | |
| *** bdemers has quit IRC | 16:09 | |
| *** dosaboy has joined #openstack-infra | 16:09 | |
| *** e0ne has quit IRC | 16:09 | |
| *** mrunge has joined #openstack-infra | 16:11 | |
| *** enikanorov2 has quit IRC | 16:12 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch nova temprevert to a cherrypick https://review.openstack.org/213714 | 16:12 |
| *** sabeen2 has joined #openstack-infra | 16:13 | |
| *** unicell1 has quit IRC | 16:15 | |
| *** david-ly_ is now known as david-lyle | 16:15 | |
| *** flepied has quit IRC | 16:18 | |
| *** ashleighfarnham has joined #openstack-infra | 16:18 | |
| *** jistr has quit IRC | 16:19 | |
| *** ihrachyshka has joined #openstack-infra | 16:19 | |
| *** yamamoto has quit IRC | 16:20 | |
| *** spzala has joined #openstack-infra | 16:20 | |
| *** flepied has joined #openstack-infra | 16:20 | |
| *** moravec1 has joined #openstack-infra | 16:20 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix init-scripts element path munging and deps https://review.openstack.org/213180 | 16:20 |
| nibalizer | what was the issue with gerrit slowness last night? | 16:21 |
| *** moravec1 has quit IRC | 16:21 | |
| *** Swami has joined #openstack-infra | 16:21 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Install-static depends on rsync https://review.openstack.org/213309 | 16:21 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: create growroot element https://review.openstack.org/206636 | 16:21 |
| nibalizer | ML says 'we think we know' but the reason wasn't specified | 16:21 |
| *** che-arne has quit IRC | 16:21 | |
| *** moravec has quit IRC | 16:21 | |
| *** jeadr has quit IRC | 16:22 | |
| openstackgerrit | Nicko Glayre proposed openstack-infra/jenkins-job-builder: Stash publisher's credentials configuration https://review.openstack.org/212004 | 16:22 |
| *** jeadr has joined #openstack-infra | 16:22 | |
| *** alexsyip has joined #openstack-infra | 16:24 | |
| *** stevemar has joined #openstack-infra | 16:24 | |
| mordred | nibalizer: "java" | 16:24 |
| * nibalizer nods sagely | 16:25 | |
| mordred | nibalizer: I believe something to do with jetty - https://code.google.com/p/gerrit/wiki/Scaling#Jetty | 16:25 |
| nibalizer | oh wow we can run gerrit in tomcat if we want | 16:26 |
| mordred | yah | 16:26 |
| *** sthillma has joined #openstack-infra | 16:26 | |
| mordred | I think we've never investigated that because we haven't had any need to and it's extra complication/overhead | 16:26 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Update nodepool to use secure.conf file https://review.openstack.org/189762 | 16:27 |
| clarkb | its memory leak leading to high gc overhead | 16:27 |
| mordred | hrm | 16:27 |
| nibalizer | so last night what I was seeing was extremely long clone times for specifically nova | 16:28 |
| mordred | clarkb: https://gerrit-review.googlesource.com/#/c/35010/6/Documentation/install-tomcat.txt is written with installign things not-in /home/gerrit but instead in normal system locations (weird) | 16:28 |
| nibalizer | a couple users (probably ci systems) were cloning for an hour+ | 16:28 |
| nibalizer | so my thought was memory contention because gerrit was packing up nova for them, which is a big repo | 16:28 |
| nibalizer | but I'm not sure that completely follows | 16:28 |
| *** moravec has joined #openstack-infra | 16:29 | |
| *** pcrews has joined #openstack-infra | 16:29 | |
| *** fhubik is now known as fhubik_brb | 16:31 | |
| *** fhubik_brb is now known as fhubik | 16:31 | |
| *** olaph has joined #openstack-infra | 16:32 | |
| *** yamahata has quit IRC | 16:35 | |
| *** sthillma has quit IRC | 16:35 | |
| timrc | Howdy | 16:36 |
| *** sergmelikyan has quit IRC | 16:37 | |
| *** matbu has quit IRC | 16:37 | |
| *** sergmelikyan has joined #openstack-infra | 16:38 | |
| *** ildikov has joined #openstack-infra | 16:38 | |
| *** tqtran-afk has joined #openstack-infra | 16:38 | |
| *** tqtran-afk is now known as tqtran | 16:38 | |
| *** moravec has quit IRC | 16:40 | |
| *** haneef has joined #openstack-infra | 16:41 | |
| openstackgerrit | Thai Tran proposed openstack-infra/project-config: Horizon linting should be voting https://review.openstack.org/213344 | 16:41 |
| openstackgerrit | Thai Tran proposed openstack-infra/project-config: Horizon linting should be voting https://review.openstack.org/213344 | 16:41 |
| *** olaph has quit IRC | 16:42 | |
| *** matbu has joined #openstack-infra | 16:42 | |
| haneef | Can any one let me know what does gate-grenade-dvsm do? I'm getting failure, but not sure why is that happening. Looks like it doesn't like piple line changes. http://logs.openstack.org/79/213379/4/check/gate-grenade-dsvm/b27d428/ | 16:42 |
| zaro | anybody heard of drone ci? CI with containers? it looks really nice. | 16:45 |
| *** yamamoto has joined #openstack-infra | 16:45 | |
| *** unicell has joined #openstack-infra | 16:45 | |
| *** earlephilhower has joined #openstack-infra | 16:45 | |
| *** pcrews has quit IRC | 16:46 | |
| *** devvesa has quit IRC | 16:46 | |
| *** puranamr has joined #openstack-infra | 16:47 | |
| *** moravec has joined #openstack-infra | 16:47 | |
| *** unicell has quit IRC | 16:50 | |
| *** skolekonov has quit IRC | 16:51 | |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Add MultiJob Project Documentation. https://review.openstack.org/212666 | 16:51 |
| *** bharath has joined #openstack-infra | 16:52 | |
| *** unicell has joined #openstack-infra | 16:53 | |
| jasondotstar | zaro: seems like it. haven't seen it b4 tho | 16:54 |
| *** lucas-brb is now known as lucasagomes | 16:54 | |
| *** jschwarz has quit IRC | 16:55 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Fix timeout function and add capturing of messages on failure https://review.openstack.org/211776 | 16:55 |
| *** smccully has quit IRC | 16:56 | |
| pelix | zaro: that should finally sort out the problem with the random failures for python-jenkins ^^^ | 16:56 |
| *** smccully has joined #openstack-infra | 16:56 | |
| *** tonytan4ever has quit IRC | 16:56 | |
| timothyb89 | What would be the best way to adjust served content types on logs.openstack.org? For example it is serving .json.gz as application/x-gzip rather than application/json with gzip encoding | 16:56 |
| *** fhubik has quit IRC | 16:56 | |
| *** sergmelikyan has quit IRC | 16:57 | |
| *** derekh has quit IRC | 16:57 | |
| clarkb | btw yesterday evening (pdt) upgrade of zanata on translate dev went well | 16:57 |
| clarkb | stevenk is lookinh inyo using the recommended newer version of wildfly but current versiob appears to work | 16:57 |
| zaro | jasondotstar: in the early stages i believe but seems very promising. too bad it's Go :( | 16:58 |
| jasondotstar | ah | 16:58 |
| *** yamahata has joined #openstack-infra | 16:58 | |
| jasondotstar | me no know go | 16:58 |
| zaro | neither do me | 16:58 |
| *** sergmelikyan has joined #openstack-infra | 16:58 | |
| *** notnownikki has quit IRC | 17:00 | |
| *** bdemers has joined #openstack-infra | 17:00 | |
| *** enikanorov2 has joined #openstack-infra | 17:00 | |
| *** sergmelikyan has quit IRC | 17:00 | |
| *** sergmelikyan has joined #openstack-infra | 17:01 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/project-config: Add election project https://review.openstack.org/213785 | 17:01 |
| clarkb | we cant use containers (in the form they normally take) for many of our tests | 17:01 |
| *** alex_xu has quit IRC | 17:01 | |
| krotscheck | Anyone have thoughts on the release-slave patch I have up? There seems to be an accumulation of positive votes. https://review.openstack.org/#/c/210601/ | 17:01 |
| zaro | clarkb: what's the problem with that? | 17:02 |
| clarkb | zaro with using containers? neutron/cinder/nova all have weird corner cases with namespaces when run in containers | 17:03 |
| tristanC | Hi folks, following on the elections tooling discussion I've proposed https://review.openstack.org/213785 to create the election project... Please let me know if it looks correct | 17:03 |
| *** serg_melikyan has joined #openstack-infra | 17:03 | |
| clarkb | you can work around them but by then you lose most of the benefits to having a container | 17:03 |
| *** smccully has quit IRC | 17:03 | |
| *** alex_xu has joined #openstack-infra | 17:03 | |
| zaro | clarkb: won't those get fixed? i mean openstack should be able to run in containers right? | 17:04 |
| clarkb | zaro kernel has said no to at least one of them | 17:04 |
| *** mrunge has quit IRC | 17:04 | |
| clarkb | so no I dont expect them to be fixed | 17:04 |
| *** serg_melikyan has quit IRC | 17:05 | |
| *** sergmelikyan has quit IRC | 17:06 | |
| zaro | ohh wow, so does that mean that openstack doesn't plan to support running in containers? | 17:06 |
| *** zhenguo has quit IRC | 17:06 | |
| *** yamamoto has quit IRC | 17:06 | |
| *** zhenguo has joined #openstack-infra | 17:06 | |
| *** sergmelikyan has joined #openstack-infra | 17:06 | |
| * fungi chuckles at zuul-related humor... http://i.imgur.com/kfWQujZ.jpg | 17:07 | |
| *** larainema has quit IRC | 17:07 | |
| pabelanger | clarkb: fungi: what do you think about using restorecon -v over chcon? https://review.openstack.org/#/c/211703/1/nodepool/elements/nodepool-base/finalise.d/99-unbound | 17:07 |
| *** madong_ has joined #openstack-infra | 17:07 | |
| *** zhenguo has quit IRC | 17:07 | |
| *** hdd has quit IRC | 17:07 | |
| clarkb | zaro not if you want to run services outside of the root network and process namespaces (I thibk those are the two that give us trouble) | 17:07 |
| *** smccully has joined #openstack-infra | 17:07 | |
| clarkb | pabelanger why wouldnt you just change the comment? | 17:08 |
| clarkb | # dib always enables selinux, fix unbound something something | 17:08 |
| *** lucasagomes has left #openstack-infra | 17:09 | |
| *** hdd has joined #openstack-infra | 17:09 | |
| *** dguerri is now known as dguerri` | 17:09 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Fix timeout function and add capturing of messages on failure https://review.openstack.org/211776 | 17:09 |
| pelix | zaro: oops missed one last bug, but at least this time it's deterministic ;) | 17:09 |
| clarkb | zaro so it depends on your definution of a what a container is | 17:09 |
| zaro | pelix: cool, will take a look today. | 17:10 |
| *** sergmelikyan has quit IRC | 17:10 | |
| fungi | clarkb: pabelanger: on, restorecon vs chcon i really don't have an opinion. if it fixes this to support selinux on both centos and fedora (assuming it's on by default upstream for both distros) then i'm cool with it | 17:10 |
| clarkb | oh does chcon not exist on fedora? in that case sure I guess I missed the intent of the originak change | 17:11 |
| *** tonytan4ever has joined #openstack-infra | 17:11 | |
| pabelanger | clarkb: no, it exists. But for some reason, I get this error: | 17:11 |
| pabelanger | chcon system_u:object_r:named_conf_t:s0 /etc/unbound/forwarding.conf | 17:11 |
| zaro | clarkb: haha, technicalities. i have no idea what exists besides a docker container running some OS or possibility ironic. | 17:11 |
| pabelanger | chcon: invalid context: system_u:object_r:named_conf_t:s0: No such file or directory | 17:11 |
| pabelanger | cannot figure out why | 17:12 |
| *** sputnik13 has joined #openstack-infra | 17:12 | |
| fungi | clarkb: pabelanger: that aside, i still think (as i stated in my review comment) that the comment should be replaced by something dib-relevant | 17:12 |
| pabelanger | so, testing restorecon -v now, to see if I can restore context to whole directory | 17:12 |
| pabelanger | fungi: ack. I can change that too | 17:12 |
| *** regXboi has joined #openstack-infra | 17:13 | |
| clarkb | maybe fedora dropped that context? | 17:13 |
| clarkb | I dont know enough about selinux to hwlp here | 17:13 |
| *** armax has joined #openstack-infra | 17:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/project-config: Add election project https://review.openstack.org/213785 | 17:14 |
| *** salv-orl_ has joined #openstack-infra | 17:14 | |
| pabelanger | clarkb: No, it looks to be valid. Ya, this my issue too, don't work much with selinux | 17:14 |
| *** mestery has quit IRC | 17:15 | |
| *** salv-orlando has quit IRC | 17:16 | |
| *** sergmelikyan has joined #openstack-infra | 17:18 | |
| greghaynes | clarkb: DIB doesnt have to enable selinux, we kind of decided thats probably a good thing keep doing though | 17:19 |
| greghaynes | pabelanger: Where were you testing that? | 17:20 |
| fungi | greghaynes: i think it should just do whatever upstream fedora and centos default installs do, wrt selinux | 17:20 |
| clarkb | ya we want selinux to work since it does currently | 17:20 |
| fungi | which seems to be selinux on and in enforcing mode | 17:20 |
| clarkb | disabling it is weird after we went through th trouble of getting it enforcing | 17:20 |
| *** sdake_ has joined #openstack-infra | 17:20 | |
| greghaynes | ++ | 17:21 |
| pabelanger | greghaynes: local ubuntu 12.04 VM | 17:22 |
| *** jerryz has joined #openstack-infra | 17:23 | |
| *** hdd has quit IRC | 17:23 | |
| greghaynes | pabelanger: hrm, I wonder if thats what happens when you run chcon in a chroot on a host that doesnt have selinux enabled | 17:24 |
| clarkb | 12.04 is not supported for crossbuilds | 17:24 |
| greghaynes | heh, that too | 17:24 |
| *** sdake has quit IRC | 17:24 | |
| clarkb | greghaynes we do chcon on trusty | 17:24 |
| greghaynes | oh | 17:25 |
| clarkb | when building centos and older fedora images | 17:25 |
| *** hdd has joined #openstack-infra | 17:25 | |
| greghaynes | ok then | 17:25 |
| greghaynes | also http://docs.openstack.org/developer/diskimage-builder/user_guide/supported_distros.html for shiney docs on supported distros | 17:25 |
| *** mestery has joined #openstack-infra | 17:26 | |
| clarkb | fungi can I ask you to rereview https://review.openstack.org/#/c/212698/ ? I made the commit message more clear based on your feedback | 17:26 |
| *** achanda has joined #openstack-infra | 17:27 | |
| clarkb | oh hrm shoukd I quote the 1 to be like the other checks near it? | 17:27 |
| clarkb | bash shouldnt care for -eq | 17:27 |
| greghaynes | mordred: Did we ever get some input on likely cause of the error 396 import failed when you try to task-create an image to import in rax? | 17:28 |
| greghaynes | I am hitting that a *TON* right now | 17:28 |
| *** armax has quit IRC | 17:29 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:29 | |
| *** ivar-lazzaro has quit IRC | 17:30 | |
| *** moravec has quit IRC | 17:30 | |
| *** kevinbenton has quit IRC | 17:30 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:31 | |
| *** sdake has joined #openstack-infra | 17:31 | |
| *** kevinbenton has joined #openstack-infra | 17:32 | |
| pabelanger | greghaynes: clarkb: my mistake, this is 14.04 | 17:32 |
| *** dannywilson has joined #openstack-infra | 17:32 | |
| pabelanger | so, not sure what is happening | 17:32 |
| *** enikanorov2 has quit IRC | 17:32 | |
| *** mestery has quit IRC | 17:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Use restorecon over chcon for unbound selinux https://review.openstack.org/211703 | 17:34 |
| *** tiswanso has left #openstack-infra | 17:34 | |
| *** sputnik13 has quit IRC | 17:34 | |
| pabelanger | clarkb: fungi: greghaynes: ^ that's what I am thinking now. | 17:34 |
| *** dannywilson has quit IRC | 17:34 | |
| *** sdake_ has quit IRC | 17:35 | |
| *** sputnik13 has joined #openstack-infra | 17:35 | |
| *** markus_z has quit IRC | 17:38 | |
| pabelanger | testing centos6 now, but first glace seems to work as expected | 17:39 |
| *** Sukhdev has quit IRC | 17:40 | |
| *** Sukhdev has joined #openstack-infra | 17:40 | |
| SpamapS | greghaynes: are you at the ops meetup? | 17:41 |
| greghaynes | SpamapS: I will be tomorrow | 17:41 |
| SpamapS | greghaynes: cool. Today, I want to coordinate some actions to try and stand up HP1 entirely within the next 10 days. | 17:41 |
| SpamapS | greghaynes: it occurred to me this weekend that we really have nothing standing in our way now. | 17:41 |
| SpamapS | greghaynes: and we can even just build our own hiera so we don't have to use XXX as all the passwords ;) | 17:42 |
| greghaynes | SpamapS: Yep, I havent banged my head against the vlan stuff yet because other distractions, but that was the next big block | 17:42 |
| SpamapS | greghaynes: I believe mordred has made us a patch, which looks like it should at least try to work. ;) | 17:42 |
| greghaynes | SpamapS: Yes, we need to make bifrost support it now though | 17:42 |
| greghaynes | which isnt hard, its just a wall of jinja though | 17:43 |
| SpamapS | greghaynes: we can update the nodes after if need be | 17:43 |
| greghaynes | hah | 17:43 |
| SpamapS | greghaynes: but yeah, its just some stuff to add to the node info. :) | 17:43 |
| greghaynes | yes, we can | 17:43 |
| SpamapS | greghaynes: anyway, how about we do a little bit of war room planning this afternoon? I feel like we've been bumping into eachother instead of marching along side toward the same goal. | 17:44 |
| *** dangers_away is now known as dangers | 17:44 | |
| greghaynes | Sounds good | 17:44 |
| *** xyang1 has quit IRC | 17:44 | |
| SpamapS | greghaynes: free at 12:00? | 17:44 |
| greghaynes | Anytime really is good for me | 17:44 |
| greghaynes | sure | 17:44 |
| austin81 | mtreinish: Do you know if nodepool images are available for download/testing use anywhere? Or know who I should ask? | 17:44 |
| SpamapS | greghaynes: ok, 12:00. I suggest we do hangouts and a shared etherpad | 17:45 |
| SpamapS | anybody else want to get in on that for infra-cloud? | 17:45 |
| * SpamapS expects crickets | 17:45 | |
| greghaynes | ok. crinkle ^ | 17:45 |
| crinkle | o/ | 17:45 |
| greghaynes | SpamapS: I paired with crinkle the other day while I did some stuff, showed her what we had set up | 17:45 |
| SpamapS | no crinkle can't come because we're going to need to assign the bulk of work to her. | 17:45 |
| greghaynes | hahaa | 17:45 |
| crinkle | >.< | 17:45 |
| SpamapS | best if it just comes down like an ACME safe on her head. ;) | 17:46 |
| TheJulia | SpamapS: wait, you can't do that, crinkle knows now :) | 17:46 |
| nibalizer | hangouts instead of asterisk server? | 17:46 |
| * nibalizer is that guy today | 17:46 | |
| SpamapS | TheJulia: she knows what, but not _when_ ;) | 17:46 |
| crinkle | @.@ | 17:46 |
| SpamapS | Oh I'd LOVE to do the asterisk server | 17:46 |
| SpamapS | seriously | 17:46 |
| SpamapS | that sounds fun | 17:46 |
| TheJulia | SpamapS: evil... very evil. | 17:46 |
| *** xyang1 has joined #openstack-infra | 17:46 | |
| greghaynes | ++ to asterisk | 17:47 |
| clarkb | austin81 they are not available for download but you can build your own using dib | 17:47 |
| *** ZZelle-off has quit IRC | 17:47 | |
| greghaynes | who here knows how to set up a room on there? | 17:47 |
| SpamapS | is the way to use that in ci.openstack.org somewhere? | 17:47 |
| clarkb | austin81 in project-config/tools there is a build-image.sh script to make it easy | 17:47 |
| *** pvaneck has joined #openstack-infra | 17:48 | |
| austin81 | clarkb: ah perfect, thank you! | 17:48 |
| *** sthillma has joined #openstack-infra | 17:48 | |
| *** angdraug has quit IRC | 17:49 | |
| greghaynes | oh, that reminds me, I need to hack on the dib for raspi deal so we can asterisk room in tokyo | 17:50 |
| *** afazkas has joined #openstack-infra | 17:50 | |
| greghaynes | SpamapS: https://wiki.openstack.org/wiki/Infrastructure/Conferencing | 17:52 |
| *** rkukura has quit IRC | 17:53 | |
| *** dims_ has joined #openstack-infra | 17:53 | |
| *** ybathia has joined #openstack-infra | 17:56 | |
| *** zul has joined #openstack-infra | 17:56 | |
| *** dims has quit IRC | 17:57 | |
| *** sergmelikyan has quit IRC | 17:58 | |
| SpamapS | greghaynes: ty | 17:59 |
| *** sridhar_ram has joined #openstack-infra | 18:00 | |
| *** afazkas has quit IRC | 18:01 | |
| pabelanger | nibalizer: greghaynes: Actually, we should setup a freeswitch server, they support video MCU now | 18:02 |
| greghaynes | heh | 18:02 |
| asselin_ | wznoinsk, see zaro & clarkb discussion ^^ for using containers to run openstack about an hour ago | 18:02 |
| devananda | (c) header question for folks -- if a vendor/driver author "forgot" to add the (c) lines in their files early on and wants to do it now, is there any reason to object? | 18:02 |
| devananda | patch in question: https://review.openstack.org/#/c/212973/ | 18:02 |
| pabelanger | russellb: looks like I have a new project to get freeswitch going to do video conferencing | 18:02 |
| clarkb | devananda if its accurate data and the holder eants it I dont think so, but they do tend to get stale over time | 18:03 |
| clarkb | I need to run to visit the dentist now back in a couple hours | 18:03 |
| devananda | clarkb: thanks. and good luck | 18:04 |
| *** angdraug has joined #openstack-infra | 18:05 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/project-config: Publish ironic-python-agent releases to pypi https://review.openstack.org/213749 | 18:06 |
| *** yamamoto has joined #openstack-infra | 18:06 | |
| *** sthillma has quit IRC | 18:07 | |
| *** sthillma has joined #openstack-infra | 18:07 | |
| openstackgerrit | Merged openstack-dev/pbr: Expose a 'rpm_version' extra command https://review.openstack.org/196191 | 18:08 |
| *** krtaylor has quit IRC | 18:08 | |
| *** stevemar has quit IRC | 18:08 | |
| SpamapS | weird, pidgin has no basic SIP mode | 18:10 |
| *** stevemar has joined #openstack-infra | 18:10 | |
| greghaynes | SpamapS: ekiga? | 18:10 |
| clarkb | I just use pots | 18:10 |
| SpamapS | I was going to try jitsi since it is recommended on the page | 18:10 |
| *** bpokorny_ has joined #openstack-infra | 18:11 | |
| SpamapS | just surprised Pidgin won't work | 18:11 |
| clarkb | because my phone works well as a phone and sip clients are all pretty braindead and wont do unauthed sip | 18:11 |
| *** pelix has quit IRC | 18:11 | |
| SpamapS | clarkb: perhaps we should add a password? ;-) | 18:11 |
| *** yamamoto has quit IRC | 18:11 | |
| *** bpokorny has quit IRC | 18:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/project-config: Add election project https://review.openstack.org/213785 | 18:12 |
| *** jeadr has quit IRC | 18:12 | |
| *** jeadr has joined #openstack-infra | 18:13 | |
| pabelanger | clarkb: fungi: patch looks good under centos6 | 18:13 |
| SpamapS | hrm | 18:14 |
| pabelanger | last I check, no native SIP support for pidgin. | 18:14 |
| pabelanger | recommend, icanblink.com | 18:15 |
| SpamapS | it does have native SIP | 18:16 |
| SpamapS | calls it "SIMPLE" | 18:16 |
| SpamapS | for some reason | 18:16 |
| SpamapS | but it won't allow no password | 18:16 |
| greghaynes | There has to be a rule about software with simple in the name | 18:16 |
| greghaynes | and how it never can be | 18:16 |
| pabelanger | SpamapS: doubt SIMPLE will work with asterisk 1.8. Would be surprised if it does | 18:17 |
| SpamapS | haha | 18:17 |
| SpamapS | Ok, we'll be in conference #7123 | 18:18 |
| *** btully has quit IRC | 18:18 | |
| *** davideagnello has quit IRC | 18:18 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Use dnf download to cache packages https://review.openstack.org/211294 | 18:19 |
| SpamapS | pabelanger: I'm pretty asterisk-stupid so.. is there a newer one that would support SIMPLE? | 18:20 |
| SpamapS | anyway, jitsi seems fine | 18:20 |
| pabelanger | SpamapS: new what? version of asterisk? | 18:20 |
| *** rraja has quit IRC | 18:20 | |
| SpamapS | pabelanger: yeah, I don't know what SIMPLE is | 18:21 |
| SpamapS | _WORST_ acronym ever btw | 18:21 |
| *** krtaylor has joined #openstack-infra | 18:21 | |
| SpamapS | seeing as anything with 6 acronymable words in its description is probably _NOT_. | 18:21 |
| timrc | Maybe it's not an acronym. Maybe the authors are yelling at you SpamapS | 18:22 |
| * greghaynes thinks of GSSAPI | 18:22 | |
| pabelanger | SpamapS: no, asterisk doesn't include SIMPLE support IIRC. We'd likely have to add some code to support pidgin. Mind you, pidgin isn't a phone, so expecting it to work is flakey at best. | 18:22 |
| pabelanger | another reason I just use blink | 18:22 |
| pabelanger | is a SIP phone, and is simple to use / configure | 18:22 |
| SpamapS | pabelanger: well pidgin has numerous abilities to provide voice support, so I think it's reasonable to think SIP would be easy. | 18:22 |
| pabelanger | SpamapS: sure, but asterisk is a phone system. | 18:23 |
| timrc | pabelanger, I think "expecting it to work is flakey at best" pretty much describes Pidgin, in general. | 18:23 |
| pabelanger | not IM | 18:23 |
| *** sharikapongubala has joined #openstack-infra | 18:23 | |
| *** kutija has joined #openstack-infra | 18:23 | |
| *** afazkas has joined #openstack-infra | 18:24 | |
| *** bharath has quit IRC | 18:24 | |
| *** kushal has quit IRC | 18:25 | |
| SpamapS | pabelanger: so, we need to hold on to the anachronism of the past just because asterisk does? | 18:26 |
| greghaynes | clarkb: fungi the dib growroot element just worked for me in hpcloud | 18:26 |
| greghaynes | clarkb: fungi SpamapS Reviews on https://review.openstack.org/#/c/206636/ welcome so we can move ahead with ubuntu-trusty nodes | 18:27 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add config flags for data source configuration https://review.openstack.org/213811 | 18:28 |
| *** yamahata has quit IRC | 18:28 | |
| pabelanger | SpamapS: seems that way. Newer version of asterisk may have better support, but like I said, I would be surprise. Most production installs, people still use hard phones. It is hard enough to try and get people to move to an softphone let alone something like pidgin. | 18:30 |
| *** afazkas has quit IRC | 18:30 | |
| *** sergmelikyan has joined #openstack-infra | 18:30 | |
| SpamapS | pabelanger: I'm pretty accustomed to using hangouts and skype.. so I actually am pretty averse to picking up my phone if I have a soft client that I can use. | 18:31 |
| *** annegentle has quit IRC | 18:32 | |
| pabelanger | SpamapS: hard phones won't die any time soon sadly. Either way, if we want to support pidgin, work could be done to support it. | 18:34 |
| clarkb | SpamapS huh I use hard phone with those things on it | 18:35 |
| clarkb | then I dont have to fight creative and whatever the built in sound card is on my desktop | 18:35 |
| *** MarkAtwood has joined #openstack-infra | 18:36 | |
| *** mriedem has joined #openstack-infra | 18:37 | |
| *** ayoung has quit IRC | 18:38 | |
| pabelanger | clarkb: buuut webrtc | 18:38 |
| clarkb | pabrlanger need working mic for that to help | 18:38 |
| clarkb | and dont get me started on pulse choosing hdmi out when no hdmi cable is plugged in | 18:38 |
| pabelanger | foiled again | 18:38 |
| clarkb | I have 3 sound devices in one machinr and none work right | 18:39 |
| *** spzala has quit IRC | 18:39 | |
| *** sshnaidm has joined #openstack-infra | 18:39 | |
| fungi | greghaynes: awesome! | 18:40 |
| *** e0ne has joined #openstack-infra | 18:40 | |
| fungi | clarkb: 212698 lgtm, thanks | 18:41 |
| *** olaph has joined #openstack-infra | 18:41 | |
| clarkb | so I just plug headset into phone and use it, works great | 18:41 |
| clarkb | also means no skyoe or hangouts on machines I care about | 18:42 |
| *** d34dh0r53 is now known as VD | 18:42 | |
| *** matbu has quit IRC | 18:43 | |
| SpamapS | I have a USB mic that is fantastic | 18:43 |
| *** VD is now known as Guest69442 | 18:43 | |
| SpamapS | generally there's no fighting. :) | 18:43 |
| SpamapS | it has a plug for monitor too, so you can just use it as your output | 18:43 |
| clarkb | SpamapS hp gave us all one that would only echo mic input to the headphones never got it working wither | 18:43 |
| fungi | relearning a lesson i've learned once already... not sure why it didn't stick: when moving, hire professionals. it's been 5 weeks since we closed on the house and just today i finally removed the last of the cleaning supplies, tools and bulk trash from the rental. tomorrow i can 100% openstack, then babysit carpet cleaners on wednesday and drop off the keys. will be _so_ glad to have this done with | 18:43 |
| SpamapS | clarkb: that sux | 18:43 |
| SpamapS | I have a GoMic | 18:43 |
| clarkb | fungi woot | 18:44 |
| SpamapS | only echo I get is with hangouts when the other side isn't using headphones. | 18:44 |
| clarkb | fungi now you can start all the work on the new place | 18:44 |
| SpamapS | fungi: oh god yes. Professionals are required for grown-up moves | 18:44 |
| fungi | clarkb: don't remind me. already have to schedule a plumber and an electrician later this week | 18:45 |
| SpamapS | I got pro's for my last one.. and I still went back to the old place for a week grabbing the stuff they missed. | 18:45 |
| fungi | oh, and someone to restring the flag pole so i can hoist my colors! | 18:45 |
| *** bharath has joined #openstack-infra | 18:45 | |
| fungi | avast ye land lubbers | 18:45 |
| clarkb | arrr | 18:45 |
| clarkb | SpamapS it was lync certified too :) | 18:46 |
| fungi | i have a 6' jolly roger waiting to be raised as soon as the 30' flag pole is in working condition | 18:46 |
| *** Guest69442 is now known as d34dh0r53 | 18:47 | |
| fungi | (i guess that's a 2m flag and 10m flag pole for most people on this channel) | 18:48 |
| *** olaph has quit IRC | 18:51 | |
| fungi | greghaynes: on 213180 why does $scripts_dir have a trailing "."? | 18:51 |
| greghaynes | fungi: thats how you make cp -R descend ibto the dir rather than copy the parent dir itself | 18:52 |
| greghaynes | (I learned that today) | 18:53 |
| fungi | greghaynes: oh, does $scripts_dir embed a trailing /? | 18:53 |
| greghaynes | yes | 18:53 |
| *** olaph has joined #openstack-infra | 18:53 | |
| fungi | right! i see you adding it above | 18:53 |
| fungi | okay, +1 (i'm not core on dib) | 18:53 |
| greghaynes | woot | 18:53 |
| *** nadya has joined #openstack-infra | 18:54 | |
| SpamapS | greghaynes: 6 minutes to SIP call. Conference 7123 | 18:55 |
| SpamapS | crinkle: ^ | 18:55 |
| greghaynes | ok | 18:55 |
| fungi | greghaynes: with 206636 i assume it's safe for us to stop installing cloud-utils which is dragging in the python-requests deb? | 18:55 |
| *** krtaylor has quit IRC | 18:55 | |
| fungi | (or rather, any time since it wasn't doing anything for us anyway?_ | 18:56 |
| morgan_2549 | can we include MIT license stuff in g-r? | 18:56 |
| *** pcrews has joined #openstack-infra | 18:56 | |
| greghaynes | fungi: is that being pulled by cloud-initramfs-growroot? | 18:56 |
| morgan_2549 | or is MIT also not-friendly? [I think MIT is friendly enough] | 18:56 |
| clarkb | yes mit/x11 is compat with apache2 | 18:56 |
| morgan_2549 | cool | 18:56 |
| fungi | morgan_2549: totally, yes (though "mit license" is vague. i assume you mean the expat license?) | 18:56 |
| greghaynes | fungi: if so, im working on a dib patch to remove it | 18:56 |
| fungi | mit x11 license != mit expat license | 18:56 |
| morgan_2549 | fungi: https://github.com/zoldar/python-ldap-test/blob/master/LICENSE.txt | 18:56 |
| morgan_2549 | that version | 18:56 |
| morgan_2549 | clarkb, fungi: I might ask for a JRE on some nodes [will be highly specific] -- will discuss later. | 18:57 |
| fungi | morgan_2549: looks like expat though i'm comparing it now | 18:57 |
| morgan_2549 | fungi: ty | 18:57 |
| clarkb | all the nodes have a jre already | 18:57 |
| morgan_2549 | fantastic | 18:57 |
| clarkb | for jenkins | 18:58 |
| morgan_2549 | that makes life easy | 18:58 |
| morgan_2549 | that python-ldap-test [makes an in-mem LDAP server] | 18:58 |
| morgan_2549 | vs. needing to spin up a whole slapd etc | 18:58 |
| morgan_2549 | useful for validating our LDAP code | 18:58 |
| fungi | morgan_2549: yep, looks like http://directory.fsf.org/wiki/License:Expat to me | 18:58 |
| fungi | it's basically ~= isc and 2-clause bsd | 18:59 |
| morgan_2549 | fungi: thnx for the confirm | 18:59 |
| morgan_2549 | makes asking that lib to join g-r for testing easier | 19:00 |
| morgan_2549 | :) | 19:00 |
| morgan_2549 | will make sure to mention this convo when proposing | 19:00 |
| *** spzala has joined #openstack-infra | 19:02 | |
| *** sshnaidm has quit IRC | 19:02 | |
| *** e0ne has quit IRC | 19:03 | |
| *** matbu has joined #openstack-infra | 19:03 | |
| fungi | greghaynes: i still need to dig deeper into the source-repos element. i think i have evidence suggesting my change to fetch all remote branches when a ref of "*" is supplied is not working as designed | 19:04 |
| *** flepied has quit IRC | 19:04 | |
| fungi | seems to get all tags, but not all branches from the origin remote | 19:05 |
| SpamapS | greghaynes: https://etherpad.openstack.org/p/infra-cloud-bootstrap | 19:05 |
| crinkle | greghaynes: SpamapS i'm in the call but i guess my mic isn't working | 19:05 |
| *** annegentle has joined #openstack-infra | 19:05 | |
| SpamapS | crinkle: ohnoes | 19:05 |
| SpamapS | crinkle: try POTS? | 19:05 |
| * fungi usually just uses pots. old school | 19:06 | |
| fungi | (pans too, sometimes) | 19:06 |
| *** geoffarnold has quit IRC | 19:07 | |
| anteaya | TIL the us manufactures things it can convince you is money but really isn't | 19:07 |
| anteaya | as in you will be given it as change and noone else will take it | 19:08 |
| anteaya | woooooo | 19:08 |
| pabelanger | if we have selinux folk in the channel, I'd love some feedback on the following change: https://review.openstack.org/#/c/211703 | 19:08 |
| *** krtaylor has joined #openstack-infra | 19:08 | |
| *** regXboi has quit IRC | 19:08 | |
| *** e0ne has joined #openstack-infra | 19:09 | |
| fungi | anteaya: yep, most residents here just use revolving virtual currency and spend their lives as indentured servants of banking conglomerates | 19:09 |
| *** olaph has quit IRC | 19:09 | |
| anteaya | well there's that | 19:09 |
| anteaya | I was actually refrencing the dollar coin | 19:09 |
| openstackgerrit | Tim Buckley proposed openstack-infra/devstack-gate: Install and run StackViz during devstack-gate host cleanup. https://review.openstack.org/212207 | 19:10 |
| anteaya | which is worth a dollar if you can find someone who agrees with you | 19:10 |
| anteaya | and is a hopscotch lager if they don't | 19:10 |
| *** pcrews has quit IRC | 19:10 | |
| anteaya | optional money | 19:10 |
| anteaya | it is a new concept for me | 19:10 |
| *** hdd has quit IRC | 19:12 | |
| *** geoffarnold has joined #openstack-infra | 19:12 | |
| *** armax has joined #openstack-infra | 19:12 | |
| fungi | pretty sure places which accept cash are actually obligated to not refuse dollar coins, though the number of store clerks who mostly only know what a credit card looks like and never learned how to do basic math have been steadily rising in the past few decades | 19:12 |
| *** davideagnello has joined #openstack-infra | 19:14 | |
| *** hdd has joined #openstack-infra | 19:15 | |
| *** Sukhdev has quit IRC | 19:15 | |
| anteaya | interesting | 19:15 |
| *** mrunge has joined #openstack-infra | 19:15 | |
| taron | I don't think I've even seen a dollar coin since I was 10 or so | 19:16 |
| taron | at least, not in the US anyway | 19:16 |
| anteaya | the clerk in question informed me it was "my right" to refuse to have a dollar coin given to me as change | 19:16 |
| anteaya | no idea where it came from | 19:16 |
| anteaya | it was in the "us change" purse so got packed | 19:16 |
| anteaya | fungi: but thanks I will try the you are obliged to honour this as money angle next time | 19:17 |
| anteaya | not sure who I complain to if they take the no path | 19:17 |
| anteaya | interesting mindset though | 19:18 |
| *** moravec has joined #openstack-infra | 19:18 | |
| *** kushal has joined #openstack-infra | 19:20 | |
| * jeblair just dialed in | 19:21 | |
| *** serg_melikyan has joined #openstack-infra | 19:22 | |
| jeblair | greghaynes, SpamapS: what's the etherpad? | 19:22 |
| *** sputnik1_ has joined #openstack-infra | 19:22 | |
| greghaynes | jeblair: https://etherpad.openstack.org/p/infra-cloud-bootstrap | 19:22 |
| jeblair | thx | 19:22 |
| *** j_king has joined #openstack-infra | 19:23 | |
| *** regXboi has joined #openstack-infra | 19:23 | |
| *** sergmelikyan has quit IRC | 19:24 | |
| j_king | fungi: is there a way to run a gate test with a higher timeout threshold? specifically looking at http://logs.openstack.org/26/210126/4/check/gate-tempest-dsvm-neutron-full/ed46df3/ and finding a lot of tempest client timeouts and wondering if up'ing that will resolve them for this patch (deleting a VM force-waits for a notification from neutron) | 19:24 |
| rcarrillocruz | mordred: heya, jeblair suggested to ping you on https://review.openstack.org/#/c/211923/ , it's some toy project i started working on here and then to automate deployment of openstack-infra with ansible | 19:24 |
| anteaya | j_king: we find that increasing the timeout threshold just masks an underlying problem | 19:25 |
| mordred | rcarrillocruz: yes! I looked at it a little bit already and like several things about it | 19:25 |
| *** sputnik13 has quit IRC | 19:25 | |
| rcarrillocruz | seems you have plans to get rid of launch node and drive deployments with ansible etc | 19:25 |
| mordred | rcarrillocruz: my biggest feedback is that I'd like for cloud to have a global default and also be overridable on a per-host basis | 19:26 |
| anteaya | j_king: can you find what the actual problem creating the timeout is and perhaps file a bug or patch? | 19:26 |
| rcarrillocruz | very very very alpha btw, but got a nudge to just put it on openstack infra, instead of github | 19:26 |
| mordred | rcarrillocruz: (since we have hosts in different clouds) | 19:26 |
| mordred | yeah - I think just putting it in infra would be great - I think originally I was thinking it should go into system-config - but what you've got is big enough that I agree with it having its own repo | 19:26 |
| j_king | anteaya: I'm a little lost tbh; these are coming from cinder, neutron, and a few others in tests that don't seem to have much to do with my patch and more to do with teardown code failling? | 19:27 |
| mordred | rcarrillocruz: that said - I think we should pull the role out into its own repo as well (similar to ansible-puppet) | 19:27 |
| mordred | rcarrillocruz: that way we can install it with galaxy | 19:27 |
| rcarrillocruz | do we have an openstack-infra galaxy namespace yet? not sure how you manage that... | 19:27 |
| rcarrillocruz | i don't think we just pull from your e_monty namespace | 19:28 |
| rcarrillocruz | ? | 19:28 |
| j_king | thought maybe that gate test was failing due to assumptions about deleting servers being instantaneous... | 19:28 |
| mordred | no - we haven't been publishing to galaxy itself | 19:28 |
| mordred | rcarrillocruz: but you can install roles using the galaxy tool from git repos | 19:28 |
| rcarrillocruz | ah ok... gotcha | 19:28 |
| anteaya | j_king: okay so in that instance perhaps spending some time with the qa team and understanding the failures might help? | 19:28 |
| mordred | rcarrillocruz: http://git.openstack.org/cgit/openstack-infra/system-config/tree/roles.yaml | 19:28 |
| anteaya | j_king: have you met mtreinish and sdague yet? | 19:29 |
| rcarrillocruz | anyway, looking forward to have it on openstack-ci family to get your ideas and others contributions | 19:29 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add support to filter results by failure test_ids https://review.openstack.org/209712 | 19:29 |
| rcarrillocruz | jeblair: is it ok to have it under openstack-infra tho? | 19:29 |
| akraminsky | hello, is there an issue with upstream ssl certs? I'm trying to clone 'https://git.openstack.org/openstack-infra/puppet-jenkins/ and am getting a "server certification verification failed" error | 19:29 |
| j_king | anteaya: I have not but would like to know how to go about this. it's a little over-whelming. | 19:29 |
| jeblair | rcarrillocruz: yep | 19:29 |
| rcarrillocruz | or should i put it on openstack ? | 19:29 |
| anteaya | and yes if teardown code is having a problem that would be something best addressed | 19:29 |
| rcarrillocruz | k | 19:29 |
| rcarrillocruz | thx | 19:29 |
| * rcarrillocruz goes back to vaca time | 19:29 | |
| mordred | rcarrillocruz: I'd split each role into its own standalone repo just so we're ahread of the game | 19:30 |
| anteaya | j_king: I understand and yes it is | 19:30 |
| j_king | mtreinish sdague: hello | 19:30 |
| anteaya | j_king: are you also in the #openstack-qa channel? | 19:30 |
| mordred | jeblair: https://github.com/rcarrillocruz/infra-ansible/blob/master/infra_config.yml.sample | 19:30 |
| rcarrillocruz | mordred: one repo for each role? i think we should just have openstack-infra/ansible-roles and maybe put system-config in there | 19:30 |
| rcarrillocruz | ? | 19:30 |
| j_king | anteaya: i am now | 19:30 |
| jeblair | mordred: that looks like things you've been saying for a while :) | 19:30 |
| rcarrillocruz | reusability,e tc | 19:30 |
| anteaya | j_king: thanks | 19:30 |
| mordred | jeblair: yah | 19:31 |
| mordred | rcarrillocruz: no. we shoudl have one repo per role | 19:31 |
| jeblair | mordred: there are several roles in there -- all of them should be new repos? | 19:31 |
| *** mriedem1 has joined #openstack-infra | 19:31 | |
| mordred | jeblair: yes. it's like puppet modules | 19:31 |
| *** matbu has quit IRC | 19:32 | |
| jeblair | mordred: hrm, some of those i'm not convinced will live for a long time... | 19:32 |
| rcarrillocruz | hmm, ok, i think we should bring the project onboard | 19:32 |
| jeblair | mordred: would it be worthwhile to incubate this for a bit? | 19:32 |
| rcarrillocruz | do some refactor | 19:32 |
| mordred | jeblair: sure! | 19:32 |
| mordred | rcarrillocruz: agree | 19:32 |
| rcarrillocruz | and whichever roles deemed to live for long, then split | 19:32 |
| rcarrillocruz | ? | 19:32 |
| *** hdd has quit IRC | 19:33 | |
| rcarrillocruz | jeblair: ++ | 19:33 |
| mordred | yeah - we don't have to do that this second - but I think we should aim for a world in which we have role-per-repo and where each of those roles is a unit of work similar to a puppet module | 19:33 |
| *** mriedem has quit IRC | 19:33 | |
| *** hdd has joined #openstack-infra | 19:33 | |
| jeblair | mordred: yeah, i agree. maybe we can do that soon, but stabilize it a bit and make sure we know where it's heading first, so we don't end up with a bunch of throwaway git repos | 19:34 |
| *** matbu has joined #openstack-infra | 19:34 | |
| mordred | ++ | 19:34 |
| rcarrillocruz | yeah, way too alpha right now | 19:34 |
| *** sharikapongubala has quit IRC | 19:35 | |
| *** smccully has quit IRC | 19:35 | |
| nibalizer | taron: http://paste.openstack.org/show/419482/ | 19:36 |
| nibalizer | looks like we need an Httpd_mod { 'rewrite': } | 19:36 |
| nibalizer | or something like that | 19:36 |
| *** tonytan4ever has quit IRC | 19:36 | |
| *** dannywilson has joined #openstack-infra | 19:37 | |
| anteaya | from the Canadian PyCon organizers: Despite the now past deadline, if any of your colleagues would still be interested in speaking at PyCon Canada we would gladly receive their proposals! | 19:38 |
| anteaya | https://2015.pycon.ca/ | 19:38 |
| j_king | anteaya: are you involved w/ PyconCA? | 19:38 |
| anteaya | no | 19:39 |
| * j_king has been a speaker there a couple of times in the past. | 19:39 | |
| openstackgerrit | Armando Migliaccio proposed openstack/requirements: Add networking-l2gw to the list https://review.openstack.org/213833 | 19:39 |
| anteaya | I just found out there is a PyCon Canada this year and copied and pasted a talk submission | 19:39 |
| anteaya | I gave them feedback that as far as I knew noone associated with OpenStack new Canada was having a PyCon | 19:40 |
| anteaya | so maybe next year we can address that | 19:40 |
| anteaya | and was given that sentence as part of my reply | 19:40 |
| anteaya | j_king: and cool, you in Canada? | 19:40 |
| j_king | anteaya: yes | 19:41 |
| pabelanger | There is not much info about the event on the website | 19:41 |
| anteaya | pabelanger: not a lot no, feels like a last minute affair, which is fine, you have to start somewhere | 19:41 |
| anteaya | j_king: oh do tell, where are you? | 19:42 |
| anteaya | I'm north of Toronto about 3 hours and pabelanger is in Ottawa | 19:42 |
| j_king | anteaya: Hamilton, Ontario. About 35mins from Toronto. :) | 19:42 |
| *** annegentle has quit IRC | 19:42 | |
| *** annegentle has joined #openstack-infra | 19:43 | |
| anteaya | nice | 19:43 |
| j_king | i work for Dreamhost on their Openstack-powered cloud. | 19:43 |
| anteaya | been there many times | 19:43 |
| anteaya | cool, remotely I'm guessing | 19:43 |
| j_king | yes. :) | 19:43 |
| anteaya | nice | 19:43 |
| anteaya | we will have to work on meeting up sometime | 19:44 |
| j_king | will be happy to buy you a beverage of choice | 19:44 |
| anteaya | tea or hot chocolate, and yes I will take you up on thata | 19:44 |
| *** rkukura has joined #openstack-infra | 19:44 | |
| anteaya | funny thing is I'm never in Toronto anymore | 19:44 |
| j_king | np, I know some good places for hot chocolate. :) | 19:44 |
| anteaya | just to the airport and gone | 19:44 |
| anteaya | nice | 19:44 |
| anteaya | going to be nice to have a few days in TO | 19:45 |
| *** e0ne has quit IRC | 19:45 | |
| openstackgerrit | Emma Barber proposed openstack-infra/puppet-hound: Configure and run apache on top of hound https://review.openstack.org/211640 | 19:45 |
| *** davideagnello has quit IRC | 19:48 | |
| *** e0ne has joined #openstack-infra | 19:48 | |
| *** dims_ has quit IRC | 19:48 | |
| *** dims has joined #openstack-infra | 19:48 | |
| anteaya | j_king: can you also join the #openstack-neutron channel? | 19:49 |
| fungi | akraminsky: seems relatively fine... https://www.ssllabs.com/ssltest/analyze.html?d=git.openstack.org bad proxy in your network? | 19:50 |
| anteaya | j_king: regXboi mentions he is working on a tempest failure | 19:50 |
| * fungi notes we could probably stand to tighten up the apache config there | 19:50 | |
| j_king | anteaya: ah | 19:50 |
| anteaya | j_king: you might be able to compare notes | 19:50 |
| *** jeadr has quit IRC | 19:51 | |
| regXboi | j_king, anteaya: my timeout issue is not the same - and I'm pretty sure it's caused by a mistake in the patch set .... sorry about that | 19:52 |
| j_king | regXboi: np | 19:52 |
| anteaya | regXboi: thanks for looking | 19:52 |
| *** jeadr has joined #openstack-infra | 19:52 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Make it possible to not remove stack sudo https://review.openstack.org/212698 | 19:52 |
| *** ayoung has joined #openstack-infra | 19:53 | |
| anteaya | I appear to be at the wrong gate | 19:54 |
| anteaya | yay me | 19:54 |
| regXboi | anteaya: oh no! | 19:55 |
| *** davideagnello has joined #openstack-infra | 19:57 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-gerrit: Add config option for asyncLoggingBufferSize https://review.openstack.org/213848 | 20:01 |
| *** davideagnello has quit IRC | 20:02 | |
| *** otter768 has joined #openstack-infra | 20:04 | |
| *** otter768 has quit IRC | 20:08 | |
| *** doug-fish has quit IRC | 20:09 | |
| *** doug-fish has joined #openstack-infra | 20:09 | |
| openstackgerrit | Tyler Cipriani proposed openstack-infra/jenkins-job-builder: adding support for android emulator https://review.openstack.org/146990 | 20:10 |
| *** docaedo2 has joined #openstack-infra | 20:13 | |
| *** crc32 has joined #openstack-infra | 20:15 | |
| *** alivigni has quit IRC | 20:15 | |
| *** sharikapongubala has joined #openstack-infra | 20:16 | |
| jgriffith | Are folks aware that openstack repos seem busted? https://www.dropbox.com/s/02pjxvsm9v66go2/Screenshot%202015-08-17%2014.16.43.png?dl=0 | 20:17 |
| jgriffith | also getting login/pass prompts trying to clone repos | 20:17 |
| clarkb | * githubs openstack/ mirror | 20:18 |
| clarkb | I can take a look | 20:18 |
| clarkb | jgriffith: but this is why we run our own set of git servers at https://git.openstack.org | 20:18 |
| jgriffith | actually... might just be Cinder | 20:18 |
| jgriffith | clarkb: well... ok | 20:18 |
| jgriffith | clarkb: never mind then | 20:18 |
| *** thcipriani has joined #openstack-infra | 20:18 | |
| jgriffith | it's github's fault. First time in 4 years for me | 20:18 |
| clarkb | jgriffith: according to their status page they are investigating the outage | 20:19 |
| fungi | clarkb: StevenK: camunoz: thanks for working through the zanata upgrade earlier | 20:19 |
| jgriffith | but I'll switch to using openstack's servers | 20:19 |
| clarkb | jgriffith: we see them fail constantly fwiw | 20:19 |
| jgriffith | clarkb: I rarely have issues, but almost always have issues with openstack servers FWIW | 20:19 |
| jgriffith | have to change my conf files in CI setups as the constantly time out | 20:19 |
| *** mrmartin has quit IRC | 20:19 | |
| clarkb | jgriffith: are you caching the repos in your ci setup? | 20:20 |
| fungi | interesting. we set up our own git server farm because we saw such regular outages from github | 20:20 |
| jgriffith | fungi: ha! Kind of ironic | 20:20 |
| clarkb | this is the easiest way to speed up fetches and avoid timeouts | 20:20 |
| jgriffith | fungi: clarkb leave it to me to be the counter case | 20:20 |
| clarkb | we do occasionally have trouble with a lot of nova clones eating all the memory on the backends | 20:20 |
| jgriffith | clarkb: yeah.. I don't, but probably should :) | 20:20 |
| fungi | yeah, definitely make sure your ci workers aren't doing full clones from git.o.o or github over and over | 20:20 |
| clarkb | I still think we should bump the memory available to those hosts | 20:20 |
| jgriffith | clarkb: yeah, it's always nova clones so you're exactly rightt on that | 20:21 |
| fungi | git is really great at reducing the workload for incremental updates of a repo | 20:21 |
| jgriffith | clarkb: fungi I'll switch up to the openstack mirror and look at changing my CI setup | 20:21 |
| jgriffith | clarkb: fungi thanks for the info | 20:21 |
| *** sharikapongubala has quit IRC | 20:21 | |
| jgriffith | clarkb: fungi I can't believe I've been so lucky all this time :) | 20:21 |
| *** nadya has quit IRC | 20:22 | |
| clarkb | fungi: jeblair et al what do you think about giving the git backends a larger flavor so that they have more memory to process nova operations? | 20:22 |
| clarkb | and/or upgrading to centos7 to get much newer git version which should be better at all the things | 20:22 |
| jgriffith | Whoa!!! I've been missing out, openstack servers seem to be a good bit faster for me as well :( | 20:23 |
| *** ybathia has quit IRC | 20:23 | |
| jgriffith | all the time wasted over the years!!!! | 20:23 |
| *** ybathia has joined #openstack-infra | 20:23 | |
| fungi | clarkb: upgrading sounds great. bigger flavors i'm less certain about (what benefit do you see over scaling the cluster wider rather than taller?) | 20:24 |
| jeblair | clarkb: is there a recent incident so we can check logs/stats? | 20:24 |
| clarkb | fungi: just that a single nova operation takes about 1GB of memory to run | 20:25 |
| clarkb | fungi: which means you can handle about 7 nova things on a single node currentl | 20:25 |
| fungi | jgriffith: if your workers have local clones and then just tell git to update those, you'll see _much_ faster performance | 20:25 |
| clarkb | jeblair: just the persistent git fails, there is an existing e-r bug and graph | 20:25 |
| *** serg_melikyan has quit IRC | 20:25 | |
| *** serg_melikyan has joined #openstack-infra | 20:26 | |
| clarkb | jeblair: looks like the last week or so has been quiet | 20:26 |
| clarkb | e-r says ~3 runs ran into problems | 20:26 |
| jgriffith | fungi: I'll take a look at that for sure. I've been just building everything on demand from scratch. But easy enough to snapshot something with the repose and just update them. | 20:26 |
| jgriffith | Thanks folks for the info | 20:26 |
| lifeless | is our pypi mirror having update issues? | 20:27 |
| lifeless | http://logs.openstack.org/06/213606/1/check/gate-tempest-dsvm-full/a2836e2/logs/devstacklog.txt.gz#_2015-08-17_10_30_00_351 | 20:27 |
| clarkb | lifeless: I can look, have an example job I can compare against? | 20:27 |
| clarkb | perfect | 20:27 |
| lifeless | is from a generate-constraints run that failed | 20:27 |
| *** ybathia has quit IRC | 20:27 | |
| lifeless | but it was hours after the actual update was proposed due to zuul stuff, so far outside the normal latency I'd expect | 20:27 |
| *** ybathia has joined #openstack-infra | 20:28 | |
| jeblair | clarkb: we made a lot of changes recently, and it's gotten a lot better. if it's now worse again, that's news to me. i just don't want us to solve the last round again. | 20:28 |
| lifeless | https://pypi.python.org/pypi/oauthlib shows 1.0.3 | 20:28 |
| clarkb | jeblair: judging by e-r numbers the incident rate is low and 2/3 were successful jos, likely timed out and retried | 20:28 |
| *** rkukura_ has joined #openstack-infra | 20:29 | |
| clarkb | lifeless: it is missing from http://pypi.dfw.openstack.org/simple/oauthlib/ so likely bandersnatch is failing | 20:29 |
| clarkb | let me grab my key and take a look | 20:29 |
| pleia2 | home, but recovering from many hours of travel over 2 days and will be at ops meetup tomorrow-wednesday | 20:31 |
| *** davideagnello has joined #openstack-infra | 20:31 | |
| davideagnello | hello, this patch needs one more core reviewer: https://review.openstack.org/#/c/201285/2 | 20:31 |
| clarkb | pleia2: welcome back | 20:31 |
| clarkb | pleia2: we got translate-dev upgraded yesterday so things are moving along on that front (though likely not as quickly as you had them going) | 20:32 |
| pleia2 | clarkb: exciting! | 20:32 |
| *** rkukura has quit IRC | 20:32 | |
| *** rkukura_ is now known as rkukura | 20:32 | |
| fungi | i don't see any references to oauthlib in any of the past few days bandersnatch logs on pypi.dfw, at least | 20:32 |
| pleia2 | clarkb: and thanks, I'll follow up with Daisy to make sure she has what she needs | 20:32 |
| fungi | which is odd since it was supposedly uploaded yesterdat | 20:32 |
| fungi | yesterday | 20:32 |
| *** sharikapongubala has joined #openstack-infra | 20:33 | |
| lifeless | I can download it from pypi, so the s3 mirror is intact | 20:33 |
| fungi | and the logs are current | 20:34 |
| lifeless | dstufft: could the data bandersnatch runs off of be corrupt ? | 20:34 |
| fungi | so unless there were no pypi serial updates related to the oauthlib-1.0.3.tar.gz upload, i have no explanation | 20:34 |
| *** doug-fish has quit IRC | 20:35 | |
| *** doug-fish has joined #openstack-infra | 20:35 | |
| *** mwagner_lap has quit IRC | 20:35 | |
| lifeless | this however has raised us another issue | 20:36 |
| lifeless | potential issue | 20:36 |
| lifeless | generate-constraints is evidently not using our mirrors | 20:36 |
| dstufft | what's up | 20:36 |
| *** mrunge has quit IRC | 20:36 | |
| dstufft | oh bandersnatch | 20:36 |
| dstufft | it's doing something weird | 20:36 |
| lifeless | dstufft: we have a bandersnatch mirror that is missing oautlib 1.0.3 | 20:36 |
| armax | clarkb: we have a fix in place for the instability caused by https://review.openstack.org/#/c/210649/ | 20:37 |
| clarkb | hrm has my new key not made it places yet? | 20:37 |
| dstufft | is it just missing that, or is it stuck trying to get django-genesis and some other things | 20:37 |
| clarkb | armax: cool | 20:37 |
| armax | clarkb: when you get a chance | 20:37 |
| armax | clarkb: could you look at https://review.openstack.org/#/c/199672/ | 20:37 |
| armax | clarkb: the neutron fix https://review.openstack.org/#/c/212145/ depends on it | 20:37 |
| *** mdenny has joined #openstack-infra | 20:38 | |
| armax | ihrachyshka: ^^^ | 20:38 |
| lifeless | dstufft: I haven't access to the logs; fungi though can look | 20:38 |
| clarkb | fungi: https://review.openstack.org/#/c/211332/ that'll help me not need to add a different key to my ssh agent >_> | 20:38 |
| clarkb | armax: sure | 20:38 |
| armax | clarkb: many thanks | 20:38 |
| clarkb | armax: lgtm, fungi, pleia2 and jeblair are in some state of around and may be able to review as well | 20:39 |
| armax | clarkb: ok, great…much appreciated | 20:40 |
| armax | clarkb: the Depends-on tag is the best invention after sliced bread | 20:40 |
| *** spredzy|afk is now known as spredzy | 20:41 | |
| clarkb | armax: you can thank jeblair for it and yes quite handy | 20:41 |
| *** Sukhdev has joined #openstack-infra | 20:41 | |
| armax | clarkb: yup, we could test the fix without disrupting the gate | 20:41 |
| armax | kudos to you all | 20:41 |
| lifeless | dstufft: this http://paste.openstack.org/show/419542/ is what my local bandersnatch has to say about oauthlib 1.0.3 | 20:42 |
| lifeless | dstufft: prior messages were 1.0.1 and it actually gets them... | 20:43 |
| clarkb | lifeless: is your local bandersnatch also out of date? | 20:43 |
| lifeless | clarkb: appears so | 20:43 |
| fungi | yeah, just checked all our pypi mirrors and bandersnatch doesn't mention that upload at all | 20:43 |
| lifeless | clarkb: I'm just starting to dig | 20:43 |
| lifeless | 2015-08-18 06:01:06,135 ERROR: Stale serial for package twx-botapi | 20:44 |
| lifeless | 2015-08-18 06:01:06,135 ERROR: Stale serial for twx-botapi (1678790) not updating. Giving up. | 20:44 |
| dstufft | is it error on django-genesis and shit | 20:44 |
| dstufft | yea | 20:44 |
| dstufft | that's my fault | 20:44 |
| lifeless | bunch of Stale serial messages | 20:44 |
| dstufft | I'm poking at it now | 20:44 |
| fungi | anyway, i need to head out for a bit. back in a while | 20:44 |
| dstufft | the database migration made all the bandersnatches get stuck because of some schema changes I rolled into it | 20:44 |
| lifeless | whups :) | 20:44 |
| *** amitgandhinz has quit IRC | 20:45 | |
| lifeless | yeah, local mirror missing oauthlib 1.0.3 | 20:45 |
| *** angdraug has quit IRC | 20:45 | |
| *** claudiub|2 has joined #openstack-infra | 20:45 | |
| openstackgerrit | Emma Barber proposed openstack-infra/puppet-hound: Configure and run apache on top of hound https://review.openstack.org/211640 | 20:45 |
| clarkb | hrm elasticsearch is still quite unhappy I will have to look into that | 20:47 |
| *** Shrews has quit IRC | 20:47 | |
| clarkb | dstufft: if we force it to start from scratch will that fix it? | 20:47 |
| dstufft | No | 20:47 |
| dstufft | not yet anyways | 20:47 |
| clarkb | ok | 20:47 |
| dstufft | that might be required | 20:47 |
| dstufft | but it won't fix it yet | 20:48 |
| dstufft | if you can hold off on doing that, I'd like to see if it fixes itself, or if it will require a refresh | 20:48 |
| *** claudiub has quit IRC | 20:48 | |
| dstufft | once I fix my end | 20:48 |
| clarkb | can do | 20:48 |
| openstackgerrit | Tim Buckley proposed openstack-infra/devstack-gate: Collect DStat CSV logs if they exist. https://review.openstack.org/212189 | 20:49 |
| anteaya | boarded and sitting, plane says hey catering how about some food, catering said no, really? plane said really | 20:49 |
| anteaya | catering said let me look | 20:49 |
| anteaya | wait | 20:49 |
| anteaya | yup, no food | 20:49 |
| anteaya | now we wait, now that catering confirmes no food | 20:50 |
| anteaya | sigh | 20:50 |
| clarkb | anteaya: you flying transpacific? | 20:50 |
| mmmpork | anteaya: next time just bring a bucket of fried chicken | 20:50 |
| clarkb | you probably want food in that case | 20:50 |
| *** shashank_hegde has joined #openstack-infra | 20:50 | |
| *** shardy has quit IRC | 20:50 | |
| anteaya | clarkb: to sfo | 20:51 |
| anteaya | mmmpork: um, yeah not me so much | 20:51 |
| anteaya | that is worse than no food | 20:52 |
| anteaya | clarkb: and yes food for transpacific, yes | 20:52 |
| *** nadya has joined #openstack-infra | 20:52 | |
| anteaya | I'm going with there is some regulation like no food/no fly | 20:52 |
| lifeless | dstufft: is there any assistance we can offer? | 20:52 |
| *** puranamr has quit IRC | 20:52 | |
| *** hyakuhei has joined #openstack-infra | 20:52 | |
| anteaya | similar to if the toilets have no water turn around and go back to hong hong | 20:53 |
| anteaya | hong kong | 20:53 |
| *** Shrews has joined #openstack-infra | 20:53 | |
| *** Shrews has quit IRC | 20:53 | |
| sc68cal | anteaya: heh. | 20:54 |
| *** matbu has quit IRC | 20:54 | |
| anteaya | now we are playing triva | 20:54 |
| anteaya | how much does the plane weigh? | 20:55 |
| anteaya | how much fuel do we have | 20:55 |
| dstufft | lifeless: sec | 20:55 |
| hyakuhei | Hey, has there been a recent change to the TLS configuration online? I’m getting cloudflare issues | 20:55 |
| ianw | dstufft: hey, wondering if you have thoughts on https://github.com/pypa/pip/issues/2940. it's about the path getting mixed up between /usr/lib and /usr/lib64 for .so's | 20:55 |
| hyakuhei | (NVM it’s fixed now) | 20:55 |
| anteaya | how much fuel would a 737 carry chicago to sfo | 20:56 |
| anteaya | fuel in pounds? | 20:56 |
| anteaya | with some help I might win a beverage | 20:56 |
| sc68cal | anteaya: have a question for you, is there a more elegant way to get OP irc privilege for #openstack-fwaas channel than having everyone leave it? | 20:56 |
| anteaya | if catering can figure itself out | 20:56 |
| *** nadya has quit IRC | 20:57 | |
| anteaya | sc68cal: not really | 20:57 |
| sc68cal | I asked in #freenode and they said to get in contact with the #openstack group | 20:57 |
| jeblair | clarkb, fungi, anteaya: i bought a house... so i might start dropping off occasionally, fungi-style, just as fungi is coming back online :) | 20:57 |
| anteaya | sc68cal: once the person who entered the channel first leaves the channel for the first time you really don't have any options | 20:57 |
| clarkb | jeblair: wow nice! | 20:57 |
| anteaya | jeblair: yay! | 20:57 |
| jeblair | sc68cal: yeah, i can help with that | 20:57 |
| clarkb | jeblair: are you moving far? | 20:57 |
| *** jeadr has quit IRC | 20:57 | |
| sc68cal | jeblair: you are my hero | 20:57 |
| openstackgerrit | Merged openstack-infra/system-config: Add new ssh key for clarkb https://review.openstack.org/211332 | 20:58 |
| anteaya | jeblair: congratulations | 20:58 |
| jeblair | clarkb: nope, about a 12 minute drive to just over the border in oakland | 20:58 |
| anteaya | 163,000 pounds the weight of the aircraft | 20:58 |
| *** jeadr has joined #openstack-infra | 20:58 | |
| anteaya | jeblair: do you have storage? | 20:58 |
| sc68cal | jeblair: basically just trying to get op so I can register it with chanserv - so https://review.openstack.org/#/c/212607/ works | 20:58 |
| jeblair | sc68cal: do you think you can ask everyone to leave? because that still is the easiest way :) | 20:59 |
| anteaya | jeblair: awesome magic powers, yay | 20:59 |
| *** serg_melikyan has quit IRC | 20:59 | |
| mmmpork | jeblair: congrats on house ownership! i hear that is no small feat for your area | 20:59 |
| *** Shrews has joined #openstack-infra | 20:59 | |
| *** sergmelikyan has joined #openstack-infra | 20:59 | |
| *** sergmelikyan has quit IRC | 20:59 | |
| jeblair | mmmpork: thanks, yeah, i'm a little surprised myself. :) | 21:00 |
| sc68cal | jeblair: ah, the client who was on a bouncer appears to have left, let's try it one more time | 21:00 |
| *** bswartz has quit IRC | 21:00 | |
| pleia2 | jeblair: ooh, congrats | 21:01 |
| *** yamahata has joined #openstack-infra | 21:01 | |
| jeblair | pleia2: thanks. i feel like i spent 10 years getting my dsl just right. now i get to do it again! :) | 21:03 |
| pleia2 | jeblair: hah | 21:03 |
| *** sthillma_ has joined #openstack-infra | 21:04 | |
| *** sergmelikyan has joined #openstack-infra | 21:04 | |
| *** sthillma has quit IRC | 21:06 | |
| *** sthillma_ is now known as sthillma | 21:06 | |
| clarkb | jeblair: hopefully you don't have to do the powerline stuff. That was the biggest drawback of my most recent move. When I Move again real networking choices are near top of requirements list | 21:06 |
| *** bexelbie_ has joined #openstack-infra | 21:06 | |
| *** pc_m_ has joined #openstack-infra | 21:08 | |
| clarkb | greghaynes: I am about to push a change that will allow nodepool to sudo during the nodepool dsvm job | 21:08 |
| clarkb | greghaynes: with taht we should be very close to working dib images in that job | 21:08 |
| jeblair | clarkb: i should be able to get a similar setup to what i have now: bonded dsl (two lines), delivering something in the 10-20mbit range, from sonic.net (extremely clueful local isp). | 21:09 |
| *** jtomasek has quit IRC | 21:09 | |
| *** amotoki has joined #openstack-infra | 21:09 | |
| greghaynes | clarkb: noice | 21:09 |
| jeblair | clarkb: but since they are a clec operating over att's lines, it's an adventure | 21:09 |
| *** yamamoto has joined #openstack-infra | 21:09 | |
| greghaynes | clarkb: powerline stuff? | 21:09 |
| greghaynes | clarkb: you mean for inside your home? | 21:09 |
| *** thorst has quit IRC | 21:10 | |
| *** bexelbie_ has quit IRC | 21:10 | |
| *** sergmelikyan has quit IRC | 21:10 | |
| greghaynes | jeblair: congrats! | 21:10 |
| clarkb | greghaynes: ya | 21:10 |
| *** pc_m has quit IRC | 21:10 | |
| jeblair | clarkb: today i learned that the house actually has a second drop from att -- they delivered one of the lines to the cpe box i knew about, the second line was delivered to an ancient connection i didn't know existed that was hidden under some new siding. | 21:10 |
| greghaynes | clarkb: heh, well if you own the house you can just lay the cat6 yourself | 21:10 |
| clarkb | greghaynes: and if you live in it | 21:11 |
| clarkb | greghaynes: because oregon | 21:11 |
| jeblair | greghaynes: i ordered a bunch of stuff from monoprice for just that :) | 21:11 |
| clarkb | otherwise ou need a licensed contracter even for low voltage work iirc | 21:12 |
| anteaya | jeblair: hidden treasure | 21:12 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Allow stack sudo in nodepool dsvm job https://review.openstack.org/213862 | 21:13 |
| clarkb | greghaynes: ^ that should do it | 21:13 |
| *** kgiusti has quit IRC | 21:13 | |
| *** zul has quit IRC | 21:14 | |
| *** angdraug has joined #openstack-infra | 21:15 | |
| anteaya | captain is keeping is well informed, still no food but he is going to track down someone eventually, I can hear it in his voice | 21:15 |
| *** mestery has joined #openstack-infra | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** alexpilotti has quit IRC | 21:16 | |
| clarkb | anteaya: if you are bored waiting for liftoff https://review.openstack.org/#/c/213862/ would help nodepool testing a bunch :) | 21:16 |
| *** alexpilotti has joined #openstack-infra | 21:16 | |
| anteaya | that and the neutron meeting should keep me entertained, thanks :) | 21:17 |
| *** julim has quit IRC | 21:17 | |
| *** bdemers has quit IRC | 21:17 | |
| *** doug-fish has quit IRC | 21:18 | |
| *** hyakuhei has quit IRC | 21:19 | |
| *** jeadr has quit IRC | 21:19 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add controller class for Infra Cloud https://review.openstack.org/209698 | 21:19 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: First stab at making things bifrosty https://review.openstack.org/208751 | 21:19 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add openstack-modules.env https://review.openstack.org/209617 | 21:19 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Create playbook to puppet InfraCloud nodes https://review.openstack.org/209632 | 21:20 |
| dstufft | ianw: Sorry, can't look right now | 21:20 |
| dstufft | lifeless: clarkb Ok, I think I have a fix figured out, testing it now | 21:20 |
| *** FallenPegasus has joined #openstack-infra | 21:20 | |
| dstufft | so give me a few moments more | 21:20 |
| clarkb | dstufft: sure, just let me know when I can do anything to help | 21:20 |
| *** jeadr has joined #openstack-infra | 21:20 | |
| anteaya | clarkb: where is the use of this variable defined? DEVSTACK_GATE_REMOVE_STACK_SUDO | 21:21 |
| anteaya | I could look myself or ask | 21:21 |
| clarkb | anteaya: in devstack-gate, the change adding it just merged | 21:21 |
| clarkb | anteaya: let me get a link | 21:21 |
| anteaya | ah usually git remote update before I grep | 21:21 |
| clarkb | anteaya: https://git.openstack.org/cgit/openstack-infra/devstack-gate/commit/?id=3ad06311fdf4c5e8d2cbcefe7c2ea6a6934cca00 there | 21:21 |
| anteaya | tried to save the phone date plan | 21:21 |
| clarkb | anteaya: np | 21:23 |
| *** dkranz has quit IRC | 21:23 | |
| *** irdkehn_ has quit IRC | 21:24 | |
| anteaya | thanks for your patience with me, +2 | 21:25 |
| dstufft | clarkb: lifeless Ok, try running a sync and see if you still get errors about django-gensis or flickr something, or the bot api thing | 21:25 |
| *** yamamoto has quit IRC | 21:26 | |
| *** irdkehn has joined #openstack-infra | 21:26 | |
| lifeless | 2015-08-18 09:26:20,125 ERROR: Stale serial for package twx-botapi | 21:26 |
| lifeless | 2015-08-18 09:26:20,125 ERROR: Stale serial for twx-botapi (1678790) not updating. Giving up. | 21:26 |
| dstufft | ok | 21:26 |
| dstufft | it'll require a refresh then | 21:26 |
| dstufft | delete the status/generation/todo | 21:26 |
| dstufft | and then run it | 21:27 |
| dstufft | and it should be all fixed afterwords | 21:27 |
| clarkb | dstufft: that will make it do a full refresh right? | 21:27 |
| dstufft | clarkb: it won't re-download files it already has, but it'll check it's state against PyPI and fix any errors | 21:28 |
| dstufft | you might be able to get away by just deleting todo | 21:28 |
| dstufft | to kill the inprogress sync | 21:28 |
| *** FallenPegasus has quit IRC | 21:29 | |
| *** aysyd has quit IRC | 21:29 | |
| clarkb | I already removed all 3 :) | 21:29 |
| dstufft | okay | 21:29 |
| dstufft | it'll take a little while, think 10 minutes or so | 21:29 |
| clarkb | ok, ifthis fixes it I will do the same on the other 3 mirrors | 21:29 |
| *** jeadr has quit IRC | 21:30 | |
| dstufft | I moved the normalization function into the database so we could query against it directly without having to store a second column and it didn't quite agree between the Python implementation in setuptools and the SQL implementation in the DB | 21:30 |
| *** sputnik1_ has quit IRC | 21:31 | |
| *** jeadr has joined #openstack-infra | 21:31 | |
| clarkb | subprocess.CalledProcessError: Command '['bandersnatch', 'mirror']' returned non-zero exit status 1 will see if that happens again in 5 minutes during the next run | 21:31 |
| *** FallenPegasus has joined #openstack-infra | 21:33 | |
| openstackgerrit | Morgan Fainberg proposed openstack/requirements: Add python-ldap-test to global requirements https://review.openstack.org/213866 | 21:33 |
| *** achanda has quit IRC | 21:35 | |
| clarkb | yup thats persistent, now to figure out why it is breaking | 21:35 |
| *** sergmelikyan has joined #openstack-infra | 21:35 | |
| dstufft | clarkb: fwiw it's not unusual for a full sync to take once or twice to actually complete (and if you're running it every 5 minutes unconditionally I think that the first process will take a lock and block out additional ones) | 21:36 |
| clarkb | dstufft: we have our own lock wrapper. Its failing immediately, looks properly broken for some reason | 21:37 |
| *** sergmelikyan has quit IRC | 21:37 | |
| *** FallenPegasus has quit IRC | 21:37 | |
| dstufft | clarkb: ok | 21:38 |
| *** unicell1 has joined #openstack-infra | 21:38 | |
| openstackgerrit | Merged openstack/requirements: Add networking-l2gw to the list https://review.openstack.org/203275 | 21:38 |
| clarkb | dstufft: I am running it all in the foreground now and should know more soon | 21:39 |
| *** unicell has quit IRC | 21:39 | |
| anteaya | I have to poewere doewn now | 21:39 |
| clarkb | anteaya: have a good flight | 21:39 |
| anteaya | clarkb: if you can be in mentruon meeting for https://wiki.openstack.org/wiki/Network/Meetings#On_Demand_Agenda | 21:39 |
| anteaya | the infra portion | 21:39 |
| clarkb | anteaya: looking | 21:39 |
| austin81 | clarkb: So I ran build-image.sh and have the images in /tmp. How do I convert to qcow2 so I can host a snapshot on my cloud provider? | 21:41 |
| clarkb | austin81: it should make a qcow2 by default | 21:42 |
| clarkb | austin81: that should go in your PWD | 21:42 |
| austin81 | clarkb: meaning the dir that I ran build-image.sh from? | 21:44 |
| clarkb | austin81: yup | 21:44 |
| clarkb | austin81: you should have a image-name.qcow2 there | 21:45 |
| dstufft | clarkb: lifeless gonna afk for a few to plug in some stuff for my wife, will be back in a bit | 21:45 |
| clarkb | dstufft: fwiw its still going in the foreground | 21:45 |
| clarkb | so I dunno what was going on earlier | 21:45 |
| austin81 | clarkb: Hmm not there... I thought it completed but I'll check the log again | 21:46 |
| dstufft | clarkb: ok! | 21:46 |
| lifeless | dstufft: ack | 21:46 |
| openstackgerrit | Merged openstack/requirements: Bump gabbi to version 1.1.4 https://review.openstack.org/201953 | 21:47 |
| lifeless | wow bandersnatch is busy writing | 21:47 |
| openstackgerrit | Merged openstack/requirements: Add networking-odl to tracked projects list https://review.openstack.org/213084 | 21:47 |
| openstackgerrit | Merged openstack/requirements: Bumg python-manilaclient version to 1.2.0 https://review.openstack.org/213043 | 21:47 |
| asselin_ | hi, our nodepool stopped working today. Anyone aware of recent merges that could cuase this? OpenStackCloudException: Error getting compute endpoint: User e77e6b2d846444bc97e7208a1ecf53cf is unauthorized for tenant admin (Disable debug mode to suppress these details.) (HTTP 401) http://paste.openstack.org/show/419603/ | 21:49 |
| clarkb | asselin_: make sure you have latest shade installed | 21:49 |
| *** davideagnello has quit IRC | 21:49 | |
| clarkb | asselin_: there was a recent backward incompatible change to shade that we had to patch nodepool for and requires shade to be latest version | 21:50 |
| clarkb | but it can affect cloud auth | 21:50 |
| *** sergmelikyan has joined #openstack-infra | 21:50 | |
| openstackgerrit | Merged openstack/requirements: Bump pecan to 1.0.0 https://review.openstack.org/213085 | 21:50 |
| asselin_ | clarkb, thanks...will check | 21:50 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add controller class for Infra Cloud https://review.openstack.org/209698 | 21:52 |
| *** davideagnello has joined #openstack-infra | 21:52 | |
| *** jeadr has quit IRC | 21:52 | |
| openstackgerrit | Merged openstack/requirements: Bump version of bandit to 0.13.2 https://review.openstack.org/210787 | 21:52 |
| openstackgerrit | Merged openstack/requirements: Allow new XStatic Libraries for alternate Horizon theme https://review.openstack.org/206668 | 21:52 |
| openstackgerrit | Merged openstack/requirements: Updated oslo.cache to 0.4.0 https://review.openstack.org/206509 | 21:52 |
| *** jeadr has joined #openstack-infra | 21:53 | |
| *** erlon has quit IRC | 21:53 | |
| openstackgerrit | Merged openstack/requirements: Update Alembic to 0.8.0 https://review.openstack.org/203960 | 21:54 |
| openstackgerrit | Armando Migliaccio proposed openstack/requirements: Add networking-l2gw to the list https://review.openstack.org/213833 | 21:54 |
| openstackgerrit | Merged openstack/requirements: Bump websockify to 0.6.1 https://review.openstack.org/211062 | 21:54 |
| *** ZZelle has joined #openstack-infra | 21:54 | |
| *** ZZelle is now known as ZZelle-off | 21:54 | |
| clarkb | fwiw I self approved the nodepool dsvm job change because it only affects a non voting job on an infra project and is trivial | 21:55 |
| asselin_ | clarkb, still fails with the same error with latest shade. | 21:56 |
| *** mwagner_lap has joined #openstack-infra | 21:57 | |
| clarkb | asselin_: did you restart nodepool? | 21:58 |
| asselin_ | clarkb, yes | 21:58 |
| clarkb | asselin_: I dunno then | 21:58 |
| clarkb | asselin_: its definitely an auth issue, I would double check the creds in use | 21:58 |
| sridhar_ram | infra-team: please help to review: https://review.openstack.org/#/c/211789/ | 21:59 |
| *** abdelwas has joined #openstack-infra | 21:59 | |
| fungi | jeblair: i can attest to how the whole buying and then moving into a house will make you strangely unavailable | 22:01 |
| clarkb | sridhar_ram: I have to say I have no idea what tacker is based on that description but I am probably not the target audience either | 22:01 |
| *** mestery has quit IRC | 22:01 | |
| asselin_ | clarkb, yeah...those are easy to verify. clear-text in nodepool.yaml and hasn't changed in a long time | 22:02 |
| clarkb | asselin_: is it possible the cloud side changed? | 22:02 |
| clarkb | asselin_: the error coming back is definitely "this thing does not have access" | 22:02 |
| *** rfolco has quit IRC | 22:02 | |
| clarkb | fungi: I have a foregrounded bandersnatch sync running on the dfw mirror right now | 22:03 |
| asselin_ | clarkb, clouds haven't changed either. akraminsky ran into this as well this morning while setting up a new ci system. | 22:03 |
| *** sergmelikyan has quit IRC | 22:03 | |
| clarkb | fungi: it failed on the cron but is going on my foreground run so I dunno what happened there but dstufft said we need to remove the generation, status, and todo files | 22:03 |
| *** camunoz has joined #openstack-infra | 22:03 | |
| *** mestery has joined #openstack-infra | 22:04 | |
| fungi | jeblair: ftr, we moved ~3mi up the island, 4mi door to door including turns/side streets. it's 5 weeks since our closing date and we're only just now wrapping up moving our junk | 22:04 |
| sridhar_ram | clarkb: tacker is a stackforge project building a NFV orchestrator .. we finally have some critical mass in our activity to make a release. Hence this patchset to enable create branch privilege | 22:04 |
| *** Swami has quit IRC | 22:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Allow stack sudo in nodepool dsvm job https://review.openstack.org/213862 | 22:04 |
| *** tlian has quit IRC | 22:04 | |
| *** smccully has joined #openstack-infra | 22:05 | |
| *** otter768 has joined #openstack-infra | 22:05 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Delete query for Bug #1477192 https://review.openstack.org/212180 | 22:06 |
| openstack | bug 1477192 in neutron "neutron test_multi_prefix_slaac failing in the gate with ping failures starting around 7/22" [High,Confirmed] https://launchpad.net/bugs/1477192 - Assigned to Henry Gessau (gessau) | 22:06 |
| clarkb | asselin_: does that user uuid match what you are using? | 22:07 |
| clarkb | asselin_: and for some reason its griping about tenant admin which seems odd as you don't need a privileged admin account to nodepool | 22:07 |
| asselin_ | clarkb, that's just what we've been using since the beginning | 22:08 |
| clarkb | asselin_: you have been using admin? | 22:08 |
| clarkb | asselin_: or that uuid matches your account? | 22:08 |
| asselin_ | clarkb, yes, it's a private cloud just for nodepool | 22:08 |
| *** mestery has quit IRC | 22:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix up tacker stackforge projects https://review.openstack.org/211789 | 22:09 |
| clarkb | my guess is something is broken in shade with admin accounts but we don't use an admin account anywhere. not even in our dsvm tests | 22:09 |
| *** otter768 has quit IRC | 22:09 | |
| jeblair | fungi: thanks for the reality check :) i realize we have more to move (or get rid of) than we think. | 22:09 |
| clarkb | asselin_: please check if that uuid string matches the account in question | 22:09 |
| *** ihrachyshka has quit IRC | 22:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Merge tacker ACL files https://review.openstack.org/212156 | 22:10 |
| clarkb | sridhar_ram: ^ | 22:10 |
| asselin_ | clarkb, sure. I will also try a different user. | 22:10 |
| *** sdake_ has joined #openstack-infra | 22:10 | |
| openstackgerrit | Yanis Guenane proposed openstack-infra/tripleo-ci: Adding more CPU in the HA scenario https://review.openstack.org/213885 | 22:10 |
| fungi | clarkb: lifeless: dstufft: fwiw it's my experience that a bandersnatch full update takes 3-6 hours to complete. so our cron wrapper with a timeout is going to not take kindly to that (but your foreground run should be fine, just slow) | 22:10 |
| clarkb | fungi: ok, with the cron it bailed out immediately twice | 22:10 |
| clarkb | fungi: its possible there is something else broken but will figure that out next | 22:11 |
| *** yamamoto has joined #openstack-infra | 22:11 | |
| clarkb | Neutron have asked us that changes to their configs get ACKed by mestery, armax, or dougwig | 22:12 |
| clarkb | they will act as infra liasons so that we can make sure their jobs and other config related business are happy before we end up merging things that explode them before a milestone | 22:12 |
| clarkb | I will draft up a short infra list email with this info too | 22:12 |
| jeblair | clarkb: ++; do you think we can land some comments in project-config in appropriate places? | 22:13 |
| clarkb | (I did mention that in the case of an emergency we do reserve the right to merge without that ACKing but that shouldn't be common occurence) | 22:13 |
| clarkb | jeblair: maybe? not quite sure where that would go | 22:13 |
| openstackgerrit | fumihiko kakuma proposed openstack/requirements: Add networking-ofagent to projects.txt https://review.openstack.org/210773 | 22:13 |
| *** sdake has quit IRC | 22:13 | |
| *** MarkAtwood has quit IRC | 22:13 | |
| jeblair | clarkb: me neither; maybe worth a look | 22:14 |
| fungi | clarkb: thanks for the heads up. i think we mostly get mestery involved on most job config changes, but i get that the recent stable branch file filter cleanup touched voting neutron jobs in not-so-great ways | 22:14 |
| *** amotoki has quit IRC | 22:14 | |
| sridhar_ram | clarkb: thanks ! | 22:14 |
| clarkb | fungi: and there was a database related d-g cleanup that was fine for neutron itself but not advanced services | 22:14 |
| *** pradk has quit IRC | 22:14 | |
| fungi | oh, stellar. i guess i missed that particular excitement | 22:14 |
| clarkb | asselin_: if a non admin account does work you should file a bug against shade and probably ping mordred/shrews about it | 22:15 |
| *** kushal has quit IRC | 22:16 | |
| *** achanda has joined #openstack-infra | 22:16 | |
| *** achanda has quit IRC | 22:16 | |
| asselin_ | clarkb, honestly I'm also looking on how to back out the change and debug this on akraminsky's system | 22:16 |
| *** achanda has joined #openstack-infra | 22:16 | |
| asselin_ | but I don't see a recent change to nodepool or shade that is related | 22:17 |
| asselin_ | clarkb, actually maybe this is the one: http://paste.openstack.org/show/419632/ | 22:18 |
| clarkb | asselin_: yes thats the one related to latest shade | 22:19 |
| clarkb | asselin_: so you need latest shade | 22:19 |
| clarkb | 0.12.0 iirc | 22:19 |
| *** alexpilotti has quit IRC | 22:19 | |
| *** sputnik13 has joined #openstack-infra | 22:19 | |
| asselin_ | clarkb, that version works in our main ci system | 22:20 |
| clarkb | asselin_: so its fixed? | 22:20 |
| Shrews | clarkb: that was the borked version of shade that required a nodepool patch. the fix must be in nodepool | 22:21 |
| asselin_ | our internal one is broken and is now on shade==0.12.1.dev1 | 22:21 |
| clarkb | Shrews: yes the fix is in nodepool | 22:21 |
| *** Swami has joined #openstack-infra | 22:21 | |
| clarkb | asselin_: can you check what version of nodepool you are running? pbr freeze | grep nodepool | 22:21 |
| *** dprince has quit IRC | 22:21 | |
| asselin_ | nodepool==0.1.2.dev66 # git sha 7978bd9 | 22:22 |
| *** ihrachyshka has joined #openstack-infra | 22:22 | |
| asselin_ | same version on both our working ci and broken ci | 22:22 |
| fungi | that's master branch tip | 22:23 |
| fungi | asselin_: when was nodepoold last restarted? | 22:23 |
| fungi | more than 2 days ago? | 22:23 |
| asselin_ | for the broken one, just now. the working one...not sure | 22:23 |
| *** marcusvrn has quit IRC | 22:23 | |
| asselin_ | I believe last week | 22:23 |
| clarkb | its possible the fix is buggy too | 22:24 |
| clarkb | it would be really helpful to confirm that uuid ooking string is valid | 22:24 |
| asselin_ | sure back to doing that | 22:24 |
| *** dims_ has joined #openstack-infra | 22:25 | |
| *** pc_m_ has quit IRC | 22:26 | |
| clarkb | our nodepool has been running since the 12th | 22:26 |
| clarkb | crinkle's change merged on the 11th | 22:27 |
| asselin_ | clarkb, http://paste.openstack.org/show/419633/ | 22:27 |
| clarkb | so I think ours is still working | 22:27 |
| asselin_ | clarkb, ours broke 2 hours ago | 22:27 |
| fungi | in a restart i guess? | 22:28 |
| *** dims has quit IRC | 22:28 | |
| asselin_ | fungi, no...it shouldn't have been restarted then | 22:28 |
| clarkb | fungi: ya not sure who did the restart but it seems to be running after the new version was installed | 22:28 |
| clarkb | oh you are talking to asselin | 22:28 |
| clarkb | asselin_: so it stopped working without a restart of the service? that would point at the cloud in question imo | 22:29 |
| fungi | clarkb: oh, on our end, the restart on the 12th was me i think | 22:29 |
| clarkb | fungi: cool and it went fine from what I can tell so not sure the code is bugg | 22:29 |
| *** gordc has quit IRC | 22:30 | |
| asselin_ | well puppet apply could have been run, and updated dependednt libraries....maybe a coworker restarted it, but he's not around now.... | 22:30 |
| *** annegentle has quit IRC | 22:30 | |
| fungi | yeah, i checked it all out when i restarted, but there have been changes merged since then | 22:30 |
| clarkb | fungi: good point could be an interaction with a newer change | 22:30 |
| fungi | so while i'm cool with restarting ours again, might be nice first to know a little more about what we might be in for | 22:31 |
| clarkb | fungi: agreed | 22:31 |
| fungi | and plan for the ensuing firefight | 22:31 |
| *** sergmelikyan has joined #openstack-infra | 22:31 | |
| asselin_ | fungi, +1, I'm avoiding touching my other working system until this is solved | 22:31 |
| clarkb | also we do test that it works against the devstack cloud buit it uses the non admin user there | 22:32 |
| clarkb | so next step is likely: use non admin user | 22:32 |
| fungi | we have increasing amounts of testing implemented for nodepool, so if this _is_ a problem in master it's something our tests aren't catching | 22:32 |
| fungi | er, yeah, what clarkb said ;) | 22:32 |
| asselin_ | clarkb, will try that now | 22:32 |
| *** zul has joined #openstack-infra | 22:33 | |
| *** esker has quit IRC | 22:33 | |
| fungi | we don't have admin access to "our" clouds, hence testing that non-admin accounts work. it's entirely possible this is something that breaks if you give nodepool access to an admin account i guess | 22:33 |
| *** chlong has quit IRC | 22:34 | |
| clarkb | fungi: yup wouldn't surprise me if something is funny in keystone auth when admining | 22:34 |
| asselin_ | btw these clouds are juno-based. I think akraminsky was checking kilo and master | 22:34 |
| clarkb | asselin_: our testing is against master which is again good for us since the public clouds we consume chase master fairly closely | 22:35 |
| fungi | also a good point. i think our nodepool integration test is against master branch tip and both of our current providers are close to that | 22:35 |
| fungi | er, yeah again clarkb beat me to that one | 22:35 |
| *** Nepoc has joined #openstack-infra | 22:35 | |
| openstackgerrit | Merged stackforge/python-jenkins: Fix timeout function and add capturing of messages on failure https://review.openstack.org/211776 | 22:36 |
| akraminsky | asselin_, yeh same issue on all 3. | 22:36 |
| *** yamamoto has quit IRC | 22:37 | |
| asselin_ | well not sure how to change users b/c the nodes are owned by admin. I can manually delete them / create a new one for non-admin user | 22:38 |
| *** claudiub|2 has quit IRC | 22:39 | |
| *** e0ne has quit IRC | 22:41 | |
| *** Nepoc has quit IRC | 22:41 | |
| *** Sukhdev has quit IRC | 22:41 | |
| *** baoli has joined #openstack-infra | 22:45 | |
| *** docaedo has quit IRC | 22:46 | |
| *** docaedo2 is now known as docaedo | 22:47 | |
| *** dims_ has quit IRC | 22:47 | |
| *** dims has joined #openstack-infra | 22:47 | |
| clarkb | asselin_: yes, manual cleanup is likely required if switching users like that | 22:48 |
| clarkb | asselin_: but you should be able to quickl confirm auth works against a non admin user | 22:48 |
| clarkb | then go back and cleanup if it helps | 22:48 |
| *** baoli has quit IRC | 22:50 | |
| *** moravec has quit IRC | 22:51 | |
| *** markvoelker has quit IRC | 22:51 | |
| *** dims has quit IRC | 22:51 | |
| notmyname | is there the concept of "sticky comments" in gerrit? like there is with a -2 vote, a comment that would stay with a diff through another patch set | 22:52 |
| clarkb | notmyname: no, but you can diff between different patchsets to see those comments | 22:53 |
| SpamapS | what ever happened to phabricator? | 22:54 |
| notmyname | yeah, but if anything has landed on master between those patch sets, the diff is a mess | 22:54 |
| clarkb | in the top left of the side by side diff display you can select the patch to diff against | 22:54 |
| clarkb | notmyname: yes | 22:54 |
| clarkb | SpamapS: its still a thing I think, it just hasn't received much attention lately | 22:54 |
| clarkb | notmyname: the technical problem with doing what you describe is where to place the comment in cases of resolved merge conflicts | 22:56 |
| *** dimtruck is now known as zz_dimtruck | 22:56 | |
| fungi | SpamapS: it's a priority effort. mordred and ttx were both putting in some work on it but they've been recently embroiled in travel/vacation | 22:56 |
| notmyname | sure, of course | 22:56 |
| clarkb | notmyname: I am not sure that this is a solveable problem in all cases but may be doable for the trivial cases | 22:56 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 22:56 |
| clarkb | notmyname: but that case is mostly already served by the inter patchset diffing | 22:57 |
| clarkb | notmyname: as you should only really be rebasing on master when you have conflicts (usually there are other reasons to do it) | 22:57 |
| asselin_ | clarkb, image-upload to non-admin seems to work | 22:59 |
| openstackgerrit | Sridhar Ramaswamy proposed openstack-infra/irc-meetings: Update title and chairs for tacker meeting https://review.openstack.org/213926 | 22:59 |
| jhesketh | Morning | 23:00 |
| asselin_ | clarkb, I take it back. | 23:00 |
| *** sergmelikyan has quit IRC | 23:01 | |
| *** sdake_ is now known as sdake | 23:01 | |
| *** austin81 has left #openstack-infra | 23:01 | |
| openstackgerrit | Merged openstack/requirements: Add openstacksdk to the global requirements https://review.openstack.org/208079 | 23:01 |
| *** zul has quit IRC | 23:04 | |
| fungi | i'm going to restart/clean up jenkins04, 05 and 07 | 23:05 |
| *** asettle has joined #openstack-infra | 23:05 | |
| *** dewsday has joined #openstack-infra | 23:06 | |
| pleia2 | just emailed greghaynes and crinkle, but fyi for others, I'm not making it down to the ops meetup tomorrow, I seem to have brought some microbial stowaways home with me (turns out I'm not just tired, I'm sick) | 23:07 |
| crinkle | pleia2: :( feel better! | 23:07 |
| fungi | pleia2: terrible to hear, but it sounds like we'll be fairly well represented, so don't fret, just get better | 23:08 |
| pleia2 | crinkle: thanks! disappointed to miss it :\ | 23:08 |
| pleia2 | fungi: nods | 23:08 |
| greghaynes | pleia2: :( hope you get better soon | 23:08 |
| pleia2 | greghaynes: thanks | 23:08 |
| *** salv-orl_ has quit IRC | 23:09 | |
| *** asettle has quit IRC | 23:10 | |
| *** gildub has joined #openstack-infra | 23:11 | |
| jeblair | pleia2: sorry to hear that; get better soon. | 23:11 |
| pleia2 | jeblair: thanks, hoping a day of rest will fix it up | 23:12 |
| *** salv-orlando has joined #openstack-infra | 23:12 | |
| gildub | Hi, is there an issue with Jenkins? I get this error ERROR: Publisher be.certipost.hudson.plugin.SCPRepositoryPublisher aborted due to exception | 23:13 |
| *** mestery has joined #openstack-infra | 23:13 | |
| *** dewsday is now known as asettle | 23:13 | |
| *** puranamr has joined #openstack-infra | 23:13 | |
| clarkb | gildub: can you link to the change that that occured on? | 23:13 |
| gildub | This is only a cosmetic change so I'm 100% sure it's from the infra: http://logs.openstack.org/98/213598/1/check/gate-puppet-keystone-puppet-beaker-rspec-upgrade-dsvm-trusty/3a237e7/console.html | 23:13 |
| gildub | clarkb, Hi | 23:14 |
| clarkb | gildub: IO error scanning directory '/home/jenkins/workspace/gate-puppet-keystone-puppet-beaker-rspec-dsvm-trusty/logs/etc/keystone' is the issue | 23:14 |
| clarkb | gildub: thats likely something wrong with the job itself | 23:15 |
| gildub | clarkb, repos for liberty were added recently (last Friday) for beaker jobs, might be related. | 23:15 |
| clarkb | I would check with EmilienM as EmilienM had been working on log collection | 23:16 |
| ianw | gildub: above that -> cp: target ‘/home/jenkins/workspace/gate-puppet-keystone-puppet-beaker-rspec-upgrade-dsvm-trusty/logs/apache_config’ is not a directory | 23:16 |
| gildub | clarkb, ok, thanks | 23:16 |
| gildub | ianw, You're my Lynx' eyes! | 23:17 |
| clarkb | greghaynes: https://jenkins03.openstack.org/job/gate-dsvm-nodepool-nv/6/console that should run with sudo access | 23:17 |
| gildub | ianw, yeah that looks weird | 23:17 |
| ianw | gildub: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/copy_puppet_logs.sh#n107 <- this is what is copying | 23:18 |
| *** Sukhdev has joined #openstack-infra | 23:19 | |
| ianw | gildub: and it's fixed in https://review.openstack.org/#/c/213688/ -- so basically do a recheck | 23:20 |
| *** smccully has quit IRC | 23:21 | |
| gildub | ianw, clarkb, ah yeah EmilienM worked on it, although he's supposed to be on PTO! | 23:21 |
| gildub | ianw, clarkb, thanks | 23:22 |
| *** sdake_ has joined #openstack-infra | 23:22 | |
| openstackgerrit | Merged openstack/requirements: Minimal setuptools required for tests https://review.openstack.org/199788 | 23:23 |
| *** near89 has joined #openstack-infra | 23:23 | |
| *** sdake has quit IRC | 23:26 | |
| *** asettle has quit IRC | 23:29 | |
| *** sergmelikyan has joined #openstack-infra | 23:31 | |
| *** sabeen2 has quit IRC | 23:32 | |
| openstackgerrit | Merged openstack/requirements: Drop PuLP from g-r https://review.openstack.org/205695 | 23:33 |
| openstackgerrit | Merged openstack/requirements: Add networking-bagpipe-l2 to projects.txt https://review.openstack.org/199578 | 23:33 |
| asselin_ | clarkb, non admin tenant also doesn't work http://paste.openstack.org/show/419706/ | 23:33 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Switched npm-run-test to npm-test https://review.openstack.org/212154 | 23:34 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm tarball upload job https://review.openstack.org/212153 | 23:34 |
| *** markvoelker has joined #openstack-infra | 23:35 | |
| SpamapS | hm.. | 23:35 |
| SpamapS | if you were a 110 line DNS update for infra-cloud.. where would you want to be git added? | 23:35 |
| clarkb | SpamapS nowhere we dont manage dns with git | 23:35 |
| *** salv-orlando has quit IRC | 23:36 | |
| SpamapS | clarkb: kk, so do I just pastebin it and ping infra-root? ;) | 23:37 |
| *** yamamoto has joined #openstack-infra | 23:37 | |
| clarkb | asselin_ I am not sure then, would check keystoneclient next probably | 23:37 |
| *** otter768 has joined #openstack-infra | 23:38 | |
| *** spzala has quit IRC | 23:38 | |
| clarkb | SpamapS probably. are there really that many hosts | 23:38 |
| asselin_ | clarkb, it's weird. I expected something to be logged on the provider side, but I don't see anything. | 23:39 |
| clarkb | do you lg at debug level? | 23:39 |
| jeblair | SpamapS, clarkb: i think there should be an inventory file in system-config at some point... maybe it can go in there? | 23:39 |
| crinkle | --> https://review.openstack.org/#/c/208751/17/modules/openstack_project/files/bifrost/baremetal.json ? | 23:40 |
| SpamapS | clarkb: there will be ~210. We're bringing up just 40 now, but I have the space reserved for more. | 23:40 |
| asselin_ | clarkb, yes | 23:41 |
| *** regXboi has quit IRC | 23:42 | |
| *** yamamoto has quit IRC | 23:42 | |
| gildub | Just wondering, jenkins tests used to run within 15 minutes, things seems really slow since last week-end. | 23:43 |
| SpamapS | greghaynes: I can't seem to find the rc file to source on baremetal01 .. ? | 23:43 |
| gildub | 788268 | 23:43 |
| gildub | ops | 23:43 |
| SpamapS | greghaynes: usually /opt/stack/ansible/hacking/env-setup .. ? | 23:43 |
| clarkb | gildub: it all depends on load on that system | 23:44 |
| jeblair | SpamapS, crinkle: yeah, if we need a file like that, can we just do that? it would be much better to put this stuff in-tree somewhere rather than treating it as transient. | 23:44 |
| SpamapS | crinkle: ooo thats good | 23:44 |
| clarkb | gildub: there is a fixed set of resources to perform a variable amount of work | 23:44 |
| SpamapS | jeblair: thats indeed exactly how I'd like it to work. | 23:44 |
| jeblair | SpamapS: cool, so we can probably write like 5 lines of python to translate that file into a rax dns commands | 23:45 |
| gildub | clarkb, I understand but we're talking about 4 times slower, sounds like a big load all of a sudden | 23:45 |
| jeblair | (which i see as an interim step until we can have something actually manage dns using an openstack api, but mumble mumble...) | 23:45 |
| openstackgerrit | Merged openstack-infra/project-config: Puppet-dashboard is read only now https://review.openstack.org/204685 | 23:46 |
| greghaynes | SpamapS: I think it is in bifrost env-setup in ubuntu user dir | 23:47 |
| SpamapS | jeblair: whats that? You're trailing off.. was that a niner in there? | 23:47 |
| openstackgerrit | Merged openstack-infra/project-config: Make mistralclient voting for python3 https://review.openstack.org/208124 | 23:48 |
| gildub | clarkb, has anyone else noticed a big load difference? | 23:48 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Document assigned network ranges in infra-cloud https://review.openstack.org/213933 | 23:48 |
| *** HeOS_ has quit IRC | 23:48 | |
| clarkb | gildub: no I haven't noticed things being terribly abnormal | 23:49 |
| *** HeOS_ has joined #openstack-infra | 23:49 | |
| clarkb | gildub: fungi did have to start cleaning up some unhappy jenkins masters but things are moving | 23:49 |
| clarkb | gildub: http://status.openstack.org/zuul/ the running jobs graph there should be between ~600-900 if we are operating normally | 23:50 |
| akraminsky | http://paste.openstack.org/show/419715/ | 23:50 |
| akraminsky | clarkb, this is the stack trace when running image-upload for both stable/kilo and master ^^ | 23:50 |
| *** armax has quit IRC | 23:51 | |
| gildub | clarkb, ok cool, I'll keep an eye on this one. | 23:51 |
| *** crc32 has quit IRC | 23:51 | |
| fungi | yeah, they're about idle now. i should be able to restart the three i've got quiescing shortly | 23:53 |
| openstackgerrit | Merged openstack-infra/project-config: adding python27 test to the storlets project https://review.openstack.org/213613 | 23:53 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm publish to eslint-config-openstack https://review.openstack.org/199725 | 23:54 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Switched npm-run-test to npm-test https://review.openstack.org/212154 | 23:54 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm publish job. https://review.openstack.org/199715 | 23:54 |
| openstackgerrit | Merged openstack-infra/system-config: Remove feed parameter from jeepyb::openstackwatch https://review.openstack.org/208757 | 23:54 |
| *** mestery has quit IRC | 23:55 | |
| *** bharath has quit IRC | 23:58 | |
| *** amotoki has joined #openstack-infra | 23:59 | |
| *** dingyichen has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
