| *** hichihara has joined #openstack-infra | 00:00 | |
| *** yamamoto has joined #openstack-infra | 00:00 | |
| fungi | you keep your shandy ingredients with the beer. great storage optimization | 00:00 |
|---|---|---|
| *** sputnik13 has joined #openstack-infra | 00:01 | |
| clarkb | stocking beer? you live in a city where this is not required | 00:01 |
| * fungi wears beer stockings | 00:02 | |
| clarkb | notmorgan: ex novo elliot is amazing if you like fruity hoppy ipaand in your neighborhood | 00:03 |
| fungi | oh, yeah, i had that one. quite tasty | 00:03 |
| fungi | that was a fun place too | 00:03 |
| fungi | also, notmorgan moved to p'land? | 00:04 |
| *** annegentle has joined #openstack-infra | 00:04 | |
| fungi | now i have additional reasons to visit | 00:04 |
| fungi | (read: additional ways to pretend i'm not just there for the beer) | 00:05 |
| *** sflanigan has joined #openstack-infra | 00:06 | |
| *** sflanigan has joined #openstack-infra | 00:06 | |
| *** sdague has quit IRC | 00:06 | |
| clarkb | rumor is yes but I havent confirmed it in person ye | 00:06 |
| fungi | drinking at ex novo^W^W^W^Wmaking sure he's settling in! | 00:07 |
| notmorgan | clarkb: oooooooh i so want to try it | 00:07 |
| notmorgan | fungi: yes i'm NE Portland | 00:07 |
| notmorgan | fungi: i escaped SoCal | 00:07 |
| fungi | too fun | 00:07 |
| fungi | i hear escape velocity is challenging | 00:07 |
| fungi | so much gravity there | 00:08 |
| *** annegentle has quit IRC | 00:08 | |
| *** baoli has quit IRC | 00:09 | |
| clarkb | thst is why they fire rockets on the other side of the country | 00:09 |
| fungi | but yeah, as clarkb said, no need to stock up. you really shouldn't be seen drinking "yesterday's beer" | 00:09 |
| *** yamamoto has quit IRC | 00:09 | |
| mordred | fungi: what about tomorrow's? | 00:10 |
| fungi | futurebeer is acceptable | 00:10 |
| fungi | or so i hear | 00:11 |
| fungi | (though it can give you worts) | 00:12 |
| * fungi lays off any further terrible brewing puns | 00:13 | |
| *** mriedem has joined #openstack-infra | 00:13 | |
| *** suhani has joined #openstack-infra | 00:13 | |
| pleia2 | hahah | 00:14 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Remove ordereddict support from py26 https://review.openstack.org/240438 | 00:14 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix default result for reverse trigger https://review.openstack.org/241802 | 00:15 |
| *** suhani has quit IRC | 00:16 | |
| *** Abhijith has joined #openstack-infra | 00:16 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul: Add support for layout configuration split https://review.openstack.org/152290 | 00:17 |
| fungi | for some reason we have a dozen devstack-centos7 nodes sitting ready in internap-nyj01 for over a day | 00:18 |
| mgagne | fungi: anything I can do? | 00:19 |
| *** Suhani has joined #openstack-infra | 00:19 | |
| fungi | mgagne: nah, i think one of our jenkins masters has gone off the deep end | 00:20 |
| fungi | mgagne: mainly just curious they accumulated in one spot | 00:20 |
| fungi | we also have rather a lot of bare-trusty nodes booting and deleting in rax-iad | 00:20 |
| mgagne | we like to hoard VMs =) | 00:20 |
| fungi | heh | 00:20 |
| fungi | i'm going to get jenkins02 cleaned up and restarted. it's got a ton of nodes stuck in building and deleting from nodepool's perspective, and is not responding on its webui | 00:21 |
| fungi | oh, it finally loaded in a browser. i may have a shot at gracefully stopping it | 00:22 |
| fungi | okay, jenkins02 is in prepare for shutdown. i'm going to delete any remaining ready nodes for it in nodepool while it wraps up the 4 jobs it's actually managing to run | 00:24 |
| openstackgerrit | Merged openstack/requirements: Update constraint for os-client-config https://review.openstack.org/281852 | 00:24 |
| *** Qiming has quit IRC | 00:25 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add doxygen builder support https://review.openstack.org/131330 | 00:25 |
| *** tzn has joined #openstack-infra | 00:28 | |
| *** salv-orlando has quit IRC | 00:32 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements https://review.openstack.org/246114 | 00:32 |
| *** jamielennox|away is now known as jamielennox | 00:33 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for git repo not cloning 1542849 https://review.openstack.org/281130 | 00:35 |
| *** sputnik13 has quit IRC | 00:36 | |
| *** yamamoto has joined #openstack-infra | 00:38 | |
| *** Sukhdev has joined #openstack-infra | 00:38 | |
| *** chenli has joined #openstack-infra | 00:39 | |
| *** tzn has quit IRC | 00:39 | |
| *** gildub has quit IRC | 00:40 | |
| *** gildub has joined #openstack-infra | 00:40 | |
| *** roxanaghe has quit IRC | 00:40 | |
| *** xarses has quit IRC | 00:41 | |
| notmorgan | fungi, mordred: future beer is only ok as long as you know it'll be mainstream after being future beer, so you drank it before it was cool [ugh sorry for the pun]... | 00:44 |
| fungi | no worse than mine | 00:44 |
| notmorgan | if it was just icky beer to begin and became ickier beer... | 00:44 |
| notmorgan | then you're doing it wrong | 00:44 |
| * Clint squints. | 00:45 | |
| *** zz_dimtruck is now known as dimtruck | 00:46 | |
| *** thorst has joined #openstack-infra | 00:47 | |
| *** yamahata has quit IRC | 00:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Template out Centos & Fedora jobs https://review.openstack.org/282117 | 00:48 |
| *** yamamoto has quit IRC | 00:49 | |
| *** thorst has quit IRC | 00:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Common template for Centos & Fedora devstack jobs https://review.openstack.org/282117 | 00:53 |
| *** hrubi has quit IRC | 00:56 | |
| *** hrubi has joined #openstack-infra | 01:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Retry twine uploads in a loop https://review.openstack.org/282060 | 01:06 |
| *** xarses has joined #openstack-infra | 01:06 | |
| *** thorst has joined #openstack-infra | 01:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix centos support for bindep-fallback.txt https://review.openstack.org/282122 | 01:15 |
| *** ashleighfarnham has joined #openstack-infra | 01:16 | |
| *** ashleighfarnham has quit IRC | 01:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix centos support for bindep-fallback.txt https://review.openstack.org/282122 | 01:21 |
| *** jroll has quit IRC | 01:22 | |
| *** Qiming has joined #openstack-infra | 01:23 | |
| *** jroll has joined #openstack-infra | 01:24 | |
| *** Thelo has quit IRC | 01:26 | |
| *** Thelo has joined #openstack-infra | 01:29 | |
| fungi | grrr, still two jobs running on jenkins02. will check back again in a bit | 01:31 |
| *** _sarob has quit IRC | 01:31 | |
| fungi | at least those seem to have been the glut of ready nodes and will probably also account for the waste band of deletes on the graph once i can take it offline | 01:32 |
| *** Swami has quit IRC | 01:32 | |
| *** tiswanso has quit IRC | 01:33 | |
| *** MarkAtwood has quit IRC | 01:33 | |
| *** sdake_ has joined #openstack-infra | 01:34 | |
| *** FallenPegasus has quit IRC | 01:36 | |
| *** thorst has quit IRC | 01:36 | |
| *** sdake has quit IRC | 01:36 | |
| *** thorst has joined #openstack-infra | 01:36 | |
| *** sdake has joined #openstack-infra | 01:38 | |
| *** sridhar_ram has quit IRC | 01:39 | |
| *** sdake_ has quit IRC | 01:41 | |
| *** Liuqing has joined #openstack-infra | 01:44 | |
| *** zhurong has joined #openstack-infra | 01:44 | |
| *** thorst has quit IRC | 01:45 | |
| openstackgerrit | Merged openstack/diskimage-builder: Don't remove python3 & grubby in 99-remove-extra-packages https://review.openstack.org/249541 | 01:46 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Fix typo in Neutron grafana dashboards https://review.openstack.org/282129 | 01:46 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add gate-grenade-dsvm-neutron to tempest failure rate dashboard https://review.openstack.org/279326 | 01:51 |
| *** shashank_hegde has quit IRC | 01:54 | |
| pabelanger | fungi: looks like 282122 is passing on centos7 now for bindep | 01:55 |
| pabelanger | \o/ | 01:56 |
| *** chlong_ has joined #openstack-infra | 01:56 | |
| pabelanger | okay, logging for tonight | 01:56 |
| *** baoli has joined #openstack-infra | 01:57 | |
| *** asettle has joined #openstack-infra | 01:59 | |
| *** yonglihe has joined #openstack-infra | 02:01 | |
| *** shashank_hegde has joined #openstack-infra | 02:02 | |
| *** jamielennox is now known as jamielennox|away | 02:02 | |
| *** gongysh has joined #openstack-infra | 02:02 | |
| *** shashank_hegde has quit IRC | 02:04 | |
| *** david-lyle_ has joined #openstack-infra | 02:04 | |
| *** baoli has quit IRC | 02:04 | |
| *** david-lyle_ is now known as david-lyle | 02:05 | |
| *** erlon has quit IRC | 02:06 | |
| *** thorst has joined #openstack-infra | 02:07 | |
| *** asettle has quit IRC | 02:07 | |
| *** Sukhdev has quit IRC | 02:07 | |
| *** thorst_ has joined #openstack-infra | 02:08 | |
| *** annegentle has joined #openstack-infra | 02:10 | |
| *** beisner has quit IRC | 02:10 | |
| *** annegentle has quit IRC | 02:10 | |
| *** beisner has joined #openstack-infra | 02:11 | |
| *** sdake_ has joined #openstack-infra | 02:11 | |
| *** thorst has quit IRC | 02:11 | |
| *** sdake has quit IRC | 02:12 | |
| *** jamielennox|away is now known as jamielennox | 02:12 | |
| *** Abhijith has quit IRC | 02:15 | |
| *** dimtruck is now known as zz_dimtruck | 02:17 | |
| *** tiswanso has joined #openstack-infra | 02:18 | |
| *** sflanigan is now known as sflanigan_afk | 02:19 | |
| *** annegentle has joined #openstack-infra | 02:23 | |
| *** jamielennox is now known as jamielennox|away | 02:24 | |
| *** apoorvad has quit IRC | 02:25 | |
| *** jamielennox|away is now known as jamielennox | 02:25 | |
| *** jamielennox is now known as jamielennox|away | 02:26 | |
| *** Qiming has quit IRC | 02:28 | |
| *** Qiming has joined #openstack-infra | 02:28 | |
| *** dims has joined #openstack-infra | 02:28 | |
| *** dims_ has quit IRC | 02:29 | |
| openstackgerrit | Merged openstack/requirements: Update to os-brick 1.0.0 https://review.openstack.org/281523 | 02:29 |
| *** shashank_hegde has joined #openstack-infra | 02:30 | |
| *** bhunter_ has joined #openstack-infra | 02:30 | |
| *** annegentle has quit IRC | 02:33 | |
| *** jamielennox|away is now known as jamielennox | 02:33 | |
| *** markvoelker_ has quit IRC | 02:34 | |
| *** jamielennox is now known as jamielennox|away | 02:34 | |
| *** jamielennox|away is now known as jamielennox | 02:35 | |
| *** jamielennox is now known as jamielennox|away | 02:36 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for git repo not cloning 1542849 https://review.openstack.org/281130 | 02:36 |
| *** Suhani has quit IRC | 02:37 | |
| *** apoorvad has joined #openstack-infra | 02:37 | |
| *** sflanigan_afk is now known as sflanigan | 02:37 | |
| *** sdake_ is now known as sdake | 02:37 | |
| *** eil397 has quit IRC | 02:44 | |
| *** baoli has joined #openstack-infra | 02:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Common template for Centos & Fedora devstack jobs https://review.openstack.org/282117 | 02:45 |
| *** bpokorny has quit IRC | 02:46 | |
| *** harlowja_at_home has joined #openstack-infra | 02:46 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Rework functional test runner https://review.openstack.org/276563 | 02:50 |
| *** annegentle has joined #openstack-infra | 02:50 | |
| *** jamielennox|away is now known as jamielennox | 02:51 | |
| *** arxcruz has quit IRC | 02:52 | |
| *** zz_dimtruck is now known as dimtruck | 02:53 | |
| *** wolverin_ has quit IRC | 02:57 | |
| *** rossella_s has quit IRC | 02:57 | |
| *** rossella_s has joined #openstack-infra | 02:58 | |
| *** apoorvad has quit IRC | 02:58 | |
| *** shashank_hegde has quit IRC | 03:00 | |
| *** _sarob has joined #openstack-infra | 03:03 | |
| tonyb | jhesketh: Your e-r change landed and look at this: http://status.openstack.org/elastic-recheck/#1542849 | 03:03 |
| tonyb | jhesketh: all rax nodes | 03:04 |
| fungi | tonyb: also all bare-trusty nodes? because rax is the only place we boot those | 03:06 |
| fungi | as clarkb had to remind me earlier | 03:06 |
| tonyb | fungi: checking | 03:06 |
| *** asettle has joined #openstack-infra | 03:06 | |
| tonyb | fungi: yeah | 03:07 |
| tonyb | fungi: based scrolling through the first 100 hits | 03:07 |
| *** _sarob has quit IRC | 03:08 | |
| *** salv-orlando has joined #openstack-infra | 03:09 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Use working Kafka version https://review.openstack.org/282147 | 03:10 |
| *** Jeffrey4l__ has joined #openstack-infra | 03:11 | |
| ianw | fungi: can the proposal script run on not a proposal node? the devstack plugin doco update thing needs to walk all the git trees to find plugins ... it seems like it should work on a trusty node | 03:11 |
| *** asettle has quit IRC | 03:11 | |
| *** salv-orlando has quit IRC | 03:11 | |
| *** _sarob has joined #openstack-infra | 03:12 | |
| fungi | ianw: i think Clint was considering having it check the gerrit api for a list of repos and then just try http to the raw copy of that file served from cgit for each one. don't really need full clones | 03:12 |
| fungi | as for running on not proposal.slave.o.o, that machine is static specifically because we puppet gerrit account credentials onto it | 03:12 |
| ianw | fungi: oh, yeah ... and it can't run arbitrary stuff to get at them | 03:13 |
| Clint | fungi: that's https://review.openstack.org/#/c/282035/ | 03:13 |
| ianw | ok, yeah, there's a change out for the network side of things | 03:13 |
| *** _sarob has quit IRC | 03:15 | |
| *** Qiming has quit IRC | 03:16 | |
| *** tzn has joined #openstack-infra | 03:17 | |
| *** yamamoto_ has joined #openstack-infra | 03:17 | |
| *** rguillebert has quit IRC | 03:17 | |
| *** _sarob has joined #openstack-infra | 03:17 | |
| Clint | ianw: yup | 03:17 |
| *** tphummel has quit IRC | 03:18 | |
| *** krotscheck_dr is now known as krotscheck_afk | 03:18 | |
| jhesketh | tonyb: yeah I knew they were all rax nodes... but in multiple regions. I missed they were all bare-trusty | 03:19 |
| *** sdake has quit IRC | 03:21 | |
| tonyb | jhesketh: I hadn't noticed the all rax thing. I stopped looking after the 3rd or 4th recheck 'cause lazy | 03:21 |
| *** rhallisey has quit IRC | 03:22 | |
| *** tzn has quit IRC | 03:22 | |
| fungi | tonyb: jhesketh: behavior started up around the same time as the rax network maintenance on tuesday, but could be coincidence | 03:22 |
| fungi | current theory though is that network instability is causing git operations to hang. we don't see it in devstack-gate because we have timeouts and retries wrapped around git operations | 03:23 |
| fungi | jeblair: has https://review.openstack.org/282099 in the works to add --timeout and --retry options to zuul-cloner | 03:23 |
| jhesketh | fungi: right, that would make sense | 03:23 |
| *** annegentle has quit IRC | 03:27 | |
| *** esker has joined #openstack-infra | 03:27 | |
| *** links has joined #openstack-infra | 03:30 | |
| *** asettle has joined #openstack-infra | 03:30 | |
| *** dims has quit IRC | 03:32 | |
| *** asettle has quit IRC | 03:34 | |
| *** annegentle has joined #openstack-infra | 03:34 | |
| *** markvoelker has joined #openstack-infra | 03:34 | |
| *** _sarob has quit IRC | 03:34 | |
| *** dims has joined #openstack-infra | 03:34 | |
| *** dims has quit IRC | 03:37 | |
| *** markvoelker has quit IRC | 03:39 | |
| *** harlowja_at_home has quit IRC | 03:41 | |
| *** david-lyle_ has joined #openstack-infra | 03:42 | |
| *** harlowja_at_home has joined #openstack-infra | 03:42 | |
| *** david-lyle has quit IRC | 03:45 | |
| *** amotoki has quit IRC | 03:47 | |
| *** amotoki has joined #openstack-infra | 03:48 | |
| *** dingyichen has quit IRC | 03:49 | |
| *** dingyichen has joined #openstack-infra | 03:49 | |
| *** _sarob has joined #openstack-infra | 03:49 | |
| *** Qiming has joined #openstack-infra | 03:50 | |
| openstackgerrit | Bertrand Roussel proposed openstack-infra/jenkins-job-builder: Description Setter Plugin support as builder https://review.openstack.org/280416 | 03:52 |
| *** david-lyle has joined #openstack-infra | 04:00 | |
| *** annegentle has quit IRC | 04:02 | |
| *** david-lyle_ has quit IRC | 04:03 | |
| *** david_lyle__ has joined #openstack-infra | 04:03 | |
| *** harlowja_at_home has quit IRC | 04:04 | |
| *** david-lyle has quit IRC | 04:04 | |
| *** gyee has quit IRC | 04:07 | |
| *** david_lyle__ has quit IRC | 04:08 | |
| *** ganso has quit IRC | 04:09 | |
| *** yuanying_ has quit IRC | 04:10 | |
| *** _sarob has quit IRC | 04:14 | |
| *** annegentle has joined #openstack-infra | 04:15 | |
| *** shashank_hegde has joined #openstack-infra | 04:17 | |
| *** baoli has quit IRC | 04:18 | |
| *** mriedem has quit IRC | 04:18 | |
| *** wolverineav has joined #openstack-infra | 04:19 | |
| openstackgerrit | Bertrand Roussel proposed openstack-infra/jenkins-job-builder: Description Setter Plugin support as builder https://review.openstack.org/280416 | 04:21 |
| *** annegentle has quit IRC | 04:22 | |
| *** wolverineav has quit IRC | 04:23 | |
| *** annegentle has joined #openstack-infra | 04:27 | |
| *** amotoki has quit IRC | 04:28 | |
| *** amotoki has joined #openstack-infra | 04:29 | |
| *** asselin__ has quit IRC | 04:30 | |
| *** asselin_ has joined #openstack-infra | 04:30 | |
| *** dimtruck is now known as zz_dimtruck | 04:31 | |
| *** annegentle has quit IRC | 04:32 | |
| *** amotoki has quit IRC | 04:35 | |
| *** thorst_ has quit IRC | 04:37 | |
| *** thorst has joined #openstack-infra | 04:38 | |
| *** tphummel has joined #openstack-infra | 04:39 | |
| prometheanfire | for dib, does anyone know of an element that sets a root password? (use case is non-openstack VM images) | 04:40 |
| *** wolverineav has joined #openstack-infra | 04:45 | |
| *** thorst has quit IRC | 04:46 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:46 | |
| *** amotoki has joined #openstack-infra | 04:47 | |
| *** Liuqing has quit IRC | 04:47 | |
| *** vivekd has joined #openstack-infra | 04:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Common template for Centos & Fedora devstack jobs https://review.openstack.org/282117 | 04:50 |
| *** yuanying has joined #openstack-infra | 04:50 | |
| *** salv-orlando has joined #openstack-infra | 04:51 | |
| *** BobBall has joined #openstack-infra | 04:52 | |
| *** BobBall is now known as Guest17806 | 04:53 | |
| *** yamahata has joined #openstack-infra | 04:55 | |
| greghaynes | prometheanfire: one doesn't exist AFAICT - http://docs.openstack.org/developer/diskimage-builder/elements/devuser/README.html is the closest to it | 04:55 |
| *** Liuqing has joined #openstack-infra | 04:55 | |
| greghaynes | prometheanfire: if you modified that to special case if the username is root that would probably work well | 04:55 |
| *** amotoki has quit IRC | 04:56 | |
| prometheanfire | I'll test that | 04:56 |
| prometheanfire | DIB_DEV_USER_USERNAME=root | 04:56 |
| prometheanfire | and password, wonder what it'd do if the user already exists | 04:57 |
| prometheanfire | honestly I'd probably keep it as is | 04:57 |
| greghaynes | prometheanfire: yea, exactly - it might fail in that case but that'd be a good thing to fix | 04:57 |
| prometheanfire | just change the user to gentoo or something | 04:57 |
| prometheanfire | that cinder-volume nodes need DB access makes me sad | 04:59 |
| prometheanfire | wonder/hope that changes in mitaka | 05:00 |
| asselin_ | dstufft, full resync complete. One left: http://paste.openstack.org/show/487523/ | 05:00 |
| *** kzaitsev_mb has quit IRC | 05:00 | |
| asselin_ | i'll be back later | 05:01 |
| *** amotoki has joined #openstack-infra | 05:02 | |
| *** salv-orlando has quit IRC | 05:07 | |
| *** tzn has joined #openstack-infra | 05:07 | |
| *** jswarren_ has joined #openstack-infra | 05:10 | |
| *** tzn has quit IRC | 05:12 | |
| *** jswarren has quit IRC | 05:13 | |
| *** jhesketh has quit IRC | 05:14 | |
| *** jhesketh has joined #openstack-infra | 05:16 | |
| *** bhunter_ has quit IRC | 05:18 | |
| *** kushal has joined #openstack-infra | 05:22 | |
| *** zhurong has quit IRC | 05:22 | |
| *** _sarob has joined #openstack-infra | 05:22 | |
| *** salv-orlando has joined #openstack-infra | 05:22 | |
| *** zhurong has joined #openstack-infra | 05:25 | |
| *** Guest17806 has quit IRC | 05:26 | |
| *** binbincong has joined #openstack-infra | 05:26 | |
| *** shashank_hegde has quit IRC | 05:27 | |
| *** salv-orlando has quit IRC | 05:28 | |
| *** vivekd has quit IRC | 05:28 | |
| *** mtanino has quit IRC | 05:29 | |
| *** asettle has joined #openstack-infra | 05:31 | |
| *** amotoki has quit IRC | 05:31 | |
| *** _sarob has quit IRC | 05:32 | |
| *** _sarob has joined #openstack-infra | 05:32 | |
| *** zz_dimtruck is now known as dimtruck | 05:33 | |
| *** asettle has quit IRC | 05:35 | |
| *** markvoelker has joined #openstack-infra | 05:36 | |
| *** rcernin has joined #openstack-infra | 05:37 | |
| *** infra-red has joined #openstack-infra | 05:38 | |
| *** amotoki has joined #openstack-infra | 05:39 | |
| *** markvoelker has quit IRC | 05:40 | |
| *** _sarob has quit IRC | 05:41 | |
| *** _sarob has joined #openstack-infra | 05:41 | |
| *** yuanying has quit IRC | 05:43 | |
| *** thorst has joined #openstack-infra | 05:44 | |
| *** _nadya_ has quit IRC | 05:46 | |
| *** thorst has quit IRC | 05:50 | |
| prometheanfire | can confirm the devuser stuff works at least :D | 05:52 |
| prometheanfire | greghaynes: thanks for the pointer | 05:52 |
| greghaynes | prometheanfire: glad to hear | 05:52 |
| prometheanfire | greghaynes: and done https://mthode.org/posts/2016/Feb/creating-gentoo-vm-images/ | 05:53 |
| greghaynes | prometheanfire: nice! | 05:53 |
| prometheanfire | will have to update a couple of things to get dib working on gentoo hosts as well | 05:53 |
| prometheanfire | but they are relitively minor | 05:54 |
| prometheanfire | greghaynes: where should I put that gentoo hosts require a package to be installed for the 'package-installs' element to work? | 05:55 |
| prometheanfire | end users will specifically need dev-python/pyyaml installed | 05:55 |
| prometheanfire | also, think https://bpaste.net/show/19353f7d41e7 is all (other then that pyyaml thing) | 05:56 |
| greghaynes | prometheanfire: http://docs.openstack.org/developer/diskimage-builder/user_guide/installation.html | 05:56 |
| prometheanfire | k | 05:56 |
| prometheanfire | I assume qemu is doc'd already | 05:56 |
| greghaynes | Yes, it mentions the rh/ubuntu packages for that | 05:56 |
| prometheanfire | guess I'll add gentoo there too | 05:57 |
| greghaynes | sounds good | 05:57 |
| *** _sarob has quit IRC | 05:57 | |
| *** _sarob has joined #openstack-infra | 05:57 | |
| prometheanfire | line 24 of that file makes me think I should add it to the package-installs README | 05:58 |
| greghaynes | prometheanfire: there too would be good | 05:59 |
| prometheanfire | so both? | 05:59 |
| greghaynes | yes | 05:59 |
| * prometheanfire shrugs | 05:59 | |
| prometheanfire | k | 05:59 |
| greghaynes | I think the user guide is the more likely place to be seen, but its also good to document in the README | 05:59 |
| prometheanfire | ya | 05:59 |
| prometheanfire | I'm sure ianw will correct us if needed :P | 06:00 |
| greghaynes | thats what reviewers are for :) | 06:00 |
| prometheanfire | yarp | 06:00 |
| *** _sarob has quit IRC | 06:01 | |
| *** shashank_hegde has joined #openstack-infra | 06:01 | |
| *** _sarob has joined #openstack-infra | 06:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/282185 | 06:01 |
| *** _sarob has quit IRC | 06:02 | |
| *** exploreshaifali has joined #openstack-infra | 06:02 | |
| *** vivekd has joined #openstack-infra | 06:02 | |
| gus | after being burnt by this yet again, would there be any downside to just basing LIBS_FROM_GIT on ` echo $ZUUL_CHANGES | awk 'BEGIN{RS="^";FS=":"} {print $1}' ` | 06:02 |
| gus | ? | 06:02 |
| gus | ie: every project mentioned in ZUUL_CHANGES is in LIBS_FROM_GIT ? | 06:03 |
| * gus is cranky about having to "pre-declare" which set of oslo libraries he might want to depend-on. | 06:03 | |
| *** asettle has joined #openstack-infra | 06:06 | |
| *** tphummel has quit IRC | 06:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Keystone-only uwsgi job https://review.openstack.org/264991 | 06:09 |
| stevemar | gerrit is a bit sloowww | 06:10 |
| *** maishsk has joined #openstack-infra | 06:10 | |
| stevemar | another java memory leak? ^_^ | 06:10 |
| *** shashank_hegde has quit IRC | 06:10 | |
| *** asettle has quit IRC | 06:11 | |
| prometheanfire | stevemar: ya, git-review is stalling on me | 06:12 |
| prometheanfire | heh, there's the timeout | 06:13 |
| prometheanfire | fatal: The remote end hung up upon initial contact | 06:13 |
| *** resker has joined #openstack-infra | 06:13 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: fix building on gentoo hosts https://review.openstack.org/282193 | 06:14 |
| prometheanfire | there we go :P | 06:15 |
| *** esker has quit IRC | 06:15 | |
| stevemar | prometheanfire: yeah, it's so painful i may go to sleep early :) | 06:16 |
| stevemar | this problem is out of my jurisdiction | 06:17 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Add Gentoo to the list of supported distributions https://review.openstack.org/281960 | 06:19 |
| prometheanfire | stevemar: seems better now? | 06:19 |
| stevemar | prometheanfire: naw :) | 06:19 |
| prometheanfire | well, worksforme :P | 06:19 |
| prometheanfire | that was me doing two commits | 06:20 |
| stevemar | prometheanfire: oh wait... | 06:20 |
| prometheanfire | hopefully they pass though | 06:20 |
| stevemar | sorta? | 06:20 |
| stevemar | getting signs of life | 06:20 |
| *** Sukhdev has joined #openstack-infra | 06:21 | |
| stevemar | prometheanfire: i'm calling it a night, earliest night i've had all week! | 06:21 |
| prometheanfire | greghaynes: you probably want to wait on gate to pass (not that triple-o is reliable) but done, again :P | 06:21 |
| stevemar | good luck with gerrit | 06:21 |
| tonyb | gus: yes but I'm not goign to tell you why you can't do that ;P | 06:21 |
| prometheanfire | stevemar: nn | 06:21 |
| *** Liuqing has quit IRC | 06:22 | |
| prometheanfire | need to figure out how to add gentoo to the list of image builds tested, the only problem with that is the time it takes, maybe a nightly? | 06:22 |
| tonyb | gus: ... because not everything in ZUUL_CHANGES is a LIB (thing requirements, nova, cinder etc) | 06:22 |
| gus | yes, but what's the downside with having a non-lib mentioned in LIBS_FROM_GIT? | 06:23 |
| tonyb | gus: I don't know if that'd have a bad outcome other than to be slightly confusing. | 06:23 |
| *** maishsk has quit IRC | 06:24 | |
| *** Liuqing has joined #openstack-infra | 06:24 | |
| gus | oh, I see check_libs_from_git is "clever" and verifies that everything mentioned there is in fact pip installed with a version including "git" | 06:25 |
| tonyb | gus: :/ | 06:25 |
| gus | well, I can certainly improve it by deleting check_libs_from_git. | 06:26 |
| gus | What I'll _actually_ do is quit for the weekend, because seriously testing against a bugfix in oslo.concurrency shouldn't require an approval round-trip through project-config :/ | 06:28 |
| *** sam_wan has joined #openstack-infra | 06:28 | |
| tonyb | gus: fair call. | 06:29 |
| *** asettle has joined #openstack-infra | 06:29 | |
| *** infra-red has quit IRC | 06:29 | |
| *** resker has quit IRC | 06:30 | |
| *** asettle has quit IRC | 06:34 | |
| *** salv-orlando has joined #openstack-infra | 06:34 | |
| *** flip214_ is now known as flip214 | 06:35 | |
| *** flip214 has joined #openstack-infra | 06:36 | |
| *** salv-orlando has quit IRC | 06:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/281661 | 06:39 |
| *** aeng has quit IRC | 06:41 | |
| *** jaosorior has joined #openstack-infra | 06:48 | |
| *** thorst has joined #openstack-infra | 06:49 | |
| *** shashank_hegde has joined #openstack-infra | 06:51 | |
| *** esker has joined #openstack-infra | 06:53 | |
| *** vgridnev has joined #openstack-infra | 06:54 | |
| *** dimtruck is now known as zz_dimtruck | 06:54 | |
| *** vgridnev has quit IRC | 06:55 | |
| *** thorst has quit IRC | 06:56 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:56 | |
| *** tzn has joined #openstack-infra | 06:56 | |
| *** woodster_ has quit IRC | 06:56 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/282185 | 06:56 |
| *** zhurong has quit IRC | 06:57 | |
| *** zhurong has joined #openstack-infra | 06:57 | |
| *** vgridnev has joined #openstack-infra | 06:57 | |
| *** rossella_s has quit IRC | 06:57 | |
| *** rossella_s has joined #openstack-infra | 06:58 | |
| *** gildub has quit IRC | 06:58 | |
| *** armax has quit IRC | 06:59 | |
| *** abhishekk has joined #openstack-infra | 07:00 | |
| *** tzn has quit IRC | 07:01 | |
| abhishekk | hi, websockify 0.8.0 is released (ttps://github.com/kanaka/websockify/releases) but it is not available on pypi (https://pypi.python.org/pypi?%3Aaction=search&term=websockify&submit=search), how can I proceed | 07:02 |
| *** rlandy has quit IRC | 07:02 | |
| *** jtomasek has joined #openstack-infra | 07:02 | |
| abhishekk | in requirement.txt websockify version is marked as websockify>=0.6.1 but since pypi has 0.7.0 version its using that only and not 0.8.0 | 07:03 |
| *** shashank_hegde has quit IRC | 07:04 | |
| *** shashank_hegde has joined #openstack-infra | 07:08 | |
| *** _nadya_ has joined #openstack-infra | 07:09 | |
| *** vgridnev has quit IRC | 07:09 | |
| *** shashank_hegde has quit IRC | 07:13 | |
| *** shashank_hegde has joined #openstack-infra | 07:14 | |
| *** camunoz has quit IRC | 07:14 | |
| *** infra-red has joined #openstack-infra | 07:16 | |
| AJaeger | abhishekk: we use constraints for the devstack jobs, see http://docs.openstack.org/developer/requirements/ for details | 07:25 |
| AJaeger | abhishekk: and then check today's proposal to updtae it: https://review.openstack.org/#/c/281661/4/upper-constraints.txt | 07:25 |
| AJaeger | so, once that passed our tests and gets approved, your jobs will use 0.8.0 | 07:25 |
| *** |-paul-| has joined #openstack-infra | 07:26 | |
| *** yuanying has joined #openstack-infra | 07:28 | |
| *** ianw has quit IRC | 07:30 | |
| *** esikachev_afk is now known as esikachev | 07:30 | |
| *** ianw has joined #openstack-infra | 07:31 | |
| *** esker has quit IRC | 07:31 | |
| *** markvoelker has joined #openstack-infra | 07:36 | |
| *** exploreshaifali has quit IRC | 07:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul: Add timeout/retries to zuul cloner https://review.openstack.org/282099 | 07:38 |
| *** HeOS has quit IRC | 07:39 | |
| *** markvoelker has quit IRC | 07:40 | |
| *** chlong_ has quit IRC | 07:41 | |
| * esikachev is now away: Away from keyboard | 07:43 | |
| *** esikachev is now known as esikachev_afk | 07:43 | |
| *** pcaruana has joined #openstack-infra | 07:45 | |
| AJaeger | yolanda: https://review.openstack.org/#/c/281563/ is an important ACL update, could you check it later, please? | 07:45 |
| *** _nadya_ has quit IRC | 07:45 | |
| yolanda | hi AJaeger | 07:46 |
| yolanda | god morning | 07:46 |
| *** kzaitsev_mb has quit IRC | 07:47 | |
| yolanda | AJaeger, why are we doing that acl per acl instead of adding that to some common acl and inherit from it? | 07:48 |
| *** |-paul-| has quit IRC | 07:48 | |
| *** e0ne has joined #openstack-infra | 07:49 | |
| *** esker has joined #openstack-infra | 07:50 | |
| AJaeger | yolanda: read my corresponding docs update as well: https://review.openstack.org/281712 | 07:51 |
| AJaeger | So, what we do is: Take away the - let's use abandon here as example - abandon right from the normal core team and give it to stable team. | 07:52 |
| AJaeger | Since we have inheritiance in refs/heads/stable from refs/heads, we need a way to remove the abandor from stable team. | 07:52 |
| AJaeger | to do this, we use exclusivepermissions (or whatever it's called) which resets the permission, and then add the stable team. | 07:53 |
| *** scheuran has joined #openstack-infra | 07:53 | |
| *** zhurong has quit IRC | 07:53 | |
| *** Liuqing has quit IRC | 07:53 | |
| AJaeger | But we inhereited from the default aCL also that Change Owner and Project bootstrapper (=gerrit admins - so, you, fungi etc) can abandon. | 07:53 |
| AJaeger | Let's add that back in... | 07:53 |
| *** thorst has joined #openstack-infra | 07:54 | |
| AJaeger | Currently if you submit a change to one of these touched projects on a stable branch, you - even if you're owner - cannot abandon, or set WIP. | 07:54 |
| AJaeger | fungi run into this - and others including myself in the past as well - and he finally wanted to fix it. | 07:54 |
| yolanda | AJaeger , i see, but why not add these extra settings to a "openstack-stable-rules" or whatever, and make all these acls that are updated from it? | 07:55 |
| AJaeger | yolanda: In my understanding that's not possible - but I'm happy to be proven wrong, so if you have an idea on how to do it, please bring it up and best discuss with fungi | 07:56 |
| AJaeger | yolanda: if you're not sure, let's leave it and let others review... | 07:57 |
| AJaeger | yolanda: restarting gerrit might be needed, there're lots of timeouts, see also scrollback ;( | 07:57 |
| yolanda | AJaeger, there is the inheritFrom clause that lets you reuse an specific acl | 07:57 |
| nibalizer | good morning yolanda AJaeger | 07:59 |
| yolanda | hi nibalizer | 07:59 |
| AJaeger | morning, nibalizer | 08:00 |
| *** thorst has quit IRC | 08:01 | |
| AJaeger | yolanda: we're not using that in project-config - I suggest you leave a question as review comment and follow up with fungi on that | 08:02 |
| nibalizer | how are you guys? | 08:02 |
| AJaeger | nibalizer: fine, discussing gerrit ACLs ;) | 08:02 |
| nibalizer | ogod i should not have pressed 'open all' on 281563 | 08:02 |
| nibalizer | firefox is displeased | 08:02 |
| yolanda | AJaeger yep, i'll leave the comment in the review. We use that clause in downstream, so we can reuse same settings in different acls | 08:02 |
| yolanda | AJaeger, and as the entries are common for all the projects, i think it can be worth trying it | 08:03 |
| AJaeger | yep, making that easier would be nice... | 08:03 |
| *** boris-42 has joined #openstack-infra | 08:04 | |
| AJaeger | nibalizer: it's the same change everywhere, reviewing one file is enough | 08:04 |
| AJaeger | enough to understand what's going on - the other files need review that it's done the same way | 08:05 |
| yolanda | nibalizer, i reviewed several of your changes, and i will need your help to review mine :) | 08:05 |
| yolanda | i started the same steps as you, to add infra cloud east to puppet, but there are some things that needs to be solved for nodepool | 08:07 |
| nibalizer | yolanda: sure | 08:08 |
| nibalizer | AJaeger: yolanda so would you like workflow +1 on 281563? I might go to bed so you'd be on the hook if it breaks later today | 08:08 |
| nibalizer | looks fine to me though | 08:09 |
| nibalizer | AJaeger: your docs patch made it much easier to understand why thats happening | 08:09 |
| nibalizer | so thanks for that | 08:09 |
| yolanda | nibalizer, i have that concern of the sahred acl | 08:09 |
| yolanda | because we are adding same entries in all projects, why not create a common acl and inherit from it? | 08:10 |
| openstackgerrit | Merged openstack-infra/infra-manual: Document gerrit ACL for stable team https://review.openstack.org/281712 | 08:10 |
| nibalizer | well its because the stable maintenace is done differently by different teams I think | 08:10 |
| *** shardy has joined #openstack-infra | 08:10 | |
| nibalizer | and because exclusivegrouppermissions is in play, inhertance doesn't work the way we'd like it to | 08:11 |
| *** HeOS has joined #openstack-infra | 08:11 | |
| AJaeger | yes, it's different teams. | 08:11 |
| nibalizer | also at the end of the day this change only adds powers to the admins, mostly | 08:11 |
| nibalizer | so the users shouldn't even feel it | 08:11 |
| AJaeger | the only think that can be reused is the ChangeOwner and admin | 08:11 |
| yolanda | Ajaeger, i see that all the entries are common, for refs/heads/stable/* ... the change you propose here, adds the same entries to same team in all projects | 08:12 |
| yolanda | so i'd like at least to throw the idea of reusing the acl, because it will be clear and easier to maintain. There may be good reasons for not doing that, but i'd like to be sure | 08:13 |
| AJaeger | yolanda: they're not the same, compare aodh and barbican, one uses aodh-stable-maint, the other barbican-stable-maint | 08:15 |
| *** sflanigan has quit IRC | 08:15 | |
| yolanda | yes i mean, the new entries. Are all under refs/heads/stable/* , and talking about change owner, project bootstrappers | 08:15 |
| yolanda | not the full acl but the new bits | 08:15 |
| AJaeger | yolanda: if something can be reused, I'm all for it - I just don't understand how exactly. Could you write up a change perhaps? | 08:15 |
| AJaeger | yolanda: will inheritfrom and exclusive work together? | 08:16 |
| yolanda | that's what i need to double check as Spencer mentioned that this may not work as expected | 08:16 |
| AJaeger | yolanda: we need exclusive here to remove the access from the normal team | 08:16 |
| AJaeger | yolanda: please double check and if it works, send an example patch and let's discsus with fungi. What you say sounds easier than what fungi does, so I'd like to see the implementation ;) | 08:17 |
| yolanda | ok going to send a dummy patch. Not sure if i can easily test as i don't have a gerrit sandbox ready now | 08:18 |
| AJaeger | thanks, yolanda | 08:18 |
| yolanda | looking at documentation i don't see it talks about any conflict with exclusive group perms, but it can happen | 08:18 |
| *** |-paul-| has joined #openstack-infra | 08:20 | |
| *** salv-orlando has joined #openstack-infra | 08:22 | |
| openstackgerrit | xiangxinyong proposed openstack-infra/project-config: Add dashboard and client for Smaug https://review.openstack.org/281019 | 08:22 |
| yolanda | mm, gerrit restart is needed for sure | 08:24 |
| *** kozhukalov has joined #openstack-infra | 08:24 | |
| *** arxcruz has joined #openstack-infra | 08:24 | |
| yolanda | weird, i'm not seeing jetty errors on gerrit this time.. but it has a heavy load | 08:25 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Alternative change for https://review.openstack.org/#/c/281563 https://review.openstack.org/282223 | 08:27 |
| yolanda | AJaeger, i'm thinking in something like that | 08:27 |
| yolanda | ^ | 08:27 |
| *** kushal has quit IRC | 08:28 | |
| *** asettle has joined #openstack-infra | 08:30 | |
| *** Sukhdev has quit IRC | 08:31 | |
| *** esker has quit IRC | 08:31 | |
| *** salv-orlando has quit IRC | 08:31 | |
| *** jlanoux has joined #openstack-infra | 08:32 | |
| yolanda | #status notice Gerrit is going to be restarted due to performance problems | 08:32 |
| openstackstatus | yolanda: sending notice | 08:32 |
| -openstackstatus- NOTICE: Gerrit is going to be restarted due to performance problems | 08:33 | |
| *** jlanoux has quit IRC | 08:34 | |
| *** asettle has quit IRC | 08:35 | |
| openstackstatus | yolanda: finished sending notice | 08:35 |
| *** ildikov has quit IRC | 08:35 | |
| *** jlanoux has joined #openstack-infra | 08:35 | |
| AJaeger | yolanda: if that works, it would be great! | 08:35 |
| AJaeger | yolanda: let's see what fungi thinks about 28223... | 08:36 |
| yolanda | i need to test it somehow, but i don't have my gerrit env ready, i had one but gerrit is not booting :( | 08:37 |
| *** piet has joined #openstack-infra | 08:37 | |
| *** exploreshaifali has joined #openstack-infra | 08:38 | |
| *** Hunner has quit IRC | 08:40 | |
| *** bmjen has quit IRC | 08:40 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Add note about unit testing with mox and python 3 compat https://review.openstack.org/257972 | 08:40 |
| *** esker has joined #openstack-infra | 08:41 | |
| *** maishsk has joined #openstack-infra | 08:43 | |
| *** dizquierdo has joined #openstack-infra | 08:45 | |
| *** piet has quit IRC | 08:45 | |
| *** kushal has joined #openstack-infra | 08:45 | |
| *** e0ne_ has joined #openstack-infra | 08:46 | |
| *** tiswanso_ has joined #openstack-infra | 08:46 | |
| *** matrohon has joined #openstack-infra | 08:47 | |
| *** dtantsur|afk is now known as dtantsur | 08:47 | |
| *** piet has joined #openstack-infra | 08:47 | |
| *** maishsk_ has joined #openstack-infra | 08:48 | |
| *** e0ne has quit IRC | 08:48 | |
| *** mikelk has joined #openstack-infra | 08:48 | |
| *** tiswanso has quit IRC | 08:48 | |
| *** e0ne_ has quit IRC | 08:48 | |
| *** ifarkas has joined #openstack-infra | 08:49 | |
| *** piet has quit IRC | 08:49 | |
| *** yuanying has quit IRC | 08:49 | |
| *** maishsk has quit IRC | 08:50 | |
| *** maishsk_ is now known as maishsk | 08:50 | |
| *** piet has joined #openstack-infra | 08:55 | |
| *** bauzas is now known as bauwser | 08:56 | |
| *** xiexs___ has joined #openstack-infra | 08:56 | |
| kozhukalov | guys, please review this https://review.openstack.org/#/c/279074/ | 08:57 |
| *** thorst has joined #openstack-infra | 08:58 | |
| *** amotoki has quit IRC | 09:00 | |
| *** hashar has joined #openstack-infra | 09:02 | |
| *** zeih has joined #openstack-infra | 09:03 | |
| *** salv-orlando has joined #openstack-infra | 09:03 | |
| *** amotoki has joined #openstack-infra | 09:04 | |
| *** zeih has quit IRC | 09:05 | |
| *** zeih has joined #openstack-infra | 09:05 | |
| xiexs___ | is there a issue with gate-nova-pep8 and python27 CI? | 09:06 |
| *** thorst has quit IRC | 09:06 | |
| *** wolverineav has quit IRC | 09:06 | |
| *** achanda has quit IRC | 09:07 | |
| *** piet has quit IRC | 09:07 | |
| *** piet has joined #openstack-infra | 09:08 | |
| AJaeger | xiexs___: we have some timeouts right now - did you run into one? | 09:08 |
| xiexs___ | AJaeger: sounds some patches are failed with them, but no error logs delivered. | 09:10 |
| *** gildub has joined #openstack-infra | 09:12 | |
| *** achanda has joined #openstack-infra | 09:12 | |
| *** piet has quit IRC | 09:13 | |
| *** shashank_hegde has quit IRC | 09:14 | |
| AJaeger | xiexs___: no error logs? Yeah, that's a known issue that the team hasbeen working on yesterday and will continue today. | 09:14 |
| AJaeger | xiexs___: please recheck | 09:14 |
| xiexs___ | AJaeger:OK,thanks.. | 09:15 |
| *** piet has joined #openstack-infra | 09:15 | |
| *** piet has quit IRC | 09:16 | |
| yolanda | AJaeger, i need to step out for a pair of hours, i hope everything works well ... | 09:18 |
| *** derekh has joined #openstack-infra | 09:18 | |
| *** maishsk has quit IRC | 09:19 | |
| *** jordanP has joined #openstack-infra | 09:20 | |
| *** eantyshev has left #openstack-infra | 09:20 | |
| abhishekk | AJaeger:Thank you for info | 09:21 |
| *** piet has joined #openstack-infra | 09:21 | |
| *** xiangxinyong has joined #openstack-infra | 09:22 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/project-config: Add docs-on-rtfd hook for fuel-noop-fixtures project https://review.openstack.org/281266 | 09:24 |
| *** _nadya_ has joined #openstack-infra | 09:25 | |
| *** hichihara has quit IRC | 09:26 | |
| *** tinyhippo has quit IRC | 09:29 | |
| nibalizer | AJaeger: got a question for ya | 09:30 |
| *** maishsk has joined #openstack-infra | 09:30 | |
| *** achanda has quit IRC | 09:30 | |
| *** BobBall has joined #openstack-infra | 09:30 | |
| *** d0ugal has quit IRC | 09:30 | |
| nibalizer | in jenkins/jobs/projects.yaml why do some of the -specs repos have the tarballs-site variable set? | 09:30 |
| *** esker has quit IRC | 09:30 | |
| nibalizer | and some do not? | 09:31 |
| *** d0ugal has joined #openstack-infra | 09:31 | |
| *** d0ugal has quit IRC | 09:31 | |
| *** d0ugal has joined #openstack-infra | 09:31 | |
| *** d0ugal has quit IRC | 09:31 | |
| openstackgerrit | Merged openstack-infra/project-config: Disable puppet tests for fuel-library 4.x branches https://review.openstack.org/280286 | 09:31 |
| *** d0ugal has joined #openstack-infra | 09:32 | |
| *** d0ugal has quit IRC | 09:32 | |
| *** d0ugal has joined #openstack-infra | 09:32 | |
| *** esker has joined #openstack-infra | 09:34 | |
| *** chenli has quit IRC | 09:34 | |
| *** vgridnev has joined #openstack-infra | 09:34 | |
| *** piet has quit IRC | 09:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:36 | |
| *** dizquierdo has quit IRC | 09:37 | |
| *** markvoelker has joined #openstack-infra | 09:37 | |
| *** piet has joined #openstack-infra | 09:38 | |
| *** zeih has quit IRC | 09:39 | |
| *** Hal has joined #openstack-infra | 09:41 | |
| *** Hal is now known as Guest58505 | 09:41 | |
| *** zeih_ has joined #openstack-infra | 09:42 | |
| *** markvoelker has quit IRC | 09:42 | |
| *** kzaitsev_mb has quit IRC | 09:44 | |
| *** zeih_ has quit IRC | 09:46 | |
| bauwser | AJaeger: yolanda: I can see you discussing on a jenkins problem, is this related to http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%3A%20Publisher%20be.certipost.hudson.plugin.SCPRepositoryPublisher%20aborted%20due%20to%20exception%5C%22 ? | 09:46 |
| *** esikachev_afk is now known as esikachev | 09:49 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:49 | |
| *** maishsk has quit IRC | 09:49 | |
| *** tzn has joined #openstack-infra | 09:50 | |
| *** tzn has quit IRC | 09:51 | |
| *** tzn has joined #openstack-infra | 09:51 | |
| *** ihrachys has joined #openstack-infra | 09:54 | |
| *** kzaitsev_mb has quit IRC | 09:55 | |
| *** piet has quit IRC | 09:56 | |
| *** dtantsur is now known as dtantsur|brb | 09:56 | |
| AJaeger | nibalizer: setting it might be cargo-cult - unless they include python-jobs as well | 09:58 |
| AJaeger | nibalizer: feel free to send a patch to clean it up... | 09:58 |
| AJaeger | bauwser: yes, that's the same issue | 09:59 |
| AJaeger | fungi, jeblair, and clarkb have been working on that yesterday but did not finish... | 10:00 |
| *** asettle has joined #openstack-infra | 10:01 | |
| bauwser | AJaeger: I see, do we have a bug report to follow ? | 10:01 |
| *** ildikov has joined #openstack-infra | 10:01 | |
| *** piet has joined #openstack-infra | 10:02 | |
| bauwser | AJaeger: I can still see some errors, so indeed I can see it's not finished :( | 10:02 |
| AJaeger | bauwser: I'm not sure - didn't read the full backlog. Feel free to check it (lsat 20 hours) | 10:03 |
| *** salv-orl_ has joined #openstack-infra | 10:03 | |
| *** thorst has joined #openstack-infra | 10:04 | |
| *** Qiming has quit IRC | 10:05 | |
| bauwser | AJaeger: meh, it's a bit unclear when reading the backlog | 10:05 |
| *** salv-orlando has quit IRC | 10:06 | |
| bauwser | AJaeger: but we had jetty issues this morning on jenkins04 | 10:06 |
| bauwser | actually, they're all impacted as per logstash | 10:07 |
| *** wolverineav has joined #openstack-infra | 10:07 | |
| AJaeger | bauwser: If you want to help, get https://review.openstack.org/282099 into shape. | 10:07 |
| *** piet has quit IRC | 10:07 | |
| bauwser | AJaeger: good to know, pulling it | 10:08 |
| *** piet has joined #openstack-infra | 10:08 | |
| *** zeih_ has joined #openstack-infra | 10:08 | |
| *** zeih_ has quit IRC | 10:08 | |
| *** zeih_ has joined #openstack-infra | 10:09 | |
| *** wolverineav has quit IRC | 10:11 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Recreate workspace in zuul-cloner cleanup https://review.openstack.org/282034 | 10:11 |
| *** thorst has quit IRC | 10:12 | |
| *** Liuqing has joined #openstack-infra | 10:12 | |
| openstackgerrit | Timur Sufiev proposed openstack-infra/project-config: Specify DEVSTACK_GATE_SETTINGS for Horizon i9n tests job https://review.openstack.org/269578 | 10:12 |
| AJaeger | nibalizer, yolanda: let's get 283034 in, please ^ | 10:12 |
| *** Liuqing has quit IRC | 10:14 | |
| AJaeger | typo, it should be 282034 | 10:14 |
| AJaeger | I'll be back later... | 10:15 |
| *** ganso has joined #openstack-infra | 10:19 | |
| *** ihrachys has quit IRC | 10:20 | |
| *** ihrachys has joined #openstack-infra | 10:20 | |
| *** rguillebert has joined #openstack-infra | 10:21 | |
| *** dtardivel has joined #openstack-infra | 10:21 | |
| *** exploreshaifali has quit IRC | 10:24 | |
| *** yuanying has joined #openstack-infra | 10:25 | |
| *** piet has quit IRC | 10:27 | |
| *** piet has joined #openstack-infra | 10:27 | |
| *** sdague has joined #openstack-infra | 10:28 | |
| *** tpsilva has joined #openstack-infra | 10:31 | |
| *** yamamoto_ has quit IRC | 10:32 | |
| *** shardy has quit IRC | 10:34 | |
| openstackgerrit | Sylvain Bauza proposed openstack-infra/zuul: Add timeout/retries to zuul cloner https://review.openstack.org/282099 | 10:35 |
| bauwser | meh, we still have timeouts, it's confirmed | 10:37 |
| *** dizquierdo has joined #openstack-infra | 10:38 | |
| *** haypo has joined #openstack-infra | 10:39 | |
| *** asettle has quit IRC | 10:40 | |
| *** asettle has joined #openstack-infra | 10:40 | |
| *** ildikov has quit IRC | 10:43 | |
| *** gildub has quit IRC | 10:44 | |
| yolanda | AJaeger, back | 10:45 |
| yolanda | reading backlog | 10:45 |
| *** ociuhandu has quit IRC | 10:47 | |
| *** tzn has quit IRC | 10:48 | |
| *** exploreshaifali has joined #openstack-infra | 10:48 | |
| *** salv-orl_ has quit IRC | 10:48 | |
| *** yuanying has quit IRC | 10:51 | |
| *** dtantsur|brb is now known as dtantsur | 10:52 | |
| *** abhishekk has quit IRC | 10:52 | |
| *** notnownikki has joined #openstack-infra | 10:53 | |
| bauwser | yolanda: made a small update for https://review.openstack.org/282099 but my zuul powers are very nits | 10:53 |
| bauwser | yolanda: if the above patch helps, could you please see if we could merge it soon ? | 10:54 |
| yolanda | bauwser yep, checking the review queue now | 10:55 |
| *** aysyd has joined #openstack-infra | 10:55 | |
| *** ldnunes has joined #openstack-infra | 10:55 | |
| *** zz_ja has quit IRC | 10:56 | |
| *** zz_ja has joined #openstack-infra | 10:57 | |
| *** rossella_s has quit IRC | 10:57 | |
| *** rossella_s has joined #openstack-infra | 10:58 | |
| *** aysyd has quit IRC | 10:59 | |
| yolanda | bauwser, added a comment | 11:00 |
| *** Qiming has joined #openstack-infra | 11:00 | |
| *** aysyd has joined #openstack-infra | 11:00 | |
| bauwser | yolanda: I just left a comment also, I'm not a super-skilled guy in zuul | 11:00 |
| dtantsur | hey folks! is Depends-On tag expected to work on patch https://review.openstack.org/#/c/281362/ ? it does not look so judging by the logs.. | 11:00 |
| bauwser | yolanda: I'm just helping to see how we can merge that soon, given that we have some job timeouts atm | 11:00 |
| *** achanda has joined #openstack-infra | 11:01 | |
| yolanda | bauwser, i agree that this is needed | 11:01 |
| bauwser | I'm actually refraining myself to ask us to prevent +Wing until that's fixed | 11:01 |
| bauwser | to the -dev list I mean | 11:01 |
| yolanda | bauwser, have you seen that timeouts being worse lately? | 11:02 |
| bauwser | +Wing other changes I mean too | 11:02 |
| bauwser | yolanda: well, logstash is not super happy | 11:02 |
| *** kzaitsev_mb has joined #openstack-infra | 11:02 | |
| yolanda | i put a -1 there, let's ping jeblair this afternoon | 11:03 |
| *** kozhukalov has quit IRC | 11:03 | |
| *** piet has quit IRC | 11:03 | |
| bauwser | yolanda: yeah, fair enoguh | 11:03 |
| bauwser | yolanda: but looking at http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:\%22ERROR:%20Publisher%20be.certipost.hudson.plugin.SCPRepositoryPublisher%20aborted%20due%20to%20exception\%22 for the last 7 days | 11:03 |
| *** piet has joined #openstack-infra | 11:04 | |
| bauwser | yolanda: it seems the issue was there the last week, but the number of failures raised in the last 2 days | 11:04 |
| AJaeger | dtantsur: did you read http://docs.openstack.org/infra/manual/developers.html#limitations-and-caveats ? | 11:04 |
| AJaeger | dtantsur: Is your test job using futurist package or futurist from source? Depends-On will only work if you check against source | 11:05 |
| *** achanda has quit IRC | 11:06 | |
| dtantsur | AJaeger, you're right, I should read the docs :) it's from package | 11:06 |
| *** piet has quit IRC | 11:06 | |
| dtantsur | which is pity, cause it means I can only tests these 2 changes together locally.. | 11:06 |
| *** piet has joined #openstack-infra | 11:07 | |
| *** kzaitsev_mb has quit IRC | 11:07 | |
| *** sam_wan has quit IRC | 11:09 | |
| *** thorst has joined #openstack-infra | 11:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Add gate-grenade-dsvm-neutron to tempest failure rate dashboard https://review.openstack.org/279326 | 11:10 |
| *** dabukalam has joined #openstack-infra | 11:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Add reno job for oslo.log https://review.openstack.org/282066 | 11:14 |
| openstackgerrit | Merged openstack-infra/project-config: Fix typo in Neutron grafana dashboards https://review.openstack.org/282129 | 11:15 |
| *** thorst has quit IRC | 11:16 | |
| *** trash has quit IRC | 11:17 | |
| *** rfolco has joined #openstack-infra | 11:17 | |
| *** jcoufal has joined #openstack-infra | 11:18 | |
| *** rfolco has quit IRC | 11:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Common template for Centos & Fedora devstack jobs https://review.openstack.org/282117 | 11:20 |
| *** zeih_ has quit IRC | 11:20 | |
| *** rfolco has joined #openstack-infra | 11:20 | |
| *** vivekd has quit IRC | 11:21 | |
| *** mikelk has quit IRC | 11:22 | |
| *** mikelk has joined #openstack-infra | 11:22 | |
| *** zz_ja has quit IRC | 11:23 | |
| *** vivekd has joined #openstack-infra | 11:24 | |
| *** zz_ja has joined #openstack-infra | 11:25 | |
| *** [1]Thelo has joined #openstack-infra | 11:25 | |
| *** vivekd_ has joined #openstack-infra | 11:26 | |
| *** Thelo has quit IRC | 11:28 | |
| *** [1]Thelo is now known as Thelo | 11:28 | |
| *** vivekd has quit IRC | 11:29 | |
| *** vivekd_ is now known as vivekd | 11:29 | |
| *** salv-orlando has joined #openstack-infra | 11:29 | |
| *** matrohon has quit IRC | 11:29 | |
| *** trash has joined #openstack-infra | 11:29 | |
| openstackgerrit | Jens proposed openstack-infra/git-review: Make it possible to configure draft as default push mode https://review.openstack.org/220426 | 11:30 |
| *** d0ugal has quit IRC | 11:31 | |
| *** shardy has joined #openstack-infra | 11:31 | |
| *** esker has quit IRC | 11:31 | |
| *** d0ugal has joined #openstack-infra | 11:31 | |
| *** d0ugal has quit IRC | 11:31 | |
| *** d0ugal has joined #openstack-infra | 11:31 | |
| *** HeOS has quit IRC | 11:32 | |
| *** piet has quit IRC | 11:33 | |
| *** piet has joined #openstack-infra | 11:33 | |
| *** vivekd has quit IRC | 11:35 | |
| *** yamamoto_ has joined #openstack-infra | 11:35 | |
| *** itxaka has joined #openstack-infra | 11:35 | |
| *** piet has quit IRC | 11:37 | |
| *** markvoelker has joined #openstack-infra | 11:38 | |
| *** piet has joined #openstack-infra | 11:38 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Make Ironic VM memory and count configurable https://review.openstack.org/281484 | 11:39 |
| *** kzaitsev_mb has joined #openstack-infra | 11:39 | |
| *** HeOS has joined #openstack-infra | 11:41 | |
| *** markvoelker has quit IRC | 11:42 | |
| *** esker has joined #openstack-infra | 11:43 | |
| *** tzn has joined #openstack-infra | 11:44 | |
| *** piet has quit IRC | 11:44 | |
| *** annegentle has joined #openstack-infra | 11:45 | |
| *** tzn has quit IRC | 11:48 | |
| *** tzn has joined #openstack-infra | 11:48 | |
| *** piet has joined #openstack-infra | 11:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix rally skip rules https://review.openstack.org/281665 | 11:48 |
| *** annegentle has quit IRC | 11:50 | |
| openstackgerrit | Merged openstack-infra/project-config: Initiate testing for puppet-openstack-cookiecutter https://review.openstack.org/272156 | 11:51 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/project-config: Add docs-on-rtfd hook for fuel-noop-fixtures project https://review.openstack.org/281266 | 11:51 |
| *** andreykurilin has joined #openstack-infra | 11:51 | |
| *** ihrachys has quit IRC | 11:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Recreate workspace in zuul-cloner cleanup https://review.openstack.org/282034 | 11:52 |
| openstackgerrit | Merged openstack-infra/project-config: Add pypi publish jobs to RefStack https://review.openstack.org/281720 | 11:53 |
| openstackgerrit | Merged openstack-infra/project-config: Specify DEVSTACK_GATE_SETTINGS for Horizon i9n tests job https://review.openstack.org/269578 | 11:53 |
| andreykurilin | AJaeger: hi! Can you help me to understand https://review.openstack.org/#/c/281665/ ? :) Why neutron is affected before(`project: ^openstack/rally` should avoid it)? | 11:53 |
| *** yamamoto_ has quit IRC | 11:54 | |
| *** yamamoto has joined #openstack-infra | 11:54 | |
| *** ihrachys has joined #openstack-infra | 11:54 | |
| AJaeger | andreykurilin: we have two conditions for the skip if - and the later wins | 11:56 |
| lucasagomes | apparently openstack/requirements gate is broken due: http://logs.openstack.org/61/281661/4/check/gate-requirements-integration-dsvm/41cb217/console.html#_2016-02-19_10_55_09_773 | 11:56 |
| AJaeger | we need to merge them. | 11:56 |
| lucasagomes | (just a fyi, not sure if you guys are looking at the problem already) | 11:57 |
| AJaeger | andreykurilin: see http://logs.openstack.org/84/281484/2/check/gate-project-config-layout/a8d9c92/console.html#_2016-02-19_11_48_16_188 and check the gate-rally-neutron-neutron line. | 11:57 |
| AJaeger | the big neutron block should have skipped it for neutron - but now its skipped only for rally ;( | 11:58 |
| *** yamamoto has quit IRC | 11:58 | |
| *** piet has quit IRC | 11:58 | |
| AJaeger | andreykurilin: see my discussion with armax yesterday http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-02-18.log.html#t2016-02-18T17:20:21 | 11:59 |
| andreykurilin | AJaeger: oh, got it. Can it be fixed in zuul 3? so if two conditions are not conflict, two of them will be applied | 11:59 |
| *** gongysh has quit IRC | 11:59 | |
| AJaeger | andreykurilin: moving it up, will remove the conflict ;) | 11:59 |
| AJaeger | andreykurilin: better ask jeblair for that | 11:59 |
| andreykurilin | :) | 12:00 |
| AJaeger | lucasagomes: I wasn't aware of it but don't know whom to tell - please raise on #openstack-qa as well | 12:00 |
| andreykurilin | jeblair: Can you answer the question above? | 12:00 |
| lucasagomes | AJaeger, will do | 12:00 |
| AJaeger | Lunch time, will be back later... | 12:00 |
| *** itxaka is now known as itxaka|afk | 12:00 | |
| andreykurilin | AJaeger: thanks for explanation | 12:00 |
| AJaeger | andreykurilin: jeblair is sleeping, might be hear in 4 to 5 hours. | 12:00 |
| andreykurilin | AJaeger: btw, does it possible to leave a comment like "check all" to launch all jobs? | 12:01 |
| *** ociuhandu has joined #openstack-infra | 12:03 | |
| *** ihrachys has quit IRC | 12:03 | |
| *** tzn has quit IRC | 12:05 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Allow for git base override https://review.openstack.org/151034 | 12:06 |
| *** zeih has joined #openstack-infra | 12:06 | |
| *** zeih has quit IRC | 12:07 | |
| *** bradm has quit IRC | 12:11 | |
| *** chlong_ has joined #openstack-infra | 12:11 | |
| *** mrmartin has joined #openstack-infra | 12:12 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version https://review.openstack.org/229789 | 12:12 |
| *** bradm has joined #openstack-infra | 12:13 | |
| *** thorst has joined #openstack-infra | 12:13 | |
| * esikachev is now away: Away from keyboard | 12:15 | |
| *** esikachev is now known as esikachev_afk | 12:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Add heat lbaas specific test for v1 and v2 https://review.openstack.org/281354 | 12:16 |
| *** annegentle has joined #openstack-infra | 12:19 | |
| *** rhallisey has joined #openstack-infra | 12:19 | |
| *** thorst has quit IRC | 12:21 | |
| *** piet has joined #openstack-infra | 12:23 | |
| *** annegentle has quit IRC | 12:23 | |
| *** thorst has joined #openstack-infra | 12:23 | |
| *** kushal has quit IRC | 12:24 | |
| *** _amrith_ is now known as amrith | 12:26 | |
| *** |-paul-| has quit IRC | 12:26 | |
| *** tzn has joined #openstack-infra | 12:26 | |
| *** kzaitsev_mb has quit IRC | 12:27 | |
| *** esker has quit IRC | 12:27 | |
| *** shardy has quit IRC | 12:29 | |
| *** dims_ has joined #openstack-infra | 12:29 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Ironic Role to OpenStack-Ansible https://review.openstack.org/282314 | 12:30 |
| openstackgerrit | Jens proposed openstack-infra/git-review: Make it possible to configure draft as default push mode https://review.openstack.org/220426 | 12:30 |
| *** zeih has joined #openstack-infra | 12:33 | |
| *** yamamoto has joined #openstack-infra | 12:33 | |
| *** gordc has joined #openstack-infra | 12:35 | |
| *** piet has quit IRC | 12:36 | |
| *** esker has joined #openstack-infra | 12:36 | |
| *** zeih has quit IRC | 12:38 | |
| *** vgridnev has quit IRC | 12:40 | |
| *** jaosorior has quit IRC | 12:40 | |
| *** jaosorior has joined #openstack-infra | 12:41 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:41 | |
| *** sdake has joined #openstack-infra | 12:43 | |
| *** asettle has quit IRC | 12:44 | |
| *** Daisy has joined #openstack-infra | 12:44 | |
| *** jaosorior has quit IRC | 12:46 | |
| *** jaosorior has joined #openstack-infra | 12:46 | |
| *** baoli has joined #openstack-infra | 12:48 | |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Adds PyMI as a global-requirement https://review.openstack.org/279122 | 12:49 |
| *** baoli_ has joined #openstack-infra | 12:50 | |
| openstackgerrit | Lukas Bednar proposed openstack-infra/gerritlib: Add / remove reviewer to / from patch set https://review.openstack.org/235884 | 12:51 |
| *** tzn has quit IRC | 12:51 | |
| *** baoli has quit IRC | 12:53 | |
| *** markvoelker has joined #openstack-infra | 12:54 | |
| *** vgridnev has joined #openstack-infra | 12:54 | |
| *** ybathia has quit IRC | 12:55 | |
| *** Wei_Liu has joined #openstack-infra | 12:56 | |
| *** lucasagomes is now known as lucas-hungry | 12:56 | |
| *** gongysh has joined #openstack-infra | 12:57 | |
| *** Wei_Liu has quit IRC | 12:57 | |
| *** markvoelker has quit IRC | 12:58 | |
| *** rlandy has joined #openstack-infra | 12:59 | |
| *** asettle has joined #openstack-infra | 13:00 | |
| AJaeger | sdague, yolanda: could you put 281907 on your review list, please? | 13:01 |
| *** markvoelker has joined #openstack-infra | 13:04 | |
| *** krotscheck_afk is now known as krotscheck | 13:05 | |
| sdague | so, am I basically seeing that zuul-cloner is bonkers with the nova repo? | 13:06 |
| sdague | AJaeger: should we revert the use of zuul cloner everywhere? | 13:06 |
| *** tiswanso_ has quit IRC | 13:06 | |
| *** vivekd has joined #openstack-infra | 13:07 | |
| *** kushal has joined #openstack-infra | 13:07 | |
| AJaeger | sdague: Let's WIP https://review.openstack.org/267096 for now ;) I won't recheck it again... | 13:08 |
| AJaeger | sdague: I hope that https://review.openstack.org/282099 fixes the problems - but it needs some more work. | 13:08 |
| *** MarkAtwood has joined #openstack-infra | 13:08 | |
| sdague | AJaeger: ok, though if we had an infra-root around we could revert the jobs as well | 13:09 |
| AJaeger | sdague: jeblair, fungi, and clarkb have been working on this yesterday. They think the network maintenance in RAX really increased the likelihood of running into this. But yes, we use zuul cloner much more now as well | 13:09 |
| sdague | right | 13:09 |
| AJaeger | sdague: we could - but it takes 3 hours to get it in since it will change so many jobs that jjb runs into a timeout and we need to do manual runs | 13:10 |
| sdague | it will | 13:10 |
| sdague | but right now basically no one can land changes | 13:10 |
| sdague | because this is so frequent | 13:11 |
| sdague | so everything is kind of dead in the water | 13:11 |
| AJaeger | sdague: let's wait for fungi and discuss what's the best way to move forward. | 13:11 |
| sdague | sure | 13:11 |
| * AJaeger didn't read the full backlog, so don't know what our fire brigade thinks | 13:11 | |
| *** dstufft has quit IRC | 13:11 | |
| *** moravec has quit IRC | 13:11 | |
| *** annegentle has joined #openstack-infra | 13:12 | |
| sdague | yeh, I don't know | 13:12 |
| sdague | and I didn't see any summary of the issue out there | 13:12 |
| AJaeger | sdague: I'm not opposed to reverting it, and suggest we strongly consider doing it if the other issues can't be solved quickly | 13:12 |
| AJaeger | sdague: that was yesterday's plan: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-02-18.log.html#t2016-02-18T19:39:42 | 13:13 |
| sdague | https://review.openstack.org/282099 is suspiciously failing on a timeout in unit tests that looks much like the issue exposed on the real jobs | 13:13 |
| AJaeger | sdague: but that job should fix it - so I wonder whether the fix is broken... | 13:14 |
| *** dstufft has joined #openstack-infra | 13:14 | |
| * AJaeger tried to dig into it but couldn't get a handle on it ;( | 13:14 | |
| AJaeger | have to leave now, will be back later | 13:14 |
| *** annegentle has quit IRC | 13:16 | |
| *** dizquierdo has quit IRC | 13:16 | |
| *** Daisy has quit IRC | 13:17 | |
| *** moravec has joined #openstack-infra | 13:17 | |
| *** EricGonczer_ has joined #openstack-infra | 13:17 | |
| mordred | morning sdague | 13:19 |
| *** salv-orlando has quit IRC | 13:21 | |
| *** vgridnev_ has joined #openstack-infra | 13:21 | |
| sdague | mordred: morning | 13:22 |
| mordred | sdague: I have not been around most of the week, so I have no useful deeper background to offer | 13:23 |
| *** Daisy_ has joined #openstack-infra | 13:24 | |
| *** vgridnev has quit IRC | 13:24 | |
| sdague | yeh, I've been hunting bugs in nova, me either. I just know we're basically blocked on landing any fixes at this point because things are failing a ton on all our unit/pep8 jobs because of git clone issues. | 13:24 |
| *** vgridnev_ has quit IRC | 13:24 | |
| *** vgridnev has joined #openstack-infra | 13:24 | |
| mordred | yah. that is about what I understand as well | 13:25 |
| *** doug-fish has joined #openstack-infra | 13:26 | |
| *** dtroyer has quit IRC | 13:26 | |
| mordred | I agree about getting input from fungi when he gets up due to the time reverting zuul-cloner would take | 13:26 |
| mordred | and the corresponding soon-ness about fungi waking up | 13:27 |
| *** edmondsw has joined #openstack-infra | 13:28 | |
| Clint | whee | 13:29 |
| *** annegentle has joined #openstack-infra | 13:29 | |
| *** Daisy_ has quit IRC | 13:30 | |
| *** Daisy_ has joined #openstack-infra | 13:30 | |
| *** bhunter_ has joined #openstack-infra | 13:35 | |
| sdague | ok, well I'm away from keyboard for a bit. However, we're effectively blocked from development right now. So my vote is revert using zuul-cloner to unblock development, and let the zuul-cloner fixes go on in parallel | 13:35 |
| mordred | sdague: nod | 13:36 |
| Shrews | mordred: i think yesterday's release of os-client-config broke shade. 2nd time in a row | 13:36 |
| mordred | Shrews: god, really? | 13:36 |
| *** piet has joined #openstack-infra | 13:37 | |
| *** Daisy_ has quit IRC | 13:37 | |
| mordred | Shrews: oh - we're testing mock calls in the occ layer again I think | 13:37 |
| *** amrith is now known as _amrith_ | 13:38 | |
| mordred | Shrews: patch coming if you're not already working on it - I'm pretty sur eI know the issue | 13:38 |
| Shrews | mordred: go for it | 13:38 |
| Shrews | i'm just now inserting the IV for caffeine | 13:38 |
| mordred | Shrews: t;ldr the changes to the call to swiftclient Connection | 13:39 |
| Shrews | *nod* | 13:39 |
| mordred | Shrews: I'm going to remove that test - it's a unit test that occ works, not that shade works | 13:39 |
| *** cbader has quit IRC | 13:40 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove mock testing of os-client-config for swift https://review.openstack.org/282339 | 13:40 |
| yolanda | AJaeger back, sorry but today i'm in and out preparing for the travel | 13:42 |
| *** EricGonczer_ has quit IRC | 13:42 | |
| mordred | I just got a creamer on the plane with "tastes like fresh milk' on it | 13:42 |
| Clint | is it a lie | 13:43 |
| mordred | that does not fill me with anything other than wonder about what non-milk thing it is if the closest they can say is "tastes like milk" | 13:43 |
| *** vgridnev has quit IRC | 13:46 | |
| *** tiswanso has joined #openstack-infra | 13:46 | |
| *** dtroyer has joined #openstack-infra | 13:47 | |
| *** shardy has joined #openstack-infra | 13:47 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: New Favicon https://review.openstack.org/282342 | 13:48 |
| smcginnis | fungi: ping | 13:49 |
| openstackgerrit | Merged openstack-infra/project-config: Switch gate-{name}-requirements-bindep to ubuntu-trusty https://review.openstack.org/281907 | 13:50 |
| openstackgerrit | Merged openstack-infra/project-config: Fix centos support for bindep-fallback.txt https://review.openstack.org/282122 | 13:50 |
| bauwser | AJaeger: I'm really not convinced that https://review.openstack.org/#/c/282099/ could help, because unittests are constantly failing | 13:52 |
| bauwser | AJaeger: so I agree with sdague to revert any change that would have introduced that behaviour, instead of trying to workaround the issue | 13:52 |
| *** achanda has joined #openstack-infra | 13:52 | |
| bauwser | (I was missing context that we changed something with zuul) | 13:53 |
| bauwser | because like sdague said, we can't land anything by now | 13:53 |
| *** salv-orlando has joined #openstack-infra | 13:54 | |
| *** lucas-hungry is now known as lucasagomes | 13:55 | |
| *** yamamoto has quit IRC | 13:56 | |
| *** exploreshaifali has quit IRC | 13:57 | |
| *** zeih has joined #openstack-infra | 13:57 | |
| *** haypo has left #openstack-infra | 13:57 | |
| *** itxaka|afk is now known as itxaka | 13:58 | |
| mordred | so - I'm honestly not convinced that reverting will fix anything either | 13:58 |
| mordred | if the problem is as the hypothesis has state d that we're seeing significant increases in the number of network traffic interruptions | 13:58 |
| *** hashar has quit IRC | 13:59 | |
| mordred | then replacing zuul cloner with something else to clone things will not change that - in either place a retry-on-fail system will need to deal with the network failures | 13:59 |
| AJaeger | bauwser: If the *setup* of the unittests fail, 282099 should help. If they fail during runtime, it's something else and unrelated to the git timeout problems we face | 14:00 |
| *** tzn has joined #openstack-infra | 14:01 | |
| bauwser | unfortunately, I'm not an -infra folk, but I just say that if we have timeouts due to some change, let's revert that instead of trying to workaround timeouts | 14:01 |
| *** tzn has quit IRC | 14:01 | |
| AJaeger | mordred: the difference is that right now we set *one* more repo up - requirements repo - and use zuul-cloner instead of git. My gut feeling is the same - we have networking issues, let's handle them and fix where we can... | 14:01 |
| *** tzn has joined #openstack-infra | 14:01 | |
| mordred | bauwser: it wasn't our change | 14:02 |
| mordred | bauwser: the cloud we run on rlled out a networking change | 14:02 |
| bauwser | but, if the change in question is not responsible for the timeouts, then that's another issue and I apologize | 14:02 |
| *** vgridnev has joined #openstack-infra | 14:02 | |
| *** esikachev_afk is now known as esikachev | 14:02 | |
| mordred | bauwser: no worries - it's a frustrating situation to be sure, and I appreciate your engagement :) | 14:02 |
| *** cbader has joined #openstack-infra | 14:02 | |
| mordred | bauwser: I'm not sure that I know 100% that our problems can be associated with underlying cloud rollout - so I could be wrong... but that is my understanding currently of the working theory of the cause of the problems | 14:03 |
| yolanda | mordred AJaeger, i think having timeouts and retries on zuul will help. However that change needs a bit more of work, some of the imports are wrong | 14:03 |
| bauwser | so, I just tried to pimp the timeout patch that jeblair provided, but I'm very worried by the UTs not passing | 14:03 |
| yolanda | i can fix that a bit and we can try approving | 14:03 |
| mordred | yolanda: go for it | 14:03 |
| yolanda | ++ | 14:03 |
| bauwser | yolanda: that would be awesome | 14:03 |
| yolanda | mordred, we ended having something like that downstream. Not on zuul but on an extra layer, so the idea sounds good to me | 14:04 |
| * AJaeger cheers up yolanda ;) | 14:04 | |
| *** piet has quit IRC | 14:04 | |
| yolanda | :) | 14:05 |
| *** yamamoto has joined #openstack-infra | 14:06 | |
| *** piet has joined #openstack-infra | 14:06 | |
| *** kgiusti has joined #openstack-infra | 14:07 | |
| *** esikachev has left #openstack-infra | 14:07 | |
| *** dkranz_ has joined #openstack-infra | 14:08 | |
| *** ccrouch has quit IRC | 14:08 | |
| *** yamamoto has quit IRC | 14:09 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/zuul: Add timeout/retries to zuul cloner https://review.openstack.org/282099 | 14:10 |
| *** dingyichen has quit IRC | 14:10 | |
| yolanda | that should be ok | 14:10 |
| yolanda | mm, pep8 fails wtih spacing :( | 14:10 |
| openstackgerrit | yolanda.robla proposed openstack-infra/zuul: Add timeout/retries to zuul cloner https://review.openstack.org/282099 | 14:11 |
| yolanda | better | 14:11 |
| AJaeger | " F401 'timeout_clone' imported but unused" | 14:13 |
| yolanda | unused? | 14:13 |
| *** yamamoto has joined #openstack-infra | 14:13 | |
| yolanda | oh, i see | 14:13 |
| yolanda | weird my pep8 didn't alert me | 14:13 |
| *** dingyichen has joined #openstack-infra | 14:13 | |
| AJaeger | but I wonder why the python27 tests fail all the time on the change - is that networking or is the change broken? | 14:14 |
| openstackgerrit | yolanda.robla proposed openstack-infra/zuul: Add timeout/retries to zuul cloner https://review.openstack.org/282099 | 14:14 |
| yolanda | python27 looks as failing for timeouts | 14:14 |
| yolanda | but going to run it locally | 14:14 |
| *** gongysh has quit IRC | 14:14 | |
| *** smarcet has joined #openstack-infra | 14:15 | |
| AJaeger | One failure I just got (with the previous version): http://paste.openstack.org/show/487577/ | 14:16 |
| yolanda | mm tests need to be fixed then | 14:16 |
| yolanda | AJaeger, i had the same now, i'll fix | 14:17 |
| AJaeger | zuul.lib.cloner.CloneTimeoutError is wrong | 14:17 |
| AJaeger | thanks, yolanda | 14:17 |
| openstackgerrit | Octavian Ciuhandu proposed openstack/requirements: Bumps os-win to 0.2.2 https://review.openstack.org/282351 | 14:18 |
| *** vgridnev has quit IRC | 14:19 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/zuul: Add timeout/retries to zuul cloner https://review.openstack.org/282099 | 14:19 |
| *** smarcet has quit IRC | 14:20 | |
| *** smarcet has joined #openstack-infra | 14:21 | |
| *** vgridnev has joined #openstack-infra | 14:21 | |
| openstackgerrit | Octavian Ciuhandu proposed openstack/requirements: Bumps os-win to 0.2.2 https://review.openstack.org/282351 | 14:22 |
| *** vgridnev has quit IRC | 14:22 | |
| *** mriedem has joined #openstack-infra | 14:23 | |
| *** achanda has quit IRC | 14:24 | |
| yolanda | i fixed that error,let's see if other tests pass | 14:24 |
| *** achanda has joined #openstack-infra | 14:24 | |
| *** achanda has quit IRC | 14:25 | |
| *** annegentle has quit IRC | 14:25 | |
| AJaeger | yolanda: one step at a time ;) | 14:25 |
| *** hashar has joined #openstack-infra | 14:25 | |
| *** esker has quit IRC | 14:26 | |
| *** vgridnev has joined #openstack-infra | 14:27 | |
| *** doug-fish is now known as doug_fish | 14:27 | |
| *** links has quit IRC | 14:27 | |
| AJaeger | if anybody wants to review a change to run less jobs for neutron and others, please - this would be nice: https://review.openstack.org/281665 | 14:28 |
| *** piet has quit IRC | 14:29 | |
| *** jsavak has joined #openstack-infra | 14:30 | |
| *** _amrith_ is now known as amrith | 14:30 | |
| * AJaeger will be back later... | 14:30 | |
| sdague | mordred: why isn't cloning failing on the devstack jobs? | 14:33 |
| *** vivekd has quit IRC | 14:34 | |
| sdague | that's why zuul-cloner issues seems the most likely hypothesis to me | 14:34 |
| *** doug_fish is now known as doug-fish | 14:34 | |
| *** dansmith is now known as superdan | 14:34 | |
| *** piet has joined #openstack-infra | 14:34 | |
| *** ddieterly has joined #openstack-infra | 14:34 | |
| *** amotoki has quit IRC | 14:35 | |
| *** amotoki has joined #openstack-infra | 14:36 | |
| *** ddieterly has quit IRC | 14:37 | |
| *** ddieterly has joined #openstack-infra | 14:37 | |
| *** ddieterly has quit IRC | 14:38 | |
| *** ddieterly has joined #openstack-infra | 14:38 | |
| *** gongysh has joined #openstack-infra | 14:39 | |
| *** amotoki has quit IRC | 14:40 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add support for provider network options https://review.openstack.org/280417 | 14:41 |
| *** gordc has quit IRC | 14:42 | |
| *** openstackgerrit_ has quit IRC | 14:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add support for provider network options https://review.openstack.org/280417 | 14:43 |
| *** gordc has joined #openstack-infra | 14:43 | |
| *** ccrouch has joined #openstack-infra | 14:44 | |
| *** openstackgerrit_ has joined #openstack-infra | 14:44 | |
| *** bmjen has joined #openstack-infra | 14:45 | |
| *** Hunner has joined #openstack-infra | 14:45 | |
| *** Hunner has quit IRC | 14:45 | |
| *** Hunner has joined #openstack-infra | 14:45 | |
| *** roxanaghe has joined #openstack-infra | 14:48 | |
| *** openstackgerrit_ has quit IRC | 14:49 | |
| *** jpr has joined #openstack-infra | 14:49 | |
| *** erlon has joined #openstack-infra | 14:53 | |
| *** vivekd has joined #openstack-infra | 14:53 | |
| *** annegentle has joined #openstack-infra | 14:54 | |
| openstackgerrit | Volodymyr Samotiy proposed openstack-infra/project-config: Add broadview-collector project https://review.openstack.org/282365 | 14:55 |
| *** ddieterl_ has joined #openstack-infra | 14:56 | |
| *** roxanaghe has quit IRC | 14:57 | |
| *** roxanagh_ has joined #openstack-infra | 14:57 | |
| *** rossella_s has quit IRC | 14:57 | |
| *** openstackgerrit_ has joined #openstack-infra | 14:57 | |
| yolanda | AJaeger, mordred , so zuul python27 test, is still running.. is taking long | 14:58 |
| *** rbrndt_ has joined #openstack-infra | 14:58 | |
| *** rossella_s has joined #openstack-infra | 14:58 | |
| *** ddieterly has quit IRC | 14:58 | |
| *** openstackgerrit_ has quit IRC | 14:59 | |
| AJaeger | sdague: note, it's bare-trusty that's the problem - I saw the timeouts earlier this morning on some documentation jobs that do not use zuul-cloner at all. | 15:00 |
| *** openstackgerrit_ has joined #openstack-infra | 15:00 | |
| AJaeger | sdague: I'll dig that one out in 30+ mins... | 15:00 |
| yolanda | hi fungi, i saw your comments on the acl. I don't have time to dedicate now to the ACL inheritance, not in the next days.. that inheritance however can simplify things, i think that is a good pattern, and we can study common acls on all projects to setup a proper hierarchy | 15:01 |
| *** amitgandhinz has joined #openstack-infra | 15:02 | |
| *** nwkarsten has joined #openstack-infra | 15:02 | |
| *** tzn has quit IRC | 15:03 | |
| fungi | yolanda: if you want to test acl behaviors, you can do it on review-dev. i do that all the time (though i just use the webui to compose acls there for testing) | 15:06 |
| yolanda | fungi seems that something is broken for me on review-dev, i'm unable to login | 15:06 |
| *** Jeffrey4l__ has quit IRC | 15:07 | |
| yolanda | fungi my idea was to go with something like that https://review.openstack.org/282223 , but haven't tested and i have no time these days... | 15:07 |
| yolanda | i agree that exclusiveGroupPermissions may be a problem in that case | 15:07 |
| *** abregman has joined #openstack-infra | 15:09 | |
| *** eharney has joined #openstack-infra | 15:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Unshadow All-Projects in exclusiveGroupPermissions https://review.openstack.org/281563 | 15:10 |
| fungi | mordred: sdague: another complication is that the affected jobs all run on bare-trusty nodes, which we can only build in rackspace because they're not dib-able and we're using dib everywhere else | 15:10 |
| *** eharney has quit IRC | 15:10 | |
| mordred | fungi: wow | 15:10 |
| *** eharney has joined #openstack-infra | 15:10 | |
| mordred | fungi: that is a fascinating set of issues | 15:10 |
| *** vivekd has quit IRC | 15:12 | |
| BobBall | asselin: grrrrr... Can you help me? Every time I do a puppet apply, my ~zuul/.ssh/known_hosts gets corrupted because the initial content does not have a trailing newline; meaning that when the next content is added (by starting zuul-merger) the first line is actually corrupted. | 15:12 |
| *** xarses has quit IRC | 15:12 | |
| *** wolverineav has joined #openstack-infra | 15:12 | |
| BobBall | asselin: May not be the trailing newline, but adding one manually (after puppet apply) fixes it | 15:12 |
| asselin_ | BobBall, are you overriding it in the heira data? | 15:13 |
| BobBall | asselin: I'm assuming that this is something that's specific to my env for some obscure reason since I can't figure out how it would work for anyone otherwise so I don't want to propose just adding a newline | 15:13 |
| BobBall | asselin: nope, no override | 15:13 |
| *** piet has quit IRC | 15:13 | |
| *** roxanagh_ has quit IRC | 15:13 | |
| *** links has joined #openstack-infra | 15:13 | |
| asselin_ | BobBall, then copy what you do manually to heira and re=run puppet. Then puppet will make sure it stays there | 15:13 |
| *** mtanino has joined #openstack-infra | 15:13 | |
| asselin_ | and not use the default value (which doesn't work for you) | 15:13 |
| clarkb | I am confused how a change yesterday that would not be included on new images when the fuss was going on (before image builds for the day start) could affect anything | 15:14 |
| *** claudiub has joined #openstack-infra | 15:14 | |
| BobBall | asselin_: I've not found a value that works - every time I start zuul-merger after a puppet apply it will append a new entry to the known_hosts file (using |1|+E3+9S+bILO+oP8W2kI0tVK7hpQ=|lwz3sme++2qrfQX1zKwAaoRdVbQ= instead of a hostname) | 15:15 |
| yolanda | so python test for zuul failed with timeout https://jenkins05.openstack.org/job/gate-zuul-python27/463/console | 15:15 |
| BobBall | asselin_: forcing the |1|+E3+9S+bILO+oP8W2kI0tVK7hpQ=|lwz3sme++2qrfQX1zKwAaoRdVbQ= entry seems very wrong to me :) | 15:15 |
| clarkb | all that to say yesterdays zuul change being reverted shouldnt affevt anything righ | 15:15 |
| fungi | clarkb: which zuul change? i thought the discussion was about reverting the project-config change which started using zuul-cloner everywhere | 15:15 |
| *** dhellmann has quit IRC | 15:16 | |
| asselin_ | BobBall, yes b/c that is the hashed version. But try it first to see if you can get it to stop updating and keep the same value. then you can change it to the unhashed version in a 2nd step. | 15:16 |
| fungi | on the assumption that gerrit-git-prep was dealing with this better (which is not actually known, the issues cropped up well after we switched) | 15:16 |
| *** zz_dimtruck is now known as dimtruck | 15:16 | |
| clarkb | ph I misread scrollback | 15:16 |
| fungi | clarkb: but yeah, as for the zuul-cloner timeout/retry improvements, we'd need new venvs on our images and then the jobs themselves would also need updating to take advantage of the new features | 15:16 |
| clarkb | everyone said revert the zuul.change and timeouts so I thought it was the addition of timeouts to zuul cloner | 15:16 |
| *** wolverineav has quit IRC | 15:17 | |
| yolanda | clarkb that change to add timeouts hasn't ever passed, i was fixing some pep8 and test errors, but still fails | 15:17 |
| fungi | and even then, it's just a workaround, doesn't directly solve the network issues which are presumably causing this | 15:18 |
| openstackgerrit | Matthew Kassawara proposed openstack-infra/project-config: Fix logic for networking-ovn jobs https://review.openstack.org/282378 | 15:18 |
| clarkb | yup | 15:18 |
| BobBall | asselin_: Nope; setting that key doesn't work either - just a host key verification failed. | 15:18 |
| *** dhellmann has joined #openstack-infra | 15:19 | |
| asselin_ | BobBall, I'm confused.... | 15:19 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:19 | |
| *** piet has joined #openstack-infra | 15:20 | |
| *** esker has joined #openstack-infra | 15:20 | |
| fungi | so as for the best way forward, i guess it's to keep moving on all available fronts (workarounds in zuul-cloner and/or in job configuration, finishing getting off bare-trusty, and hoping the rash of network instability gets fixed by the provider) | 15:21 |
| mordred | fungi: agree | 15:21 |
| BobBall | so am I. I added that hashed one to common.yaml and restarted zuul-merger and it added a different hash |1|osqaB6ce97MZNd6YdyyGa3Ykqqw=|YS7ZF8WnKy1Z9pGyMBzPNDmOmT0= instead | 15:22 |
| *** ociuhandu has quit IRC | 15:22 | |
| asselin_ | why is zuul merger adding hashes? | 15:23 |
| *** infra-red has quit IRC | 15:24 | |
| *** ociuhandu has joined #openstack-infra | 15:24 | |
| *** amotoki has joined #openstack-infra | 15:24 | |
| BobBall | asselin_: http://paste.openstack.org/show/487586/ | 15:24 |
| BobBall | I'm assuming because the hash in the file isn't the host it's looking at, so it makes a new one | 15:25 |
| *** esker has quit IRC | 15:25 | |
| *** nwkarsten has quit IRC | 15:25 | |
| *** achanda has joined #openstack-infra | 15:25 | |
| *** nwkarsten has joined #openstack-infra | 15:26 | |
| BobBall | asselin_: Weirdly the hash seems to keep changing (see above when the key it was adding was +E3..., then it added osqa, now it's added Hv0IG) | 15:26 |
| asselin_ | BobBall, or an issue with newlines? | 15:26 |
| BobBall | asselin_: if I stop zuul merger, add the new line before |1| and restart it, everything is fine. | 15:26 |
| BobBall | asselin_: But, of course, it breaks again on the next puppet apply when puppet rewrites the file. | 15:26 |
| asselin_ | BobBall, ok...what about this change: https://review.openstack.org/#/c/266041/2/contrib/single_node_ci_site.pp | 15:27 |
| AJaeger | I was wrong - these failure on docu projects were also zuul-cloner related | 15:27 |
| *** claudiub has quit IRC | 15:27 | |
| *** dims_ has quit IRC | 15:27 | |
| BobBall | asselin_: Already tried that - no joy | 15:27 |
| asselin_ | BobBall, can you use that unhashed key? Also delete the zuul .ssh folder altogethere and let puppet recreate it | 15:27 |
| *** abregman is now known as abregman|nb | 15:28 | |
| *** achanda has quit IRC | 15:30 | |
| *** nwkarsten has quit IRC | 15:30 | |
| *** sdake_ has joined #openstack-infra | 15:30 | |
| *** sdake has quit IRC | 15:31 | |
| *** jsavak has quit IRC | 15:32 | |
| BobBall | asselin_: same problem | 15:32 |
| *** jsavak has joined #openstack-infra | 15:32 | |
| *** amotoki has quit IRC | 15:33 | |
| *** alaski is now known as lascii | 15:33 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on Username/Password trailling whitespace https://review.openstack.org/282389 | 15:34 |
| *** zeih has quit IRC | 15:34 | |
| *** vgridnev has quit IRC | 15:34 | |
| *** gongysh has quit IRC | 15:34 | |
| mtreinish | mordred, Shrews: have you seen this before?: http://logs.openstack.org/16/278816/8/check/shade-coverage/5221b72/console.html#_2016-02-19_05_23_39_728 | 15:35 |
| openstackgerrit | Octavian Ciuhandu proposed openstack/requirements: Bumps os-win to 0.2.1 https://review.openstack.org/282351 | 15:35 |
| *** xarses has joined #openstack-infra | 15:35 | |
| mtreinish | it's blocking my download image patch | 15:35 |
| *** vgridnev has joined #openstack-infra | 15:36 | |
| mtreinish | mordred, Shrews, notmorgan: oh, I bet it's related to the keystoneauth release the other day | 15:37 |
| mordred | mtreinish: yes. I pushed up a patch to fix earlier today | 15:37 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on Username/Password trailling whitespace https://review.openstack.org/282389 | 15:38 |
| mtreinish | mordred: ok, cool | 15:38 |
| mordred | gah. pep9 | 15:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove mock testing of os-client-config for swift https://review.openstack.org/282339 | 15:40 |
| *** mrmartin has quit IRC | 15:40 | |
| mordred | mtreinish: that | 15:40 |
| *** amotoki has joined #openstack-infra | 15:41 | |
| *** vgridnev has quit IRC | 15:41 | |
| *** vgridnev has joined #openstack-infra | 15:42 | |
| *** jaosorior has quit IRC | 15:44 | |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on Username/Password trailling whitespace https://review.openstack.org/282389 | 15:45 |
| *** vgridnev has quit IRC | 15:46 | |
| fungi | pabelanger: i wonder if the issue on http://logs.openstack.org/27/281427/1/experimental/gate-python-swiftclient-python34-bindep/0590ce7/console.html#_2016-02-17_21_37_20_580 compared to the normal python34 run is that the latter is using upper-constraints | 15:46 |
| fungi | going into the bindep job migration, i'm going to be making it really hard to revert the constraints-everywhere change because i'm going to be rearranging all the same job configs | 15:47 |
| *** pblaho has quit IRC | 15:47 | |
| *** pblaho has joined #openstack-infra | 15:47 | |
| fungi | also, all my experimental testing was with gerrit-git-prep because the constraints change landed while i was testing this | 15:47 |
| fungi | pabelanger: as for that job failure, the subunit2html errors are not the reason the job failed. it failed several unit tests prior to what you saw | 15:49 |
| *** claudiub has joined #openstack-infra | 15:49 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config: Use Ironic as Devstack plugin in Rally gates https://review.openstack.org/274824 | 15:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Extract ansible playbook running to a library https://review.openstack.org/282397 | 15:51 |
| *** ganso has left #openstack-infra | 15:51 | |
| *** EricGonczer_ has joined #openstack-infra | 15:52 | |
| *** vgridnev has joined #openstack-infra | 15:54 | |
| *** xyang1 has joined #openstack-infra | 15:55 | |
| *** tzn has joined #openstack-infra | 15:55 | |
| *** EricGonc_ has joined #openstack-infra | 15:55 | |
| *** Liuqing has joined #openstack-infra | 15:57 | |
| *** eantyshev has joined #openstack-infra | 15:57 | |
| *** vgridnev has quit IRC | 15:57 | |
| *** jsavak has quit IRC | 15:57 | |
| *** regXboi has joined #openstack-infra | 15:58 | |
| *** vgridnev has joined #openstack-infra | 15:58 | |
| *** EricGonczer_ has quit IRC | 15:59 | |
| *** jsavak has joined #openstack-infra | 15:59 | |
| pabelanger | fungi: Ya, I looked more last night and seen that too. Didn't notice the uppet-constraints however | 16:00 |
| *** mriedem is now known as mriedem_meeting | 16:00 | |
| pabelanger | fungi: I am doing a check experimental atm to test the bindep requirements job now that it merged | 16:00 |
| AJaeger | yolanda: that zuul-cloner patch does not terminate the tests locally for me - so something is fishy there. The timeout seems to be an implementation/testsuite issue | 16:00 |
| *** tzn has quit IRC | 16:00 | |
| yolanda | AJaeger, failed for me as well | 16:01 |
| *** piet has quit IRC | 16:01 | |
| fungi | pabelanger: thanks. i'm still thinking through how to go about writing the production job config transition | 16:01 |
| BobBall | asselin_: If I inject a newline into https://review.openstack.org/#/c/266041/2/contrib/single_node_ci_site.pp it works (see comment) | 16:01 |
| yolanda | i was retrying but failed for 2 times | 16:01 |
| *** dizquierdo has joined #openstack-infra | 16:02 | |
| pabelanger | fungi: are you thinking of keeping bindep in the name, or replacing the current? | 16:02 |
| fungi | pabelanger: i think i'm going to end up hard-coding node labels into most of the affected job-templates, though at the moment the per-project node defaults don't really accomplish much so i'm not sure that's a real problem | 16:02 |
| *** amitgandhinz has quit IRC | 16:02 | |
| fungi | pabelanger: it would be a change to the main job templates | 16:02 |
| fungi | like gate-{name}-pep8 | 16:02 |
| pabelanger | right | 16:02 |
| *** coolsvap|away has quit IRC | 16:03 | |
| AJaeger | fungi, what about adding openstack-manuals to your testing as well? Or is that a third or fourth step? | 16:03 |
| pabelanger | what about if a project also wanted to test bindep on centos / fedora? how do you see exposing node for that? | 16:03 |
| pabelanger | gate-{name}-pep8-{node} ? | 16:03 |
| pabelanger | or is that overkill | 16:04 |
| *** amotoki has quit IRC | 16:04 | |
| fungi | the problem is that to make this transition, builders in jobs have to be changed and reordered, so unless i have one massive change (which will conflict with evert project-config change under the sun and be terribly hard to back out if we need to) i'll end up hard-coding node parameters in the templates which i modify as i go | 16:04 |
| *** cschwede has quit IRC | 16:04 | |
| fungi | AJaeger: we could add it, though i was focusing on python projects and testing python jobs initially | 16:04 |
| *** cschwede has joined #openstack-infra | 16:05 | |
| fungi | seemed like the unit tests, pep8, coverage and requirements check were a digestible chunk to bite off initially | 16:05 |
| AJaeger | fungi: if you want to do this in steps, it's fine by me and I consider that a good approach | 16:05 |
| * AJaeger was just alarmed hearing changing it "everywhere" | 16:05 | |
| fungi | i mean, i can't really test everything. we have close to 1000 repos now | 16:05 |
| fungi | so some things quite possibly will break at some point during this transition | 16:05 |
| *** piet has joined #openstack-infra | 16:06 | |
| AJaeger | fungi: let's do those python jobs and once they are stable ping me and I'll help with documentation repo jobs | 16:06 |
| pabelanger | I am game for the bandaid approach, happy to help fix issue :) | 16:06 |
| *** salv-orlando has quit IRC | 16:06 | |
| fungi | but so far i've tested 7 server projects, 7 client projects, including both supported stable branches on each of them, and 7 different jobs for each of those... the matrix gets huge quickly | 16:07 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add support for provider network options https://review.openstack.org/280417 | 16:07 |
| fungi | 14*3*7 ~= 300 jobs i've validated in experimental so far | 16:08 |
| fungi | and that covers maybe half the potentially impacted jobs for about 1.5% of the repos we host | 16:08 |
| AJaeger | fungi: yeah, noticed - we had 8800 jobs last week, then 8001 on Monday - and then you came ;) - now it's 8300 again | 16:09 |
| *** tphummel has joined #openstack-infra | 16:09 | |
| fungi | well, those will go away pretty soon | 16:09 |
| AJaeger | I know - no worries. | 16:09 |
| *** coolsvap|away has joined #openstack-infra | 16:09 | |
| *** pcaruana has quit IRC | 16:10 | |
| *** jsavak has quit IRC | 16:11 | |
| *** jsavak has joined #openstack-infra | 16:12 | |
| AJaeger | fungi: we only have 120 open changes for project-config - so go for it ;) | 16:12 |
| AJaeger | 120 is an all time low AFAIR | 16:12 |
| fungi | wow! | 16:12 |
| *** vgridnev has quit IRC | 16:12 | |
| AJaeger | 116 actually - wow! | 16:13 |
| *** vivekd has joined #openstack-infra | 16:15 | |
| sdague | hey, was afk for a while. Was a plan of attack sorted on the zuul-cloner issue | 16:15 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Added bower install to post-install hook https://review.openstack.org/278057 | 16:15 |
| *** woodster_ has joined #openstack-infra | 16:15 | |
| mordred | sdague: 15:21:00 fungi | so as for the best way forward, i guess it's to keep moving on all available fronts (workarounds in zuul-cloner and/or in job configuration, finishing getting off bare-trusty, and | 16:15 |
| mordred | | hoping the rash of network instability gets fixed by the provider) | 16:15 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Corrected license https://review.openstack.org/278058 | 16:15 |
| *** nwkarsten has joined #openstack-infra | 16:16 | |
| fungi | yeah, i'm currently focusing on the bindep migration so we can run these jobs on ubuntu-trusty and therefore not only in rackspace | 16:16 |
| mordred | sdague: tl;dr - there are non-zuul-cloner jobs affected, it's only on bare-trusty - so there are several unfortunately complicating angels | 16:16 |
| mordred | angles. not angels | 16:16 |
| mordred | there are no angels related to this | 16:16 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Corrected repository field. https://review.openstack.org/278059 | 16:16 |
| sdague | only complex angels | 16:16 |
| bauwser | mrodden: AJaeger said that docs jobs were zuul-cloner too | 16:17 |
| bauwser | oops s/mrodden/mordred | 16:17 |
| fungi | also, if we _do_ want to switch those jobs back to gerrit-git-prep and not use constraints for now (which isn't guaranteed to solve this), we should do it before i switch the same jobs over to using bindep since i'll be changing and rearranging the configuration of the same job-templates | 16:17 |
| *** vgridnev has joined #openstack-infra | 16:18 | |
| mordred | fungi: yolanda updated the zuul clner patch | 16:18 |
| sdague | well, whatever you all think gets us moving forward the quickest | 16:18 |
| AJaeger | mordred: but it fails the testsuite - times out. Even locally ;( | 16:18 |
| fungi | i saw. did she get the unit tests for the new options working? | 16:18 |
| AJaeger | fungi: not yet AFAIU | 16:19 |
| yolanda | fungi is not the new tests, i commented those and fail anyway | 16:19 |
| yolanda | i'm testing locally | 16:19 |
| fungi | sdague: network instability going away probably gets us working quickest, but we have no direct control over that | 16:19 |
| *** piet has quit IRC | 16:19 | |
| sdague | yeh, cloud and all | 16:19 |
| mordred | yolanda: let me know if you need help - I have spotty wifi but I can be useful if you point me at things | 16:19 |
| *** amotoki has joined #openstack-infra | 16:20 | |
| sdague | fungi: is there a rax ticket? | 16:20 |
| fungi | next best things we can do are add workarounds for the instability and try to get as many of our eggs as possible out of this one basket | 16:20 |
| sdague | can we get johnthetubaguy to poke? | 16:20 |
| fungi | sdague: well, they've been doing impactful planned network maintenance in the regions we're using all week | 16:20 |
| *** bpokorny has joined #openstack-infra | 16:20 | |
| fungi | also because the problem isn't easily reproducible we don't really have any details to give them | 16:20 |
| mordred | yah. once we get bare-trusty gone, we can actually turn off regions during times like this | 16:21 |
| mordred | but unfortunately this one image type only exists in rax | 16:21 |
| mordred | so there is no remediation until we get off bare-trusty | 16:21 |
| yolanda | mordred i'm trying to manually checkwhat fails with testr, still i don't know | 16:21 |
| *** mriedem_meeting is now known as mriedem | 16:21 | |
| *** Daisy_ has joined #openstack-infra | 16:22 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Switch experimental bindep jobs to constraints https://review.openstack.org/282409 | 16:23 |
| fungi | pabelanger: ^ | 16:23 |
| fungi | once that lands, i'll recheck my 40+ test changes | 16:23 |
| *** piet has joined #openstack-infra | 16:24 | |
| *** Qiming has quit IRC | 16:24 | |
| *** xarses has quit IRC | 16:25 | |
| *** xarses has joined #openstack-infra | 16:26 | |
| openstackgerrit | Matthew Kassawara proposed openstack-infra/project-config: Rename networking-ovn job https://review.openstack.org/282412 | 16:26 |
| AJaeger | fungi: LGTM. | 16:26 |
| fungi | sdague: really the only correlation we have at the moment is that we started seeing an orders of magnitude increase in git remote call hangs/timeouts coincident with the network maintenance in iad on tuesday, but without being able to trigger one on demand and get a packet capture on the server and client sides i'm afraid that's a bit vague | 16:26 |
| *** achanda has joined #openstack-infra | 16:27 | |
| fungi | also they were doing more network maintenance last night, so there was faint hope that was going to magically solve this | 16:28 |
| *** nwkarsten has quit IRC | 16:28 | |
| mordred | fungi: I like magic | 16:28 |
| *** vgridnev has quit IRC | 16:29 | |
| pabelanger | fungi: sure, looking now | 16:29 |
| *** nwkarsten has joined #openstack-infra | 16:29 | |
| sdague | fungi: ok, I have yet to have a nova patch get a good clone on all 5 of the bare jobs in the last 24 hours | 16:29 |
| sdague | usually 2 of 5 fail | 16:29 |
| sdague | so the incident rate is pretty high | 16:29 |
| AJaeger | could somebody review https://review.openstack.org/281665 and https://review.openstack.org/281097 , please? | 16:29 |
| fungi | sdague: painful, i understand. we could turn off rackspace, but at the moment that means none of these jobs would run since they are written to run on a node type which now exists only there and nowhere else | 16:30 |
| sdague | fungi: well, you were asking about reproduce | 16:30 |
| *** nwkarste_ has joined #openstack-infra | 16:30 | |
| sdague | if we can trigger this on 40% of nova clones | 16:30 |
| sdague | it seems like a direct reproduce shouldn't be too hard, right? | 16:30 |
| fungi | yeah, if someone with a vm in rax-iad wants to try cloning in a loop and tcpdump it that would help | 16:31 |
| mordred | sdague, fungi: perhaps clonova in a loop over and over again with packeet capture running? | 16:31 |
| *** austin81 has joined #openstack-infra | 16:31 | |
| mordred | fungi: clone from git.o.o ? | 16:32 |
| fungi | only reason i say iad is because anecdotal evidence pointed to more of these happening there, but that also may simply be because we have more workers in iad (i don't recall the relative quota proportions between regions) | 16:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Move centos / ubuntu bindep jobs voting https://review.openstack.org/282413 | 16:32 |
| AJaeger | yolanda: I can run the testsuite successfully for zuul locally without jeblair's change in | 16:32 |
| pabelanger | fungi: AJaeger: that will help too ^ | 16:32 |
| yolanda | AJaeger, me too | 16:32 |
| yolanda | so is something with the change but haven't noticed what's the failure | 16:32 |
| mordred | fungi: I can work on that | 16:32 |
| fungi | mordred: if that gets it reproduced, sure. alternatively might need to test checking out a zuul ref from one of the zuul mergers to trigger this | 16:32 |
| mordred | yah | 16:33 |
| sdague | the best I can tell it explodes before the zuul refs | 16:33 |
| fungi | basically we're seeing it at least mostly impact zuul-cloner usage, so that's hitting both git.o.o and zmXX.o.o i think | 16:33 |
| *** achanda has quit IRC | 16:33 | |
| *** nwkarsten has quit IRC | 16:33 | |
| sdague | or at least quite often | 16:33 |
| mordred | fungi: any preference for how to packet capture? I usually leave that to you smart people | 16:33 |
| *** armax has joined #openstack-infra | 16:34 | |
| AJaeger | pabelanger: we run this on bindep and project-config - why on project-config as well? | 16:34 |
| fungi | sudo tcpdump -i eth0 -r git-client.pcap host git.openstack.org | 16:34 |
| sdague | I guess the follow on question, the "File Not Found" issue on jobs reporting failure, is that related, or a different fixed issue? | 16:34 |
| pabelanger | AJaeger: because we have bindep-fallback.txt stored in project-config. | 16:34 |
| sdague | just trying to go back through fails I've got in my patch list for patterns | 16:35 |
| fungi | sdague: that's a side effect. git removes the job workspace when the clone fails, as part of its cleanup | 16:35 |
| pabelanger | AJaeger: we could set it up to be like other zuul patterns however, to only run when we change the file | 16:35 |
| sdague | fungi: okie dokie | 16:35 |
| mordred | I have held a rax-iad bare-trusty node | 16:35 |
| AJaeger | fungi: that workspace mkdir chnage merged a few hours ago | 16:35 |
| sdague | guess both good and bad. Good that it's the same bug. Bad that it's still a bug. | 16:35 |
| fungi | sdague: yeah, so the change AJaeger just mentioned should work around that error and allow us to collect logs from those fails | 16:36 |
| AJaeger | pabelanger: separate change... | 16:36 |
| AJaeger | anybody to +2 fungi's bindep change? https://review.openstack.org/282409 | 16:36 |
| fungi | one +2 is enough for that. it's just experimental jobs | 16:37 |
| fungi | i just went ahead and kicked it into the gate so we can have these validated sometime today | 16:38 |
| fungi | i'll go ahead and put together the wip change to switch the pep8 and unit test jobs over | 16:39 |
| fungi | AJaeger: the gate-{name}-docs job-template in python-jobs, is that going to impact openstack-manuals? you have different jobs for testing your documentation builds right? | 16:39 |
| *** jcooley_ has joined #openstack-infra | 16:40 | |
| AJaeger | fungi: we have different ones | 16:40 |
| fungi | okay, thanks. then should be safe to include in my initial migration patch | 16:40 |
| AJaeger | fungi: so, go ahead | 16:40 |
| *** salv-orlando has joined #openstack-infra | 16:40 | |
| *** dtantsur is now known as dtantsur|afk | 16:41 | |
| *** piet has quit IRC | 16:42 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: add reno jobs for oslo.config https://review.openstack.org/282417 | 16:43 |
| openstackgerrit | Merged openstack-infra/project-config: Switch experimental bindep jobs to constraints https://review.openstack.org/282409 | 16:44 |
| *** vgridnev has joined #openstack-infra | 16:44 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: add reno jobs for oslo.config https://review.openstack.org/282417 | 16:44 |
| *** rkukura has quit IRC | 16:45 | |
| AJaeger | requirements jobs are completely broken right now. | 16:45 |
| AJaeger | "error: can't copy 'sahara_dashboard/content/data_processing/locale': doesn't exist or not a regular file" - it's a symbolic link | 16:46 |
| mordred | fungi: where does git-client.pcap come from? | 16:46 |
| austin81 | mtreinish, ianw, greghaynes: thoughts on deploying stackviz via puppet module rather than dib element? | 16:46 |
| fungi | mordred: oh, sorry, meant -w | 16:47 |
| fungi | -r is for reading an existing pcap file, -w is for writing a new one | 16:47 |
| clarkb | AJaeger: I belive that id a bug in sahara_dashboard they cant use symlinks and setuptools | 16:47 |
| yolanda | so for zuul change, i see it fails after the job is even finished | 16:48 |
| *** gyee has joined #openstack-infra | 16:48 | |
| *** scheuran has quit IRC | 16:48 | |
| *** mikelk has quit IRC | 16:49 | |
| yolanda | fungi, mordred, that's output of a test that hangs , on test_one_branch -> http://paste.ubuntu.com/15133225/ | 16:49 |
| *** ddieterl_ is now known as ddieterly | 16:50 | |
| AJaeger | clarkb: https://review.openstack.org/282422 | 16:50 |
| *** kzaitsev_mb has quit IRC | 16:50 | |
| mordred | fungi, sdague: I am cloning nova in a loop with tcpdump running | 16:53 |
| *** kzaitsev_mb has joined #openstack-infra | 16:53 | |
| AJaeger | lucasagomes: 282422 should fix the requirements problems | 16:53 |
| *** links has quit IRC | 16:53 | |
| lucasagomes | AJaeger, awesome! Thanks for looking into it | 16:54 |
| mordred | bandwidth certainly is variable | 16:54 |
| mordred | Receiving objects: 100% (378925/378925), 112.43 MiB | 7.62 MiB/s, done. | 16:54 |
| mordred | Receiving objects: 100% (378925/378925), 112.67 MiB | 2.87 MiB/s, done. | 16:54 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Switch most python jobs to ubuntu-trusty https://review.openstack.org/282425 | 16:54 |
| fungi | mordred: pabelanger: AJaeger: ^ going to wip until we're done validating, but somethign like that | 16:54 |
| *** tzn has joined #openstack-infra | 16:55 | |
| *** sputnik13 has joined #openstack-infra | 16:55 | |
| mordred | fungi: lgtm | 16:56 |
| *** jsavak has quit IRC | 16:56 | |
| yolanda | mordred, zuul tests hang on self.merge_server.join() on shutdown() for base.py. for some reason that thread is not finishing | 16:56 |
| *** pblaho has quit IRC | 16:57 | |
| *** ifarkas has quit IRC | 16:57 | |
| *** jsavak has joined #openstack-infra | 16:57 | |
| yolanda | i need to go now, sorry that i cannot help more | 16:57 |
| AJaeger | lucasagomes: change is approved - but not tested yet ;) | 16:58 |
| lucasagomes | AJaeger, gate will test it for us | 16:58 |
| *** jpr has quit IRC | 16:58 | |
| *** rcernin has quit IRC | 16:58 | |
| AJaeger | dhellmann, all of requirements fails right now - 282422 is a fix for it. Just FYI. | 16:58 |
| *** kzaitsev_mb has quit IRC | 16:59 | |
| dhellmann | AJaeger : it looks like that's on the way to merging, thanks for the heads-up | 16:59 |
| *** tzn has quit IRC | 16:59 | |
| zaro | morning | 17:00 |
| *** vgridnev has quit IRC | 17:00 | |
| *** jaypipes is now known as leakypipes | 17:00 | |
| *** vgridnev has joined #openstack-infra | 17:00 | |
| mordred | fungi: I could not get it to hang with git command line - I am now retrying with zuul-cloner | 17:01 |
| *** vgridnev has quit IRC | 17:01 | |
| AJaeger | fungi, you always had the same lines in the builder, is there a way to combine these into one builder? Can be a followup as well... | 17:01 |
| *** tphummel has quit IRC | 17:01 | |
| *** e0ne has joined #openstack-infra | 17:02 | |
| fungi | mordred: yeah, one possibility is that the 40% of the time we see this it's not because a random 40% of clones fails, but rather because a random 40% of the machines are getting booted in a broken part of the network | 17:02 |
| pleia2 | good morning | 17:02 |
| *** yamamoto_ has joined #openstack-infra | 17:03 | |
| *** yamamoto has quit IRC | 17:03 | |
| fungi | AJaeger: i'd actually like to strip those down. the bulk of that is the 4 database setup macros, which ideally we can eventually have on their own job so that not every project waits for mysql and postgres to be configured | 17:03 |
| *** kzaitsev_mb has joined #openstack-infra | 17:03 | |
| fungi | i can actually probably take them out of the pep8, docs and requirements job-templates actually | 17:03 |
| fungi | actually actually | 17:03 |
| fungi | just wanted to abuse that word a few more times | 17:04 |
| fungi | anyway, i'll do that now | 17:04 |
| AJaeger | fungi: actually, that's a good idea | 17:04 |
| *** jlanoux has quit IRC | 17:04 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Switch most python jobs to ubuntu-trusty https://review.openstack.org/282425 | 17:05 |
| fungi | AJaeger: ^ there (that's actually how they're done in the experimental versions anyway) | 17:05 |
| *** jcooley_ has joined #openstack-infra | 17:05 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Move Neutron Grenade multi from experimental to check queue https://review.openstack.org/282428 | 17:06 |
| AJaeger | fungi: great! | 17:07 |
| fungi | honestly, probably only nova, ironic and a couple other projects even need the db setup macros, but until they get pushed onto a separate job-template from other projects which don't need preconfigured databases to run their unit tests, that's just a wart other projects are also going to be stuck with | 17:07 |
| fungi | don't want to go changing too many things at once during this transition | 17:07 |
| *** annegentle has quit IRC | 17:07 | |
| AJaeger | fungi: I agree, we can optimize later... | 17:07 |
| *** esker has joined #openstack-infra | 17:08 | |
| *** arxcruz has quit IRC | 17:10 | |
| *** kzaitsev_mb has quit IRC | 17:10 | |
| fungi | anyway, i need to go run some lunchtime errands. hopefully by the time i get back the constraints config for experimental bindep jobs will be live and i can check experimental a bunch and make sure these are still valid | 17:10 |
| fungi | bbiaw | 17:10 |
| *** zeih_ has joined #openstack-infra | 17:12 | |
| *** asettle has quit IRC | 17:12 | |
| *** annegentle has joined #openstack-infra | 17:13 | |
| *** wolverineav has joined #openstack-infra | 17:14 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Corrected package author https://review.openstack.org/278060 | 17:16 |
| *** annegentle has quit IRC | 17:16 | |
| *** annegentle has joined #openstack-infra | 17:16 | |
| *** jcooley_ has quit IRC | 17:17 | |
| *** jcooley_ has joined #openstack-infra | 17:17 | |
| *** Daisy_ has quit IRC | 17:18 | |
| *** wolverineav has quit IRC | 17:19 | |
| *** hashar has quit IRC | 17:20 | |
| *** annegentle has quit IRC | 17:22 | |
| *** annegentle has joined #openstack-infra | 17:22 | |
| notmorgan | pleia2: mornin! | 17:24 |
| *** jsavak has quit IRC | 17:25 | |
| *** annegentle has quit IRC | 17:27 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on AVG feedback calculation https://review.openstack.org/282438 | 17:27 |
| *** sc68cal has joined #openstack-infra | 17:27 | |
| *** itxaka has quit IRC | 17:27 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on AVG feedback calculation https://review.openstack.org/282438 | 17:28 |
| *** jsavak has joined #openstack-infra | 17:28 | |
| *** bdemers_ has joined #openstack-infra | 17:28 | |
| *** esker has quit IRC | 17:29 | |
| *** bdemers_ has left #openstack-infra | 17:29 | |
| *** yamahata has quit IRC | 17:29 | |
| *** rcernin has joined #openstack-infra | 17:30 | |
| *** doug-fish is now known as doug_fish | 17:31 | |
| *** doug_fish is now known as doug-fish | 17:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:33 | |
| *** dims has joined #openstack-infra | 17:35 | |
| *** nwkarste_ has quit IRC | 17:36 | |
| *** jsavak has quit IRC | 17:37 | |
| *** BobBall is now known as BobBall_AWOL | 17:37 | |
| *** jsavak has joined #openstack-infra | 17:38 | |
| *** vgridnev has joined #openstack-infra | 17:40 | |
| *** jsavak has quit IRC | 17:41 | |
| *** jsavak has joined #openstack-infra | 17:42 | |
| *** Swami has joined #openstack-infra | 17:42 | |
| *** kzaitsev_mb has quit IRC | 17:43 | |
| *** dizquierdo has quit IRC | 17:43 | |
| *** Guest58505 has quit IRC | 17:44 | |
| *** yamahata has joined #openstack-infra | 17:45 | |
| *** bpokorny_ has joined #openstack-infra | 17:46 | |
| *** bpokorny_ has quit IRC | 17:47 | |
| *** bpokorny_ has joined #openstack-infra | 17:47 | |
| *** eliqiao_ has quit IRC | 17:48 | |
| *** zeih_ has quit IRC | 17:48 | |
| *** bpokorny has quit IRC | 17:50 | |
| *** eliqiao_ has joined #openstack-infra | 17:50 | |
| *** achanda has joined #openstack-infra | 17:50 | |
| *** _nadya_ has quit IRC | 17:52 | |
| *** bnemec has quit IRC | 17:52 | |
| *** tzn has joined #openstack-infra | 17:56 | |
| *** jordanP has quit IRC | 17:56 | |
| *** lucasagomes is now known as lucas-dinner | 17:57 | |
| *** derekh has quit IRC | 17:57 | |
| *** exploreshaifali has joined #openstack-infra | 18:00 | |
| *** jsavak has quit IRC | 18:01 | |
| *** matrohon has joined #openstack-infra | 18:02 | |
| *** tzn has quit IRC | 18:02 | |
| *** jsavak has joined #openstack-infra | 18:03 | |
| *** rlandy has quit IRC | 18:05 | |
| *** shashank_hegde has joined #openstack-infra | 18:06 | |
| *** SumitNaiksatam has quit IRC | 18:07 | |
| *** asettle has joined #openstack-infra | 18:07 | |
| *** jcoufal has quit IRC | 18:07 | |
| *** bnemec has joined #openstack-infra | 18:08 | |
| *** markusz has quit IRC | 18:08 | |
| *** jpr has joined #openstack-infra | 18:09 | |
| *** _nadya_ has joined #openstack-infra | 18:10 | |
| *** ccrouch1 has joined #openstack-infra | 18:10 | |
| *** tphummel has joined #openstack-infra | 18:11 | |
| *** vgridnev has quit IRC | 18:11 | |
| *** Liuqing has quit IRC | 18:12 | |
| *** ccrouch has quit IRC | 18:12 | |
| *** asettle has quit IRC | 18:12 | |
| *** vgridnev has joined #openstack-infra | 18:12 | |
| *** vgridnev has quit IRC | 18:13 | |
| *** austin81 has left #openstack-infra | 18:13 | |
| *** amotoki has quit IRC | 18:13 | |
| *** vgridnev has joined #openstack-infra | 18:14 | |
| *** BobBall_AWOL has quit IRC | 18:14 | |
| *** vgridnev has quit IRC | 18:14 | |
| *** wolverineav has joined #openstack-infra | 18:15 | |
| *** rlandy has joined #openstack-infra | 18:15 | |
| *** vgridnev has joined #openstack-infra | 18:15 | |
| *** vivekd has quit IRC | 18:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Extract ansible playbook running to a library https://review.openstack.org/282397 | 18:16 |
| AJaeger | fungi, I do not see any conflicts for your change 282425 - cool! | 18:16 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Move Neutron Grenade multi from experimental to check queue https://review.openstack.org/282428 | 18:17 |
| *** chlong_ has quit IRC | 18:18 | |
| *** wolverineav has quit IRC | 18:19 | |
| *** smarcet has quit IRC | 18:19 | |
| *** matrohon has quit IRC | 18:20 | |
| *** wolverineav has joined #openstack-infra | 18:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable coverage non-voting for networking-cisco https://review.openstack.org/281956 | 18:22 |
| *** dims_ has joined #openstack-infra | 18:22 | |
| *** _sarob has joined #openstack-infra | 18:22 | |
| *** annegentle has joined #openstack-infra | 18:23 | |
| *** smarcet has joined #openstack-infra | 18:23 | |
| *** ddieterly has quit IRC | 18:24 | |
| *** ddieterly has joined #openstack-infra | 18:24 | |
| *** achanda has quit IRC | 18:25 | |
| *** dims has quit IRC | 18:25 | |
| *** imcsk8 has quit IRC | 18:25 | |
| *** kuromagi has quit IRC | 18:25 | |
| *** chem has quit IRC | 18:25 | |
| *** taron2 has quit IRC | 18:25 | |
| *** vponomaryov has quit IRC | 18:25 | |
| *** bodepd has quit IRC | 18:25 | |
| *** tonyb has quit IRC | 18:25 | |
| *** jesusaur has quit IRC | 18:25 | |
| *** cinerama has quit IRC | 18:25 | |
| *** olaph has quit IRC | 18:25 | |
| *** abramley has quit IRC | 18:25 | |
| *** krotscheck has quit IRC | 18:25 | |
| *** acabot has quit IRC | 18:25 | |
| *** skolekonov has quit IRC | 18:25 | |
| *** mikal has quit IRC | 18:25 | |
| *** Aegil has quit IRC | 18:25 | |
| *** dmellado has quit IRC | 18:25 | |
| *** cp16net has quit IRC | 18:25 | |
| *** Mmike has quit IRC | 18:25 | |
| *** pasquier-s has quit IRC | 18:25 | |
| *** bookwar has quit IRC | 18:25 | |
| *** clayton has quit IRC | 18:25 | |
| *** ruhe has quit IRC | 18:25 | |
| *** _ody has quit IRC | 18:25 | |
| *** mkoderer__ has quit IRC | 18:25 | |
| *** MCoLo has quit IRC | 18:25 | |
| *** lyarwood has quit IRC | 18:25 | |
| *** wznoinsk has quit IRC | 18:25 | |
| *** denisra has quit IRC | 18:25 | |
| *** psanchez has quit IRC | 18:25 | |
| *** Aegil has joined #openstack-infra | 18:25 | |
| *** _ody has joined #openstack-infra | 18:25 | |
| *** cp16net has joined #openstack-infra | 18:25 | |
| *** cinerama has joined #openstack-infra | 18:25 | |
| *** skolekonov has joined #openstack-infra | 18:25 | |
| *** bodepd has joined #openstack-infra | 18:25 | |
| *** mikal has joined #openstack-infra | 18:25 | |
| *** bookwar has joined #openstack-infra | 18:25 | |
| *** imcsk8 has joined #openstack-infra | 18:25 | |
| *** olaph has joined #openstack-infra | 18:25 | |
| *** denisra has joined #openstack-infra | 18:25 | |
| *** krotscheck has joined #openstack-infra | 18:25 | |
| *** acabot has joined #openstack-infra | 18:25 | |
| *** MCoLo has joined #openstack-infra | 18:26 | |
| *** wznoinsk has joined #openstack-infra | 18:26 | |
| *** dmellado has joined #openstack-infra | 18:26 | |
| *** pasquier-s has joined #openstack-infra | 18:26 | |
| *** abramley has joined #openstack-infra | 18:26 | |
| *** vponomaryov has joined #openstack-infra | 18:26 | |
| *** ruhe has joined #openstack-infra | 18:26 | |
| *** Mmike has joined #openstack-infra | 18:26 | |
| *** jesusaur has joined #openstack-infra | 18:26 | |
| *** taron2 has joined #openstack-infra | 18:26 | |
| *** kuromagi has joined #openstack-infra | 18:26 | |
| *** clayton has joined #openstack-infra | 18:26 | |
| *** claudiub has quit IRC | 18:26 | |
| *** mkoderer__ has joined #openstack-infra | 18:27 | |
| *** Daisy has joined #openstack-infra | 18:27 | |
| *** gordc has quit IRC | 18:27 | |
| *** psanchez has joined #openstack-infra | 18:27 | |
| *** rakhmerov has quit IRC | 18:27 | |
| *** mkj_wk has quit IRC | 18:27 | |
| *** pc_m has quit IRC | 18:27 | |
| *** bradjones has quit IRC | 18:27 | |
| *** raginbajin has quit IRC | 18:27 | |
| *** rhedlind has quit IRC | 18:27 | |
| *** SamYaple has quit IRC | 18:27 | |
| *** jkt has quit IRC | 18:27 | |
| *** _diana_ has quit IRC | 18:27 | |
| *** f1ller has quit IRC | 18:27 | |
| *** frickler has quit IRC | 18:27 | |
| *** nihilifer has quit IRC | 18:27 | |
| *** blogan has quit IRC | 18:27 | |
| *** mrodden has quit IRC | 18:27 | |
| *** eantyshev has quit IRC | 18:27 | |
| *** cfarquhar has quit IRC | 18:27 | |
| *** lindsayk has quit IRC | 18:27 | |
| *** jmccrory has quit IRC | 18:27 | |
| *** tobiash has quit IRC | 18:27 | |
| *** lifeless has quit IRC | 18:27 | |
| *** purp has quit IRC | 18:27 | |
| *** sshnaidm has quit IRC | 18:27 | |
| *** christiana has quit IRC | 18:27 | |
| *** laurelm has quit IRC | 18:27 | |
| *** frickler has joined #openstack-infra | 18:27 | |
| *** jkt has joined #openstack-infra | 18:27 | |
| *** jmccrory_ has joined #openstack-infra | 18:27 | |
| *** SamYaple has joined #openstack-infra | 18:27 | |
| *** mkj_wk has joined #openstack-infra | 18:27 | |
| *** mrodden_ has joined #openstack-infra | 18:27 | |
| *** pc_m has joined #openstack-infra | 18:27 | |
| *** cfarquhar has joined #openstack-infra | 18:27 | |
| *** cfarquhar has quit IRC | 18:27 | |
| *** cfarquhar has joined #openstack-infra | 18:27 | |
| *** nihilifer has joined #openstack-infra | 18:27 | |
| *** f1ller has joined #openstack-infra | 18:27 | |
| *** raginbaj- has joined #openstack-infra | 18:27 | |
| *** lifeless has joined #openstack-infra | 18:27 | |
| *** purp_too has joined #openstack-infra | 18:27 | |
| *** blogan has joined #openstack-infra | 18:27 | |
| *** david-lyle has joined #openstack-infra | 18:27 | |
| *** mrodden_ is now known as mrodden | 18:27 | |
| *** rakhmerov has joined #openstack-infra | 18:27 | |
| *** laurelm has joined #openstack-infra | 18:28 | |
| *** eantyshev has joined #openstack-infra | 18:28 | |
| *** jmccrory_ is now known as jmccrory | 18:28 | |
| *** bradjones has joined #openstack-infra | 18:28 | |
| *** bradjones has quit IRC | 18:28 | |
| *** bradjones has joined #openstack-infra | 18:28 | |
| *** rhedlind has joined #openstack-infra | 18:28 | |
| *** clayton_ has joined #openstack-infra | 18:28 | |
| *** clayton has quit IRC | 18:28 | |
| *** _diana_ has joined #openstack-infra | 18:28 | |
| *** purp_too is now known as purp | 18:28 | |
| *** tobiash has joined #openstack-infra | 18:28 | |
| *** raginbaj- is now known as raginbajin | 18:29 | |
| *** flaper87 has quit IRC | 18:29 | |
| *** flaper87 has joined #openstack-infra | 18:29 | |
| *** annegentle has quit IRC | 18:29 | |
| *** gordc has joined #openstack-infra | 18:29 | |
| *** clayton_ is now known as clayton | 18:29 | |
| *** lyarwood has joined #openstack-infra | 18:30 | |
| *** tonyb has joined #openstack-infra | 18:31 | |
| *** Daisy has quit IRC | 18:31 | |
| *** HeOS has quit IRC | 18:33 | |
| *** HeOS has joined #openstack-infra | 18:34 | |
| *** lindsayk has joined #openstack-infra | 18:34 | |
| pleia2 | tsk, missed Daisy (been working on the puppet-translation_checksite this week) | 18:35 |
| *** kushal has quit IRC | 18:35 | |
| *** apoorvad has joined #openstack-infra | 18:36 | |
| *** nwkarsten has joined #openstack-infra | 18:37 | |
| *** sridhar_ram has joined #openstack-infra | 18:38 | |
| *** mriedem has quit IRC | 18:39 | |
| pleia2 | AJaeger: sorry for not following up on the scripts email thread (and I wish more of this was on list, the private i18n email threads we always have make my brain sad) | 18:39 |
| *** jpr has quit IRC | 18:40 | |
| openstackgerrit | Merged openstack/requirements: bump upper constraint for keystonemiddleware https://review.openstack.org/281028 | 18:40 |
| *** annegentle has joined #openstack-infra | 18:40 | |
| openstackgerrit | Merged openstack/requirements: Constrain eventlet to 0.18.2 for now https://review.openstack.org/281479 | 18:40 |
| AJaeger | pleia2: there're so few working on this that I go to small list by default - feel free to kick me everytime and ask for public discussion ;/ | 18:40 |
| *** mriedem has joined #openstack-infra | 18:41 | |
| pleia2 | AJaeger: I think it would help all of us if we could nudge these back into public | 18:41 |
| AJaeger | pleia2: -dev or -infra? | 18:41 |
| *** electrofelix has quit IRC | 18:41 | |
| pleia2 | AJaeger: -infra | 18:41 |
| *** nwkarsten has quit IRC | 18:41 | |
| AJaeger | Ok, I'll write an email | 18:41 |
| pleia2 | thank you :) | 18:41 |
| *** infra-red has joined #openstack-infra | 18:42 | |
| *** achanda has joined #openstack-infra | 18:43 | |
| *** jpr has joined #openstack-infra | 18:44 | |
| pleia2 | AJaeger: right-o, no need to add us! | 18:45 |
| *** amitgandhinz has joined #openstack-infra | 18:46 | |
| *** ccrouch1 has quit IRC | 18:46 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Increase undercloud swap to 2 GB https://review.openstack.org/282462 | 18:47 |
| *** dtardivel has quit IRC | 18:48 | |
| *** kushal has joined #openstack-infra | 18:48 | |
| *** doug-fish has quit IRC | 18:48 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements https://review.openstack.org/278654 | 18:49 |
| *** infra-re_ has joined #openstack-infra | 18:50 | |
| *** doug-fish has joined #openstack-infra | 18:50 | |
| *** sdake_ is now known as sdake | 18:50 | |
| *** infra-red has quit IRC | 18:51 | |
| *** amitgandhinz has quit IRC | 18:51 | |
| *** slogan_r has joined #openstack-infra | 18:51 | |
| *** mrmartin has joined #openstack-infra | 18:52 | |
| *** nwkarsten has joined #openstack-infra | 18:52 | |
| *** doug-fish has quit IRC | 18:55 | |
| *** _nadya_ has quit IRC | 18:56 | |
| *** rcernin has quit IRC | 18:56 | |
| slogan_r | AJaeger, yolanda: if you have a moment, https://review.openstack.org/#/c/282365/ is looking for +2 - has Jenkins +1, and Monasca project PTL +1 already. Thanks! | 18:57 |
| slogan_r | (I hope I have the correct yolanda) | 18:57 |
| *** rossella_s has quit IRC | 18:58 | |
| slogan_r | Ajaeger, yolanda, all: thanks for helping make https://github.com/openstack/broadview-lib a reality! | 18:58 |
| *** wolverineav has quit IRC | 18:58 | |
| *** rossella_s has joined #openstack-infra | 18:58 | |
| *** annegentle has quit IRC | 18:58 | |
| *** annegentle has joined #openstack-infra | 18:59 | |
| *** mrmartin has quit IRC | 18:59 | |
| slogan_r | anteaya: thanks for the helpful comments as well, much appreciated, and sorry I was a pest bugging you for help :-) | 18:59 |
| AJaeger | slogan_r: why monasca PTL +1? Will it be part of monasca? | 18:59 |
| slogan_r | It will be generating metrics directly to monasca. We are hoping to have a joint presentation on the results in Austin, if we get enough votes that is | 19:00 |
| AJaeger | slogan_r: I'll review it eventually but not anymore today | 19:00 |
| *** annegentle has quit IRC | 19:00 | |
| slogan_r | AJaeger: no problem, I'm a few days from doing the cookiecutter and having the code ready anyway | 19:01 |
| slogan_r | s/I'm/We're/ | 19:01 |
| *** vkmc has joined #openstack-infra | 19:01 | |
| *** annegentle has joined #openstack-infra | 19:01 | |
| *** bpokorny_ has quit IRC | 19:01 | |
| *** bpokorny has joined #openstack-infra | 19:02 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Get more info from ps command https://review.openstack.org/282508 | 19:02 |
| *** jsavak has quit IRC | 19:02 | |
| *** jsavak has joined #openstack-infra | 19:03 | |
| slogan_r | AJeager: there will be a third project that extends Horizon to configure all of this. Anyway, I'll ping again in a week or so, no worries. | 19:03 |
| AJaeger | slogan_r: no need to ping any of us - reviews happen eventually ;) | 19:04 |
| *** infra-re_ has quit IRC | 19:04 | |
| *** bpokorny has quit IRC | 19:04 | |
| *** bpokorny has joined #openstack-infra | 19:04 | |
| slogan_r | AJaeger: nod, that's good to know | 19:04 |
| *** sdague has quit IRC | 19:05 | |
| *** dimtruck is now known as zz_dimtruck | 19:05 | |
| * slogan_r hates being a pest | 19:05 | |
| *** zz_dimtruck is now known as dimtruck | 19:05 | |
| *** slogan_r is now known as slogan_r_lab | 19:05 | |
| AJaeger | slogan_r: if it gets urgent, that's different - but we always plan in buffer, don't we? ;) | 19:06 |
| *** annegentle has quit IRC | 19:06 | |
| slogan_r_lab | yeah, we would have teed these last to project creates up earlier, but it was our first experience with it. We learned exactly what to do with openstack/broadview-lib and are doing the same config for the next two, so, hopefully just one patchset this time (instead of 7) | 19:07 |
| slogan_r_lab | s/to project/two project/ | 19:07 |
| AJaeger | ;) | 19:08 |
| slogan_r_lab | clear to me that infra has a giant job on its shoulders, so buffering is a really good idea | 19:08 |
| apoorvad | Hello! On the following review https://review.openstack.org/#/c/268370/ I am trying to see what exactly failed in pep8 but I get "File Not Found" | 19:09 |
| apoorvad | Here http://logs.openstack.org/70/268370/5/check/gate-cinder-pep8/b8625c8/ | 19:09 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove redundant access configuration https://review.openstack.org/281461 | 19:10 |
| AJaeger | apoorvad: a known problem we're trying to solve right now. Please "recheck" in this case. | 19:10 |
| apoorvad | AJaeger: I did and it failed again | 19:11 |
| *** _sarob has quit IRC | 19:12 | |
| AJaeger | we fixed the issue of the disappearing log files since then - try once more, please | 19:12 |
| AJaeger | the change merged 5 hours ago or so - but your recheck was earlier | 19:13 |
| *** shashank_hegde has quit IRC | 19:13 | |
| apoorvad | AJaeger: cool thanks! I will do it once aagin | 19:13 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Move Neutron Grenade multi from experimental to check queue https://review.openstack.org/282428 | 19:13 |
| *** eil397 has joined #openstack-infra | 19:13 | |
| sc68cal | AJaeger: ^ fixed depends-on line | 19:14 |
| *** achanda has quit IRC | 19:14 | |
| *** matrohon has joined #openstack-infra | 19:15 | |
| *** wolverineav has joined #openstack-infra | 19:15 | |
| *** esker has joined #openstack-infra | 19:15 | |
| *** erlon has quit IRC | 19:16 | |
| fungi | back now, catching up | 19:16 |
| AJaeger | sc68cal: thanks | 19:17 |
| sc68cal | np thank you | 19:17 |
| AJaeger | fungi: could you humour zaro and +2A 281461 before we have to rebase it the third time, please? | 19:17 |
| *** _sarob has joined #openstack-infra | 19:18 | |
| *** ildikov has joined #openstack-infra | 19:18 | |
| *** amitgandhinz has joined #openstack-infra | 19:18 | |
| fungi | sure, i never find enough humor in my day | 19:18 |
| *** e0ne has quit IRC | 19:18 | |
| AJaeger | do we have any stats about the zuul-cloner/network status? Is it getting better or worse? | 19:19 |
| fungi | AJaeger: it's just cleaning up the acl for the openstack/openstack repo i guess? awesome | 19:19 |
| AJaeger | fungi: yes | 19:20 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: support 2.0 branch telemetry jobs https://review.openstack.org/282511 | 19:20 |
| fungi | AJaeger: if we're no longer losing console logs on git clone failures for constraints-based tox jobs, then probably we can do some more useful stats via logstash now | 19:21 |
| *** mrmartin has joined #openstack-infra | 19:22 | |
| *** mrmartin has quit IRC | 19:23 | |
| *** rcernin has joined #openstack-infra | 19:23 | |
| *** e0ne has joined #openstack-infra | 19:23 | |
| *** e0ne has quit IRC | 19:25 | |
| *** ddieterly has quit IRC | 19:28 | |
| *** jcooley_ has quit IRC | 19:28 | |
| *** matrohon has quit IRC | 19:29 | |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: Fix Gerrit test site generation https://review.openstack.org/282515 | 19:32 |
| *** sdague has joined #openstack-infra | 19:33 | |
| *** ddieterly has joined #openstack-infra | 19:33 | |
| pleia2 | "A box of t-shirts just came in for you" :D | 19:33 |
| pleia2 | we have sprint swag | 19:33 |
| *** ddieterl_ has joined #openstack-infra | 19:34 | |
| *** ildikov has quit IRC | 19:35 | |
| *** ddieterly has quit IRC | 19:36 | |
| *** andreykurilin__ has joined #openstack-infra | 19:36 | |
| *** |-paul-| has joined #openstack-infra | 19:37 | |
| fungi | now we just need a sprint! | 19:37 |
| fungi | fingers crossed the universe doesn't end over the weekend | 19:37 |
| patrickeast | anyone have ideas on why jobs wouldn't be getting registered with gearman after restarting zuul? i've got nodepool nodes attached to jenkins that are still running from before the restart, not sure what else might prevent them from being registered | 19:37 |
| patrickeast | in the past i've only seen this when there aren't any jenkins slaves... but there are a bunch of them | 19:38 |
| fungi | patrickeast: i think in the past we've seen jobs only get registered when new workers are added to jenkins or when the gearman plugin is disabled and enabled again | 19:38 |
| sdake | quick q, why can't I vote on this review: https://review.openstack.org/#/c/275840/14 | 19:38 |
| patrickeast | fungi: ahh ok, lemmie check those, thanks! | 19:38 |
| sdake | I am able to vote on every other review in the queue | 19:38 |
| pleia2 | taking lunch | 19:38 |
| sdake | is it because its going through a check atm? | 19:38 |
| fungi | patrickeast: i think it doesn't end up exposed for us very often just due to the rate of node churn we get | 19:38 |
| patrickeast | fungi: oh yea, sure enough as soon as nodepool created a new one and attached it they got registered | 19:39 |
| fungi | sdake: try voting on the latest patchset | 19:39 |
| sdake | oh duh | 19:39 |
| fungi | sdake: that url is going to an older patchset (14) | 19:39 |
| fungi | latest patchset is 15 | 19:39 |
| sdake | thanks fungi | 19:40 |
| fungi | np | 19:40 |
| *** ociuhandu has quit IRC | 19:41 | |
| *** annegentle has joined #openstack-infra | 19:42 | |
| *** asettle has joined #openstack-infra | 19:44 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Update InfraCloud SSL configuration https://review.openstack.org/279375 | 19:45 |
| *** rcernin has quit IRC | 19:45 | |
| *** ccrouch has joined #openstack-infra | 19:47 | |
| *** ddieterl_ has quit IRC | 19:48 | |
| *** ddieterly has joined #openstack-infra | 19:48 | |
| *** asettle has quit IRC | 19:48 | |
| *** jsavak has quit IRC | 19:50 | |
| *** burgerk has joined #openstack-infra | 19:50 | |
| *** jsavak has joined #openstack-infra | 19:51 | |
| *** ddieterl_ has joined #openstack-infra | 19:51 | |
| *** asettle has joined #openstack-infra | 19:51 | |
| *** asettle has quit IRC | 19:51 | |
| fungi | crinkle: do i need to add keys to hiera for 279375 or are they already in place? | 19:52 |
| fungi | commit message says they're needed, so i assume that's still the case | 19:53 |
| *** ddieterly has quit IRC | 19:53 | |
| fungi | oh, also depends on 279286 | 19:53 |
| crinkle | fungi: yes they are needed | 19:53 |
| crinkle | fungi: sorry should have clarified in that comment | 19:54 |
| fungi | which is not approved yet, but will review that one too | 19:54 |
| *** EricGonc_ has quit IRC | 19:54 | |
| fungi | did you have content somewhere you need copied into hiera, or shall i generate fresh values for this? | 19:54 |
| *** annegentle has quit IRC | 19:55 | |
| *** annegentle has joined #openstack-infra | 19:55 | |
| *** mrmartin has joined #openstack-infra | 19:55 | |
| *** EricGonczer_ has joined #openstack-infra | 19:55 | |
| crinkle | fungi: i generated a new cert/key pair that I sent to nibalizer but I don't think he's had time to add them yet | 19:56 |
| fungi | in that case i'll coordinate with him | 19:56 |
| fungi | thanks! | 19:56 |
| nibalizer | i have them decrypted | 19:56 |
| nibalizer | i haven't slottend them into hiera yet | 19:56 |
| nibalizer | i can do that if it is GoTime(tm) | 19:56 |
| AJaeger | pc_m: do you want to abandon https://review.openstack.org/#/c/255992/ and https://review.openstack.org/#/c/256391 ? I don't see you continuing with them - correct? | 19:57 |
| fungi | nibalizer: i can make it gotime, though 279375 will need workflow +1 toggled/amended since i didn't spot it had a cross-repo dependency that wasn't approved yet | 19:57 |
| fungi | i'm still reviewing the dependency | 19:57 |
| nibalizer | fungi: please find a cert/key in ~root/infracloud on puppetmaster | 19:58 |
| pc_m | AJaeger: Yeah. Will do. I'm redoing things following a different procedure (using non-constraint names for constraints targets) | 19:58 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Update InfraCloud SSL configuration https://review.openstack.org/279375 | 19:58 |
| AJaeger | pc_m: and https://review.openstack.org/256702 as well | 19:58 |
| *** flepied1 has joined #openstack-infra | 19:58 | |
| AJaeger | pc_m: yes, that was my impression... | 19:58 |
| *** jsavak has quit IRC | 19:59 | |
| fungi | nibalizer: indeed--i see it | 19:59 |
| *** tzn has joined #openstack-infra | 19:59 | |
| *** flepied has quit IRC | 19:59 | |
| *** asettle has joined #openstack-infra | 19:59 | |
| *** jsavak has joined #openstack-infra | 20:00 | |
| pc_m | AJaeger: I think I got them all. | 20:00 |
| pc_m | Thanks | 20:00 |
| *** davidlenwell has quit IRC | 20:01 | |
| *** annegent_ has joined #openstack-infra | 20:02 | |
| *** gordc has quit IRC | 20:02 | |
| *** annegentle has quit IRC | 20:02 | |
| *** annegent_ has quit IRC | 20:02 | |
| *** max_lobur has quit IRC | 20:02 | |
| *** annegentle has joined #openstack-infra | 20:03 | |
| *** tzn has quit IRC | 20:03 | |
| *** annegentle has quit IRC | 20:04 | |
| fungi | nibalizer: crinkle: according to the commit message on 279375 only hpuswest_ssl_cert_file_contents needs to be in the infracloud group, but ssl_key_file_contents goes in the shared master list? or where? | 20:05 |
| fungi | if ssl_key_file_contents is a global hiera key, it's sort of ambiguous | 20:05 |
| *** davidlenwell has joined #openstack-infra | 20:06 | |
| fungi | name-wise | 20:06 |
| *** annegentle has joined #openstack-infra | 20:06 | |
| *** dahoo has joined #openstack-infra | 20:06 | |
| *** dahoo is now known as jerryz | 20:06 | |
| crinkle | fungi: it should go in the fqdn file since it only need to be known to that one server | 20:06 |
| *** ddieterl_ has quit IRC | 20:07 | |
| fungi | aha, thanks. the controller00.hpuswest.ic.openstack.org file then? | 20:07 |
| crinkle | fungi: yes | 20:07 |
| crinkle | fungi: OR i can rename it to be less ambiguous and then it can go anywhere | 20:07 |
| *** ddieterly has joined #openstack-infra | 20:07 | |
| fungi | nah, this is fine | 20:08 |
| *** ddieterl_ has joined #openstack-infra | 20:08 | |
| AJaeger | thanks, pc_m | 20:08 |
| *** kushal has quit IRC | 20:08 | |
| fungi | crinkle: we can always adjust later if it's really confusing anyone | 20:08 |
| *** burgerk has quit IRC | 20:08 | |
| crinkle | fungi: okay | 20:08 |
| pc_m | AJaeger: np. I was being lazy. Needed to clean up. Once I get VPN done, I'm going to work on the other *aaS and neutron-lib. Having issues with functional test passing. Will have to dig into it Monday. | 20:09 |
| fungi | hrm, these have been hand-edited and are not alphabetized | 20:09 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Update SSL configuration https://review.openstack.org/279286 | 20:09 |
| AJaeger | pc_m: might this be a difference between using constraints and not using them? So, getting a different version breaking you? | 20:10 |
| pc_m | AJaeger: Not sure. The func test targets were converted to have -constraints suffix. I set up the non-constraints to do the same thing as constraints, and then removed the suffix (in post_test), but seeing test failures. | 20:12 |
| pc_m | AJaeger: There are some configuration scripts, so I'm not sure if I'm missing something. Will look into it Monday, when I'm fresh. It's been a long day already. | 20:13 |
| AJaeger | have a restfull weekend, pc_m ! | 20:14 |
| pc_m | AJaeger: Thanks. And Thanks for the reviews this week! | 20:15 |
| *** EricGonc_ has joined #openstack-infra | 20:17 | |
| *** ldnunes has quit IRC | 20:17 | |
| pc_m | AJaeger: Actually, I think there is still something odd in the gate. 281914, which fixes config script has been in the gate for a while and seems to keep getting re-run. | 20:17 |
| *** eharney has quit IRC | 20:18 | |
| *** sc68cal has quit IRC | 20:18 | |
| pc_m | AJaeger: It passed Jenkins yesterday, and now in gate, but I see it running again and again. :( | 20:18 |
| fungi | pc_m: what do you mean rerun? | 20:19 |
| *** ccrouch has quit IRC | 20:19 | |
| fungi | like failed to merge, then approved again? | 20:19 |
| pc_m | I see it in gate queue passing tests.... and then later I see it in the queue re-running tests it had already passed, so must be retrying? | 20:19 |
| fungi | looks like it wasn't approved until a couple hours ago | 20:19 |
| *** MarkAtwood has quit IRC | 20:20 | |
| pc_m | fungi: Yeah, but I saw all tests passed and full running. Now full, and the two functional are running. | 20:20 |
| fungi | pc_m: were there changes ahead of it in the gate failing? if so, they were removed from the testing scenario for it and tests had to be rerun to see if they still passed with those other changes removed | 20:20 |
| pc_m | fungi: Ah. Could be. | 20:20 |
| crinkle | fungi: nibalizer apache failed during the puppet run on the controller, i'm looking at it | 20:20 |
| *** burgerk has joined #openstack-infra | 20:20 | |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: Fix Gerrit test site generation https://review.openstack.org/282515 | 20:21 |
| *** EricGonczer_ has quit IRC | 20:21 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:22 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove redundant access configuration https://review.openstack.org/281461 | 20:22 |
| crinkle | fungi: nibalizer oh, because the system-config change wasn't merged yet when puppet ran | 20:23 |
| *** sridhar_ram has quit IRC | 20:23 | |
| *** rkukura has joined #openstack-infra | 20:26 | |
| *** kzaitsev_mb has quit IRC | 20:26 | |
| *** ccrouch has joined #openstack-infra | 20:28 | |
| *** burgerk has quit IRC | 20:29 | |
| *** burgerk has joined #openstack-infra | 20:29 | |
| *** dims_ has quit IRC | 20:31 | |
| fungi | https://review.openstack.org/282409 merged almost 4 hours ago and is still not live on (at least) jenkins01. i'm going to start digging into jjb runs on our jenkins masters, but will likely end up needing to manually update them again | 20:32 |
| *** asettle has quit IRC | 20:32 | |
| *** BobBall_AWOL has joined #openstack-infra | 20:33 | |
| zaro | fungi: thanks for suggestion on https://review.openstack.org/#/c/282515/, is that your preference? or just an alternative? | 20:34 |
| *** ccrouch has quit IRC | 20:34 | |
| *** christiana has joined #openstack-infra | 20:34 | |
| fungi | zaro: well, it avoids importing yet another module to do random number generation | 20:34 |
| *** jsavak has quit IRC | 20:34 | |
| *** jsavak has joined #openstack-infra | 20:35 | |
| fungi | i'm really not sure why the address selection was written the way it was, but it looked like the original author was reinventing a modulus operator poorly | 20:35 |
| fungi | zaro: but ultimately, whatever works (and avoids collisions to a small enough margin of error) is fine | 20:35 |
| *** achanda has joined #openstack-infra | 20:35 | |
| zaro | ok. i'll leave as is then. | 20:36 |
| fungi | yeah, bit-shifting the pid to get an octet was sort of a strange choice | 20:36 |
| zaro | fungi: the tests for git-review doesn't work on my mac due to my network config. was wondering if you could help? | 20:37 |
| *** ccrouch has joined #openstack-infra | 20:37 | |
| fungi | i don't really know much about networking on a mac | 20:37 |
| fungi | does installing gerrit on a mac usually work? | 20:37 |
| *** _sarob has quit IRC | 20:38 | |
| zaro | it seems like something to do with netmask setting and maybe not mac specific. basically my problem is here: https://groups.google.com/forum/#!topic/repo-discuss/I6j_guUcuEQ | 20:38 |
| zaro | fungi: yeah, just about all the gerrit guys dev on a mac. | 20:39 |
| *** annegentle has quit IRC | 20:39 | |
| zaro | fungi: i've checked my netmask and it seems like it's something given out by the dhcp server, not something you can set yourself correct? | 20:40 |
| *** tzn has joined #openstack-infra | 20:40 | |
| *** rhallisey has quit IRC | 20:40 | |
| *** aysyd has quit IRC | 20:40 | |
| fungi | i'd be surprised if that gerrit functional test did anything to your network interfaces besides the loopback interface | 20:41 |
| *** annegentle has joined #openstack-infra | 20:41 | |
| fungi | zaro: this may be a linuxism | 20:42 |
| fungi | modern linux kernels have all of 127.*.*.* mapped to the loopback interface | 20:42 |
| *** _sarob has joined #openstack-infra | 20:42 | |
| openstackgerrit | Merged openstack-infra/system-config: Update InfraCloud SSL configuration https://review.openstack.org/279375 | 20:42 |
| fungi | mac os x/darwin (and perhaps most/all *bsds) only use 127.0.0.1 on the loopback | 20:43 |
| zaro | on my VM the 'lo' interface is set to 127.0.0.1 and mask is '255.0.0.0' | 20:43 |
| fungi | i should say _may_ only use | 20:43 |
| fungi | i'm looking for confirmation now | 20:43 |
| AJaeger | project config cores, could one more review https://review.openstack.org/281665 and https://review.openstack.org/281097 , please? | 20:44 |
| fungi | like, if you ping 127.1.47.7 on a bsd do you get a response? you do on linux | 20:44 |
| *** ccrouch has quit IRC | 20:44 | |
| *** sdake has quit IRC | 20:44 | |
| *** eharney has joined #openstack-infra | 20:44 | |
| fungi | i have some openbsd boxes here i'll test on. closest thing to mac os x within reach | 20:44 |
| *** rfolco has quit IRC | 20:44 | |
| zaro | ahh, your right. | 20:45 |
| fungi | ping: sendto: Network is unreachable | 20:45 |
| *** annegentle has quit IRC | 20:45 | |
| fungi | so yeah, i wouldn't expect that job, as designed, to work anywhere besides linux | 20:45 |
| zaro | i see i have a 'lo0' interface on my mac netmask is set to 0xff000000 | 20:45 |
| fungi | the netmask alone doesn't mean it's listening on that entire address range | 20:46 |
| fungi | on linux, it's a special behavior in the lo driver | 20:46 |
| fungi | not really a network configuration | 20:46 |
| *** claudiub has joined #openstack-infra | 20:47 | |
| *** tzn has quit IRC | 20:47 | |
| *** ccrouch has joined #openstack-infra | 20:47 | |
| zaro | hmm, ok. no way to make test work on mac then? | 20:47 |
| *** _sarob has quit IRC | 20:47 | |
| fungi | probably not unless it grows an option to serialize the tests so that only one gerrit is running at a time, and always on 127.0.0.1 | 20:48 |
| fungi | though that will probably make it take a lot longer | 20:48 |
| fungi | or some way to make all the gerrits listen on 127.0.0.1 but on different port numbers... that would likely get a lot more complex to configure git-review for in each separate scenario | 20:49 |
| pabelanger | AJaeger: add some +1s :) | 20:49 |
| AJaeger | thanks, pabelanger ! | 20:49 |
| *** _sarob has joined #openstack-infra | 20:50 | |
| zaro | argh! thanks fungi. | 20:52 |
| *** ayoung has quit IRC | 20:52 | |
| *** bauwser is now known as bauzas | 20:52 | |
| asselin_ | what's the best way to add nodes for a new provider to production in nodepool? Is there a way to stage it? | 20:52 |
| *** apoorvad has quit IRC | 20:53 | |
| *** flepied has joined #openstack-infra | 20:54 | |
| *** regXboi has quit IRC | 20:54 | |
| *** thorst has quit IRC | 20:54 | |
| *** thorst has joined #openstack-infra | 20:54 | |
| asselin_ | or just set max servers to 1 to limit possible errors? | 20:55 |
| *** _sarob has quit IRC | 20:55 | |
| *** tzn has joined #openstack-infra | 20:55 | |
| *** _sarob has joined #openstack-infra | 20:55 | |
| *** flepied1 has quit IRC | 20:56 | |
| *** mrmartin has quit IRC | 20:56 | |
| *** burgerk has quit IRC | 20:56 | |
| fungi | asselin_: that's basically what we do | 20:57 |
| asselin_ | fungi, thanks | 20:57 |
| openstackgerrit | Merged openstack-infra/shade: Remove mock testing of os-client-config for swift https://review.openstack.org/282339 | 20:58 |
| fungi | asselin_: you can also set it so that you generate nodes with an otherwise unused label there, and then run manual tests with the assurance they won't get assigned any real jobs | 20:58 |
| *** asettle has joined #openstack-infra | 20:58 | |
| *** asettle has quit IRC | 20:58 | |
| fungi | configure them identically to your normal production nodes, just with a different label | 20:59 |
| *** ccrouch has quit IRC | 20:59 | |
| asselin_ | ack | 20:59 |
| *** thorst has quit IRC | 20:59 | |
| fungi | aha! i think https://review.openstack.org/282034 is the reason why jjb hadn't updated on any of our jenkins masters for hours | 21:01 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Redirect undercloud install output to file https://review.openstack.org/282531 | 21:01 |
| fungi | also explains why the manual updates i'm babysitting now are taking so long | 21:01 |
| *** amrith is now known as _amrith_ | 21:01 | |
| *** zeih has joined #openstack-infra | 21:02 | |
| *** ccrouch has joined #openstack-infra | 21:02 | |
| *** sarob_ has joined #openstack-infra | 21:02 | |
| fungi | yolanda: AJaeger: job config changes like that which touch a bunch of jobs take way longer to apply than the timeout we put on automated jjb updates | 21:03 |
| AJaeger | fungi: argh, yes ;( sure that touches lots of projects and we should have checked directly. Sorry, didn't think to flag it | 21:03 |
| fungi | no worries, was pretty easy to spot once i started thinking about it | 21:03 |
| * AJaeger has this on the radar recently - but totally forgot on this change ;( | 21:03 | |
| AJaeger | fungi: time to call it a weekend! | 21:04 |
| * AJaeger waves good-bye... | 21:04 | |
| fungi | well, would have helped if the commit message included a reminder | 21:04 |
| fungi | have a good weekend AJaeger! | 21:04 |
| *** markvan has quit IRC | 21:04 | |
| AJaeger | thanks, fungi. You as well! | 21:04 |
| *** _sarob has quit IRC | 21:04 | |
| *** taron2 is now known as taron | 21:05 | |
| *** ddieterl_ has quit IRC | 21:05 | |
| *** ddieterly has quit IRC | 21:05 | |
| *** tzn has quit IRC | 21:05 | |
| *** markvan has joined #openstack-infra | 21:05 | |
| *** matrohon has joined #openstack-infra | 21:05 | |
| *** nwkarsten has quit IRC | 21:06 | |
| zaro | fungi: it seems you can set an alias to the loopback interface using ifconfig -alias command | 21:06 |
| *** BobBall_AWOL has quit IRC | 21:07 | |
| fungi | zaro: yeah, that's a possible workaround. directly bind every address you need to the loopback, though that involves knowing the addresses in advance and doing things with root permissions | 21:07 |
| fungi | i think the current version of that job manages to get away with not doing anything as root/with sudo | 21:08 |
| zaro | fungi: yeah, planning to just add doc to git-review to let users know and they can do it manually | 21:08 |
| *** exploreshaifali has quit IRC | 21:08 | |
| zaro | that is mac users | 21:08 |
| fungi | my wife has two macbooks and two iphones (one each for business vs personal use), and she's been instructed in no uncertain terms that i am not to be asked to provide her with technical support ;) | 21:10 |
| zaro | fungi: not sure how unlikely, but not even if she was hacking on openstack? | 21:12 |
| *** shashank_hegde has joined #openstack-infra | 21:12 | |
| *** mrmartin has joined #openstack-infra | 21:12 | |
| fungi | probably not, no | 21:12 |
| *** notmorgan is now known as morgan | 21:13 | |
| fungi | <- not a fan of proprietary operating systems | 21:13 |
| bkero | fungi: you likely have a nice thinkpad waiting for her when the macbooks break or she wants to play a flac file? :) | 21:14 |
| fungi | bkero: no thinkpad. i tend to travel with netbooks and use desktops the rest of the time | 21:15 |
| bkero | Ahh ok | 21:16 |
| bkero | fungi: ever use a Vaio P? | 21:16 |
| *** e0ne has joined #openstack-infra | 21:16 | |
| fungi | i had a vaio slim some years ago. one of the 11" models. before there were netbooks | 21:17 |
| *** jcooley_ has joined #openstack-infra | 21:17 | |
| fungi | ran debian just great | 21:17 |
| bkero | nice | 21:17 |
| *** apoorvad has joined #openstack-infra | 21:17 | |
| bkero | I remember that z505 slim being the thinnest laptop in the world | 21:17 |
| bkero | might still be...superwedge | 21:18 |
| fungi | indeed, that was the model i had | 21:18 |
| fungi | i think it was actually thicker than my wife's macbook air though | 21:18 |
| *** nwkarsten has joined #openstack-infra | 21:18 | |
| *** nwkarsten has quit IRC | 21:18 | |
| *** achanda has quit IRC | 21:18 | |
| fungi | but probably not by more than a mm or two | 21:18 |
| *** KimJ has joined #openstack-infra | 21:18 | |
| *** |-paul-| has quit IRC | 21:19 | |
| *** nwkarsten has joined #openstack-infra | 21:19 | |
| *** abregman|nb is now known as abregman|afk | 21:19 | |
| * morgan enjoys a lunch beer. clarkb we shall need to beer and food soon. | 21:20 | |
| *** e0ne has quit IRC | 21:21 | |
| *** achanda has joined #openstack-infra | 21:21 | |
| *** jcooley_ has quit IRC | 21:22 | |
| *** jcooley_ has joined #openstack-infra | 21:22 | |
| bkero | fungi: I've been kind of sad by netbooks of late. Always with the gimped keyboards. | 21:22 |
| bkero | morgan: just lunchbeered. Always a good decision in Portland. :) | 21:23 |
| morgan | bkero: YES! | 21:23 |
| bkero | The People's Pig had an Ex Novo IPA on tap today. Very nice with a pile of smoked meat. | 21:24 |
| morgan | I'm actually doing the left over pizza, warmed in a skillet and some... deschuttes beer from my beer cabinet. | 21:24 |
| bkero | Mmm | 21:25 |
| bkero | homelunchbeer also good | 21:25 |
| *** Guest8259 has joined #openstack-infra | 21:25 | |
| *** ddieterly has joined #openstack-infra | 21:25 | |
| *** ddieterl_ has joined #openstack-infra | 21:25 | |
| *** [1]Thelo has joined #openstack-infra | 21:25 | |
| morgan | bkero: it seems very windy out, so i just felt the "stay home" vibe. also i just got news that my couch will be delivered on monday. yay nice furniture | 21:25 |
| bkero | Dwellings are always improved with the presence of a couch :) | 21:26 |
| morgan | bkero: yes! | 21:26 |
| *** rlandy has quit IRC | 21:26 | |
| morgan | and in this case, a couch *and* a nice chair | 21:26 |
| bkero | A nice compliment | 21:26 |
| morgan | yup. now i just need an end table and i'm all set. furniture wise. ok maybe i need another couple wall-arts | 21:27 |
| morgan | but otherwise i has a home, and it's nice :) | 21:27 |
| crinkle | glance on infracloud is sad, "*** stack smashing detected ***: /usr/bin/python terminated" | 21:27 |
| pleia2 | I have not seen that one before | 21:27 |
| *** Thelo has quit IRC | 21:27 | |
| *** [1]Thelo is now known as Thelo | 21:28 | |
| morgan | crinkle: whoa | 21:28 |
| morgan | crinkle: thats... a new one | 21:28 |
| bkero | import hax; import glibc; glibc.getaddrinfo(hax) | 21:28 |
| morgan | bkero: import ctypes; ctypes.do_evil_things(); | 21:29 |
| *** lucas-dinner has quit IRC | 21:29 | |
| *** shardy has quit IRC | 21:31 | |
| *** |-paul-| has joined #openstack-infra | 21:31 | |
| *** lucasagomes has joined #openstack-infra | 21:35 | |
| *** vgridnev has quit IRC | 21:35 | |
| *** wolverineav has quit IRC | 21:37 | |
| *** verdurin has quit IRC | 21:38 | |
| *** wolverineav has joined #openstack-infra | 21:43 | |
| *** edmondsw has quit IRC | 21:44 | |
| *** sridhar_ram has joined #openstack-infra | 21:46 | |
| *** tpsilva has quit IRC | 21:46 | |
| *** e0ne has joined #openstack-infra | 21:47 | |
| *** wolverineav has quit IRC | 21:48 | |
| *** e0ne has quit IRC | 21:48 | |
| *** nwkarsten has quit IRC | 21:50 | |
| nibalizer | crinkle: nice | 21:50 |
| nibalizer | crinkle: so is that every time glance tries to start ? | 21:51 |
| crinkle | nibalizer: yes, and the upstart script is freaking out so it tries to start about once a second | 21:52 |
| nibalizer | eeep | 21:54 |
| crinkle | it definitely has to do with the new cert, its aborting somewhere in the middle of trying to sslwrap the port | 21:54 |
| crinkle | er socket | 21:55 |
| nibalizer | hrm | 21:56 |
| nibalizer | so does glance run behind apache? | 21:56 |
| nibalizer | looks kinda like | 21:56 |
| nibalizer | no | 21:56 |
| *** kgiusti has left #openstack-infra | 21:56 | |
| crinkle | no | 21:57 |
| *** taron has quit IRC | 21:57 | |
| openstackgerrit | Merged openstack-infra/shade: Add a method to download an image from glance https://review.openstack.org/278816 | 21:57 |
| fungi | is it possible i garbled the cert when adding it to hiera? | 21:58 |
| fungi | is it truncated or anything (missing begin/end tags)? | 21:58 |
| fungi | also openssl x509 should be able to read the cert i think | 21:59 |
| fungi | testing now with the copy on puppetmaster | 21:59 |
| *** taron has joined #openstack-infra | 22:00 | |
| nibalizer | i can read the cert on the controller00 with openssl | 22:00 |
| crinkle | the cert looks fine, verified with openssl x509, plus this isn't an issue with nova-api or neutron-server | 22:00 |
| fungi | Subject: C=US, ST=OpenStack, O=OpenStack Infrastructure Team, CN=controller00.hpuswest.ic.openstack.org/emailAddress=openstack-infra@lists.openstack.org | 22:00 |
| fungi | lgtm | 22:00 |
| fungi | just making sure. never attribute to bugs what could be explained by my sloppiness at the keyboard, after all | 22:01 |
| nibalizer | crinkle: the key is -rw------ and the cert is -r--r--r- is that intentional? | 22:01 |
| nibalizer | -rw------- 1 glance root even | 22:01 |
| nibalizer | which is sortof an odd perm | 22:02 |
| crinkle | 600 and 444 is intentional | 22:02 |
| fungi | jenkins05 finished jjb updates in just over 80 minutes. the others are still running | 22:03 |
| *** salv-orl_ has joined #openstack-infra | 22:03 | |
| *** matrohon has quit IRC | 22:03 | |
| crinkle | user glance is intentional, user root is sort of a side effect of not caring about the group | 22:03 |
| *** salv-orlando has quit IRC | 22:06 | |
| fungi | 87 minutes for jjb updates to complete on jenkins07 | 22:06 |
| *** asettle has joined #openstack-infra | 22:07 | |
| *** calebb has quit IRC | 22:07 | |
| nibalizer | crinkle: how can i see the error you are seeing? | 22:07 |
| *** calebb has joined #openstack-infra | 22:08 | |
| *** jsavak has quit IRC | 22:09 | |
| *** maximov_ has quit IRC | 22:09 | |
| fungi | 91 minutes for jjb updates to complete on jenkins03 | 22:09 |
| *** ybathia has joined #openstack-infra | 22:09 | |
| crinkle | nibalizer: dmesg | tail, tail /var/log/upstart/glance-api.log | 22:09 |
| *** maximov_ has joined #openstack-infra | 22:09 | |
| crinkle | this is where it's happening http://paste.openstack.org/show/487635/ | 22:10 |
| *** moravec has quit IRC | 22:10 | |
| crinkle | so something wrong with the key | 22:10 |
| openstackgerrit | Merged openstack/requirements: Use working Kafka version https://review.openstack.org/282147 | 22:11 |
| *** Guest8259 has quit IRC | 22:11 | |
| *** asettle has quit IRC | 22:11 | |
| *** annegentle has joined #openstack-infra | 22:12 | |
| *** changbl has quit IRC | 22:12 | |
| *** annegentle has quit IRC | 22:12 | |
| *** annegentle has joined #openstack-infra | 22:12 | |
| *** annegentle has quit IRC | 22:13 | |
| *** jsavak has joined #openstack-infra | 22:13 | |
| nibalizer | til /var/log/upstart is a thing | 22:13 |
| *** annegentle has joined #openstack-infra | 22:14 | |
| fungi | 94 minutes for jenkins04 | 22:14 |
| fungi | still waiting for jenkins01, 02 and 06 to finish | 22:15 |
| nibalizer | fungi: did you use the ansible playbook? | 22:15 |
| nibalizer | we have one of those now | 22:15 |
| nibalizer | also 'ugh' | 22:16 |
| fungi | nibalizer: haven't used it yet, and also with yolanda noticing corrupt jjb caches several times in the past week i wanted to make sure i didn't start it if there was already a jjb update in progress | 22:16 |
| nibalizer | ok | 22:16 |
| fungi | i'll give it a try sometime when it's less time-sensitive | 22:17 |
| nibalizer | crinkle: huh | 22:18 |
| *** notnownikki_ has joined #openstack-infra | 22:18 | |
| *** mrmartin has quit IRC | 22:18 | |
| nibalizer | i mean, I don't know how to debug that | 22:19 |
| fungi | openssl should be able to tell you if the key and cert correspond | 22:19 |
| clarkb | nibalizer: that playbook doesnt get installed last I checked | 22:19 |
| nibalizer | clarkb: oh? we didn't shove that in /playbooks in system-config? | 22:19 |
| clarkb | so I had a thing running before I eealized where it was hiding | 22:19 |
| fungi | though i don't recall which openssl subcommand you need for that | 22:20 |
| nibalizer | fungi: i did this thing | 22:20 |
| clarkb | nibalizer: yes it is in the repo but not in /etc/ansible/playbooks | 22:20 |
| nibalizer | wait no i ran openssl rsa -text -noout -in | 22:20 |
| nibalizer | which verified that its a real cert | 22:20 |
| openstackgerrit | Merged openstack/requirements: Bump python-muranoclient to a 0.8.2 mitaka version https://review.openstack.org/281544 | 22:20 |
| nibalizer | also in crinkle's paste there is verification that the key doesn't work,the cert isn't even recognized | 22:20 |
| *** ccrouch1 has joined #openstack-infra | 22:22 | |
| fungi | nibalizer: i don't see where in that paste there's anything at all going on with the cert. that's entirely just the file with the key | 22:22 |
| *** notnownikki has quit IRC | 22:22 | |
| *** lascii is now known as alaski | 22:22 | |
| nibalizer | fungi: yea thats what i was trying to say | 22:23 |
| fungi | tries to use the key to sign a uuid, *bam* crashes | 22:23 |
| nibalizer | i just bad at english | 22:23 |
| nibalizer | 'isnt even recognized'probably should have been 'isnt even considered' | 22:23 |
| fungi | sure, i was mostly trying to come up with ways to confirm the key itself is a usable key | 22:23 |
| fungi | as you say, checking it with openssh rsa should have been sufficient | 22:24 |
| *** ccrouch has quit IRC | 22:24 | |
| *** sarob_ has quit IRC | 22:24 | |
| fungi | though maybe trying to use openssl rsa to sign something with that key would be a way to validate, or get better debugging | 22:25 |
| fungi | looks like it's tickling something in libopenssl and python is falling over before it can convey any helpful details | 22:25 |
| crinkle | i wonder if the key is too big? | 22:27 |
| fungi | nibalizer: what keysize did you use when you generated it? | 22:27 |
| fungi | 4096? | 22:27 |
| crinkle | it is 8192 bytes | 22:27 |
| *** zeih has quit IRC | 22:28 | |
| *** Daisy has joined #openstack-infra | 22:30 | |
| pabelanger | that is big | 22:30 |
| fungi | no obvious bug reports anywhere about crashing crypto.sign() with an 8192-bit rsa key | 22:32 |
| pabelanger | okay, calling it for today. If I don't pop on to work tomorrow looking forward to seeing eveybody in Fort Collins monday morning | 22:34 |
| fungi | see you there, pabelanger! | 22:34 |
| *** sdake has joined #openstack-infra | 22:34 | |
| *** nwkarsten has joined #openstack-infra | 22:34 | |
| *** nwkarsten has quit IRC | 22:35 | |
| *** Daisy has quit IRC | 22:35 | |
| *** amitgandhinz has quit IRC | 22:35 | |
| rm_work | man, pip_install is killing me: https://gist.github.com/rm-you/9e8d889d458b0164b693 T_T | 22:36 |
| *** dims has joined #openstack-infra | 22:36 | |
| rm_work | builds numpy like four times, each at almost 5 minutes <_< | 22:36 |
| rm_work | anyone have any tricks to make it cache? | 22:36 |
| *** _sarob has joined #openstack-infra | 22:36 | |
| *** _sarob has quit IRC | 22:37 | |
| fungi | rm_work: this is one reason we build a wheel mirror | 22:37 |
| *** _sarob has joined #openstack-infra | 22:37 | |
| rm_work | hmm | 22:37 |
| *** nwkarsten has joined #openstack-infra | 22:37 | |
| *** _sarob has quit IRC | 22:37 | |
| rm_work | so if I set up my own wheel mirror, I can point devstack at it? | 22:37 |
| *** mriedem has quit IRC | 22:37 | |
| fungi | yeah | 22:37 |
| rm_work | is that a localrc option or something | 22:37 |
| rm_work | ? | 22:37 |
| fungi | it's pip configuration | 22:37 |
| rm_work | ah. yeah makes sense. can I ... use yours? is it public? | 22:37 |
| *** _sarob has joined #openstack-infra | 22:37 | |
| *** KimJ has quit IRC | 22:37 | |
| *** notnownikki_ has quit IRC | 22:38 | |
| rm_work | seems like that'd be the best option anyway as it'd be guaranteed closer to the gate | 22:38 |
| fungi | it's a distributed mirror in the various cloud providers where we have workers. systems other than ours shouldn't depend on those since we could rename/move/reorganize them at any time without warning | 22:38 |
| rm_work | hmm k | 22:38 |
| rm_work | how likely is that actually to happen? | 22:38 |
| rm_work | because i can just adjust my scripts if it does | 22:38 |
| rm_work | not putting anything in any upstream repos that'd use that | 22:39 |
| crinkle | fungi: nibalizer i think it is the key size, i generated a 4096 one and it seems okay | 22:39 |
| fungi | rm_work: pretty likely. we had lots of third-party ci operators confused this week because they were using some old configuration pointed to an old pypi mirror we just took offline | 22:39 |
| rm_work | heh | 22:39 |
| fungi | it becomes a support burden on us when we have to help people who were depending on things we didn't mean to be used outside our systems | 22:40 |
| *** changbl has joined #openstack-infra | 22:40 | |
| rm_work | well ... if we were to assume i'd be willing to take that risk personally and sign away any rights to complain, where could I find the url for that? :P | 22:40 |
| nibalizer | crinkle: ah ok | 22:40 |
| nibalizer | well im fine with 4096 bit encryption | 22:40 |
| fungi | crinkle: i wonder if that was an issue in old pyopenssl versions? you said that environment's on kilo, yeah? | 22:41 |
| fungi | though in theory it's libopenssl handling that call | 22:41 |
| fungi | so maybe not | 22:41 |
| crinkle | fungi: yes it is kilo | 22:41 |
| *** nwkarsten has quit IRC | 22:41 | |
| fungi | rm_work: i thought devstack's pip usage was able to create and take advantage of a local wheel cache anyway | 22:42 |
| rm_work | hmm | 22:42 |
| fungi | rm_work: are you maybe on an older version of pip? | 22:42 |
| rm_work | well it doesn't seem to be able to | 22:42 |
| rm_work | hmmmmmmmmmmm actually i wonder | 22:42 |
| rm_work | what version of pip would that require? | 22:42 |
| fungi | rm_work: it was a non-default behavior of pip until a version or two ago i think. checking but 8 should definitely do it, 7 may as well | 22:43 |
| rm_work | also, I assume this is the basics: https://github.com/openstack-infra/project-config/blob/master/nodepool/scripts/configure_mirror.sh#L24 | 22:43 |
| rm_work | yeah seem to be on 8 | 22:43 |
| crinkle | fungi: nibalizer do one of you want to generate a new key/cert to put in hiera or want me to send you the one i just generated? | 22:43 |
| rm_work | 8.0.2 | 22:43 |
| rm_work | i wonder if there's a flag i have to set in localrc to let it do that | 22:43 |
| fungi | rm_work: yeah. you might want to see why it's not caching (or not reusing) a wheel for numpy on your system. #openstack-qa might have people who can help figure that out too | 22:43 |
| rm_work | cool, thanks fungi :) really appreciate the advice | 22:44 |
| *** achanda has quit IRC | 22:44 | |
| *** tzn has joined #openstack-infra | 22:44 | |
| fungi | crinkle: i'm fine doing either, though need to know the special constraint config you were using for that (was with ca=true i think?) | 22:44 |
| *** achanda has joined #openstack-infra | 22:45 | |
| *** abregman|afk has quit IRC | 22:46 | |
| crinkle | fungi: i haven't changed anything in my openssl.cnf from the default configuration on my system, i see basicConstraints=CA:FALSE in mine | 22:46 |
| crinkle | fungi: oh! i see basicConstraints = CA:true in the [v3_ca] section, maybe that's the magic one | 22:47 |
| fungi | crinkle: oh, i thought yolanda's testing indicated you needed that true | 22:47 |
| fungi | ahh, yep, in v3 | 22:47 |
| *** tzn has quit IRC | 22:48 | |
| *** smarcet has quit IRC | 22:49 | |
| *** achanda has quit IRC | 22:50 | |
| *** ccrouch1 has quit IRC | 22:50 | |
| *** david-lyle has quit IRC | 22:50 | |
| *** david-lyle has joined #openstack-infra | 22:51 | |
| *** nwkarsten has joined #openstack-infra | 22:51 | |
| *** ccrouch has joined #openstack-infra | 22:52 | |
| nibalizer | pabelanger: https://twitter.com/roidelapluie/status/700815071585419268 | 22:52 |
| nibalizer | facter 3 comming to el | 22:52 |
| *** tzn has joined #openstack-infra | 22:52 | |
| *** ociuhandu has joined #openstack-infra | 22:53 | |
| *** david-lyle has quit IRC | 22:54 | |
| *** david-lyle has joined #openstack-infra | 22:54 | |
| *** nwkarsten has quit IRC | 22:56 | |
| *** davideagnello has quit IRC | 23:00 | |
| cody-somerville | hmm... the fact that our pypi mirrors are in a subdirectory is kind of annoying. | 23:01 |
| cody-somerville | Is there a particular reason we did that? | 23:01 |
| fungi | cody-somerville: because they coexist beside wheel mirrors, deb mirrors, soon rpm and npm mirrors... | 23:02 |
| fungi | cody-somerville: so it was either sub-trees or a bunch of additional dns names to separate vhosts | 23:03 |
| *** annegentle has quit IRC | 23:04 | |
| cody-somerville | subdomain would be nice so that paths are the same as pypi.python.org though it isn't a big deal -- I'll just have to special case some code I'm writing. | 23:04 |
| *** achanda has joined #openstack-infra | 23:04 | |
| *** annegentle has joined #openstack-infra | 23:04 | |
| *** dimtruck is now known as zz_dimtruck | 23:05 | |
| clarkb | the paths are the same just at different roots | 23:05 |
| fungi | cody-somerville: could probably look up the mirror path in the pip.conf if this is something running on a worker | 23:05 |
| bkero | name-based vhosts are hard | 23:05 |
| *** |-paul-| has quit IRC | 23:06 | |
| fungi | it's not so much that name-based vhosts are hard, it's that updating dns for every new type of content we want to host on those machines is a questionable burden for a minimal amount of return | 23:06 |
| cody-somerville | Yea. I'm just using existing code to do something that expects path to be the same across all hosts. Will just need to work around it. | 23:06 |
| bkero | fungi: how often do you add new content types? | 23:06 |
| fungi | bkero: at least recently, every week or two | 23:07 |
| fungi | and we don't have orchestrated dns management | 23:07 |
| fungi | the volume will probably die down, but off the top of my head there's at least 5 vhosts we'd be serving on each of our mirror hosts (we're up to 8 of those i think? i'm probably missing one or two) | 23:09 |
| fungi | jenkins06 jjb update completed at 101 minutes. jenkins01 and 02 are still going but look close to finishing | 23:10 |
| *** annegentle has quit IRC | 23:10 | |
| fungi | gah! 158 minutes into updating, jjb crashed on a jenkins api error on jenkins02 | 23:14 |
| *** ddieterl_ has quit IRC | 23:14 | |
| *** ddieterly has quit IRC | 23:14 | |
| fungi | though jenkins01 finally completed at 155 minutes | 23:14 |
| fungi | and hopefully 02 won't be very long on its rerun | 23:15 |
| cody-somerville | One thing I've noticed though is that it seems like pip takes consistently around 100-300ms longer to run pip install --ignore-installed on a single package that is already in download cache when using our pypi mirrors vs. pypi.python.org despite no meaningful difference in ping time or download time. | 23:15 |
| *** jpr has quit IRC | 23:16 | |
| cody-somerville | Does pypi do anything special to speed up serving index information? | 23:16 |
| fungi | it's possible that's due to the fact that we're using a 2gb virtual machine running apache in front of an afs filesystem cache rather than a large caching cdn (pypi.p.o uses fastly) | 23:16 |
| cody-somerville | fungi: yea but it isn't downloading anything in this case except indexes. and I did tests of ping and download time for files from pypi and that don't stand out as noticeably faster or anything. | 23:18 |
| *** jtomasek has quit IRC | 23:19 | |
| clarkb | the index is served the same way | 23:19 |
| *** spiffxp_ has joined #openstack-infra | 23:21 | |
| *** spiffxp has quit IRC | 23:21 | |
| *** sdague has quit IRC | 23:21 | |
| *** vipul- has joined #openstack-infra | 23:22 | |
| *** SlickNik has quit IRC | 23:22 | |
| *** roxanaghe has joined #openstack-infra | 23:22 | |
| *** xyang1 has quit IRC | 23:22 | |
| *** vipuls has quit IRC | 23:22 | |
| *** annegentle has joined #openstack-infra | 23:22 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:22 | |
| *** SlickNik has joined #openstack-infra | 23:23 | |
| *** ociuhandu has quit IRC | 23:23 | |
| *** annegentle has quit IRC | 23:25 | |
| *** spiffxp_ has quit IRC | 23:25 | |
| *** tzn has quit IRC | 23:25 | |
| *** EricGonc_ has quit IRC | 23:25 | |
| *** jsavak has quit IRC | 23:26 | |
| crinkle | fungi: nibalizer the cloud is still broken because of the bad key, how can i help get a new one into hiera? | 23:26 |
| *** vipul- has quit IRC | 23:27 | |
| nibalizer | crinkle: if you send me a new one i will put it in hiera | 23:27 |
| *** SlickNik has quit IRC | 23:27 | |
| *** spiffxp has joined #openstack-infra | 23:27 | |
| *** roxanagh_ has joined #openstack-infra | 23:27 | |
| crinkle | nibalizer: okay | 23:27 |
| *** EricGonczer_ has joined #openstack-infra | 23:27 | |
| fungi | crinkle: sorry, was trying to get expense reports caught up. i'll do key generation now and then get back to that, sorry1 | 23:27 |
| fungi | oh, nibalizer is on the case. nm! | 23:27 |
| crinkle | okay | 23:27 |
| *** EricGonczer_ has quit IRC | 23:27 | |
| *** vipuls has joined #openstack-infra | 23:27 | |
| clarkb | Ishoulddo expense reports but this baby plague has me hidinf from computers | 23:28 |
| *** SlickNik has joined #openstack-infra | 23:28 | |
| *** |-paul-| has joined #openstack-infra | 23:28 | |
| fungi | there are probably faster ways to expense receipts than this old flat-bed scanner | 23:29 |
| *** david-lyle_ has joined #openstack-infra | 23:29 | |
| *** EricGonczer_ has joined #openstack-infra | 23:29 | |
| crinkle | nibalizer: sent to your email | 23:29 |
| cody-somerville | interestingly, if I set http://pypi.python.org/simple/ as index-url in pip config it takes longer like our mirrors as well. | 23:29 |
| *** david-lyle has quit IRC | 23:30 | |
| *** roxanaghe has quit IRC | 23:30 | |
| cody-somerville | so weird. must be something in pip that is taking longer when you manually set index-url. | 23:31 |
| fungi | pip's default indexing may be something other than crawling the simple index | 23:31 |
| nibalizer | cody-somerville: ask | 23:32 |
| fungi | i believe pypi.p.o has a "smart index" of some sort | 23:32 |
| nibalizer | infra-root taking the hiera lock | 23:32 |
| nibalizer | crinkle: are neutron and glance and nova all the same key? | 23:33 |
| nibalizer | and cert | 23:33 |
| crinkle | nibalizer: yes | 23:33 |
| *** zz_dimtruck is now known as dimtruck | 23:33 | |
| nibalizer | so can we store that just once | 23:33 |
| fungi | the other hiera keys for those are cruft now, right? | 23:34 |
| crinkle | nibalizer: you should only need to change two values, hpuswest_ssl_cert_file_contents and ssl_key_file_contents | 23:34 |
| fungi | and also the one for the chain certs? | 23:34 |
| crinkle | nibalizer: fungi yes the others are cruft now | 23:34 |
| nibalizer | so i can zero the chain? | 23:34 |
| fungi | chain certs key being cruft also, i mean | 23:34 |
| nibalizer | or the chain is now the cert that you sent me? | 23:34 |
| crinkle | the chain is cruft | 23:34 |
| fungi | there is no chain, only snakeoil | 23:34 |
| *** sdake has quit IRC | 23:35 | |
| *** rhallisey has joined #openstack-infra | 23:36 | |
| *** pradk has quit IRC | 23:37 | |
| nibalizer | so all we need is 'ssl_key_file_contents' and and 'hpuswest_ssl_cert_file_contents' | 23:37 |
| nibalizer | ? | 23:37 |
| crinkle | nibalizer: yes | 23:37 |
| fungi | yeah, that's all i installed last time | 23:37 |
| nibalizer | so why did we neamespace only one of those | 23:38 |
| fungi | because there's going to be one key shared by both east and west | 23:38 |
| crinkle | nibalizer: well the key only belongs to one server | 23:38 |
| *** SlickNik has quit IRC | 23:38 | |
| fungi | oh! | 23:38 |
| fungi | i totally misunderstood why it wasn't namespaced. now i get it | 23:38 |
| fungi | because it's only going in the fqdn hiera file | 23:38 |
| crinkle | nibalizer: fungi if it is too confusing we can change it | 23:39 |
| *** spiffxp has quit IRC | 23:39 | |
| *** vipuls has quit IRC | 23:39 | |
| *** vipul- has joined #openstack-infra | 23:39 | |
| fungi | while the infracloud group file will have certs for more than one cloud | 23:39 |
| nibalizer | ohhhh | 23:39 |
| nibalizer | well yea we have this cert file | 23:39 |
| nibalizer | which is public data | 23:39 |
| *** SlickNik has joined #openstack-infra | 23:39 | |
| nibalizer | and the keyfile which is not | 23:39 |
| fungi | the cert could in theory even go in public hiera | 23:39 |
| nibalizer | what would you say to putting the cert into public hiera | 23:39 |
| fungi | heh | 23:39 |
| crinkle | hahaha | 23:40 |
| nibalizer | "now that ive started to understand this id like to add more complexity" | 23:40 |
| *** spiffxp has joined #openstack-infra | 23:40 | |
| fungi | eventually i'd like to do that for all certs | 23:40 |
| fungi | but let's save that for later | 23:40 |
| nibalizer | fungi: well note that this cert file has to go into system-config /anyways/ | 23:40 |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: Add documentation about running tests on Macs https://review.openstack.org/282571 | 23:40 |
| nibalizer | or multiple places since nodepool and puppetmaster need it | 23:40 |
| fungi | great point | 23:41 |
| fungi | i forgot about needing to puppet it onto nodepool | 23:41 |
| *** dimtruck is now known as zz_dimtruck | 23:44 | |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: enable test with ODL release of boron-snapshot https://review.openstack.org/282572 | 23:44 |
| *** Daisy has joined #openstack-infra | 23:44 | |
| *** harlowja_at_home has joined #openstack-infra | 23:44 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add hpuswest public certificate to public hiera https://review.openstack.org/282573 | 23:46 |
| nibalizer | okay infra-root hiera lock released | 23:47 |
| nibalizer | fungi: crinkle 282573 adds it to public hiera | 23:47 |
| *** Daisy has quit IRC | 23:49 | |
| crinkle | nibalizer: is it currently also in private hiera? | 23:49 |
| nibalizer | yes | 23:49 |
| nibalizer | no | 23:49 |
| nibalizer | the private key is in private hiera | 23:50 |
| *** yamamoto_ has quit IRC | 23:50 | |
| *** yamamoto has joined #openstack-infra | 23:51 | |
| crinkle | nibalizer: should we rename it to 'infracloud_hpuswest_ssl_cert_file_contents' if it's going in common.yaml? | 23:51 |
| nibalizer | ya probably | 23:52 |
| nibalizer | sec | 23:52 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add hpuswest public certificate to public hiera https://review.openstack.org/282573 | 23:53 |
| *** yamamoto has quit IRC | 23:56 | |
| *** rbrndt_ has quit IRC | 23:56 | |
| *** sridhar_ram1 has joined #openstack-infra | 23:57 | |
| *** jcooley_ has quit IRC | 23:57 | |
| *** sridhar_ram has quit IRC | 23:58 | |
| crinkle | nibalizer: commented | 23:58 |
| nibalizer | heh | 23:58 |
| nibalizer | compute servers | 23:58 |
| nibalizer | who needs em | 23:58 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add hpuswest public certificate to public hiera https://review.openstack.org/282573 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!