| *** baoli has quit IRC | 00:02 | |
| *** tiswanso has quit IRC | 00:04 | |
| *** achanda has joined #openstack-infra | 00:06 | |
| openstackgerrit | Kai Qiang Wu(Kennan) proposed openstack-infra/project-config: Add docs and publish jobs for kolla https://review.openstack.org/218137 | 00:07 |
|---|---|---|
| jamielennox | lifeless: when using pip install .[dep] syntax here https://github.com/openstack/keystone/blob/master/tox.ini#L10-L14 my (previously ~empty) 3gb /tmp partition gets filled and then the process crashes, works if i do those deps normally | 00:07 |
| jamielennox | lifeless: is this something you've seen before? | 00:07 |
| lifeless | jamielennox: you're filling 3GB installing keystone! | 00:10 |
| lifeless | jamielennox: so a) no. b) what uses the sapce up ? | 00:11 |
| jamielennox | lifeless: it appears to be a /tmp/pip-XXX-build | 00:11 |
| jamielennox | if i swap out the .[XXX] syntax with the packages they actually point to it doesn't happen though | 00:12 |
| lifeless | yeah - whats in there? | 00:12 |
| *** stevemar has quit IRC | 00:13 | |
| *** yushiro has joined #openstack-infra | 00:13 | |
| *** dimtruck is now known as zz_dimtruck | 00:15 | |
| jamielennox | gah, tmp fills everything dies | 00:16 |
| jamielennox | sec | 00:16 |
| *** sflanigan has joined #openstack-infra | 00:19 | |
| *** sflanigan has joined #openstack-infra | 00:19 | |
| *** oomichi has joined #openstack-infra | 00:21 | |
| jamielennox | lifeless: ok, http://paste.openstack.org/show/434120/ | 00:22 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 00:22 | |
| lifeless | 994M./pip-rqs5G_-build/.testrepository | 00:23 |
| lifeless | thats odd | 00:23 |
| jamielennox | lifeless: so i ran that after rm -rf /tmp/pip-* | 00:23 |
| jamielennox | lifeless: right - and it's creating 3 of them | 00:23 |
| lifeless | something is running tox within that | 00:23 |
| lifeless | and checking out from git... | 00:24 |
| lifeless | jamielennox: so this is super weird :) | 00:24 |
| jamielennox | oh good :) | 00:25 |
| lifeless | jamielennox: possibly your pip debug log would help, but I'm mostly worried about wtf tox is doing in there | 00:25 |
| jamielennox | this change seems to have happened a while ago, i haven't heard of it as a problem from other keystone people | 00:25 |
| lifeless | like | 00:25 |
| jamielennox | but it's been hitting me for a while | 00:25 |
| lifeless | I wonder | 00:25 |
| lifeless | you're installing .[foo] right ? | 00:26 |
| lifeless | ok, I think I know | 00:26 |
| lifeless | don't do .[foo] .[bar] | 00:26 |
| lifeless | thats bonkers | 00:26 |
| lifeless | .[foo,bar] and you should remove requirements.txt entirely | 00:26 |
| lifeless | since its implied by keystone itself, which is what . refers to | 00:27 |
| lifeless | .[foo] is 'keystone + the keystone foo extra' | 00:27 |
| lifeless | I am not sure why you've got a full copy of your current dir being made | 00:27 |
| jamielennox | so it's doing 3 keystone installs to figure out what is specified in each of the [XXX] reqs | 00:27 |
| jamielennox | ? | 00:27 |
| lifeless | approximately | 00:28 |
| lifeless | and copying the working tree as part of that | 00:28 |
| lifeless | which is where your space usage is coming from | 00:28 |
| lifeless | fixing the redundancy will make things much snappier up front | 00:28 |
| jamielennox | ah, yes my keystone/.testrepository is apparently 994M | 00:28 |
| lifeless | you might want to trim or reset your testr database too | 00:28 |
| jamielennox | nfi why | 00:29 |
| lifeless | there's no built in gc | 00:29 |
| jamielennox | ok, so tox requirements should simply be "-r test-requirements .[XXX,YYY,ZZZ]" and let requirements be sourced from "." | 00:30 |
| jamielennox | and kill my .testrepository | 00:30 |
| jamielennox | lifeless: https://bugs.launchpad.net/keystone/+bug/1490354 if you want to comment on anything | 00:40 |
| openstack | Launchpad bug 1490354 in Keystone "Tox exhausting /tmp partition" [Undecided,New] | 00:40 |
| jamielennox | lifeless: thanks | 00:40 |
| lifeless | yes | 00:43 |
| *** achanda has quit IRC | 00:44 | |
| yushiro | Hi all :) | 00:49 |
| yushiro | I'd like to review https://review.openstack.org/#/c/218067/ . | 00:49 |
| *** bswartz has joined #openstack-infra | 00:52 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add DIB_KEEP_PACKAGE_CACHE variable https://review.openstack.org/208321 | 00:55 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add DIB_USE_PACKAGE_CACHE variable https://review.openstack.org/211434 | 00:55 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add #!/bin/bash to library functions https://review.openstack.org/218100 | 00:55 |
| *** chlong has joined #openstack-infra | 00:55 | |
| *** vivekd has quit IRC | 01:04 | |
| *** vivekd has joined #openstack-infra | 01:05 | |
| *** tbarron has quit IRC | 01:07 | |
| *** legoktm has quit IRC | 01:07 | |
| *** salv-orlando has joined #openstack-infra | 01:07 | |
| *** yuanying-alt has joined #openstack-infra | 01:10 | |
| *** salv-orlando has quit IRC | 01:13 | |
| *** baoli has joined #openstack-infra | 01:13 | |
| *** salv-orlando has joined #openstack-infra | 01:13 | |
| *** yuanying-alt has quit IRC | 01:14 | |
| openstackgerrit | Merged openstack-infra/project-config: Promote horizon selenium test job https://review.openstack.org/218068 | 01:16 |
| *** baoli has quit IRC | 01:19 | |
| *** zz_dimtruck is now known as dimtruck | 01:24 | |
| *** zhenguo has joined #openstack-infra | 01:24 | |
| *** salv-orlando has quit IRC | 01:26 | |
| *** ddieterly has quit IRC | 01:28 | |
| *** dimtruck is now known as zz_dimtruck | 01:29 | |
| *** tiswanso has joined #openstack-infra | 01:29 | |
| *** vivekd has quit IRC | 01:32 | |
| *** baoli has joined #openstack-infra | 01:33 | |
| *** marzif__ has joined #openstack-infra | 01:33 | |
| *** tiswanso has quit IRC | 01:34 | |
| *** kambiz has quit IRC | 01:37 | |
| *** eantyshev has quit IRC | 01:37 | |
| *** marzif_ has quit IRC | 01:37 | |
| *** achanda has joined #openstack-infra | 01:37 | |
| *** flepied has joined #openstack-infra | 01:38 | |
| *** flepied1 has quit IRC | 01:38 | |
| *** jeadre has quit IRC | 01:39 | |
| *** jeadre has joined #openstack-infra | 01:40 | |
| Kennan | hi infra-cores | 01:40 |
| Kennan | Could anyone take some to help review it https://review.openstack.org/#/c/218137/ ? | 01:41 |
| *** kambiz has joined #openstack-infra | 01:43 | |
| *** baoli has quit IRC | 01:46 | |
| *** chlong has quit IRC | 01:49 | |
| *** baoli has joined #openstack-infra | 02:00 | |
| *** armax has joined #openstack-infra | 02:10 | |
| *** yamamoto has quit IRC | 02:12 | |
| *** tbarron has joined #openstack-infra | 02:20 | |
| *** achanda has quit IRC | 02:22 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 02:23 | |
| *** achanda has joined #openstack-infra | 02:24 | |
| *** stevemar has joined #openstack-infra | 02:25 | |
| *** salv-orlando has joined #openstack-infra | 02:28 | |
| *** ddieterly has joined #openstack-infra | 02:29 | |
| *** stevemar has quit IRC | 02:30 | |
| *** ddieterly has quit IRC | 02:33 | |
| *** gildub has joined #openstack-infra | 02:33 | |
| mordred | lifeless: should we spend any mental energy on getting pip to not copy .testrepository as part of copying the working tree? | 02:34 |
| *** asettle has quit IRC | 02:40 | |
| *** salv-orlando has quit IRC | 02:41 | |
| *** asettle has joined #openstack-infra | 02:44 | |
| *** tiswanso has joined #openstack-infra | 02:45 | |
| *** FallenPegasus has joined #openstack-infra | 02:45 | |
| *** tiswanso has quit IRC | 02:50 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 02:53 | |
| *** gildub has quit IRC | 02:56 | |
| *** ddieterly has joined #openstack-infra | 02:56 | |
| *** yamamoto has joined #openstack-infra | 02:58 | |
| *** yuanying-alt has joined #openstack-infra | 02:59 | |
| *** gildub has joined #openstack-infra | 02:59 | |
| *** gildub has quit IRC | 03:00 | |
| openstackgerrit | Merged openstack-infra/puppet-apps_site: Generate assets file from assets.yaml https://review.openstack.org/205882 | 03:02 |
| *** yuanying-alt has quit IRC | 03:04 | |
| *** gildub has joined #openstack-infra | 03:08 | |
| *** larainema has joined #openstack-infra | 03:11 | |
| *** FallenPe_ has joined #openstack-infra | 03:12 | |
| *** FallenPegasus has quit IRC | 03:13 | |
| *** d34dh0r53 has quit IRC | 03:25 | |
| *** eglute has quit IRC | 03:25 | |
| *** Trozz_ has quit IRC | 03:25 | |
| *** sigmavirus24_awa has quit IRC | 03:25 | |
| lifeless | mordred: possibly | 03:25 |
| lifeless | mordred: the issue is that if you don't copy dotfiles, for instance, .git is missing and boom | 03:26 |
| *** jroll has quit IRC | 03:26 | |
| lifeless | mordred: I don't recall why its copied in the first place, which would be the obvious optimisation | 03:26 |
| jhesketh | marzif__: sorry for the delay, I've reviewed 193125 for you | 03:26 |
| jhesketh | Kennan: I've taken a look fyi | 03:27 |
| *** sputnik13 has joined #openstack-infra | 03:27 | |
| mordred | lifeless: yeah | 03:27 |
| Kennan | thanks jhesketh: | 03:27 |
| mordred | lifeless: or, at some point wasn't pip going to move to doing an sdist first then an install of the sdist as part of install? | 03:27 |
| mordred | lifeless: because that way the manifest structure already in place should handle copying/not-copying the right things | 03:27 |
| *** apmelton has quit IRC | 03:28 | |
| *** apmelton has joined #openstack-infra | 03:30 | |
| *** jroll has joined #openstack-infra | 03:31 | |
| *** eglute has joined #openstack-infra | 03:31 | |
| *** Trozz_ has joined #openstack-infra | 03:31 | |
| *** larainema has quit IRC | 03:31 | |
| *** madong_ has joined #openstack-infra | 03:31 | |
| *** d34dh0r53 has joined #openstack-infra | 03:32 | |
| *** sigmavirus24_awa has joined #openstack-infra | 03:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch Horizon to Javascript-jobs https://review.openstack.org/214373 | 03:36 |
| *** links has joined #openstack-infra | 03:40 | |
| *** salv-orlando has joined #openstack-infra | 03:43 | |
| *** FallenPe_ has quit IRC | 03:46 | |
| Kennan | hi jhesketh: | 03:47 |
| Kennan | not sure if it is right channel to ask question about openstack/requirements project | 03:48 |
| Kennan | I found another my code change to just add project to requirement namespace, it hit | 03:48 |
| Kennan | AssertionError: False is not true : Invalid mac address detected. | 03:48 |
| Kennan | it seems nova issue | 03:48 |
| Kennan | I did not think it is related with project added change | 03:49 |
| Kennan | http://logs.openstack.org/44/218144/3/check/gate-tempest-dsvm-full/9a7dc63/console.html | 03:49 |
| *** cody-somerville has joined #openstack-infra | 03:50 | |
| *** cody-somerville has quit IRC | 03:50 | |
| *** cody-somerville has joined #openstack-infra | 03:50 | |
| *** Piet has joined #openstack-infra | 03:55 | |
| *** Piet has quit IRC | 03:56 | |
| *** salv-orlando has quit IRC | 03:56 | |
| *** Piet has joined #openstack-infra | 03:56 | |
| *** legoktm has joined #openstack-infra | 03:58 | |
| *** legoktm has quit IRC | 03:58 | |
| *** legoktm has joined #openstack-infra | 03:58 | |
| *** Piet has quit IRC | 03:59 | |
| *** sputnik13 has quit IRC | 04:04 | |
| *** ddieterly has quit IRC | 04:06 | |
| jhesketh | Kennan: It doesn't look related, but there could be some subtleties in novas dependencies | 04:06 |
| *** baoli has quit IRC | 04:07 | |
| *** bharathm has joined #openstack-infra | 04:12 | |
| openstackgerrit | Steve Baker proposed openstack/requirements: netaddr!=0.7.16 due to MAC validation regression https://review.openstack.org/218720 | 04:17 |
| stevebaker | this change ^ should unwedge gate-heat-python27, which may mean it deserves being put at the head of the queue | 04:19 |
| jhesketh | Kennan: looks like it's hitting a few projects in the last day | 04:20 |
| *** sputnik13 has joined #openstack-infra | 04:21 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/requirements: Encap netaddr-0.7.16 https://review.openstack.org/218721 | 04:22 |
| *** stevemar has joined #openstack-infra | 04:23 | |
| jhesketh | stevebaker, oomichi: looks like you both have fixes for netaddr ^ | 04:24 |
| oomichi | jhesketh: oh, thanks | 04:27 |
| oomichi | jhesketh: I send a tempest side patch also, https://review.openstack.org/#/q/status:open+branch:master+topic:bug/1490368,n,z | 04:27 |
| oomichi | jhesketh, stevebaker: I abondoned my requirement patch and update tempest side based on that. | 04:28 |
| jhesketh | oomichi: cool, thanks | 04:30 |
| oomichi | jhesketh: tempest side patch is done on https://review.openstack.org/#/c/218712/ | 04:32 |
| oomichi | jhesketh: thanks for the notice :) | 04:33 |
| *** sputnik13 has quit IRC | 04:36 | |
| *** sabeen1 has joined #openstack-infra | 04:40 | |
| yushiro | good afternoon, infra-team. | 04:43 |
| lifeless | mordred: its kindof-planned | 04:43 |
| lifeless | mordred: but all the attempts so far have rolled back, IIRC | 04:43 |
| stevebaker | oomichi: beat you by 5 minutes :) | 04:44 |
| *** tiswanso has joined #openstack-infra | 04:46 | |
| oomichi | stevebaker: yeah, nice job ;) | 04:48 |
| *** yuanying-alt has joined #openstack-infra | 04:48 | |
| *** sabeen1 has quit IRC | 04:49 | |
| *** deepakcs has joined #openstack-infra | 04:49 | |
| *** sergmelikyan has joined #openstack-infra | 04:50 | |
| *** tiswanso has quit IRC | 04:51 | |
| *** salv-orlando has joined #openstack-infra | 04:52 | |
| *** yuanying-alt has quit IRC | 04:52 | |
| *** yfried|afk has quit IRC | 04:53 | |
| *** Sree has joined #openstack-infra | 04:53 | |
| *** Sree has quit IRC | 04:54 | |
| *** Sree has joined #openstack-infra | 04:55 | |
| *** abregman has joined #openstack-infra | 04:55 | |
| *** Sukhdev has joined #openstack-infra | 04:56 | |
| *** salv-orlando has quit IRC | 04:57 | |
| *** salv-orlando has joined #openstack-infra | 04:57 | |
| *** ig0r__ has joined #openstack-infra | 04:59 | |
| *** ig0r_ has quit IRC | 04:59 | |
| *** Sree has quit IRC | 05:03 | |
| *** puranamr has joined #openstack-infra | 05:05 | |
| *** ddieterly has joined #openstack-infra | 05:06 | |
| *** asettle has quit IRC | 05:06 | |
| *** asettle has joined #openstack-infra | 05:08 | |
| *** sergmelikyan has quit IRC | 05:09 | |
| *** salv-orlando has quit IRC | 05:10 | |
| *** ddieterly has quit IRC | 05:11 | |
| *** sabeen1 has joined #openstack-infra | 05:13 | |
| *** asettle has quit IRC | 05:14 | |
| *** asettle has joined #openstack-infra | 05:14 | |
| *** Sree has joined #openstack-infra | 05:21 | |
| *** sdake_ has quit IRC | 05:25 | |
| *** sdake has joined #openstack-infra | 05:26 | |
| openstackgerrit | Stanislaw Pitucha proposed openstack-infra/project-config: Add pypy build to Anchor https://review.openstack.org/218729 | 05:27 |
| *** oomichi has quit IRC | 05:27 | |
| *** sdake has quit IRC | 05:29 | |
| *** sdake has joined #openstack-infra | 05:30 | |
| *** baoli has joined #openstack-infra | 05:38 | |
| *** sabeen2 has joined #openstack-infra | 05:39 | |
| *** sabeen1 has quit IRC | 05:39 | |
| *** baoli has quit IRC | 05:42 | |
| *** sdake_ has joined #openstack-infra | 05:45 | |
| *** sdake has quit IRC | 05:49 | |
| *** markvoelker has joined #openstack-infra | 05:51 | |
| *** markvoelker_ has joined #openstack-infra | 05:53 | |
| *** matbu has joined #openstack-infra | 05:53 | |
| *** markvoelker has quit IRC | 05:56 | |
| *** jeadre has quit IRC | 06:01 | |
| *** jeadre has joined #openstack-infra | 06:02 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/218737 | 06:07 |
| *** ddieterly has joined #openstack-infra | 06:07 | |
| *** ParsectiX has joined #openstack-infra | 06:07 | |
| *** jeadre has quit IRC | 06:08 | |
| *** jeadre has joined #openstack-infra | 06:09 | |
| *** flepied has quit IRC | 06:09 | |
| *** salv-orlando has joined #openstack-infra | 06:10 | |
| *** ddieterly has quit IRC | 06:12 | |
| *** Sree_ has joined #openstack-infra | 06:14 | |
| *** Sree_ is now known as Guest41722 | 06:14 | |
| *** stevemar has quit IRC | 06:15 | |
| *** davideagnello has joined #openstack-infra | 06:16 | |
| *** Sree has quit IRC | 06:16 | |
| *** oomichi has joined #openstack-infra | 06:18 | |
| *** afazekas_ has joined #openstack-infra | 06:19 | |
| *** markvoelker has joined #openstack-infra | 06:22 | |
| *** markvoelker_ has quit IRC | 06:25 | |
| *** yfried|afk has joined #openstack-infra | 06:26 | |
| *** yushiro has quit IRC | 06:26 | |
| *** markvoelker_ has joined #openstack-infra | 06:27 | |
| *** sergmelikyan has joined #openstack-infra | 06:27 | |
| *** armax has quit IRC | 06:28 | |
| *** arxcruz has joined #openstack-infra | 06:28 | |
| *** stevemar has joined #openstack-infra | 06:29 | |
| *** markvoelker has quit IRC | 06:29 | |
| *** davideagnello has quit IRC | 06:29 | |
| *** yfried has joined #openstack-infra | 06:30 | |
| *** yfried|afk has quit IRC | 06:31 | |
| Kennan | jhesketh: did we have a bug for that jenkins failure issue? | 06:31 |
| jhesketh | Kennan: yes, it's related to here https://bugs.launchpad.net/neutron/+bug/1490380 | 06:31 |
| openstack | Launchpad bug 1490380 in neutron "netaddr 0.7.16 causes gate havoc" [Critical,In progress] - Assigned to Kevin Benton (kevinbenton) | 06:31 |
| jhesketh | a fix has gone in | 06:31 |
| *** sergmelikyan has quit IRC | 06:32 | |
| Kennan | ok jhesketh: seems it is still under review | 06:33 |
| *** jeadre has quit IRC | 06:33 | |
| *** jeadre has joined #openstack-infra | 06:34 | |
| *** sdake_ is now known as sdake | 06:36 | |
| *** yuanying-alt has joined #openstack-infra | 06:37 | |
| *** nadya has joined #openstack-infra | 06:37 | |
| *** mrunge has joined #openstack-infra | 06:40 | |
| *** flepied has joined #openstack-infra | 06:40 | |
| *** flepied1 has joined #openstack-infra | 06:41 | |
| *** yuanying-alt has quit IRC | 06:41 | |
| *** flepied has quit IRC | 06:45 | |
| *** tiswanso has joined #openstack-infra | 06:47 | |
| *** yushiro has joined #openstack-infra | 06:48 | |
| *** ParsectiX has quit IRC | 06:51 | |
| *** tiswanso has quit IRC | 06:51 | |
| *** ParsectiX has joined #openstack-infra | 06:52 | |
| *** stevemar has quit IRC | 06:53 | |
| *** puranamr has quit IRC | 06:58 | |
| *** Guest41722 has quit IRC | 06:58 | |
| *** Sree has joined #openstack-infra | 06:59 | |
| *** yushiro has quit IRC | 06:59 | |
| *** bexelbie_ has joined #openstack-infra | 06:59 | |
| *** Sukhdev has quit IRC | 07:00 | |
| *** stevemar has joined #openstack-infra | 07:00 | |
| *** Sree has quit IRC | 07:03 | |
| *** nadya has quit IRC | 07:05 | |
| *** achanda has quit IRC | 07:05 | |
| *** ddieterly has joined #openstack-infra | 07:08 | |
| *** peristeri has joined #openstack-infra | 07:09 | |
| *** jtomasek has quit IRC | 07:11 | |
| *** jtomasek has joined #openstack-infra | 07:11 | |
| *** ddieterly has quit IRC | 07:13 | |
| *** sabeen2 has quit IRC | 07:18 | |
| *** jeadre has quit IRC | 07:18 | |
| *** jeadre has joined #openstack-infra | 07:19 | |
| *** zeih has joined #openstack-infra | 07:19 | |
| openstackgerrit | Sachi King proposed openstack-infra/project-config: Add constraints jobs to Jenkins https://review.openstack.org/207262 | 07:20 |
| *** marcusvrn_ has quit IRC | 07:25 | |
| *** tmorin has joined #openstack-infra | 07:29 | |
| *** asettle has quit IRC | 07:30 | |
| *** Ala has joined #openstack-infra | 07:33 | |
| *** kaisers has joined #openstack-infra | 07:33 | |
| *** doude has joined #openstack-infra | 07:34 | |
| *** fhubik has joined #openstack-infra | 07:36 | |
| *** Sree has joined #openstack-infra | 07:37 | |
| *** jeadre has quit IRC | 07:40 | |
| *** zeih_ has joined #openstack-infra | 07:41 | |
| *** jeadre has joined #openstack-infra | 07:41 | |
| *** b10n1k_ has quit IRC | 07:44 | |
| *** zeih has quit IRC | 07:44 | |
| *** dizquierdo has joined #openstack-infra | 07:44 | |
| *** samuelBartel has joined #openstack-infra | 07:45 | |
| *** sshnaidm has joined #openstack-infra | 07:52 | |
| *** hashar has joined #openstack-infra | 07:55 | |
| *** baoli has joined #openstack-infra | 07:58 | |
| gmann | oomichi: mtreinish : lifeless : https://review.openstack.org/#/c/218736/ (needed for https://review.openstack.org/#/c/218720/) is failing on neutron kilo job (invalid MAC) | 08:01 |
| *** jerryz has quit IRC | 08:02 | |
| *** sshnaidm has quit IRC | 08:02 | |
| fbo | clarkb, Hi, about your discussion on puppet-httpd and http_mod on centos7 please have a look to https://review.openstack.org/#/c/216318/ and https://review.openstack.org/#/c/216319/. I think this is a bit like what we want : avoid a2enmod to be run on centos, as modules are enabled as soon as they are installed. | 08:03 |
| *** baoli has quit IRC | 08:03 | |
| gmann | oomichi: mtreinish : lifeless : which i think requirements updates ( netaddr>=0.7.12,!=0.7.16) needed for kilo branch too but that again need https://review.openstack.org/#/c/218736/ fix | 08:03 |
| gmann | seems like deadlock here ? | 08:03 |
| *** ifarkas has joined #openstack-infra | 08:03 | |
| gmann | sdake: ^^ | 08:04 |
| *** gildub has quit IRC | 08:08 | |
| *** ddieterly has joined #openstack-infra | 08:09 | |
| *** claudiub has joined #openstack-infra | 08:10 | |
| *** hichihara has quit IRC | 08:13 | |
| *** ddieterly has quit IRC | 08:14 | |
| oomichi | gmann: the latest https://pypi.python.org/pypi/netaddr is released | 08:14 |
| *** jistr has joined #openstack-infra | 08:15 | |
| oomichi | gmann: netaddr-0.7.17 | 08:15 |
| *** sshnaidm has joined #openstack-infra | 08:15 | |
| *** kiran-r has joined #openstack-infra | 08:15 | |
| *** derekh has joined #openstack-infra | 08:15 | |
| gmann | oomichi: i see, thats cool - https://github.com/drkjam/netaddr/commit/75eee70655597da60123aae7835afb8f66760149 | 08:17 |
| gmann | oomichi: I will recheck our fixes | 08:17 |
| oomichi | gmann: yeah, cool :) | 08:18 |
| lifeless | gmann: oomichi we should be able to fix kilo first, then roll forward to fix liberty/master | 08:19 |
| gmann | lifeless: tempest test failing on https://review.openstack.org/#/c/218720/ will fail for kilo too | 08:20 |
| gmann | lifeless: ohk, you mean requirements update. yea https://review.openstack.org/#/c/218736/ should pass with 0.7.17 and then we can update kilo requirement also | 08:22 |
| *** salv-orlando has quit IRC | 08:24 | |
| *** salv-orlando has joined #openstack-infra | 08:25 | |
| *** rossella_s has quit IRC | 08:25 | |
| *** yuanying-alt has joined #openstack-infra | 08:25 | |
| *** abalutoiu has joined #openstack-infra | 08:27 | |
| *** rossella_s has joined #openstack-infra | 08:27 | |
| *** dizquierdo has quit IRC | 08:27 | |
| *** pblaho has joined #openstack-infra | 08:28 | |
| *** markvoelker_ has quit IRC | 08:28 | |
| *** yuanying-alt has quit IRC | 08:30 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Tripleo CI reshuffle https://review.openstack.org/205479 | 08:31 |
| *** kushal has joined #openstack-infra | 08:32 | |
| *** ihrachys has joined #openstack-infra | 08:41 | |
| *** amotoki has joined #openstack-infra | 08:41 | |
| *** ParsectiX_ has joined #openstack-infra | 08:41 | |
| *** ParsectiX_ has quit IRC | 08:42 | |
| *** dguitarbite has joined #openstack-infra | 08:42 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new SF repository for Fuel ScaleIO plugin integration https://review.openstack.org/218769 | 08:47 |
| *** dingyichen has quit IRC | 08:47 | |
| *** tiswanso has joined #openstack-infra | 08:48 | |
| *** sdake has quit IRC | 08:50 | |
| *** fhubik is now known as fhubik_brb | 08:51 | |
| *** tiswanso has quit IRC | 08:53 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 08:57 |
| *** e0ne has joined #openstack-infra | 08:58 | |
| *** yushiro has joined #openstack-infra | 08:59 | |
| yushiro | Hi, | 08:59 |
| lifeless | gmann: 218736 is abandoned | 09:00 |
| *** ildikov has joined #openstack-infra | 09:01 | |
| gmann | lifeless: yea, now with netaddr-0.7.17 that test should pass without any issue | 09:02 |
| gmann | lifeless: https://review.openstack.org/#/c/218720/ should merge now without any tempest changes | 09:03 |
| *** rossella_s has quit IRC | 09:05 | |
| openstackgerrit | Merged openstack-infra/release-tools: Document library release process https://review.openstack.org/217307 | 09:06 |
| *** rossella_s has joined #openstack-infra | 09:06 | |
| *** e0ne has quit IRC | 09:09 | |
| *** fhubik_brb is now known as fhubik | 09:09 | |
| *** e0ne has joined #openstack-infra | 09:09 | |
| *** flip214 has joined #openstack-infra | 09:09 | |
| *** ddieterly has joined #openstack-infra | 09:10 | |
| *** bharathm has quit IRC | 09:12 | |
| *** abregman has quit IRC | 09:13 | |
| *** ddieterly has quit IRC | 09:14 | |
| *** bexelbie_ is now known as bexelbie | 09:19 | |
| derekh | Is there anybody else around who can take a look at a reshuffle of tripleo jobs/repos please https://review.openstack.org/#/c/205479/ https://review.openstack.org/#/c/215186/ | 09:23 |
| *** markvoelker has joined #openstack-infra | 09:23 | |
| *** salv-orlando has quit IRC | 09:24 | |
| *** markvoelker_ has joined #openstack-infra | 09:26 | |
| *** pelix has joined #openstack-infra | 09:27 | |
| marzif__ | here: please if anyone can +A this review: https://review.openstack.org/#/c/193125/ | 09:29 |
| *** markvoelker has quit IRC | 09:30 | |
| *** markvoelker_ has quit IRC | 09:31 | |
| *** hashar_ has joined #openstack-infra | 09:34 | |
| *** yushiro has quit IRC | 09:36 | |
| *** ig0r_ has joined #openstack-infra | 09:37 | |
| *** hashar has quit IRC | 09:37 | |
| *** markvoelker has joined #openstack-infra | 09:37 | |
| *** marzif__ has quit IRC | 09:38 | |
| *** marzif__ has joined #openstack-infra | 09:38 | |
| bauzas | hey folks | 09:38 |
| bauzas | can see http://status.openstack.org/zuul/ logstash's job queue is spiking | 09:39 |
| *** ig0r__ has quit IRC | 09:39 | |
| bauzas | anyone could confirm that logstash is a bit delayed? | 09:39 |
| *** yfried is now known as yfried|afk | 09:43 | |
| *** dizquierdo has joined #openstack-infra | 09:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul: Support pagination in simplequery for Gerrit >=2.9 https://review.openstack.org/209906 | 09:46 |
| *** yfried|afk is now known as yfried | 09:53 | |
| *** vivekd has joined #openstack-infra | 09:54 | |
| *** pc_m has joined #openstack-infra | 09:57 | |
| *** sshnaidm has quit IRC | 09:57 | |
| *** Sree__ has joined #openstack-infra | 09:58 | |
| *** pc_m_ has joined #openstack-infra | 09:58 | |
| *** tnovacik has joined #openstack-infra | 09:59 | |
| *** pc_m_ has quit IRC | 09:59 | |
| *** Sree has quit IRC | 10:01 | |
| *** bharathm has joined #openstack-infra | 10:01 | |
| *** pc_m has quit IRC | 10:01 | |
| *** bharathm has quit IRC | 10:06 | |
| *** sshnaidm has joined #openstack-infra | 10:10 | |
| *** ddieterly has joined #openstack-infra | 10:10 | |
| *** marzif__ has quit IRC | 10:12 | |
| *** pc_m has joined #openstack-infra | 10:12 | |
| *** yuanying-alt has joined #openstack-infra | 10:14 | |
| *** john-davidge has joined #openstack-infra | 10:15 | |
| *** ddieterly has quit IRC | 10:15 | |
| *** john-davidge_ has joined #openstack-infra | 10:18 | |
| openstackgerrit | Ramakrishnan G proposed openstack/diskimage-builder: Add proliant-tools element https://review.openstack.org/163754 | 10:19 |
| *** yuanying-alt has quit IRC | 10:19 | |
| *** fhubik is now known as fhubik_brb | 10:19 | |
| *** john-davidge has quit IRC | 10:20 | |
| *** john-davidge_ is now known as john-davidge | 10:20 | |
| *** Sree__ has quit IRC | 10:23 | |
| *** jeadre has quit IRC | 10:23 | |
| *** jeadre has joined #openstack-infra | 10:24 | |
| openstackgerrit | Ramakrishnan G proposed openstack/diskimage-builder: Add proliant-tools element https://review.openstack.org/163754 | 10:31 |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds os-win repository https://review.openstack.org/209204 | 10:31 |
| *** ParsectiX_ has joined #openstack-infra | 10:37 | |
| *** akraminsky has joined #openstack-infra | 10:46 | |
| *** teran has joined #openstack-infra | 10:46 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack/requirements: Add version cap for netaddr < 0.7.16 https://review.openstack.org/218805 | 10:46 |
| *** marzif__ has joined #openstack-infra | 10:47 | |
| *** tiswanso has joined #openstack-infra | 10:48 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: copy_puppet_logs: copy dstat logs if exist https://review.openstack.org/218586 | 10:50 |
| *** tiswanso has quit IRC | 10:53 | |
| *** jaypipes has joined #openstack-infra | 10:58 | |
| *** jtomasek has quit IRC | 10:59 | |
| *** bharathm has joined #openstack-infra | 11:02 | |
| *** dguitarbite has quit IRC | 11:04 | |
| *** bharathm has quit IRC | 11:07 | |
| marzif__ | please if anyone give +A to: https://review.openstack.org/#/c/193125/ we are currently blocked, thanks | 11:08 |
| *** soren has joined #openstack-infra | 11:09 | |
| *** e0ne has quit IRC | 11:10 | |
| *** ddieterly has joined #openstack-infra | 11:11 | |
| *** ddieterly has quit IRC | 11:16 | |
| *** nadya has joined #openstack-infra | 11:17 | |
| *** dguitarbite has joined #openstack-infra | 11:20 | |
| *** stevemar has quit IRC | 11:22 | |
| *** fhubik_brb is now known as fhubik | 11:24 | |
| sdague | clarkb: so, the DHCPNAK issue is probably not a direct relationship to the failure | 11:27 |
| sdague | however... | 11:27 |
| *** jcoufal has joined #openstack-infra | 11:28 | |
| sdague | it does show that we are bleeding dhcp requests between test nodes all the time - http://logstash.openstack.org/#eyJzZWFyY2giOiJcImZhOjE2OjNlOjhlOjkxOjJiXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE0NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MTAyMDM3MzQzN30= | 11:29 |
| sdague | I tracked back through and matched up mac address entries | 11:29 |
| marzif__ | sdague, if you have one moment, we need +A as we are blocked currently by https://review.openstack.org/#/c/193125/ | 11:32 |
| sdague | done | 11:34 |
| *** wenlock has quit IRC | 11:34 | |
| *** jaypipes has quit IRC | 11:35 | |
| *** jaypipes has joined #openstack-infra | 11:36 | |
| *** ldnunes has joined #openstack-infra | 11:38 | |
| *** markus_z has joined #openstack-infra | 11:39 | |
| *** amotoki has quit IRC | 11:39 | |
| *** amotoki has joined #openstack-infra | 11:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Splitting stackforge/freezer on multiple repos https://review.openstack.org/193125 | 11:39 |
| *** marzif__ has quit IRC | 11:42 | |
| *** gordc has joined #openstack-infra | 11:43 | |
| *** yfried is now known as yfried|afk | 11:43 | |
| *** fhubik is now known as fhubik_brb | 11:43 | |
| *** trown|outttypeww is now known as trown | 11:44 | |
| daemontool_ | sdague, ty | 11:45 |
| *** fhubik_brb is now known as fhubik | 11:45 | |
| *** openstackgerrit has quit IRC | 11:46 | |
| *** jtomasek has joined #openstack-infra | 11:46 | |
| *** yfried|afk is now known as yfried | 11:47 | |
| *** openstackgerrit has joined #openstack-infra | 11:47 | |
| *** k4n0 has joined #openstack-infra | 11:48 | |
| *** markvoelker_ has joined #openstack-infra | 11:51 | |
| *** markvoelker has quit IRC | 11:51 | |
| *** akraminsky has quit IRC | 11:51 | |
| *** markvoelker_ has quit IRC | 11:55 | |
| *** ihrachys has quit IRC | 11:56 | |
| *** ihrachys has joined #openstack-infra | 11:56 | |
| *** jtomasek has quit IRC | 11:56 | |
| *** jtomasek has joined #openstack-infra | 11:56 | |
| *** markvoelker has joined #openstack-infra | 11:56 | |
| *** sergmelikyan has joined #openstack-infra | 11:58 | |
| *** mwagner_lap has quit IRC | 11:59 | |
| *** e0ne has joined #openstack-infra | 11:59 | |
| *** nadya has quit IRC | 12:01 | |
| *** sergmelikyan has quit IRC | 12:02 | |
| *** yuanying-alt has joined #openstack-infra | 12:03 | |
| *** vivekd has quit IRC | 12:07 | |
| *** yuanying-alt has quit IRC | 12:07 | |
| *** dprince has joined #openstack-infra | 12:08 | |
| *** alivigni has joined #openstack-infra | 12:09 | |
| openstackgerrit | Chris Burroughs proposed openstack-infra/jenkins-job-builder: support for the phabricator plugin https://review.openstack.org/215750 | 12:10 |
| *** ddieterly has joined #openstack-infra | 12:12 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new SF repository for Fuel ScaleIO plugin integration https://review.openstack.org/218769 | 12:13 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jenkins: Don't configure memory swap bytes on cgroups for Trusty https://review.openstack.org/218227 | 12:13 |
| *** aysyd has joined #openstack-infra | 12:14 | |
| *** ddieterly has quit IRC | 12:16 | |
| *** dkranz has joined #openstack-infra | 12:17 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: Make gate-ceilometer-dsvm-functional-postgresql voting https://review.openstack.org/215485 | 12:17 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config: Add python-gnocchiclient https://review.openstack.org/218826 | 12:18 |
| *** trown is now known as trown|relocating | 12:19 | |
| *** rfolco has joined #openstack-infra | 12:19 | |
| *** ParsectiX_ has quit IRC | 12:21 | |
| *** ParsectiX has quit IRC | 12:21 | |
| *** ParsectiX has joined #openstack-infra | 12:22 | |
| *** cburroughs has joined #openstack-infra | 12:23 | |
| ajo | ping | 12:23 |
| ajo | can we prioritize https://review.openstack.org/#/c/218723/ to unbreak python unit tests? | 12:24 |
| ajo | kibana link: http://bit.ly/1hr8blF | 12:24 |
| ajo | we're seeing 1000s of failures due to a netaddr change | 12:24 |
| ajo | sdague, anteaya , does it make sense to ping anyone about this? | 12:25 |
| ajo | ihrachys ^? | 12:25 |
| sdague | ajo: it needs tests to come back first | 12:25 |
| ajo | hmm, I guess it's not that super urgent, I just realized it's only neutron-unit tests | 12:26 |
| ajo | hmm, woops sdague , I see | 12:26 |
| ajo | what happened? | 12:26 |
| *** hashar_ has quit IRC | 12:26 | |
| ihrachys | ajo: I guess sdague means we need a +1 report from jenkins first | 12:27 |
| sdague | ihrachys: yes | 12:27 |
| ihrachys | ajo: now that it is about to fail on grenade, how about respinning the patch to clear some nits you got? | 12:28 |
| ajo | ah, ok, I thought we had a worse issue | 12:28 |
| ajo | ihrachys, we could, if we get the +2's again quickly | 12:28 |
| ajo | I didn't want to clear those out | 12:28 |
| ajo | and delay the thing | 12:28 |
| ihrachys | ajo: obviously we will get it quick | 12:29 |
| ajo | ihrachys, Ok, I will handle the comment about validate_mac | 12:29 |
| *** baoli has joined #openstack-infra | 12:29 | |
| ihrachys | and rosella's | 12:29 |
| ajo | the other changes could deserve more discussion, | 12:29 |
| ajo | let he mech her's | 12:29 |
| ajo | check | 12:29 |
| sdague | also - Bug 1355573 - grenade volume scenarios periodically fail /w SSHTimeout seems to have gotten massively worse in the last 24 hours | 12:29 |
| openstack | bug 1355573 in grenade "grenade volume scenarios periodically fail /w SSHTimeout" [Undecided,New] https://launchpad.net/bugs/1355573 | 12:29 |
| *** baoli_ has joined #openstack-infra | 12:31 | |
| *** baoli_ has quit IRC | 12:31 | |
| *** baoli_ has joined #openstack-infra | 12:32 | |
| *** soren has quit IRC | 12:33 | |
| *** k4n0 has quit IRC | 12:34 | |
| *** baoli has quit IRC | 12:34 | |
| *** tiswanso has joined #openstack-infra | 12:34 | |
| *** krtaylor has quit IRC | 12:34 | |
| *** Sree has joined #openstack-infra | 12:35 | |
| *** tiswanso has quit IRC | 12:39 | |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: added query for netaddr 0.7.16 mac addr parsing bug https://review.openstack.org/218834 | 12:39 |
| *** annegentle has joined #openstack-infra | 12:44 | |
| openstackgerrit | will soula proposed openstack-infra/jenkins-job-builder: Add Ability to use the AWS Cloudformation Plugin https://review.openstack.org/202300 | 12:44 |
| openstackgerrit | will soula proposed openstack-infra/jenkins-job-builder: Add support for whitesource plugin https://review.openstack.org/204615 | 12:45 |
| *** ddieterly has joined #openstack-infra | 12:46 | |
| *** tlian has joined #openstack-infra | 12:46 | |
| *** krtaylor has joined #openstack-infra | 12:47 | |
| *** ifarkas has quit IRC | 12:48 | |
| *** k4n0 has joined #openstack-infra | 12:48 | |
| *** ifarkas has joined #openstack-infra | 12:49 | |
| *** burgerk has joined #openstack-infra | 12:50 | |
| *** bharathm has joined #openstack-infra | 12:51 | |
| openstackgerrit | Merged openstack-infra/puppet-accessbot: Basic acceptance tests https://review.openstack.org/206438 | 12:52 |
| *** amuller has joined #openstack-infra | 12:53 | |
| *** bharathm has quit IRC | 12:56 | |
| *** afazekas__ has joined #openstack-infra | 12:56 | |
| *** cbader has joined #openstack-infra | 12:58 | |
| *** afazekas_ has quit IRC | 12:58 | |
| *** hashar has joined #openstack-infra | 13:01 | |
| *** tmorin has quit IRC | 13:01 | |
| *** mjturek1 has joined #openstack-infra | 13:05 | |
| *** rossella_s has quit IRC | 13:05 | |
| *** vivekd has joined #openstack-infra | 13:06 | |
| *** rossella_s has joined #openstack-infra | 13:06 | |
| *** dkranz has quit IRC | 13:07 | |
| *** madong_ has quit IRC | 13:08 | |
| *** jeadre has quit IRC | 13:08 | |
| *** tmorin has joined #openstack-infra | 13:08 | |
| *** ddieterly has quit IRC | 13:09 | |
| *** jeadre has joined #openstack-infra | 13:09 | |
| tmorin | looking for help on an issue with jenkins checks not starting on a change ( https://review.openstack.org/#/c/218366/ ) | 13:10 |
| tmorin | anyone ? | 13:10 |
| *** sabeen1 has joined #openstack-infra | 13:11 | |
| *** vinay_yadhav has joined #openstack-infra | 13:11 | |
| *** kgiusti has joined #openstack-infra | 13:11 | |
| *** rlandy has joined #openstack-infra | 13:13 | |
| *** glauco_ has joined #openstack-infra | 13:13 | |
| *** xyang1 has joined #openstack-infra | 13:16 | |
| *** alivigni has quit IRC | 13:16 | |
| *** jcoufal has quit IRC | 13:17 | |
| *** wenlock has joined #openstack-infra | 13:18 | |
| *** dkranz has joined #openstack-infra | 13:20 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config: Add python-gnocchiclient https://review.openstack.org/218826 | 13:20 |
| *** e0ne has quit IRC | 13:22 | |
| *** wenlock has quit IRC | 13:22 | |
| *** matrohon has joined #openstack-infra | 13:23 | |
| tmorin | ping fungi ? (about the jenkins-related issue above) | 13:23 |
| *** ivar-lazzaro has joined #openstack-infra | 13:23 | |
| *** ivar-lazzaro has quit IRC | 13:24 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: added query for netaddr 0.7.16 mac addr parsing bug https://review.openstack.org/218834 | 13:24 |
| *** ivar-lazzaro has joined #openstack-infra | 13:24 | |
| *** glauco_ has quit IRC | 13:26 | |
| *** k4n0 has quit IRC | 13:27 | |
| *** eharney has joined #openstack-infra | 13:27 | |
| openstackgerrit | Merged openstack-infra/groups: Add ambassador application form https://review.openstack.org/217833 | 13:28 |
| *** e0ne has joined #openstack-infra | 13:29 | |
| *** mtanino has joined #openstack-infra | 13:30 | |
| *** links has quit IRC | 13:30 | |
| *** jcoufal has joined #openstack-infra | 13:30 | |
| *** ayoung has joined #openstack-infra | 13:31 | |
| *** jcoufal has quit IRC | 13:31 | |
| *** thorst1 has joined #openstack-infra | 13:32 | |
| *** jcoufal has joined #openstack-infra | 13:32 | |
| *** burgerk has quit IRC | 13:34 | |
| *** stevemar has joined #openstack-infra | 13:34 | |
| *** trown|relocating is now known as trown | 13:35 | |
| *** superdan is now known as dansmith | 13:35 | |
| *** salv-orlando has joined #openstack-infra | 13:36 | |
| *** dims has joined #openstack-infra | 13:36 | |
| *** hashar_ has joined #openstack-infra | 13:39 | |
| *** doug-fish has joined #openstack-infra | 13:39 | |
| *** stevemar has quit IRC | 13:39 | |
| *** k4n0 has joined #openstack-infra | 13:40 | |
| *** ivar-lazzaro has quit IRC | 13:41 | |
| *** tiswanso has joined #openstack-infra | 13:41 | |
| *** hashar has quit IRC | 13:41 | |
| *** ldnunes has quit IRC | 13:41 | |
| *** soren has joined #openstack-infra | 13:41 | |
| *** ddieterly has joined #openstack-infra | 13:43 | |
| *** k4n0 has quit IRC | 13:44 | |
| tmorin | ping anteaya ?... I have an issue with jenkins checks not getting started on a change ( https://review.openstack.org/#/c/218366/ ), blocking progress of the said patch | 13:44 |
| *** k4n0 has joined #openstack-infra | 13:45 | |
| svasheka | fungi, mordred : hi guys, could you please review https://review.openstack.org/#/c/215026/15 , it's been waiting for a long time | 13:46 |
| *** ldnunes has joined #openstack-infra | 13:46 | |
| openstackgerrit | Pranav Salunke proposed openstack-infra/project-config: Create new project labs https://review.openstack.org/218345 | 13:46 |
| *** Sree has quit IRC | 13:46 | |
| *** Sree has joined #openstack-infra | 13:47 | |
| *** esker has quit IRC | 13:47 | |
| *** esker has joined #openstack-infra | 13:48 | |
| *** doude_ has joined #openstack-infra | 13:48 | |
| *** kiran-r has quit IRC | 13:48 | |
| *** doude has quit IRC | 13:48 | |
| *** doude_ has quit IRC | 13:49 | |
| pabelanger | morning | 13:49 |
| *** cdoron has joined #openstack-infra | 13:49 | |
| *** annegentle has quit IRC | 13:49 | |
| *** k4n0 has quit IRC | 13:50 | |
| mordred | svasheka: done | 13:51 |
| mhu | mordred: hi, I was away on holidays, is phabricator + cauth still a thing, and if so, anything needed on our side to make it work ? | 13:51 |
| mordred | mhu: oh hey! | 13:51 |
| mordred | Clint: ^^ mhu is our favorite person | 13:51 |
| *** Sree has quit IRC | 13:51 | |
| *** mwagner_lap has joined #openstack-infra | 13:52 | |
| mordred | mhu: I've been mostly away as well - just getting back in to it this week | 13:52 |
| *** yuanying-alt has joined #openstack-infra | 13:52 | |
| mhu | yay, I am popular ! \o/ | 13:52 |
| svasheka | mordred, thank you! | 13:52 |
| mordred | mhu: it's definitely still a thing, I believe our next step is "figure out how to run cauth in front of phab" | 13:52 |
| mordred | mhu: which will probably be easier with you :) | 13:52 |
| mhu | mordred, alright, happy to help | 13:53 |
| Clint | mordred: woot! | 13:53 |
| *** glauco has joined #openstack-infra | 13:54 | |
| *** pradk has joined #openstack-infra | 13:54 | |
| pabelanger | Wanted to see if there was any issue add myself into the trystack-site-core group? Some team mates have some review up and looking for help getting them merged into upstream. | 13:54 |
| fungi | tmorin: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n7713 shows that stackforge/networking-bagpipe-l2 doesn't have any gate pipeline jobs (except for a job which only runs when you're changing requirements lists) | 13:56 |
| *** glauco has quit IRC | 13:56 | |
| *** yuanying-alt has quit IRC | 13:56 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:56 | |
| tmorin | fungi: yes, that's true, we haven't enabled unit tests yet (we'll soon enable python-jobs) | 13:57 |
| *** spzala has joined #openstack-infra | 13:57 | |
| *** k4n0 has joined #openstack-infra | 13:57 | |
| fungi | tmorin: without a job to run on a change, zuul won't enqueue it into the corresponding pipeline | 13:57 |
| tmorin | fungi: however, that shouldn't block changes being merged right ? but right now, a workflow+1 does nothing | 13:57 |
| tmorin | fungi: this wasn't true last week | 13:57 |
| fungi | tmorin: did the change last week modify a requirements list? | 13:58 |
| tmorin | fungi: e.g. https://review.openstack.org/#/c/218366/ was merged without a jenkins vote | 13:58 |
| tmorin | sorry wrong example | 13:58 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: add query for misuse of netaddr https://review.openstack.org/218865 | 13:58 |
| openstackgerrit | Merged openstack-infra/project-config: Add copy-screenshots publisher https://review.openstack.org/215026 | 13:58 |
| openstackgerrit | Ryan Rossiter proposed openstack/requirements: Bump oslo.versionedobjects version to 0.9.0 https://review.openstack.org/216848 | 13:58 |
| *** mriedem has joined #openstack-infra | 13:59 | |
| tmorin | fungi: a change was merged last week, and yes it did change requirements.txt -- would that block subsequent changes ? | 13:59 |
| tmorin | https://review.openstack.org/#/c/213932/ | 13:59 |
| fungi | tmorin: no, the only jobs your project has in the check and gate pipelines are jobs which specifically limit their running to changes which modify requirements.txt and test-requirements.txt (the requirements check jobs) | 14:00 |
| tmorin | fungi: understood | 14:01 |
| tmorin | fungi: what would cause a workflow+1 to do nothing then ? ( https://review.openstack.org/#/c/218366/ ) | 14:01 |
| fungi | tmorin: correct | 14:02 |
| fungi | tmorin: without a job to run on a change, zuul won't enqueue it into the corresponding pipeline | 14:02 |
| fungi | (repeating my earlier statement) | 14:02 |
| *** doude has joined #openstack-infra | 14:02 | |
| *** doude has quit IRC | 14:02 | |
| *** doude has joined #openstack-infra | 14:02 | |
| fungi | tmorin: i recommend starting with something like adding some unit test jobs, and then modifying tox.ini so that they always pass | 14:03 |
| *** Kennan has quit IRC | 14:03 | |
| *** Kennan2 has joined #openstack-infra | 14:03 | |
| *** dteselkin has quit IRC | 14:03 | |
| *** asilenkov has quit IRC | 14:03 | |
| fungi | then you can introduce a change to your tox.ini later which runs some actual tests in your unit test job | 14:03 |
| *** deepakcs has quit IRC | 14:03 | |
| tmorin | fungi: yes, we plan to do that, but other changes to the project (before we added the check-requirements) were merging fine | 14:03 |
| *** ParsectiX has quit IRC | 14:04 | |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: WIP: Removes posix-ipc from global-requirements https://review.openstack.org/217824 | 14:04 |
| fungi | tmorin: have an example? | 14:04 |
| *** ayoung has quit IRC | 14:04 | |
| tmorin | https://review.openstack.org/#/c/199952/ | 14:04 |
| tmorin | fungi: oh, I get it | 14:04 |
| tmorin | fungi: we had noop before, but we haven't it now | 14:05 |
| fungi | tmorin: we have a noop job which you can run if you don't have any check or gate jobs. that's what you were running previously | 14:05 |
| fungi | yes | 14:05 |
| *** bnemec has joined #openstack-infra | 14:05 | |
| tmorin | fungi: somebody suggested to remove it when we added check-requirements.... | 14:05 |
| * fungi is in a meeting right now or would explain options in more detail | 14:05 | |
| fungi | tmorin: yes, we don't want anyone running noop if they have other jobs in check or gate pipelines | 14:06 |
| fungi | it causes them to share a gate queue with other projects which only run the noop job | 14:06 |
| *** asilenkov has joined #openstack-infra | 14:06 | |
| tmorin | fungi: yes, but in our case, that blocks any merge from happening right ? | 14:06 |
| fungi | most projects start with non-conditional jobs and don't run into this corner case | 14:06 |
| *** wfoster has joined #openstack-infra | 14:07 | |
| *** dteselkin has joined #openstack-infra | 14:07 | |
| fungi | the requirements jobs are conditional, and the reviewers of the change to your jobs didn't consider that | 14:07 |
| *** burgerk has joined #openstack-infra | 14:07 | |
| tmorin | fungi: does it makes sense to re-add noop-job until we enable python-jobs ? | 14:07 |
| tmorin | fungi: ok, I understand the reason for the mistake | 14:08 |
| *** dkranz has quit IRC | 14:08 | |
| fungi | tmorin: only if you remove the requirements job template when switching back to the noop template | 14:08 |
| *** afazekas__ has quit IRC | 14:08 | |
| *** jroll has quit IRC | 14:09 | |
| *** jroll has joined #openstack-infra | 14:09 | |
| *** markvoelker has quit IRC | 14:09 | |
| tmorin | fungi: you mean I can't have both the noop-jobs and the check-requirements job in zuul ? (even temporarily until we add python-jobs) | 14:10 |
| fungi | tmorin: yes, we don't want anyone running noop if they have other jobs in check or gate pipelines | 14:10 |
| fungi | it causes them to share a gate queue with other projects which only run the noop job | 14:10 |
| tmorin | fungi: ok, understood | 14:11 |
| *** eharney has quit IRC | 14:11 | |
| *** boris-42 has joined #openstack-infra | 14:11 | |
| tmorin | fungi: then I have an incentive to not delay enabling python-jobs, which is not a bad thing ! ;) | 14:11 |
| *** zz_dimtruck is now known as dimtruck | 14:11 | |
| fungi | tmorin: silver lining! | 14:11 |
| tmorin | fungi: ;-) thanks a lot for your clear explanations | 14:12 |
| pabelanger | any infra core mind +2 https://review.openstack.org/#/c/216725/ its adding a .gitreview files into trystack-site | 14:12 |
| fungi | htm... jenkins01 is so starved for memory i can't run `free` (and can't even ssh in without many attempts) | 14:14 |
| anteaya | fungi: :( | 14:15 |
| *** wenlock has joined #openstack-infra | 14:17 | |
| *** mpaolino has joined #openstack-infra | 14:17 | |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: update 1490374 for tempest hits as well https://review.openstack.org/218876 | 14:18 |
| openstackgerrit | Merged openstack-infra/system-config: Upgrade etherpad-dev to Ubuntu Trusty https://review.openstack.org/215243 | 14:19 |
| openstackgerrit | Brianna Poulos proposed openstack/requirements: Add Castellan to global requirements https://review.openstack.org/184874 | 14:19 |
| *** annegentle has joined #openstack-infra | 14:19 | |
| *** ihrachys has quit IRC | 14:20 | |
| *** ildikov has quit IRC | 14:20 | |
| *** ihrachys has joined #openstack-infra | 14:21 | |
| *** zeih_ has quit IRC | 14:22 | |
| *** tonytan4ever has joined #openstack-infra | 14:22 | |
| *** r-daneel has joined #openstack-infra | 14:22 | |
| *** cdoron has quit IRC | 14:24 | |
| *** dkranz has joined #openstack-infra | 14:24 | |
| *** thiagop has joined #openstack-infra | 14:24 | |
| sdague | fungi: when you get a chance, can you promote this test skip - https://review.openstack.org/#/c/218871/3 | 14:24 |
| sdague | that's causing quite a lot of carnage atm, and we need breathing room to figure it out | 14:25 |
| *** vinay_yadhav has quit IRC | 14:26 | |
| *** vivekd has quit IRC | 14:27 | |
| *** smoser has quit IRC | 14:27 | |
| *** dkranz has quit IRC | 14:28 | |
| fungi | sdague: done | 14:28 |
| *** baoli_ has quit IRC | 14:29 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: add query for misuse of netaddr https://review.openstack.org/218865 | 14:29 |
| *** baoli has joined #openstack-infra | 14:30 | |
| *** jswarren has joined #openstack-infra | 14:30 | |
| *** jswarren_ has joined #openstack-infra | 14:30 | |
| openstackgerrit | Victor Ryzhenkin proposed openstack/requirements: Add murano-specs repository to projects.txt https://review.openstack.org/218878 | 14:31 |
| *** abalutoiu has quit IRC | 14:31 | |
| openstackgerrit | John Trowbridge proposed openstack/diskimage-builder: Move service enable to install https://review.openstack.org/218879 | 14:31 |
| *** dkranz has joined #openstack-infra | 14:32 | |
| *** claudiub has quit IRC | 14:32 | |
| *** alivigni has joined #openstack-infra | 14:34 | |
| *** jswarren_ has quit IRC | 14:35 | |
| openstackgerrit | Victor Ryzhenkin proposed openstack/requirements: Add murano-specs repository to projects.txt https://review.openstack.org/218878 | 14:36 |
| *** ildikov has joined #openstack-infra | 14:36 | |
| sdague | fungi: thank yuo | 14:37 |
| *** peristeri has quit IRC | 14:38 | |
| *** ayoung has joined #openstack-infra | 14:38 | |
| *** mriedem is now known as mriedem_away | 14:39 | |
| *** bharathm has joined #openstack-infra | 14:40 | |
| clarkb | sdague so >1 VM fight for the same ip? | 14:41 |
| *** dkranz has quit IRC | 14:43 | |
| *** jaosorior has joined #openstack-infra | 14:44 | |
| openstackgerrit | John Trowbridge proposed openstack/diskimage-builder: Move service enable to install https://review.openstack.org/218879 | 14:45 |
| *** bharathm has quit IRC | 14:45 | |
| sdague | clarkb: nope | 14:46 |
| sdague | it's other test nodes | 14:46 |
| clarkb | but we firewall | 14:46 |
| sdague | if you look at that search by MAC you'll find it's on 2 jobs at the same time | 14:46 |
| *** MarkAtwood has joined #openstack-infra | 14:47 | |
| sdague | clarkb: apparently pooly | 14:47 |
| sdague | poorly | 14:47 |
| clarkb | hrm | 14:47 |
| clarkb | I wonder if nova/neutron just wipe out our default deny | 14:47 |
| clarkb | oh except its l2 broadcast | 14:48 |
| sdague | clarkb: could be, they do assume they own the iptables | 14:48 |
| clarkb | hrm ebtables may be needed | 14:48 |
| *** dkranz has joined #openstack-infra | 14:48 | |
| sdague | let me get a good example | 14:48 |
| *** eharney has joined #openstack-infra | 14:48 | |
| fungi | oh, that's sinister | 14:48 |
| *** bmjen has joined #openstack-infra | 14:48 | |
| *** stevemar has joined #openstack-infra | 14:48 | |
| mtreinish | yikes, that's something | 14:49 |
| *** achanda has joined #openstack-infra | 14:49 | |
| fungi | i guess we can just block on the host interfaces at least and fix that | 14:50 |
| clarkb | we can use last octet of test node to set second octet for test vm range | 14:50 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiZmE6MTY6M2U6YjU6ZmE6YjZcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MTAzMjU4MzY3Mn0= | 14:50 |
| clarkb | that should isolate on /24s | 14:50 |
| *** sputnik13 has joined #openstack-infra | 14:50 | |
| sdague | clarkb: well, except, it shouldn't matter | 14:50 |
| sdague | because we're doing the right thing and not handing that address to that node | 14:50 |
| sdague | and, our default lease to cirros is 1d | 14:50 |
| *** stevemar has quit IRC | 14:50 | |
| *** wenlock1 has joined #openstack-infra | 14:51 | |
| clarkb | ya but it is extra noise | 14:51 |
| sdague | so, from the Nova side, it's handled correctly | 14:51 |
| *** dims has quit IRC | 14:51 | |
| sdague | sure, so the other option is *always* run multi host | 14:51 |
| sdague | becaus nova-net then protects against this | 14:51 |
| sdague | and makes dnsmasq only listen on internal interface | 14:52 |
| clarkb | ah | 14:52 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiZmE6MTY6M2U6YjU6ZmE6YjZcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MTAzMjU4MzY3Mn0= | 14:52 |
| *** sputnik13 has quit IRC | 14:52 | |
| sdague | so there is a DHCPNAK on http://logs.openstack.org/11/208511/12/check/gate-grenade-dsvm-partial-ncpu/dc137b4/logs/syslog.txt | 14:52 |
| *** sputnik13 has joined #openstack-infra | 14:53 | |
| sdague | at exactly the same time the same MAC address is getting an ACK from it's test node - http://logs.openstack.org/22/217522/5/check/gate-tempest-dsvm-full/be0fb79/logs/syslog.txt | 14:53 |
| sdague | 14:53 | |
| sdague | different patch different test jobs | 14:53 |
| *** markvoelker has joined #openstack-infra | 14:53 | |
| sdague | but time corolated exactly | 14:53 |
| *** Ala has quit IRC | 14:53 | |
| sdague | I got suspicious when I found out how much we DHCPNAK but succeed in runs | 14:54 |
| *** markvoelker has quit IRC | 14:54 | |
| sdague | and eventually dug through enough logs to realize the mac address in question wasn't one that existed in the job at all | 14:54 |
| *** markvoelker has joined #openstack-infra | 14:55 | |
| *** hdd has joined #openstack-infra | 14:55 | |
| *** samuelBartel has quit IRC | 14:56 | |
| openstackgerrit | John Trowbridge proposed openstack/diskimage-builder: [ironic-agent] Move service enable to install https://review.openstack.org/218879 | 14:56 |
| *** spzala has quit IRC | 14:57 | |
| *** jistr is now known as jistr|call | 14:57 | |
| sdague | also, in the land of promoting things we need: https://review.openstack.org/#/c/218852/ and https://review.openstack.org/#/c/218723/ - in that order. Which is stable/kilo and master patches to neutron to fix the netaddr issue with their l3 agent | 14:57 |
| sdague | grenade w/ neutron is currently 100% fail | 14:57 |
| sdague | so blocking grenade, devstack, tempest in addition to neutron | 14:58 |
| fungi | sdague: i guess as soon as 218871,3 finishes gate jobs, i'll pump those through after it | 14:58 |
| sdague | fungi: cool, the kilo patch has to go first, so that the master one will pass grenade | 14:58 |
| sdague | the master patch is currently jenkins -1 because of that | 14:58 |
| fungi | also i may just need to reboot jenkins01 straight up. not even enough memory to fork a ps | 15:00 |
| sdague | fungi: yeh, I found it was super slow this weekend | 15:00 |
| fungi | though on a positive note, the server's so bad off that nodepool hasn't been able to add any slaves to it for a while | 15:00 |
| sdague | heh | 15:00 |
| fungi | so i don't think this will interrupt any jobs at least | 15:00 |
| sdague | is there some kind of monit definition that could be used to auto kill things? | 15:01 |
| sdague | jenkins seems to go south at regular enough a clip | 15:01 |
| *** baoli has quit IRC | 15:01 | |
| fungi | for jenkins? probably not. under normal circumstances we've been putting them into a graceful shutdown (can be done with an authenticated api call) and then waiting for all running jobs to complete and deleting any ready nodes from them while that happens, then stopping the service and editing the global configuration to clear out remaining slaves | 15:02 |
| fungi | i'd be hesitant to automate that | 15:02 |
| *** baoli has joined #openstack-infra | 15:02 | |
| fungi | however, there's a recent commit that fixes a jenkins bug which may be related to the behavior clarkb reported, so upgrading is something we should look into soon | 15:03 |
| *** dims has joined #openstack-infra | 15:03 | |
| fungi | since it might make this stop being necessary again | 15:03 |
| sdague | would be nice | 15:03 |
| *** dizquierdo has quit IRC | 15:03 | |
| clarkb | I did respond to the bug to clarify again that we dont use a cloud plugin... | 15:03 |
| fungi | yeah, i saw | 15:04 |
| clarkb | but ya maybe we can get a build with that fix and try it real soon now | 15:04 |
| fungi | okay, jenkins01 is rebooting now | 15:04 |
| *** bnemec has quit IRC | 15:04 | |
| *** achanda has quit IRC | 15:05 | |
| *** flepied has joined #openstack-infra | 15:05 | |
| *** eantyshev has joined #openstack-infra | 15:05 | |
| fungi | i caught it before it came up enough to start running any jobs, stopped the service and cleaned the remaining slaves out of its config | 15:06 |
| clarkb | I do enjoy that the response to people thinking you use cloud plugins is that you should expect such broken behavior | 15:06 |
| *** gyee has joined #openstack-infra | 15:07 | |
| fungi | also that if you're not using long-lived slaves you must by default be using one of those broken cloud plugins | 15:07 |
| fungi | pretty sure they're going to come back and say we're not using jenkins slaves in the way that they're meant to be used, should use a cloud plugin instead | 15:08 |
| *** flepied1 has quit IRC | 15:08 | |
| *** dkranz has quit IRC | 15:08 | |
| derekh | Is there anybody else around who can take a look at a reshuffle of tripleo jobs/repos please https://review.openstack.org/#/c/205479/ https://review.openstack.org/#/c/215186/ | 15:09 |
| *** bnemec has joined #openstack-infra | 15:09 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 15:09 | |
| *** yfried has quit IRC | 15:11 | |
| *** tjones has joined #openstack-infra | 15:11 | |
| *** dkranz has joined #openstack-infra | 15:11 | |
| *** larainema has joined #openstack-infra | 15:11 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: update 1490374 for tempest hits as well https://review.openstack.org/218876 | 15:14 |
| *** tjones has left #openstack-infra | 15:14 | |
| *** wznoinsk1 is now known as wznoinsk | 15:15 | |
| *** stevemar has joined #openstack-infra | 15:16 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:17 | |
| *** woodster_ has joined #openstack-infra | 15:18 | |
| *** jtomasek has quit IRC | 15:20 | |
| *** stevemar has quit IRC | 15:21 | |
| fungi | nodepool seems to have ceased booting new nodes. looking into it now | 15:21 |
| *** zeih has joined #openstack-infra | 15:22 | |
| *** thedodd has joined #openstack-infra | 15:23 | |
| anteaya | fungi: :| | 15:23 |
| *** claudiub has joined #openstack-infra | 15:23 | |
| openstackgerrit | Evgeny Antyshev proposed openstack-infra/devstack-gate: Third-party rhel-based oses in devstack-gate https://review.openstack.org/215029 | 15:24 |
| *** FallenPegasus has joined #openstack-infra | 15:24 | |
| *** wshao_ has joined #openstack-infra | 15:24 | |
| *** ivar-lazzaro has joined #openstack-infra | 15:24 | |
| flip214 | Hi. Looking at the drbd tempest runs I can see that *all* failed during some 6-hour window today. was that some general problem, or do I start do dig deeper? | 15:25 |
| flip214 | picture link would be http://graphite.openstack.org/render/?width=600&height=344&_salt=1434709688.361&from=-3days&xFormat=%25b%25d-%25T&title=DRBD%20Cinder%2FDevstack%20stats&colorList=red%2Cgreen%2Cblue&target=stats_counts.zuul.pipeline.check.job.gate-tempest-dsvm-full-drbd-devstack-nv.FAILURE&target=stats_counts.zuul.pipeline.check.job.gate-tempest-dsvm-full-drbd-devstack-nv.SUCCESS | 15:26 |
| flip214 | sorry for the long line, but it's easier to see that way | 15:26 |
| *** jaypipes has quit IRC | 15:26 | |
| *** vtapia has joined #openstack-infra | 15:26 | |
| flip214 | hmmm, perhaps that's better: http://links.linbit.com/openstack-ci-results | 15:27 |
| fungi | last mention of demand from gearman was at 14:51 utc, which roughly corresponds to when it looks like nodepool may have ceased building new nodes | 15:27 |
| *** zeih has quit IRC | 15:27 | |
| openstackgerrit | Merged openstack/requirements: netaddr!=0.7.16 due to MAC validation regression https://review.openstack.org/218720 | 15:27 |
| *** ivar-lazzaro has quit IRC | 15:27 | |
| clarkb | flip214: if you use neutron it is ucrrently broken by a new netaddr release | 15:27 |
| sdague | fungi / clarkb - the tempest patch merged | 15:27 |
| openstackgerrit | Christopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory https://review.openstack.org/218898 | 15:28 |
| *** ivar-lazzaro has joined #openstack-infra | 15:28 | |
| sdague | so if we could get a promote on https://review.openstack.org/#/c/218852/ and https://review.openstack.org/#/c/218723/ - in that order | 15:28 |
| clarkb | flip214: if you don't use neutron then I think you may need to do more investgating | 15:28 |
| clarkb | sdague: once we figure out why we aren't building nodes we can do that | 15:28 |
| sdague | clarkb: ok | 15:28 |
| *** jsavak has joined #openstack-infra | 15:28 | |
| flip214 | clarkb: thanks for the hint, "AssertionError: False is not true : Invalid mac address detected." | 15:29 |
| *** k4n0 has quit IRC | 15:29 | |
| fungi | rax-ord: Forbidden: Quota exceeded for ram: Requested 8192, but already used 507904 of 512000 ram | 15:29 |
| *** flepied has quit IRC | 15:29 | |
| zaro | morning | 15:29 |
| fungi | though that's probably unrelated | 15:29 |
| anteaya | fungi: still not what we like to hear | 15:30 |
| anteaya | morning zaro | 15:30 |
| *** tiswanso has quit IRC | 15:30 | |
| clarkb | fungi: the gearman connection appears to be up and functioning, both sides report it ot be established | 15:30 |
| *** tiswanso has joined #openstack-infra | 15:31 | |
| *** david-ly_ has joined #openstack-infra | 15:31 | |
| clarkb | fungi: we did only calcuate demand 10 times during the 1400 hour | 15:32 |
| *** david-ly_ is now known as david-lyle_ | 15:33 | |
| clarkb | we hould be doing it every few seconds iirc | 15:33 |
| fungi | all the traceback noise in the debug log from broken tripleo-test-cloud-hp1 is tough to wade through | 15:33 |
| *** harlowja_at_home has joined #openstack-infra | 15:33 | |
| *** david-lyle has quit IRC | 15:33 | |
| clarkb | fungi: I think we should stack dump then lookto see what the main thrad is doing as that is where we calculate iirc | 15:33 |
| sdague | flip214: Bug 1490380 - netaddr 0.7.16 causes gate havoc | 15:34 |
| openstack | bug 1490380 in neutron "netaddr 0.7.16 causes gate havoc" [Critical,In progress] https://launchpad.net/bugs/1490380 - Assigned to Miguel Angel Ajo (mangelajo) | 15:34 |
| fungi | agreed, getting one now | 15:34 |
| sdague | flip214: http://status.openstack.org//elastic-recheck/#1490380 - you can see how we got wrecked by that as well | 15:34 |
| *** fedexo has joined #openstack-infra | 15:34 | |
| fungi | clarkb: SIGUSR2 YEAH? | 15:34 |
| fungi | er, capslockfail | 15:34 |
| *** david-lyle_ is now known as david-lyle | 15:35 | |
| Clint | more like capslock success | 15:35 |
| *** puranamr has joined #openstack-infra | 15:35 | |
| fungi | it's the toggle of AWESOME | 15:35 |
| flip214 | sdague: thanks, right. | 15:35 |
| flip214 | we're running in the openstack CI, so "your" problems are "our" problems as well ;/ | 15:35 |
| flip214 | thanks for all the hard work, BTW | 15:36 |
| clarkb | flip214: I think so but let me check | 15:36 |
| fungi | clarkb: yeah, that was it | 15:36 |
| clarkb | er fungi ^ and yes confirmed | 15:36 |
| sdague | flip214: so, if you use neutron, you need to wait for another set of patches to come in | 15:36 |
| clarkb | pabelanger: the centos7 image build issue apperas to possibly be a difference between rax image sand hpcloud images :/ | 15:37 |
| *** ifarkas has quit IRC | 15:37 | |
| flip214 | it already started working a few hours ago. | 15:37 |
| clarkb | pabelanger: they build fine on rax but not hpcloud so that is the next place to start looking | 15:37 |
| *** fhubik has quit IRC | 15:37 | |
| sdague | flip214: ok, yeh, then that was just the invalid macaddr thing | 15:37 |
| sdague | that was 0.7.16 having a big bug | 15:37 |
| mordred | clarkb: snapshot or dib? | 15:37 |
| clarkb | mordred: snapshot | 15:37 |
| sdague | there is still a behavior change in 0.7.17 that there are patches to fix | 15:37 |
| mordred | clarkb: lovely | 15:37 |
| clarkb | mordred: there is no different between the providers with dib | 15:37 |
| mordred | clarkb: I know - but I've long since stopped asusming | 15:38 |
| fungi | clarkb: 2015-08-31 15:35:43,851 DEBUG nodepool.stack_dump | 15:38 |
| fungi | clarkb: and i did a second one at 2015-08-31 15:36:47,214 DEBUG nodepool.stack_dump | 15:38 |
| fungi | for comparison purposes | 15:38 |
| mordred | clarkb: when you upload dibs manually - what have you been doing to get it to work consistenly. anything in particular? or just tons of retries? | 15:38 |
| mordred | s/consistently// | 15:39 |
| *** jtomasek has joined #openstack-infra | 15:39 | |
| clarkb | fungi: I see the issue | 15:39 |
| fungi | clarkb: Thread: Gearman client connect (140579170150144) seems to be spinning in waiter.acquire() | 15:39 |
| clarkb | fungi: I don't think thats the problem Thread: NodePool (140581185824512) is the main thread | 15:40 |
| clarkb | fungi: and its in reconfigure managers and its sitting on a jenkins open | 15:40 |
| clarkb | we might need to make that smarter considering our jenkins woes and fal late rather than early on that | 15:40 |
| fungi | oh... i wonder whether jenkins01 tanked it | 15:40 |
| clarkb | ya I think it did | 15:40 |
| fungi | okay, so any objection to me restarting nodepoold now? | 15:40 |
| clarkb | no let me paste this thread stack so we can debug it after but don't need you to wait | 15:41 |
| fungi | okay, nodepoold restarted | 15:41 |
| *** glauco has joined #openstack-infra | 15:42 | |
| fungi | it's booting a ton of nodes | 15:42 |
| clarkb | http://paste.openstack.org/show/435058/ | 15:42 |
| clarkb | we either need a timeout there or need to defer that action to later | 15:42 |
| fungi | the test nodes graph already indicates the change. i'll go ahead and promote those changes sdague mentioned now | 15:42 |
| *** glauco has quit IRC | 15:43 | |
| clarkb | fungi: but run loop is basically update config then check for needed nodes then launch nodes | 15:43 |
| *** glauco has joined #openstack-infra | 15:43 | |
| clarkb | so if update config hangs on jenkins we stop checking demand | 15:43 |
| clarkb | we have already calculated demand more in the last couple minutes than we did in the whole 1400 hour | 15:44 |
| TheJulia | SpamapS: greghaynes: wrt building your own IPA images, 218879 is required to fix the dib element | 15:44 |
| *** Piet has joined #openstack-infra | 15:44 | |
| clarkb | fungi: I am going to investigate the simple act of adding a timeout there | 15:46 |
| fungi | clarkb: thanks, sounds promising anyway | 15:46 |
| *** soren has quit IRC | 15:47 | |
| dims | fungi: clarkb: anyone report "Intel PCI CI" out of control yet? https://review.openstack.org/#/c/218468/ | 15:48 |
| pabelanger | clarkb: okay, thanks for the heads up | 15:49 |
| clarkb | fungi: looks like its doable so once I get breakfast I can whip something up for that then investigate how I broke centos7 image builds on hpcloud | 15:49 |
| dims | logging failures every few mins | 15:49 |
| fungi | dims: nope, you're the first caller | 15:49 |
| dims | ccarmack reported it on #openstack-qa | 15:49 |
| *** ccarmack has joined #openstack-infra | 15:49 | |
| *** amotoki has quit IRC | 15:50 | |
| dims | fungi: looks like it's hitting other reviews too :( https://review.openstack.org/#/c/216917/ | 15:51 |
| fungi | anteaya: feel up for writing to the tpci-ann ml about this while i deactivate the account? example https://review.openstack.org/218468 | 15:51 |
| fungi | dims: yep | 15:51 |
| ccarmack | Hi, can someone help me with Intel PCI CI? I uploaded a patch set, and now that CI is going crazy with failures. https://review.openstack.org/#/c/218468/ | 15:51 |
| anteaya | fungi: yes gerrit id 15751 | 15:51 |
| *** Sukhdev has joined #openstack-infra | 15:51 | |
| fungi | https://review.openstack.org/#/q/reviewer:pci-ci%2540intel.com+status:open,n,z | 15:52 |
| *** jistr|call is now known as jistr | 15:53 | |
| openstackgerrit | Christopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory https://review.openstack.org/218898 | 15:54 |
| fungi | i've deactivated their account, flushed gerrit's caches and am seeing if i can find and disconnect any open connections they currently have | 15:54 |
| *** glauco has quit IRC | 15:56 | |
| *** glauco has joined #openstack-infra | 15:56 | |
| *** mrmartin has joined #openstack-infra | 15:56 | |
| *** stevemar has joined #openstack-infra | 15:57 | |
| anteaya | fungi: thank you | 15:57 |
| anteaya | I will send the email | 15:57 |
| anteaya | ccarmack: thank you for speaking up | 15:57 |
| fungi | i see two open connections in gerrit from the intelpcici username, from different source addresses | 15:59 |
| ccarmack | anteaya: no problem, I see that dims had the same problem, and now the emails seemed to have stopped | 15:59 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 15:59 | |
| anteaya | ccarmack: great, thank you for that feedback | 15:59 |
| *** jsavak has quit IRC | 15:59 | |
| *** annegentle has quit IRC | 15:59 | |
| *** wshao_ has quit IRC | 15:59 | |
| *** jtomasek has quit IRC | 15:59 | |
| fungi | hrm, no close-connection api command in 2.9 | 16:00 |
| *** hdd has quit IRC | 16:00 | |
| *** markvoelker_ has joined #openstack-infra | 16:00 | |
| anteaya | no? | 16:00 |
| anteaya | oh zaro backported it | 16:00 |
| anteaya | are we on 2.9? | 16:00 |
| *** markvan_ is now known as markvan | 16:00 | |
| *** jsavak has joined #openstack-infra | 16:00 | |
| anteaya | our gerrit 2.9 has zaro's backport I think | 16:00 |
| *** sshnaidm has quit IRC | 16:00 | |
| fungi | er, 2.8 i meant | 16:01 |
| *** puranamr has quit IRC | 16:01 | |
| anteaya | yeah, not in 2.8 | 16:01 |
| anteaya | is is a 2.11 feature | 16:01 |
| fungi | yeah, looking to see if there's some other way to terminate specific connections in gerrit | 16:01 |
| anteaya | but can be backported to 2.10 and 2.9 | 16:01 |
| *** markvoel_ has joined #openstack-infra | 16:01 | |
| anteaya | fungi: go you | 16:01 |
| *** sridhar_ram has joined #openstack-infra | 16:01 | |
| *** sergmelikyan has joined #openstack-infra | 16:01 | |
| fungi | may just have to wait it out, or perhaps restart gerrit | 16:01 |
| anteaya | :( | 16:01 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 16:01 | |
| clarkb | and we cant 2.9 2.10 or 2.11 until jgit works again | 16:02 |
| *** bexelbie has quit IRC | 16:03 | |
| fungi | well, anyway it looks like intelpcici hasn't left any new review comments since 1600 utc, so several minutes now | 16:03 |
| *** markvoelker has quit IRC | 16:03 | |
| *** dannywilson has joined #openstack-infra | 16:04 | |
| fungi | actually looks like 15:53 was the most recent | 16:04 |
| fungi | so about the time i deactivated them | 16:04 |
| *** ivar-lazzaro has quit IRC | 16:04 | |
| *** baoli has quit IRC | 16:04 | |
| clarkb | fungi for ssh slaves looks like latest release does include the fix. maybe we should put 04 in shutdown mode? (thats the one running newer than default already) | 16:04 |
| *** trown is now known as trown|lunch | 16:04 | |
| sdague | fungi: thanks for promoting those patches | 16:04 |
| *** baoli has joined #openstack-infra | 16:05 | |
| sdague | lets see what the dust settles / looks like after all of this | 16:05 |
| anteaya | fungi: thanks | 16:05 |
| *** markvoelker_ has quit IRC | 16:05 | |
| anteaya | clarkb: yeah jgit :( | 16:05 |
| *** baoli has quit IRC | 16:05 | |
| anteaya | http://lists.openstack.org/pipermail/third-party-announce/2015-August/000263.html | 16:06 |
| anteaya | ccarmack: ^^ | 16:06 |
| *** dkranz has quit IRC | 16:06 | |
| *** markus_z has quit IRC | 16:06 | |
| *** hdd has joined #openstack-infra | 16:07 | |
| *** akraminsky has joined #openstack-infra | 16:07 | |
| *** mriedem_away is now known as mriedem | 16:08 | |
| ccarmack | anteaya: sorry, could you explain "please address your system"? Is that a message to the intel pci owner? | 16:08 |
| *** hashar_ has quit IRC | 16:08 | |
| *** matrohon has quit IRC | 16:08 | |
| anteaya | ccarmack: yes | 16:08 |
| ccarmack | ok, cool | 16:08 |
| anteaya | it means fix yourself | 16:09 |
| ccarmack | yes! | 16:09 |
| anteaya | thanks | 16:09 |
| *** flashgordon is now known as jogo | 16:09 | |
| *** dkranz has joined #openstack-infra | 16:10 | |
| fungi | heh... "go fix yourself" | 16:10 |
| fungi | clarkb: yeah, that sounds like a great idea | 16:10 |
| fungi | want me to? | 16:10 |
| *** sridhar_ram1 has joined #openstack-infra | 16:10 | |
| anteaya | fungi: should I use that next time? | 16:10 |
| anteaya | I'm collecting phrases | 16:10 |
| anteaya | I don't seem to have hit on the winning combination yet | 16:11 |
| SpamapS | TheJulia: ACK, thanks | 16:11 |
| clarkb | fungi: I just about have a jenkins api timeout patch ready have to make it pep8 happy | 16:11 |
| *** flepied has joined #openstack-infra | 16:12 | |
| clarkb | this line is one character too long :/ | 16:12 |
| *** bharathm has joined #openstack-infra | 16:12 | |
| *** sridhar_ram has quit IRC | 16:12 | |
| fungi | clarkb: awesome. i'll start quiescing jenkins04 while you adjust your terminal width | 16:12 |
| *** kiran-r has joined #openstack-infra | 16:13 | |
| *** glauco has quit IRC | 16:13 | |
| fungi | jenkins04 is in prepare for shutdown now, eta ~1.1 hours i thinkl | 16:13 |
| *** jistr has quit IRC | 16:13 | |
| clarkb | fungi: once its ready you can just upgrade via the gui to get the latest and any dep changes if necessary | 16:14 |
| clarkb | fungi: itwill show you what it upgraded in that case whcih we can then more automagically apply to the other masters | 16:14 |
| fungi | clarkb: yep, since there's a release with it. will do | 16:14 |
| *** spzala has joined #openstack-infra | 16:15 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config: Add python-gnocchiclient https://review.openstack.org/218826 | 16:15 |
| *** glauco has joined #openstack-infra | 16:15 | |
| *** armax has joined #openstack-infra | 16:15 | |
| *** fedexo has quit IRC | 16:15 | |
| *** glauco has quit IRC | 16:16 | |
| *** mpaolino has quit IRC | 16:16 | |
| *** bharathm has quit IRC | 16:16 | |
| *** tmorin has left #openstack-infra | 16:17 | |
| *** vtapia has quit IRC | 16:17 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/puppet-httpd: Only run http_mod for Debian os family for mod/*.pp https://review.openstack.org/216318 | 16:18 |
| *** vivekd has joined #openstack-infra | 16:18 | |
| *** tiswanso has quit IRC | 16:19 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/puppet-httpd: Only run http_mod for Debian os family for mod/*.pp https://review.openstack.org/216319 | 16:20 |
| *** e0ne has quit IRC | 16:20 | |
| *** glauco has joined #openstack-infra | 16:20 | |
| openstackgerrit | Merged openstack-infra/groups: Install ruby-bundler during bootstrapping https://review.openstack.org/209517 | 16:20 |
| mmmpork | i just added some folks to the review, but if anyone has a chance, I'd love if someone would review my nodepool testing doc changes - https://review.openstack.org/#/c/214319/ | 16:21 |
| *** sridhar_ram1 is now known as sridhar_ram | 16:21 | |
| clarkb | fungi: and now pypi is being cranky at me installing deps ... anyways its being worked on | 16:21 |
| *** jaypipes has joined #openstack-infra | 16:22 | |
| *** skylerberg has joined #openstack-infra | 16:22 | |
| *** tiswanso has joined #openstack-infra | 16:23 | |
| *** glauco has quit IRC | 16:23 | |
| *** sergmelikyan has quit IRC | 16:23 | |
| *** zeih has joined #openstack-infra | 16:23 | |
| anteaya | clarkb: :( | 16:24 |
| *** Swami has joined #openstack-infra | 16:24 | |
| *** glauco_ has joined #openstack-infra | 16:24 | |
| *** salv-orl_ has joined #openstack-infra | 16:25 | |
| *** unicell has quit IRC | 16:25 | |
| *** sergmelikyan has joined #openstack-infra | 16:26 | |
| greghaynes | TheJulia: Cool, I think I got past all my PEBKAC, now on to infra-cloud specific issues | 16:26 |
| SpamapS | greghaynes: o/ | 16:27 |
| greghaynes | SpamapS: One thing im running in to now is that glean seems to be too willing to overwrite our eni confic with dhcp | 16:27 |
| *** harlowja_at_home has quit IRC | 16:28 | |
| *** zeih has quit IRC | 16:28 | |
| *** salv-orlando has quit IRC | 16:28 | |
| SpamapS | greghaynes: so if we're using inventory_dhcp, we should not add network info | 16:28 |
| SpamapS | I didn't think of that before, but it seems imperative. | 16:28 |
| greghaynes | SpamapS: Yep, I have bifrost doing that - IIRC glean assumes no config = you want dhcp | 16:28 |
| SpamapS | OHH | 16:28 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: update signature for another expression https://review.openstack.org/218940 | 16:28 |
| SpamapS | yeah lets fix that | 16:28 |
| SpamapS | thats WAY too presumptuous | 16:29 |
| greghaynes | yep | 16:29 |
| SpamapS | are you sure we're not just accidentally including dhcp-all-interfaces or something? | 16:29 |
| SpamapS | which actually respects existing configs IIRC | 16:29 |
| greghaynes | SpamapS: not 100%, I need to find it in the glean code real quick... | 16:29 |
| *** tqtran has joined #openstack-infra | 16:30 | |
| mordred | greghaynes, SpamapS: so - the reason for that | 16:30 |
| SpamapS | I think glean should just leave any interface already configured alone. | 16:30 |
| greghaynes | yea, thatll break our current HP setup | 16:31 |
| greghaynes | i think | 16:31 |
| mordred | right. | 16:31 |
| *** markvoel_ has quit IRC | 16:31 | |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: update signature for another expression https://review.openstack.org/218940 | 16:31 |
| *** ociuhandu has joined #openstack-infra | 16:31 | |
| *** ashleighfarnham has joined #openstack-infra | 16:31 | |
| mordred | we can't make an image that has dhcp interfaces configured and then also tell glean to ignore configured interfaces | 16:31 |
| *** ashleighfarnham has quit IRC | 16:31 | |
| mordred | because we have one cloud where eth0 is static and one where it's dhcp | 16:31 |
| openstackgerrit | David Lyle proposed openstack/requirements: Changing the bounds for Django https://review.openstack.org/216469 | 16:31 |
| *** ashleighfarnham has joined #openstack-infra | 16:31 | |
| mordred | and in one of those config-drive does not include any network info | 16:31 |
| *** markvoelker has joined #openstack-infra | 16:31 | |
| greghaynes | mordred: what if we change it to not overwrite in that instance, only to write out an interface which doesnt exist | 16:32 |
| openstackgerrit | David Lyle proposed openstack/requirements: Bumping django_openstack_auth https://review.openstack.org/216710 | 16:32 |
| mordred | not overwriting I think is a fine idea | 16:32 |
| mordred | becuase our current workflow is ... | 16:32 |
| mordred | oh, wait | 16:32 |
| *** markvoelker_ has joined #openstack-infra | 16:32 | |
| mordred | I tihnk the reason we weren't doing that is that udev wouldn't make the thing in the first place if there wasn't a file at least to start with | 16:32 |
| *** sridhar_ram has quit IRC | 16:32 | |
| mordred | or something wouldn't get the event triggered | 16:32 |
| greghaynes | hrm | 16:33 |
| mordred | I _fully_ support this being better | 16:33 |
| *** alexsyip has joined #openstack-infra | 16:33 | |
| mordred | just trying to vomit the reasons I remember for why it is the way it is | 16:33 |
| greghaynes | yea, I remember us adding this for a reason | 16:33 |
| mordred | greghaynes: what's the tl;dr on the current problem? | 16:33 |
| mordred | greghaynes: what's inventory_dhcp? | 16:33 |
| greghaynes | mordred: Were baking bridge + vlan eni config into our infra-cloud images so they boot and dhcp on correct interfaces | 16:33 |
| mordred | ah | 16:34 |
| *** lifeless has quit IRC | 16:34 | |
| mordred | rather than passing that info via config drive | 16:34 |
| *** vtapia has joined #openstack-infra | 16:34 | |
| greghaynes | mordred: its 'your hosts get ip from dhcp' | 16:34 |
| mordred | yeah | 16:34 |
| *** ramashri has joined #openstack-infra | 16:34 | |
| greghaynes | yes - because even with the vlan config-drive patch we need to then alter the interface post-glean so its on a bridge, not just a vlan | 16:34 |
| mordred | that's why I was originally suggesting finishing teaching glean about the dhcp and vlan options that config drive can pass, and then teaching bifrost to pass in vlan info with "please use dhcp" in the config drive stanza | 16:34 |
| mordred | ah | 16:35 |
| mordred | bridge | 16:35 |
| mordred | bother | 16:35 |
| greghaynes | which would mean some puppet that operates with networking down | 16:35 |
| SpamapS | oh, well that seems like a rather smooth way forward | 16:35 |
| *** annegentle has joined #openstack-infra | 16:35 | |
| SpamapS | except, we want to DHCP on the bridge | 16:36 |
| SpamapS | not the interface | 16:36 |
| greghaynes | yes, exactly | 16:36 |
| *** markvoelker has quit IRC | 16:36 | |
| SpamapS | greghaynes: were we struggling with glean's vlan stuff? | 16:36 |
| SpamapS | greghaynes: because maybe we are violating layers here | 16:37 |
| greghaynes | SpamapS: no, I got the bifrost patch done | 16:37 |
| greghaynes | hrm? | 16:37 |
| mordred | SpamapS: glean doesn't grok bridges at all | 16:37 |
| mordred | SpamapS: yeah | 16:37 |
| SpamapS | greghaynes: maybe we should just let glean setup vlan25, and then just always run post-glean | 16:37 |
| mordred | yeah | 16:37 |
| *** markvoelker_ has quit IRC | 16:37 | |
| jklare | hi again, anybody got a moment to help me with a periodic job that does not run at all? | 16:37 |
| mordred | like, have glean set up eth0 - and then have post-glean move the interface into the bridge | 16:37 |
| SpamapS | so add to the image a post-up.d that moves the network info to the bridge | 16:37 |
| *** Hal has joined #openstack-infra | 16:37 | |
| mordred | SpamapS: ++ | 16:38 |
| mordred | I think that's the right separation of concerns | 16:38 |
| mordred | SpamapS: while we're gleaning: https://review.openstack.org/#/c/212177/ | 16:38 |
| mordred | SpamapS: I'd love to land that and cut a new glean | 16:38 |
| SpamapS | greghaynes: perhaps chalk this up to another educational failure? ;-) | 16:38 |
| *** geoffarnold has joined #openstack-infra | 16:39 | |
| fungi | jklare: what periodic job? is it filtered (to a specific branch or certain set of files)? if so, that's the most common cause | 16:39 |
| greghaynes | ok, I wanted to do this because it is very low effort and I think itll 'just work'... how about I deploy crinkle's two hosts like this real quick and you can work on the post-up bit? | 16:39 |
| *** sergmelikyan has quit IRC | 16:39 | |
| greghaynes | and then ill poke at bifrost + vlan patch | 16:39 |
| *** Hal has quit IRC | 16:41 | |
| *** lifeless has joined #openstack-infra | 16:41 | |
| greghaynes | SpamapS: other thought - with the glean + vlan what subnet do we want the ironic agent to run over? | 16:41 |
| greghaynes | still untagged? | 16:41 |
| SpamapS | greghaynes: yeah lets get the hosts that help people move forward in parallel up for sure | 16:42 |
| SpamapS | greghaynes: that seems like the simplest thing yeah | 16:42 |
| greghaynes | I guess it has to otherwise we have to edit the ramdisk | 16:42 |
| clarkb | mmmpork: reviewed | 16:42 |
| greghaynes | ok, well then that was a lot of the work I was doing so not a big waste ;) | 16:42 |
| SpamapS | unless we want to start adding glean to IPA's | 16:42 |
| SpamapS | which I don't think we do! ;) | 16:42 |
| jklare | fungi yeah, acutally it is, but i do not understand why it is getting filtered (let me give you the link.... | 16:43 |
| krotscheck | Anyone here have opinions on languge style guides? | 16:43 |
| fungi | jklare: the periodic pipeline is changeless and branchless, so adding any filters to the job will cause it not to be selected | 16:43 |
| jklare | fungi https://review.openstack.org/#/c/194164/ | 16:43 |
| *** Hal has joined #openstack-infra | 16:43 | |
| greghaynes | SpamapS: TheJulia ok, one patch I need to write up for bifrost - in a lot of places its using ansible's default network interface to determine the controller IP, I think that needs to be exposed as a setting so we can force it to use the 10.x address rather than 15.x | 16:43 |
| mmmpork | clarkb: thanks! i'll look over the changes and submit revisions! | 16:43 |
| krotscheck | If so, go look at these and weigh in :) https://review.openstack.org/#/q/status:open+project:openstack/eslint-config-openstack,n,z | 16:43 |
| mmmpork | i figured there'd be a few problems ;D | 16:43 |
| *** arxcruz has quit IRC | 16:44 | |
| greghaynes | SpamapS: TheJulia Right now I have that hacked in, but that needs to happen with either config_drive or inventory_dhcp | 16:44 |
| *** jsavak has quit IRC | 16:44 | |
| clarkb | mmmpork: I think the biggest one is dont use git review -x and use -d instead | 16:44 |
| *** jsavak has joined #openstack-infra | 16:44 | |
| clarkb | mmmpork: -x is just problematic if you don't understand what it is doing so I don't like putting it in docs as a thing people will blindly copy pasta | 16:44 |
| jklare | fungi ah, so i have to figure out a way to not filter this job... but what if it can only run for newer branches? | 16:44 |
| fungi | jklare: the way we usually solve that is to have branch-specific job names for the periodic pipeline (such as gate-openstack-chef-repo-chef-rake-integration-kilo and gate-openstack-chef-repo-chef-rake-integration-master) by using a branch-designator parameter in the job-template to vary its name | 16:45 |
| jklare | fungi ok, i will try this and put in a patch later | 16:45 |
| greghaynes | SpamapS: I'm guessing that is what you meant by 'split up bifrost networking between deploy and agent networks'? | 16:46 |
| fungi | jklare: basically, the zuul periodic pipelines, because they're not tied to proposed changes, lack context for things like branches so you have to specify the branch to run within the job definition | 16:46 |
| jklare | fungi i will add you as a reviewer if thats ok? | 16:46 |
| mmmpork | clarkb: thanks! i'm glad that the mistakes were things i brought over from the README and not my actual things haha | 16:46 |
| fungi | jklare: sure, that's fine, though i think most of the project-config-core reviewers are familiar with the concept | 16:46 |
| mmmpork | so thanks for taking the time to look at that, i'll make those fixes and resubmit | 16:46 |
| *** davideagnello has joined #openstack-infra | 16:47 | |
| *** jerryz has joined #openstack-infra | 16:47 | |
| SpamapS | greghaynes: Yes | 16:47 |
| SpamapS | greghaynes: in places it is trying to have them separate, but it's not complete. | 16:48 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: WIP: Removes posix-ipc from global-requirements https://review.openstack.org/217824 | 16:48 |
| greghaynes | ah, ok | 16:48 |
| *** ayoung has quit IRC | 16:48 | |
| *** davideagnello has quit IRC | 16:49 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds os-win repository https://review.openstack.org/209204 | 16:49 |
| SpamapS | greghaynes: so yeah, get crinkle some boxes to test the openstack-itself puppet on. And I'll poke at the glean vlan patch and maybe a post-up.d that will move vlan25's network info to a bridge. Or did you already have the bridge script? | 16:50 |
| *** davideagnello has joined #openstack-infra | 16:50 | |
| greghaynes | SpamapS: I dont have a bridge script, no | 16:50 |
| *** yamahata has joined #openstack-infra | 16:50 | |
| *** tqtran has quit IRC | 16:51 | |
| greghaynes | SpamapS: If you want to poke at the vlan patch - I am cycling on node 2e251e9d-0b00-477c-9069-f9dbe966b5e3 - you could cycle on any of the others | 16:51 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Add a Jenkins API timeout. https://review.openstack.org/218949 | 16:52 |
| clarkb | fungi: ^ note that is completely untested by any of our testing :/ | 16:52 |
| *** sdake_ has joined #openstack-infra | 16:52 | |
| *** jsavak has quit IRC | 16:53 | |
| *** jsavak has joined #openstack-infra | 16:53 | |
| *** wenlock_ has joined #openstack-infra | 16:54 | |
| clarkb | gah that needs voluptuous updates too | 16:55 |
| *** dims has quit IRC | 16:55 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Add a Jenkins API timeout. https://review.openstack.org/218949 | 16:56 |
| *** salv-orl_ has quit IRC | 16:56 | |
| docaedo | voluptuous updates are the best kind | 16:56 |
| zaro | ianw: any reason this hasn't merged? https://review.openstack.org/#/c/200727/ | 16:57 |
| zaro | * merged/approved | 16:58 |
| *** wenlock_ has quit IRC | 16:59 | |
| *** dizquierdo has joined #openstack-infra | 17:00 | |
| *** puranamr has joined #openstack-infra | 17:01 | |
| derekh | Are their other issues with the new netaddr package ? we're seeing this on tripleo https://bugs.launchpad.net/tripleo/+bug/1490639 | 17:01 |
| openstack | Launchpad bug 1490639 in tripleo "Floating IP's no longer working" [Undecided,New] | 17:01 |
| *** earlephilhower has joined #openstack-infra | 17:02 | |
| *** dannywilson has quit IRC | 17:02 | |
| *** bhunter71 has joined #openstack-infra | 17:02 | |
| derekh | nevermind, I see neutron is on it | 17:03 |
| SpamapS | greghaynes: +1 thanks | 17:03 |
| ihrachys | derekh: it's just fixed in neutron | 17:04 |
| derekh | ihrachys: yup, thanks, I just noticed it after I posted my message | 17:05 |
| *** ihrachys has quit IRC | 17:05 | |
| *** derekh has quit IRC | 17:05 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: add new periodic job for chef integration testing https://review.openstack.org/218955 | 17:05 |
| *** rossella_s has quit IRC | 17:05 | |
| *** trown|lunch is now known as trown | 17:06 | |
| *** rossella_s has joined #openstack-infra | 17:06 | |
| *** sridhar_ram has joined #openstack-infra | 17:09 | |
| *** kiran-r has quit IRC | 17:10 | |
| *** julim has joined #openstack-infra | 17:11 | |
| fungi | Nakato: if you need any cooperation from individual project-teams (particularly neutron!) on the tox/unit test constraints work, http://lists.openstack.org/pipermail/openstack-dev/2015-August/073302.html would be a great place to chime in | 17:11 |
| *** tonytan4ever has quit IRC | 17:12 | |
| *** unicell has joined #openstack-infra | 17:12 | |
| openstackgerrit | Augustina Ragwitz proposed openstack-infra/nodepool: Improve testing and initial installation documentation https://review.openstack.org/214319 | 17:13 |
| mmmpork | clarkb: i made the corrections you suggested! thanks so much | 17:13 |
| *** ybathia has joined #openstack-infra | 17:14 | |
| jklare | fungi is there any documentation on these 'branch designators' in job templates? | 17:14 |
| sdake_ | gmann what can I do for you | 17:14 |
| greghaynes | clarkb: Hey - so you said growroot worked and tests passed, are we just waiting on more reliable uploads for ubuntu-trusty? | 17:15 |
| clarkb | jklare: no and they are poorly named should just be called 'suffix' or similar | 17:15 |
| clarkb | jklare: they are mostly used to duplicate jobs with uniue names so that we can change behavior in zuul | 17:15 |
| clarkb | jklare: non voting, specific branch, test against source of another project, etc | 17:15 |
| clarkb | greghaynes: we are | 17:15 |
| greghaynes | clarkb: hrm, ok. I wonder if we should try another image type too now? | 17:16 |
| clarkb | greghaynes: centos7 maye | 17:16 |
| clarkb | or f22 depending on how brave you are | 17:16 |
| greghaynes | haha | 17:16 |
| greghaynes | two of my favorite things | 17:17 |
| *** sergmelikyan has joined #openstack-infra | 17:17 | |
| *** achanda has joined #openstack-infra | 17:18 | |
| fungi | jklare: clarkb: though in the current case they are aptly named since the desire seems to be to designate the branch on you want to run the job when in the periodic pipeline | 17:19 |
| fungi | (which was also the original use for that parameter, hence its name) | 17:20 |
| *** madhu_ak has joined #openstack-infra | 17:20 | |
| *** tnovacik has quit IRC | 17:22 | |
| *** vivekd has quit IRC | 17:23 | |
| sdague | it appears that rax has added some really slow nodes | 17:23 |
| *** mjturek1 has quit IRC | 17:23 | |
| *** zeih has joined #openstack-infra | 17:24 | |
| sdague | hmmm... actually, it's zm02.openstack.org being under a lot of load ? | 17:25 |
| sdague | fungi / clarkb - so workspace setup is now often taking 40 minutes? - http://logs.openstack.org/86/218286/1/check/gate-tempest-dsvm-large-ops/1044c69//logs/devstack-gate-setup-workspace-new.txt.gz | 17:25 |
| *** e0ne has joined #openstack-infra | 17:26 | |
| *** hdd has quit IRC | 17:27 | |
| fungi | yeeowch | 17:28 |
| *** zeih has quit IRC | 17:28 | |
| greghaynes | http://logs.openstack.org/86/218286/1/check/gate-tempest-dsvm-large-ops/1044c69//logs/devstack-gate-setup-workspace-new.txt.gz#_2015-08-31_16_21_41_064 a lot of time on that line... what is that doing? | 17:28 |
| fungi | sdague: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1540&rra_id=all | 17:29 |
| greghaynes | just a super slow clone? | 17:29 |
| *** dims has joined #openstack-infra | 17:29 | |
| *** amuller is now known as amuller_afk | 17:29 | |
| fungi | sdague: got so bad it was unresponsive for 20 minutes | 17:29 |
| sdague | 2015-08-31 16:48:13.542 | + git_fetch_at_ref openstack/heat refs/zuul/master/Zcd98780d74ad49518c70397c8e2c3912 - took 90 seconds, for instance | 17:29 |
| sdague | fungi: ok, yeh, that's triggering a ton of fails on jobs from timeouts | 17:29 |
| sdague | however, I don't think it's just the last 20 minutes | 17:30 |
| *** dims_ has joined #openstack-infra | 17:30 | |
| jeblair | we may need to clean out the refs | 17:30 |
| *** dims has quit IRC | 17:30 | |
| *** hdd has joined #openstack-infra | 17:30 | |
| sdague | http://status.openstack.org//elastic-recheck/#1311066 - started hitting again 4 days ago | 17:30 |
| sdague | out of the blue | 17:30 |
| *** mjturek1 has joined #openstack-infra | 17:30 | |
| fungi | compare to http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1518&rra_id=all for zm01 | 17:31 |
| sdague | right, so zm02 is way slower anyway | 17:31 |
| *** ildikov has quit IRC | 17:31 | |
| sdague | as zm01 is rarely getting above loadavg 2 | 17:31 |
| *** larainema has quit IRC | 17:32 | |
| sdague | and zm02 is averaging load avg 7 over that time period | 17:32 |
| jeblair | they have approx the same number of refs in nova | 17:33 |
| *** sdake has joined #openstack-infra | 17:33 | |
| sdague | jeblair: how about heat? | 17:34 |
| sdague | the nova fetch was actually super speedy | 17:34 |
| sdague | http://logs.openstack.org/86/218286/1/check/gate-tempest-dsvm-large-ops/1044c69//logs/devstack-gate-setup-workspace-new.txt.gz#_2015-08-31_16_30_00_159 | 17:34 |
| clarkb | it does the remote branch listing thing to prune stale branches iirc | 17:35 |
| *** sthillma has joined #openstack-infra | 17:35 | |
| clarkb | that is slow when you have a bunch of refs | 17:35 |
| *** baoli has joined #openstack-infra | 17:35 | |
| *** jaosorior has quit IRC | 17:35 | |
| jeblair | sdague: about the same too (about 25% of nova) | 17:35 |
| *** stevemar has quit IRC | 17:35 | |
| sdague | yeh, I don't know. Any chance it's just a noisy neighbor problem? | 17:36 |
| *** e0ne has quit IRC | 17:36 | |
| *** tonytan4ever has joined #openstack-infra | 17:36 | |
| jeblair | sdague: that's possible :/ | 17:36 |
| *** hdd has quit IRC | 17:36 | |
| *** hdd has joined #openstack-infra | 17:37 | |
| *** pvaneck has joined #openstack-infra | 17:37 | |
| sdague | because, it's definitely gotten a ton worse over the last 4 days | 17:37 |
| *** baoli has quit IRC | 17:37 | |
| sdague | this typically ends up triggering failures in the jobs because they run out of time | 17:37 |
| *** sdake_ has quit IRC | 17:37 | |
| sdague | like large-ops, which I think is 90 minutes total timeout | 17:37 |
| sdague | but when we spend 40 minutes in setup workspace, you're toast | 17:37 |
| *** baoli has joined #openstack-infra | 17:38 | |
| jeblair | there's a definite memory/swap spike at the same time; unsure whether that's cause or effect | 17:38 |
| *** stevemar has joined #openstack-infra | 17:38 | |
| *** e0ne has joined #openstack-infra | 17:39 | |
| openstackgerrit | Augustina Ragwitz proposed openstack/diskimage-builder: Update dpkg elements README from free text to table formatting https://review.openstack.org/218965 | 17:40 |
| *** bharathm has joined #openstack-infra | 17:41 | |
| mmmpork | greghaynes: ^^ | 17:41 |
| greghaynes | tyty | 17:41 |
| *** stevemar has quit IRC | 17:42 | |
| *** sputnik13 has quit IRC | 17:43 | |
| *** ig0r__ has joined #openstack-infra | 17:43 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: warn when workspace setup takes a long time https://review.openstack.org/218966 | 17:44 |
| *** ashleighfarnham has quit IRC | 17:44 | |
| sdague | ok, I'm proposing we add a timing warning there as well that we can directly get a signature on | 17:44 |
| sdague | the devstack one is secondary | 17:45 |
| sdague | and they are slow for different reasons | 17:45 |
| sdague | so we should warn and track them separately | 17:45 |
| openstackgerrit | Christian Berendt proposed openstack-infra/groups: Add a basic Vagrantfile for a development environment https://review.openstack.org/209577 | 17:45 |
| clarkb | the times there don't line up for the centos7 switch fwiw | 17:45 |
| clarkb | centos7 swtich happened early 29th UTC | 17:46 |
| *** ig0r_ has quit IRC | 17:46 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:46 | |
| clarkb | also I have a test change up that trims down the PROJECT list to make this less bad | 17:47 |
| *** mjturek1 has quit IRC | 17:47 | |
| clarkb | it will take a lot of work to do on the JJB side though I think | 17:47 |
| pvaneck | jeblair: Hi, when you get a chance, could you review https://review.openstack.org/#/c/214950/ ? | 17:49 |
| *** ashleighfarnham has joined #openstack-infra | 17:50 | |
| madhu_ak | dougwig: could you please have a look at this patch: https://review.openstack.org/#/c/211767/? | 17:50 |
| *** MIDENN_ has joined #openstack-infra | 17:50 | |
| jeblair | pvaneck: ack | 17:51 |
| *** doug-fish has quit IRC | 17:51 | |
| *** doug-fish has joined #openstack-infra | 17:51 | |
| *** Sukhdev has quit IRC | 17:51 | |
| *** FallenPegasus has quit IRC | 17:53 | |
| *** mjturek1 has joined #openstack-infra | 17:53 | |
| jeblair | fungi, sdague: the spike on zm02 has a good timing correlation with the several hundred nods launched when fungi restarted nodepool | 17:56 |
| sdague | jeblair: ok, except the failure window here is not limitted to just that spike | 17:57 |
| *** shayneburgess has left #openstack-infra | 17:59 | |
| *** sputnik13 has joined #openstack-infra | 18:00 | |
| *** FallenPegasus has joined #openstack-infra | 18:00 | |
| *** jeadre has quit IRC | 18:01 | |
| fungi | yeah, zm03 for example saw a spike too, but its load average only jumped to 30 not 140 | 18:01 |
| *** jeadre has joined #openstack-infra | 18:01 | |
| fungi | suggesting that zm02 is suffering some other performance degredation, perhaps | 18:02 |
| *** stevemar has joined #openstack-infra | 18:03 | |
| *** salv-orlando has joined #openstack-infra | 18:04 | |
| *** shayneburgess has joined #openstack-infra | 18:05 | |
| *** abdelwas has joined #openstack-infra | 18:06 | |
| *** FallenPegasus has quit IRC | 18:06 | |
| *** davideagnello has quit IRC | 18:06 | |
| jeblair | sdague: looking at logstash links for the "slow node" query, the jobs are not simple jobs. they are largely multinode or ironic-pxe_ssh | 18:06 |
| jeblair | it may be that "merger load spike" and "appearance of jobs triggering slow node query" are different issues | 18:07 |
| clarkb | multinode with the serial devstacking may trip that more | 18:08 |
| sdague | jeblair: yeh, unfortunately the timeouts that seem to be hurting us are the largeops jobs | 18:08 |
| sdague | anyway https://review.openstack.org/218966 | 18:08 |
| sdague | gives us the timings on the workspace create | 18:08 |
| sdague | which is what will be impacted by the zm02 slowness | 18:09 |
| *** sergmelikyan has quit IRC | 18:09 | |
| sdague | the devstack slow warning isn't really a great indicator | 18:09 |
| *** salv-orlando has quit IRC | 18:09 | |
| jeblair | sdague: oh, are you saying you have noticed in increased incidence of large-ops failing in this manner? | 18:09 |
| *** dizquierdo has quit IRC | 18:09 | |
| jeblair | i thought the only trend we had was the slow-query | 18:10 |
| sdague | jeblair: yes, sorry, that's why I went down this rabbit hole | 18:10 |
| sdague | large-ops failures | 18:10 |
| jeblair | sdague: have any more handy? | 18:11 |
| sdague | not atm | 18:11 |
| sdague | was chasing cinder unit test fails | 18:12 |
| openstackgerrit | Christian Berendt proposed openstack-infra/groups: Add a basic Vagrantfile for a development environment https://review.openstack.org/209577 | 18:12 |
| fungi | clarkb: jenkins04 is idle now. i'll give upgrading the plugin a go | 18:13 |
| clarkb | fungi: thanks let me know if Ican help | 18:13 |
| jeblair | fungi, clarkb: oh, what plugin? | 18:13 |
| clarkb | upstream puppet wildfly isn't a huge fan of my update to make upgrades possible :/ I have responded and hopeflly they turn around | 18:14 |
| clarkb | jeblair: ssh-slaves | 18:14 |
| clarkb | jeblair: there was a recent (this month) update to the plugin to fix a deadlock situation which we think may be our problem | 18:14 |
| jeblair | clarkb: is it sposed to fix the leaks? | 18:14 |
| jeblair | neat! | 18:14 |
| clarkb | jeblair: it is supposed to fix a deadlock, not really sure if its ourdeadlock | 18:14 |
| fungi | pabelanger: ianw: i just saw someone mention anecdotally on the distutils-sig ml in a packaging thread that "dnf 1.1.0 just fixed package download caching" | 18:14 |
| clarkb | but seems worht a shot | 18:14 |
| asselin_ | clarkb, fungi let me know if I can help test on my system as well | 18:14 |
| *** FallenPegasus has joined #openstack-infra | 18:15 | |
| fungi | jeblair: theory being that we delete the slave before the cleanup finishes in jenkins, and it leaves a stuck thread behind trying to ssh to the slave to complete teh cleanup | 18:15 |
| clarkb | asselin_: we hit it regularly enough taht we should be able to test it effectively on our master(s) but feel free to upgrade ssh-slaves plugin to latest and see if it fixes on your end | 18:15 |
| asselin_ | ok | 18:15 |
| jeblair | fungi: nice | 18:15 |
| fungi | jeblair: the patch in question implements a timeout for that operation, allowing the thread to terminate gracefully | 18:15 |
| pabelanger | fungi: ORLY? | 18:16 |
| pabelanger | checking it out now | 18:16 |
| *** pelix has quit IRC | 18:17 | |
| *** markvoelker has joined #openstack-infra | 18:17 | |
| *** hashar has joined #openstack-infra | 18:18 | |
| jeblair | sdague: typo in 218966 | 18:18 |
| *** julim has quit IRC | 18:18 | |
| clarkb | fungi: the thing to keep an eye on is the thread count in melody and specifically see if the forremoting thread count skyrockets | 18:19 |
| clarkb | (that is a case insensitive search string) | 18:19 |
| *** hdd has quit IRC | 18:20 | |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: add signature for cinder pyopenssl issue https://review.openstack.org/218976 | 18:20 |
| *** sergmelikyan has joined #openstack-infra | 18:20 | |
| pabelanger | fungi: ianw: looks like they added back the --downloadonly flag | 18:20 |
| *** ildikov has joined #openstack-infra | 18:20 | |
| pabelanger | https://bugzilla.redhat.com/show_bug.cgi?id=1048433 | 18:20 |
| openstack | bugzilla.redhat.com bug 1048433 in dnf "RFE --downloadonly" [Unspecified,New] - Assigned to packaging-team-maint | 18:20 |
| fungi | w00t! | 18:20 |
| pabelanger | going to give it a test now | 18:21 |
| *** markvoelker has quit IRC | 18:21 | |
| *** julim has joined #openstack-infra | 18:21 | |
| *** b10n1k_ has joined #openstack-infra | 18:21 | |
| *** FallenPegasus has quit IRC | 18:21 | |
| sdague | jeblair: looking | 18:21 |
| fungi | the road to happiness is paved with small victories | 18:21 |
| *** jtomasek has joined #openstack-infra | 18:22 | |
| fungi | (or something) | 18:22 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: warn when workspace setup takes a long time https://review.openstack.org/218966 | 18:22 |
| sdague | jeblair: good catch, thanks | 18:23 |
| fungi | clarkb: looks like the ssh slaves plugin was the only one it downloaded when updating | 18:23 |
| *** hashar has quit IRC | 18:23 | |
| clarkb | fungi: great, I know in the last upgrade it updated another plugin | 18:24 |
| *** zeih has joined #openstack-infra | 18:25 | |
| *** davideagnello has joined #openstack-infra | 18:25 | |
| *** spzala has quit IRC | 18:28 | |
| *** MarkAtwood has quit IRC | 18:28 | |
| *** dtantsur|afk is now known as dtantsur | 18:29 | |
| *** zeih has quit IRC | 18:29 | |
| *** FallenPegasus has joined #openstack-infra | 18:30 | |
| fungi | clarkb: ssh slaves plugin 1.10 was what you were expecting, right? | 18:31 |
| clarkb | fungi: ys | 18:32 |
| clarkb | fungi: it appears to have released right after that bugfix and according to changelog includes the deadlock fix | 18:32 |
| fungi | awesome. fingers crossed | 18:32 |
| fungi | uncrossed now. made typing too hard | 18:32 |
| *** gyee has quit IRC | 18:33 | |
| fungi | the clouds favor us. we're getting pretty decent node utilization for a change | 18:33 |
| openstackgerrit | Augustina Ragwitz proposed openstack/diskimage-builder: Update dpkg elements README from free text to table formatting https://review.openstack.org/218965 | 18:34 |
| clarkb | fungi: right now 04 has 8 forremoting threads | 18:34 |
| clarkb | fungi: I think <100 is normal and a few thousand is definitely broken | 18:35 |
| *** mjturek1 has quit IRC | 18:36 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config: Add python-gnocchiclient https://review.openstack.org/218826 | 18:36 |
| fungi | clarkb: cool. probably have to give it a few days for confirmation unfortunately | 18:36 |
| clarkb | ya | 18:36 |
| fungi | but here's hoping | 18:36 |
| *** sputnik13 has quit IRC | 18:37 | |
| *** FallenPegasus has quit IRC | 18:37 | |
| *** tonytan4ever has quit IRC | 18:38 | |
| *** tonytan4ever has joined #openstack-infra | 18:38 | |
| openstackgerrit | Christian Berendt proposed openstack-infra/groups: Add a basic Vagrantfile for a development environment https://review.openstack.org/209577 | 18:38 |
| *** tonytan4ever has quit IRC | 18:38 | |
| *** ayoung has joined #openstack-infra | 18:38 | |
| fungi | my dyslexia saw "add a balsamic vinaigrette file" | 18:39 |
| jeblair | fungi: hungry? | 18:39 |
| clarkb | pour over your salad | 18:39 |
| fungi | mmmvagrantsalad | 18:39 |
| *** salv-orlando has joined #openstack-infra | 18:39 | |
| anteaya | I'm actually making a salad | 18:41 |
| anteaya | blackbean and pink grapefruit | 18:41 |
| *** mrmartin has quit IRC | 18:42 | |
| fungi | that sounds like a most excellent salad | 18:43 |
| *** mjturek1 has joined #openstack-infra | 18:43 | |
| fungi | i do like fruit and bean salads | 18:43 |
| *** pcm_ has joined #openstack-infra | 18:43 | |
| fungi | a generally under-appreciated category of saladness | 18:43 |
| anteaya | agreed | 18:45 |
| anteaya | with walnuts | 18:45 |
| *** pc_m has quit IRC | 18:45 | |
| anteaya | and fresh basil if it is around | 18:45 |
| clarkb | fritos, beans and salsa | 18:45 |
| *** FallenPegasus has joined #openstack-infra | 18:46 | |
| anteaya | ha ha ha | 18:47 |
| anteaya | there's that too | 18:47 |
| *** juggler has quit IRC | 18:47 | |
| pabelanger | fungi: okay, so there is a fix for --downloadonly, but not merged into dnf yet | 18:47 |
| fungi | pabelanger: does it look like its future is promising, or are the primary dnf devs balking at it? | 18:48 |
| pabelanger | fungi: unsure | 18:48 |
| pabelanger | going to try and loop myself into the discussions | 18:48 |
| *** ig0r__ has quit IRC | 18:48 | |
| fungi | pabelanger: at any rate, --downloadonly may be separate from fixing download caching | 18:48 |
| clarkb | pabelanger: looking at puppet logs nothing about firewalld is logged | 18:49 |
| clarkb | pabelanger: so I wonder if facter is giving us different data and making things go wonky | 18:49 |
| fungi | pabelanger: since ianw indicated that dnf's current package caching is useless/nonexistent | 18:49 |
| fungi | (or was, perhaps until 1.1.0) | 18:49 |
| *** jeadre has quit IRC | 18:49 | |
| *** jeadre has joined #openstack-infra | 18:50 | |
| clarkb | pabelanger: but it does claim to install the package, the two rules files (ipv6 and ipv4) then it tries to "start" the service to apply them and that fails | 18:50 |
| *** FallenPegasus has quit IRC | 18:50 | |
| *** mjturek2 has joined #openstack-infra | 18:52 | |
| glauco_ | is there a major concern on migrating to use puppetlab official vcsrepo? https://github.com/puppetlabs/puppetlabs-vcsrepo | 18:52 |
| pabelanger | fungi: Ya. We had some thought about using squid, but unsure if that is the right approach either | 18:52 |
| glauco_ | we noticed that openstack version is quite outdated and we would like to use the depth param to speed up some configurations | 18:53 |
| pabelanger | clarkb: Odd. I don't have an HP account otherwise, I'd try and poke around too | 18:53 |
| *** mjturek1 has quit IRC | 18:53 | |
| clarkb | pabelanger: ya I intend on booting a node and poking around | 18:53 |
| *** stevemar has quit IRC | 18:54 | |
| clarkb | glauco_: it didn't work way back when and mordred made changes that were not compatible with upstream to make it work iirc | 18:54 |
| clarkb | glauco_: in general I advise against shallow clones as they come with a large number of caveats and they are broken on ubuntu trusty for example | 18:54 |
| clarkb | glauco_: its possible that the upstream module would work today, would need to test it | 18:55 |
| mordred | clarkb: aroo? | 18:55 |
| clarkb | mordred: puppet vcsrepo | 18:55 |
| clarkb | mordred: glauco_ is wondering if we can use upstream again | 18:55 |
| mordred | ah. yeah. it's designed around a different use case | 18:55 |
| mordred | _possibly_ | 18:55 |
| glauco_ | clarkb: for some context, we noticed that zuul installation is taking a long time because it is checking out the whole twitter bootstrap repo | 18:55 |
| *** edmondsw_ has joined #openstack-infra | 18:56 | |
| glauco_ | would using a tarball generated by github be a better alternative to shallow git repos? | 18:56 |
| *** edmondsw_ has quit IRC | 18:56 | |
| mordred | the upstream one did not do what we considered to be the right thing related to wanting ot always have master, a tag or a revision in place | 18:56 |
| nibalizer | glauco_: i'd like to see us get to a modern vcsrepo | 18:56 |
| *** edmondsw has joined #openstack-infra | 18:56 | |
| mordred | and goes out of its way to not overwrite a repo state if a repo is a state other than what's declared | 18:56 |
| nibalizer | I agree with clark that shallow clones save you run-time at the cost of weird-stuff-happened-weridly-debug-time | 18:56 |
| mordred | which is useful for laptops | 18:56 |
| mordred | but pretty bad for servers | 18:56 |
| fungi | perhaps the openstackinfra/zuul module should not continuously deploy bootstrap from git? we must be doing that for some reason but i don't know what the reason is | 18:57 |
| *** edmondsw has quit IRC | 18:57 | |
| mordred | we also wanted to be able to specify a branch and have it keep it at tip of that branch | 18:57 |
| mordred | AND | 18:57 |
| *** edmondsw has joined #openstack-infra | 18:57 | |
| mordred | to not report change if it had just fetched but didn't actually update the working tree | 18:57 |
| nibalizer | The git module understands our needs and is git focused and simple | 18:57 |
| mordred | if upstream can do all of those things, I'm great with using it | 18:57 |
| fungi | nibalizer: yeah, back when we had more uses of hg and bzr than we do now | 18:58 |
| openstackgerrit | Merged openstack/requirements: Bump oslo.versionedobjects version to 0.9.0 https://review.openstack.org/216848 | 18:58 |
| *** mrunge has quit IRC | 18:58 | |
| *** amuller_afk is now known as amuller | 18:58 | |
| *** sthillma_ has joined #openstack-infra | 18:58 | |
| *** ociuhandu has quit IRC | 18:58 | |
| nibalizer | so if we were gonna do a big change i'd advocate for the git module, or maybe patching our vscrepo changes into the upstream vcsrepo | 18:58 |
| nibalizer | but it also seems to work as is so | 18:58 |
| dims_ | pcm_: since it got merged in global requirements, the proposal bot has already filed reviews for other projects that use netaddr (https://review.openstack.org/#/q/owner:%22OpenStack+Proposal+Bot%22+status:open,n,z) | 18:59 |
| *** sabeen1 has quit IRC | 18:59 | |
| glauco_ | ok, we'll check how is the support for upstream vcsrepo later on. thanks for sharing the concerns | 18:59 |
| *** FallenPegasus has joined #openstack-infra | 18:59 | |
| *** sthillma has quit IRC | 19:01 | |
| *** sthillma_ is now known as sthillma | 19:01 | |
| mordred | nibalizer: ++ to git - a lot of the problems we patched around had to do with the difficulty of finding the "right" semantics to express across the different VCS systems | 19:01 |
| *** spzala has joined #openstack-infra | 19:01 | |
| mordred | which, it turns out, is VERY hard | 19:01 |
| mordred | and I give props to upstream for getting it as close as they did | 19:02 |
| *** dtantsur is now known as dtantsur|afk | 19:03 | |
| *** sdake has quit IRC | 19:03 | |
| *** jsavak has quit IRC | 19:05 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Add acceptance tests for puppet-gerrit https://review.openstack.org/214725 | 19:06 |
| *** yfried has joined #openstack-infra | 19:06 | |
| *** FallenPegasus has quit IRC | 19:06 | |
| *** jsavak has joined #openstack-infra | 19:06 | |
| *** hashar has joined #openstack-infra | 19:07 | |
| *** Rockyg has joined #openstack-infra | 19:08 | |
| clarkb | yolanda: nibalizer ^ that change failed both of its rspec jobs | 19:09 |
| *** sergmelikyan has quit IRC | 19:09 | |
| clarkb | yolanda: nibalizer should we be merging tests that dont pass? | 19:09 |
| *** sdake has joined #openstack-infra | 19:10 | |
| *** sdake has quit IRC | 19:10 | |
| *** sdake has joined #openstack-infra | 19:10 | |
| *** ZZelle has joined #openstack-infra | 19:13 | |
| *** fitoduarte has joined #openstack-infra | 19:14 | |
| *** mjturek1 has joined #openstack-infra | 19:14 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: update signature for another expression https://review.openstack.org/218940 | 19:15 |
| *** FallenPegasus has joined #openstack-infra | 19:15 | |
| *** mjturek2 has quit IRC | 19:15 | |
| rm_work | Hey, possible to get this workflowed today? Running out of time for testing for Liberty. :( https://review.openstack.org/#/c/217458/ | 19:15 |
| crinkle | clarkb: probably not when the patch is for adding tests... | 19:17 |
| *** hdd has joined #openstack-infra | 19:18 | |
| openstackgerrit | John Trowbridge proposed openstack/diskimage-builder: [ironic-agent] Use svc-map for enabling agent https://review.openstack.org/218879 | 19:18 |
| *** FallenPegasus has quit IRC | 19:20 | |
| mordred | clarkb: heads up - glanceclient 1.0 breaks shade | 19:22 |
| clarkb | woo | 19:23 |
| mordred | clarkb: it broke jlk doing something else - I'm investgating - but for now, let's not upgrade glanceclient on nodepool :) | 19:23 |
| greghaynes | :( | 19:24 |
| clarkb | ok http://logs.openstack.org/22/218822/1/check/gate-ceilometer-dsvm-functional-mongodb/f3bc93b/console.html#_2015-08-31_12_54_42_818 and http://logs.openstack.org/86/218286/1/gate/gate-tempest-dsvm-neutron-full/f070942/console.html#_2015-08-31_12_46_38_128 are fun ones | 19:24 |
| *** baoli has quit IRC | 19:24 | |
| asselin_ | oh no... :( | 19:24 |
| clarkb | thats two differnet hosts in ord getting the same IP at the same time | 19:24 |
| clarkb | lifeless: ^ | 19:24 |
| clarkb | I suppose we can only file a bug against rax for that? | 19:24 |
| *** baoli has joined #openstack-infra | 19:24 | |
| greghaynes | wow | 19:25 |
| clarkb | and hope they can figure out what nova/neutron did | 19:25 |
| jesusaurus | impressive | 19:25 |
| *** mrmartin has joined #openstack-infra | 19:25 | |
| clarkb | such cloud many IPs wow | 19:25 |
| sdague | mordred: glanceclient 1.0 broken cinder unit tests, but only because they were implicitly using it's pyopenssl dependency | 19:25 |
| greghaynes | I wonder if cloud-utils does the dhcp-like thing where it tries to ping the IP before assigning it | 19:25 |
| *** zeih has joined #openstack-infra | 19:25 | |
| anteaya | rm_work: done, nice work | 19:25 |
| greghaynes | er, cloud-init | 19:25 |
| sdague | clarkb: it's neutron with a custom backend there | 19:25 |
| sdague | greghaynes: no, cloud-init doesn't care | 19:26 |
| greghaynes | nice | 19:26 |
| sdague | cloud-init does what you told it | 19:26 |
| clarkb | and in this case its their weird agent setting the IP | 19:26 |
| greghaynes | oh even better | 19:26 |
| mordred | sdague, clarkb: apparently there is now a problem with the "insecure" argument | 19:26 |
| clarkb | sdague: so not sure how we would fingerprint that other than look for same IP in use at the same time with some fuzzyness | 19:26 |
| clarkb | sdague: because we do safely reuse IPs over time | 19:26 |
| glauco_ | clarkb: gerrit-puppet-beaker-rspec-dsvm-trusty fails because beaker sees a pending test as a failure | 19:26 |
| Piet | We posted a draft version of the cloud personas to invision | 19:27 |
| Piet | Feel free to comment - https://invis.io/VM41Q203B | 19:27 |
| sdague | clarkb: so... does this mean that we've got 2 ssh client connections running 2 stack.sh at the same time? | 19:27 |
| clarkb | sdague: yes | 19:27 |
| rm_work | anteaya: thanks | 19:27 |
| clarkb | sdague: and as far as nodepool is concerned they are different hosts | 19:27 |
| sdague | so, could we stamp a file in d-g | 19:27 |
| sdague | when it starts | 19:27 |
| clarkb | we can probably defend against this ins nodepool by treating a dup IP in the db as an error | 19:27 |
| sdague | and look for it when another one starts | 19:27 |
| clarkb | I may work on that patch today | 19:27 |
| sdague | yeh, or that, in nodepool | 19:28 |
| sdague | anyway, get defensive about it | 19:28 |
| nibalizer | clarkb: argh dangit, I thought those were the upgrade jobs that never pass | 19:28 |
| *** FallenPegasus has joined #openstack-infra | 19:29 | |
| glauco_ | clarkb: gerrit-puppet-beaker-rspec-dsvm-centos7 fails to apply the existing puppet module. | 19:29 |
| clarkb | glauco_: regardless if your intent is to add tests they should make the job pass | 19:29 |
| clarkb | glauco_: otherwise its just noise | 19:29 |
| clarkb | glauco_: thats ok we expect them to pass on at least one of them right? | 19:29 |
| clarkb | (trusty) | 19:29 |
| nibalizer | glauco_: ya why isn't htat passing on trusty | 19:29 |
| nibalizer | Failure/Error: apply_manifest(default_puppet_module, catch_failures: true) it's not a pending that is failing | 19:30 |
| nibalizer | however there isn't a failure in the last 10 lines of output | 19:30 |
| *** zeih has quit IRC | 19:30 | |
| glauco_ | clarkb: sure! We could either approach it as: let's make this module idempotent or we could remove that test | 19:30 |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental job for Barbican to test updated gate https://review.openstack.org/217458 | 19:30 |
| glauco_ | nibalizer: do you mean on trusty or centos7? | 19:31 |
| glauco_ | on trusty the test that fails is the pending one | 19:31 |
| madhu_ak | can anyone workflow this patch? https://review.openstack.org/#/c/214767/ | 19:31 |
| glauco_ | on centos7 what fails is the 'apply' of the manifest | 19:31 |
| clarkb | rm_work: now that I am not debugging nodepool crazyness, what is the liberty concern? infra doesn't follow that schedule so we can get that experiemtanl test in any time | 19:32 |
| *** sthillma_ has joined #openstack-infra | 19:32 | |
| clarkb | rm_work: if however you are trying to test that particular thing before liberty cuts that different (but considering its experimental I am not sure if that is urgent) | 19:32 |
| sdague | clarkb: so before I decide I need to start drinking, because my well pump adventure is going to have a day 2 (or 3 based on counting), blocking random inbound dhcp requests to nodes.... | 19:32 |
| rm_work | clarkb: right but i need the experimental job so i can properly test a change in a project that DOES follow the freezes :P | 19:32 |
| rm_work | clarkb: but it's fine, anteaya +W'd it | 19:33 |
| *** mrmartin has quit IRC | 19:33 | |
| rm_work | clarkb: yeah, the latter | 19:33 |
| mtreinish | sdague: you can't drink and talk about blocking random inbound dhcp requests to nodes? They seem to go hand in hand :) | 19:33 |
| sdague | heh | 19:33 |
| glauco_ | clarkb: nibalizer: would you prefer removing that pending test? | 19:33 |
| clarkb | sdague: I realized that my octet idea won't work for the same reason its happening. l2 broadcast | 19:33 |
| *** FallenPegasus has quit IRC | 19:33 | |
| clarkb | sdague: so ebtables necessary or listening only on the private addr | 19:34 |
| clarkb | glauco_: I would prefer that we only merge test changes that pass some sort of testing | 19:34 |
| clarkb | glauco_: however its best to do that I defer to you and nibalizer and yolanda, I just don't want to see "add tests" and the tests fail | 19:34 |
| *** sthillma has quit IRC | 19:34 | |
| *** sthillma_ is now known as sthillma | 19:34 | |
| clarkb | sdague: or accept it as noise> would make it hard to debug real dhcp issues if/when they happen | 19:35 |
| sdague | clarkb: so, I'm wondering if - http://logs.openstack.org/74/218074/3/gate/gate-grenade-dsvm/9614128//logs/grenade.sh.txt.gz#_2015-08-31_19_31_56_328 is also double node use | 19:35 |
| clarkb | sdague: the other wierd dhcp case is multinode but it uses nova multihost so would not be affected | 19:35 |
| pcm_ | hi! does any know what the magical incantation is for python-neutronclient to specify a list as an attribute? | 19:35 |
| clarkb | sdague: oh maybe I can dig through logs for that one too | 19:35 |
| *** davideag_ has joined #openstack-infra | 19:35 | |
| sdague | because that log shows pbr being installed to 1.6.0 | 19:36 |
| *** davideagnello has quit IRC | 19:37 | |
| openstackgerrit | Assaf Muller proposed openstack-infra/elastic-recheck: Add query for Neutron + functional bug 1478847 https://review.openstack.org/218455 | 19:37 |
| openstack | bug 1478847 in neutron "test_udp_{input|output}.* fails with " AssertionError: RuntimeError not raised."" [High,In progress] https://launchpad.net/bugs/1478847 - Assigned to Jakub Libosvar (libosvar) | 19:37 |
| sdague | but pbr freeze shows it at 0.11.0 | 19:37 |
| clarkb | sdague: will be easier to check once logstash catches up tags:"console" AND message:"inet 10.0.3.189" is the query | 19:37 |
| *** salv-orlando has quit IRC | 19:37 | |
| clarkb | but let me dig through the logs and see if Ican find another ansible like lifeless did | 19:38 |
| *** ramashri has quit IRC | 19:40 | |
| clarkb | sdague: there are two worlddump files, can that happen during a single test run? | 19:41 |
| *** sharikapongubala has joined #openstack-infra | 19:41 | |
| *** jeadre has quit IRC | 19:41 | |
| anteaya | pcm_: oh that would be a neutron question I am thinking | 19:41 |
| *** jeadre has joined #openstack-infra | 19:42 | |
| *** FallenPegasus has joined #openstack-infra | 19:42 | |
| *** bknudson has joined #openstack-infra | 19:42 | |
| pcm_ | anteaya: yeah but really applies to any client. If I have python-fooclient bar-create name cidrs=? | 19:43 |
| sdague | clarkb: yes | 19:43 |
| *** hdd has quit IRC | 19:43 | |
| sdague | there can be up to 6 I think | 19:44 |
| sdague | I'm dumping at success points to try to build up data for the ping goes away issue that I don't understand | 19:44 |
| *** hdd has joined #openstack-infra | 19:44 | |
| pcm_ | anteaya: Any cli client that needs to provide an attibute that is a list (e.g. converted into JSON {'cidr': ['10.1.0.0/24', '20.1.0.0/24'] ...} | 19:44 |
| clarkb | sdague: gotcha | 19:45 |
| pcm_ | anteaya: I can't seem to figure out how to specify the command line representation for the list. | 19:45 |
| clarkb | sdague: so without more data on whta the other thing may be its hard to know if this was dup host again, but logstash should be able to tell uys when it catches up | 19:45 |
| sdague | it seemed better to sort an iptables issue by seeing the tables when it worked | 19:45 |
| sdague | and see if there was anything different when they did not | 19:45 |
| clarkb | hrm I should recheck the nodepool logs though because for the instance I did find nodepool wasn't really aware | 19:46 |
| *** andersonvom has joined #openstack-infra | 19:46 | |
| anteaya | pcm_: let's move to -dev this isn't really an -infra question | 19:46 |
| *** thorst1 has quit IRC | 19:47 | |
| *** andersonvom has quit IRC | 19:47 | |
| clarkb | the mystery gets more mysterious | 19:47 |
| clarkb | according to nodepool we booted and deleted this node yesterday | 19:47 |
| fungi | lifeless: http://curtis.lassam.net/comics/cube_drone/161.gif | 19:48 |
| fungi | (you've probably already seen it, but HA!) | 19:48 |
| clarkb | OH! | 19:48 |
| clarkb | I know what happened | 19:48 |
| clarkb | fungi: when you restarted jenkins01 the slave from the 30th was still in there | 19:48 |
| clarkb | fungi: then jenkins came back up and ssh'd to that IP address which was the new slave from today | 19:48 |
| *** FallenPegasus has quit IRC | 19:49 | |
| clarkb | fungi: it then ran a "second" job on the old host when in reality it was a second job on the new host with the shared IP | 19:49 |
| clarkb | thats magical | 19:49 |
| fungi | clarkb: wait, what?!? | 19:49 |
| clarkb | fungi: did you clera out the slave list on jenkins01 when you restarted it? | 19:49 |
| *** yfried has quit IRC | 19:49 | |
| fungi | maybe it happened in the split second before i stopped the service | 19:49 |
| fungi | but yes, i did | 19:49 |
| *** Sukhdev has joined #openstack-infra | 19:49 | |
| clarkb | oh wait no because the timestamps don't match up for the restart | 19:50 |
| *** salv-orlando has joined #openstack-infra | 19:50 | |
| fungi | i kept hammering ssh until i got in and immediately did a service stop jenkins, confirmed it wasn't running, then cleaned up the config.xml and started it again | 19:50 |
| clarkb | ya I think I am wrong because you restarted well after these jobs ran | 19:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Move glanceclient to new common interface https://review.openstack.org/219005 | 19:51 |
| mordred | SpamapS, Shrews: when you get sec ^^ | 19:51 |
| glauco_ | nibalizer: where are we defining beaker-rspec-* checks and gates? I can't find it anywhere other than project-config/zuul/layout | 19:51 |
| sdague | clarkb: so a ton of nodes started failing on the pbr contextual conflict issue | 19:51 |
| mordred | SpamapS, Shrews: that does not work yet - there are more interface changes | 19:51 |
| mordred | I just wanted to loop you in that glanceclient is different now | 19:52 |
| *** davideagnello has joined #openstack-infra | 19:52 | |
| glauco_ | nibalizer: I would like to see the command it is executing right now. Just noticed that puppet-gerrit will not going to work on centos because of the java_home settings | 19:52 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcInBrZ19yZXNvdXJjZXMuQ29udGV4dHVhbFZlcnNpb25Db25mbGljdDogKHBiciAwLjExLjBcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MTA1MDY4MzYxOH0= | 19:52 |
| *** DericHorn-HP has joined #openstack-infra | 19:52 | |
| mordred | fungi, jhesketh, pleia2, SergeyLukjanov, jeblair, clarkb: nobody upgrade glanceclient on nodepool. it'll break hard | 19:52 |
| clarkb | so from the logs what I can tell happened is yesterday we made 4636852, and deleted it that day. Then today we made 4656491, a jenkins master was able to ssh into 4656491 thinking it was 4636852, then we ran two jobs on the same host because two different slaves were both ssh'd into it | 19:53 |
| glauco_ | nibalizer: does deleting the nodeset 'nodepool-centos7.yml' from under nodesets folder makes it skip centos testing? | 19:53 |
| clarkb | glauco_: no | 19:53 |
| *** davideag_ has quit IRC | 19:53 | |
| clarkb | glauco_: it will run the job and fail, its fine if centos7 is known to fail for now | 19:53 |
| *** jeadre has quit IRC | 19:53 | |
| clarkb | glauco_: as long as those tests pass somewhere | 19:53 |
| *** jeadre has joined #openstack-infra | 19:54 | |
| clarkb | ah more info | 19:54 |
| clarkb | 4636852 didn't run a job | 19:54 |
| *** stevemar has joined #openstack-infra | 19:54 | |
| clarkb | so it was never offlined | 19:54 |
| clarkb | fungi: I have a hunch this is just jenkins derping, we try to delete the node and it doesn't actually delete | 19:54 |
| clarkb | fungi: then later when jenkins gets enough cpu cycles that aren't doing too many threads things it ssh's back into hosts | 19:55 |
| clarkb | fungi: and if they have never run a job before well its totally cool to run a job now | 19:55 |
| clarkb | so hopefully that plugin update fixes this :/ | 19:56 |
| clarkb | sdague: I would be surprised if the pbr thing is this | 19:56 |
| sdague | clarkb: ok, the pbr thing is spiking | 19:56 |
| sdague | and it's really confusing | 19:56 |
| sdague | because we install pbr 1.6.0 twice in most of those runs | 19:56 |
| sdague | and it says it's 0.11.0 at the end | 19:57 |
| clarkb | sdague: if a project has a direct 0.11.0 depends it may be sucking it down | 19:57 |
| *** bswartz has quit IRC | 19:57 | |
| sdague | and not ever tell us? | 19:57 |
| clarkb | sdague: if its via a lib dep install as a transitive thing it wouldn't surprise me | 19:58 |
| clarkb | pip logs is where I would look next | 19:58 |
| mordred | I was just about to suggest those | 19:58 |
| clarkb | we don't seem to grab them on grenade runs | 19:58 |
| sdague | it's not just grenade runs | 19:59 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcInBrZ19yZXNvdXJjZXMuQ29udGV4dHVhbFZlcnNpb25Db25mbGljdDogKHBiciAwLjExLjBcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MTA1MDY4MzYxOCwibW9kZSI6InNjb3JlIiwiYW5hbHl6ZV9maWVsZCI6ImJ1aWxkX25hbWUifQ== | 19:59 |
| openstackgerrit | Jim Rollenhagen proposed openstack/requirements: Add ironic-lib at initial release https://review.openstack.org/219011 | 19:59 |
| sdague | though those do get hit a bunch | 19:59 |
| jroll | dhellmann: ^ corresponding g-r if you're curious | 19:59 |
| *** stevemar has quit IRC | 19:59 | |
| annegentle | mordred: around? | 19:59 |
| *** teran has quit IRC | 20:00 | |
| clarkb | fungi: I am tempted to just go ahead and update the plugin across the board in the off chance it just fixes it and if it doesn't then we are no worse off (assuming its not riddled with even worse bugs too) | 20:00 |
| mordred | annegentle: yes - oh bother - is that today? | 20:01 |
| *** david-lyle has quit IRC | 20:01 | |
| * anteaya hands mordred a pot of honey | 20:01 | |
| mordred | annegentle: my mom just got in to town and I'm juggling her and my sanity | 20:01 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: WIP: Removes posix-ipc from global-requirements https://review.openstack.org/217824 | 20:01 |
| annegentle | mordred: yup! | 20:01 |
| mordred | annegentle: can we push to tomorrow?? (/me hides and apologizes profusely) | 20:02 |
| anteaya | does you mom weigh about the same as the other two oranges? | 20:02 |
| annegentle | mordred: oh sure | 20:02 |
| annegentle | mordred: no prob | 20:02 |
| mordred | annegentle: THANK YOU! | 20:03 |
| annegentle | mordred: 'course we won't conflict with TC meeting :) | 20:03 |
| *** jeadre has quit IRC | 20:04 | |
| glauco_ | nibalizer: clarkb the test seems to be failing due the apache race condition. this patch should fix the race https://review.openstack.org/#/c/216708/. could we have the workflow triggered for this one? | 20:04 |
| *** jeadre has joined #openstack-infra | 20:05 | |
| greghaynes | SpamapS: Did you have a patch or somesuch for your bifrost network splitting out? | 20:05 |
| greghaynes | SpamapS: My changes are all one offs on the bastion, about to work on formulating them into a patch | 20:05 |
| *** markvoelker has joined #openstack-infra | 20:06 | |
| *** david-lyle has joined #openstack-infra | 20:06 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: add signature for cinder pyopenssl issue https://review.openstack.org/218976 | 20:07 |
| *** glauco_ has quit IRC | 20:07 | |
| *** kgiusti has left #openstack-infra | 20:10 | |
| *** stevemar has joined #openstack-infra | 20:10 | |
| *** ddecapit has joined #openstack-infra | 20:11 | |
| *** FallenPegasus has joined #openstack-infra | 20:11 | |
| *** andersonvom has joined #openstack-infra | 20:11 | |
| *** thedodd has quit IRC | 20:11 | |
| *** andersonvom has quit IRC | 20:11 | |
| *** dims has joined #openstack-infra | 20:11 | |
| *** dims has quit IRC | 20:12 | |
| *** thedodd has joined #openstack-infra | 20:12 | |
| *** cburroughs has quit IRC | 20:13 | |
| *** ddecapit_ has joined #openstack-infra | 20:14 | |
| *** thorst1 has joined #openstack-infra | 20:14 | |
| *** puranamr has quit IRC | 20:14 | |
| *** jeadre has quit IRC | 20:14 | |
| *** dims_ has quit IRC | 20:14 | |
| *** puranamr has joined #openstack-infra | 20:15 | |
| *** dprince has quit IRC | 20:16 | |
| clarkb | madhu_ak: done | 20:16 |
| *** FallenPegasus has quit IRC | 20:16 | |
| *** mrmartin has joined #openstack-infra | 20:16 | |
| *** thedodd has quit IRC | 20:16 | |
| *** ramashri has joined #openstack-infra | 20:16 | |
| madhu_ak | clarkb: thanks | 20:16 |
| *** sergmelikyan has joined #openstack-infra | 20:17 | |
| *** ddecapit has quit IRC | 20:17 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Move glanceclient to new common interface https://review.openstack.org/219005 | 20:18 |
| mordred | dguerri`, SpamapS, Shrews, clarkb: ^^ that works for me now | 20:18 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: clarify duties in library release process https://review.openstack.org/219017 | 20:18 |
| *** tiswanso has quit IRC | 20:20 | |
| *** jeadre has joined #openstack-infra | 20:20 | |
| *** HeOS has joined #openstack-infra | 20:20 | |
| *** tiswanso has joined #openstack-infra | 20:20 | |
| *** mrmartin has quit IRC | 20:21 | |
| *** glauco has joined #openstack-infra | 20:21 | |
| fungi | mordred: thanks for the glanceclient heads up | 20:22 |
| fungi | mordred: should we stop puppet there/. | 20:22 |
| fungi | ? | 20:22 |
| clarkb | fungi: possibly, any nodepool merges will upgrade it I think | 20:23 |
| fungi | i'll do that as a prophylactic measure | 20:23 |
| fungi | clarkb: agree on upgrading the plugin everywhere. it doesn't seem to have broken jenkins04 yet, so presumably at least no worse than the current version we have on other masters | 20:24 |
| fungi | i can cycle through more of them gracefully over the course of my evening | 20:24 |
| *** austin81 has joined #openstack-infra | 20:24 | |
| clarkb | fungi: great let me know if you need help | 20:25 |
| openstackgerrit | Merged openstack-infra/project-config: Make lbaasv2 scenario API job non-voting https://review.openstack.org/214767 | 20:25 |
| *** FallenPegasus has joined #openstack-infra | 20:25 | |
| fungi | #status info puppet agent temporarily disabled on nodepool.openstack.org to avoid accidental upgrade to python-glanceclient 1.0.0 | 20:26 |
| openstackstatus | fungi: unknown command | 20:26 |
| clarkb | sdague: for the 1900ish grenade run I on;y see one job matching that IP and its in logstash now | 20:26 |
| lifeless | fungi: lol, had not | 20:26 |
| *** zeih has joined #openstack-infra | 20:26 | |
| mordred | fungi: yeah | 20:26 |
| lifeless | clarkb: syslog records ansible runs | 20:26 |
| mordred | fungi: also - that change (and the stack under it) we should land and release | 20:26 |
| lifeless | clarkb: it showed the wrong job name in the second ansible invocation | 20:26 |
| fungi | #status log puppet agent temporarily disabled on nodepool.openstack.org to avoid accidental upgrade to python-glanceclient 1.0.0 | 20:27 |
| openstackstatus | fungi: finished logging | 20:27 |
| lifeless | clarkb: oh ugh IP conflict? noice. | 20:27 |
| mordred | fungi: (although, actually, lemme rebase it so it's not held up on the neutorn change) | 20:27 |
| clarkb | lifeless: maybe? I think it may have actually been stale jenkins config then jenkins ssh'd to the same slave twice thinking it was two different machines | 20:27 |
| fungi | mordred: confirming, 219005? | 20:27 |
| mordred | yes | 20:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Move glanceclient to new common interface https://review.openstack.org/219005 | 20:28 |
| mordred | fungi: there you go - rebased | 20:28 |
| lifeless | clarkb: so we've had three jobs with the same looks-like-a-race symptom | 20:28 |
| lifeless | clarkb: I haven't dug into the other two to see if its the same | 20:28 |
| clarkb | lifeless: I think its related to an issue with jenkins that we have suffered through for a while | 20:29 |
| clarkb | we are testing a potential fix now | 20:29 |
| lifeless | clarkb: cool. thanks ! | 20:29 |
| glauco | Can we get this review https://review.openstack.org/#/c/216708/ workflowed? Right now the acceptance tests are failing due to this apache race condition. | 20:29 |
| *** FallenPegasus has quit IRC | 20:30 | |
| *** zeih has quit IRC | 20:31 | |
| clarkb | glauco: I think we are still discussing how we want to fix it on the mailing list | 20:31 |
| clarkb | and jeblair in particular is not a fan of that method | 20:31 |
| *** wenlock_ has joined #openstack-infra | 20:31 | |
| fungi | mordred: removing more lines than you add in shade means that the bug it represents is slightly fixed! | 20:31 |
| mordred | fungi: yes! | 20:32 |
| *** jsavak has quit IRC | 20:32 | |
| mordred | fungi: honestly, as much as it's a yuck for a second - it's GREAT that it's aligning | 20:32 |
| clarkb | mordred: and by aligning you mean using the sessions from keystone? | 20:32 |
| fungi | mordred: yep, this is a very good thing. shade shrinkage ftw | 20:33 |
| *** jsavak has joined #openstack-infra | 20:33 | |
| fungi | clarkb: yep | 20:33 |
| mordred | clarkb: yes. and common constructor args and patterns | 20:33 |
| *** matrohon has joined #openstack-infra | 20:33 | |
| fungi | mordred: you should have a chart of shade loc over time, with the loc axis labeled "how broken is openstack interoperability?" | 20:34 |
| mordred | fungi: hahaha | 20:34 |
| clarkb | mordred: why is service_name None in the test? | 20:34 |
| mordred | clarkb: becuase it defaults to none | 20:34 |
| mordred | clarkb: you only need to pass something if you override it | 20:34 |
| openstackgerrit | Augustina Ragwitz proposed openstack/diskimage-builder: Update dpkg elements README from free text to table formatting https://review.openstack.org/218965 | 20:35 |
| clarkb | ok if tests work then it should merge | 20:35 |
| *** wenlock_ has quit IRC | 20:35 | |
| fungi | i've got jenkins06 and 07 quiescing in preparation for ssh slaves plugin upgrade | 20:36 |
| *** achanda has quit IRC | 20:38 | |
| *** FallenPegasus has joined #openstack-infra | 20:39 | |
| *** amitgandhinz has joined #openstack-infra | 20:40 | |
| *** achanda has joined #openstack-infra | 20:41 | |
| *** achanda has quit IRC | 20:41 | |
| *** achanda has joined #openstack-infra | 20:41 | |
| *** kushal has quit IRC | 20:42 | |
| *** dprince has joined #openstack-infra | 20:42 | |
| *** jsavak has quit IRC | 20:43 | |
| openstackgerrit | Augustina Ragwitz proposed openstack/diskimage-builder: Update dpkg elements README from free text to table formatting https://review.openstack.org/218965 | 20:43 |
| *** gyee has joined #openstack-infra | 20:43 | |
| *** FallenPegasus has quit IRC | 20:43 | |
| *** alivigni has quit IRC | 20:44 | |
| *** puranamr has quit IRC | 20:46 | |
| *** puranamr has joined #openstack-infra | 20:46 | |
| clarkb | fungi: https://jenkins04.openstack.org/monitoring?part=threads still looks good fwiw | 20:48 |
| *** tiswanso has quit IRC | 20:49 | |
| *** tiswanso has joined #openstack-infra | 20:50 | |
| *** regXboi has joined #openstack-infra | 20:51 | |
| clarkb | sdague: http://logs.openstack.org/87/218987/1/check/gate-tempest-dsvm-large-ops/a32152f/logs/devstacklog.txt.gz#_2015-08-31_20_28_11_159 | 20:51 |
| *** baoli has quit IRC | 20:51 | |
| clarkb | so I think thats the issue, need keystonemiddleware to fix | 20:52 |
| clarkb | http://logs.openstack.org/87/218987/1/check/gate-tempest-dsvm-large-ops/a32152f/logs/devstacklog.txt.gz#_2015-08-31_20_28_22_222 | 20:52 |
| *** FallenPegasus has joined #openstack-infra | 20:52 | |
| clarkb | now to check on centos7 things | 20:53 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Add option to ignore ref-updated events emitted by branch deletions https://review.openstack.org/178833 | 20:53 |
| *** Swami has quit IRC | 20:53 | |
| *** ildikov has quit IRC | 20:54 | |
| *** xyang1 has quit IRC | 20:55 | |
| *** yamahata has quit IRC | 20:55 | |
| *** mwagner_lap has quit IRC | 20:56 | |
| *** teran has joined #openstack-infra | 20:56 | |
| *** markvoelker has quit IRC | 20:56 | |
| *** rfolco has quit IRC | 20:57 | |
| *** FallenPegasus has quit IRC | 20:57 | |
| *** yamahata has joined #openstack-infra | 20:57 | |
| *** dims has joined #openstack-infra | 20:57 | |
| *** ihrachys has joined #openstack-infra | 20:57 | |
| *** jpeeler has quit IRC | 21:00 | |
| *** ccarmack has left #openstack-infra | 21:00 | |
| *** xyang1 has joined #openstack-infra | 21:01 | |
| *** esker has quit IRC | 21:01 | |
| *** hichihara has joined #openstack-infra | 21:01 | |
| *** shashank_hegde has joined #openstack-infra | 21:01 | |
| greghaynes | SpamapS: https://review.openstack.org/#/c/219026/ is a straightforwardish way to I think get what we want | 21:02 |
| greghaynes | SpamapS: not as pretty as it could be, but its probably a good start | 21:02 |
| greghaynes | TheJulia: ^ | 21:02 |
| *** jpeeler has joined #openstack-infra | 21:03 | |
| clarkb | update on hpcloud centos7, there is no facter installed? | 21:03 |
| *** jpeeler has quit IRC | 21:03 | |
| *** jpeeler has joined #openstack-infra | 21:03 | |
| clarkb | hrm maybe on that one host, thats weird | 21:04 |
| *** glauco__ has joined #openstack-infra | 21:04 | |
| clarkb | definitely no facter on the two hpcloud-b1 images I checked | 21:04 |
| clarkb | b3 is fine though, so weird | 21:04 |
| *** annegentle has quit IRC | 21:04 | |
| *** dizquierdo has joined #openstack-infra | 21:05 | |
| fungi | different image ages there? | 21:05 |
| clarkb | fungi: ya looking at that next | 21:05 |
| *** rossella_s has quit IRC | 21:05 | |
| *** glauco has quit IRC | 21:06 | |
| *** FallenPegasus has joined #openstack-infra | 21:06 | |
| clarkb | two days newer in b1 | 21:06 |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Block sqlalchemy-migrate 0.9.8, cap at 0.10.0 https://review.openstack.org/219027 | 21:06 |
| mriedem | dhellmann: sdague: mtreinish: ^ | 21:06 |
| *** rossella_s has joined #openstack-infra | 21:06 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Fix race condition in httpd install and config https://review.openstack.org/216708 | 21:08 |
| *** stevemar has quit IRC | 21:08 | |
| ryanpetrello | clarkb: http://lists.openstack.org/pipermail/openstack-dev/2015-August/073341.html | 21:09 |
| ryanpetrello | I'm literally banging my head against this atm | 21:09 |
| ryanpetrello | (the pbr issue) | 21:09 |
| clarkb | pabelanger: firewalld is not installed on the centos7 image in hpcloud | 21:09 |
| ryanpetrello | have spent the last 30 minutes trying to figure out wtf I'm doing wrong :) | 21:10 |
| clarkb | ryanpetrello: keystonemiddleware needs a new release that isn't broken | 21:10 |
| clarkb | and or we need to exclude it from testing | 21:10 |
| *** regXboi has quit IRC | 21:10 | |
| ryanpetrello | yep, right now it's totally breaking kilo devstack for me | 21:10 |
| *** FallenPegasus has quit IRC | 21:10 | |
| *** doug-fish has quit IRC | 21:11 | |
| pabelanger | clarkb: Right, so it looks like we are trying to control a service we are not properly managing. | 21:12 |
| pabelanger | guess we need to add some logic for firewalld correctly | 21:13 |
| clarkb | pabelanger: well its not trivial to do it properly with puppet, we can install it everywhere which is silly since we want it off | 21:13 |
| pabelanger | ya | 21:13 |
| clarkb | but we can't stop it unless its installed | 21:13 |
| clarkb | and we can't uninstall it until after its stopped | 21:13 |
| clarkb | we may want to move the logic out of puppet and into run_puppet.sh or something | 21:13 |
| clarkb | or wirte a fact to check if it is there and do things based on that | 21:14 |
| pabelanger | Ya | 21:14 |
| *** julim has quit IRC | 21:14 | |
| pabelanger | I can try sometime locally tomorrow | 21:14 |
| clarkb | this is tricky, is there any way to tell puppet its ok if this resource fails? | 21:14 |
| clarkb | we might get around this with some exec hacks | 21:15 |
| *** rmoats has joined #openstack-infra | 21:15 | |
| pabelanger | Right, I was thinking of some exec hack | 21:15 |
| clarkb | exec systemctl stop firwalld onlyif ps -elf | grep -q firewalld | 21:15 |
| *** ddecapit_ has quit IRC | 21:15 | |
| clarkb | pabelanger: ^ something like that? then have a package ensure absent on firewalld require that exec? | 21:15 |
| *** puranamr has quit IRC | 21:16 | |
| clarkb | pabelanger: I'm guessing that firewalld became a default in a more recent point release of centos7 | 21:16 |
| clarkb | and that hpcloud still lags on the base image point release | 21:16 |
| *** thcipriani has left #openstack-infra | 21:16 | |
| *** puranamr has joined #openstack-infra | 21:16 | |
| clarkb | I am going to throw up that patch really quick and we can argue over how terrible it is :) | 21:16 |
| pabelanger | okay | 21:16 |
| pabelanger | WFM | 21:16 |
| *** Hal has quit IRC | 21:17 | |
| *** dprince has quit IRC | 21:18 | |
| *** tiswanso has quit IRC | 21:19 | |
| *** esker has joined #openstack-infra | 21:19 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/infra-manual: Add warnings about tagging a release https://review.openstack.org/219030 | 21:20 |
| *** FallenPegasus has joined #openstack-infra | 21:20 | |
| *** mayurig has joined #openstack-infra | 21:20 | |
| *** trown is now known as trown|outttypeww | 21:20 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-iptables: Really stop using firewalld https://review.openstack.org/219031 | 21:23 |
| clarkb | pabelanger: ^ I am gonna WIP that until I can test it | 21:23 |
| *** glauco has joined #openstack-infra | 21:23 | |
| *** esker has quit IRC | 21:24 | |
| *** tmcpeak has joined #openstack-infra | 21:24 | |
| *** esker has joined #openstack-infra | 21:24 | |
| *** FallenPegasus has quit IRC | 21:24 | |
| *** john-davidge has quit IRC | 21:25 | |
| *** burgerk has quit IRC | 21:26 | |
| tmcpeak | hey all - I'm working on some automation for Bandit. We currently have some automation which can produce a list of projects like ['anchor', 'barbican'] etc. What's the best way to get the corresponding repo for that project? | 21:27 |
| *** glauco__ has quit IRC | 21:27 | |
| tmcpeak | initial unsophisticated approach would be: try openstack/ and then try stackforge/, but that seems low tech | 21:27 |
| tmcpeak | was thinking you guys might have something better :) | 21:27 |
| *** dizquierdo has quit IRC | 21:27 | |
| *** ivar-lazzaro has quit IRC | 21:28 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for sqla-migrate 0.9.8 bug 1490740 https://review.openstack.org/219033 | 21:29 |
| *** Rockyg has quit IRC | 21:29 | |
| openstack | bug 1490740 in OpenStack-Gate "sqlalchemy-migrate 0.9.8 release breaks stable/kilo because of capped pbr" [Undecided,In progress] https://launchpad.net/bugs/1490740 - Assigned to Matt Riedemann (mriedem) | 21:29 |
| clarkb | tmcpeak: gerrit project list | grep ? | 21:29 |
| clarkb | tmcpeak: the suffix is unique | 21:29 |
| *** dkranz has quit IRC | 21:30 | |
| *** mayurig has quit IRC | 21:30 | |
| tmcpeak | clarkb: ahh ok cool | 21:30 |
| *** mayurig has joined #openstack-infra | 21:31 | |
| clarkb | tmcpeak: however, maybe if you explained the use case a bit more there is a better way | 21:31 |
| clarkb | do you need to retrieve the source to run bandit against? | 21:32 |
| clarkb | maybe you can just hardcode repo paths for each name in bandit? | 21:32 |
| clarkb | that seems like it has inverted the relationship between linter and project though | 21:33 |
| tmcpeak | clarkb: yeah so we currently have a tool which parses the Jenkins Jobs and Zuul Layout to determine which projects are using Bandit gates. What we'd like to do is build additional automation that pulls the project source for each project that does use Bandit, and runs that proejct's Bandit tox setup | 21:33 |
| *** austin81 has left #openstack-infra | 21:33 | |
| *** FallenPegasus has joined #openstack-infra | 21:33 | |
| tmcpeak | clarkb: we'd also like to avoid hardcoding anything. The idea is that the list is automatically generated. The reason we'd like to get it is so that we make sure Bandit changes don't break any projects which are currently using it in a gate | 21:33 |
| clarkb | tmcpeak: thats an odd expectation for a linter | 21:34 |
| fungi | i was hoping jenkins06 and 07 would quiesce before i ran out to dinner, but they still need some time. i'll continue with them when i get back | 21:34 |
| clarkb | tmcpeak: since in theory as you add new rulkes you will be breaking projects | 21:34 |
| * fungi will bbiaw | 21:34 | |
| clarkb | tmcpeak: then at the beginning of each cycle we fix per project and bump the bandit version | 21:34 |
| clarkb | fungi: np | 21:34 |
| tmcpeak | clarkb: yeah, but in this case when we "add new rules" we are technically exposing new security issues. Would be nice to at least provide some fore-warning | 21:34 |
| *** e0ne has quit IRC | 21:34 | |
| tmcpeak | clarkb: also our current setup is for projects to not pin to a specific version of Bandit | 21:35 |
| tmcpeak | we'd like to keep it that way :) | 21:35 |
| *** annegentle has joined #openstack-infra | 21:35 | |
| clarkb | ok, that is really different than the way I tried to have you all put it in place | 21:35 |
| *** ddieterly has quit IRC | 21:36 | |
| clarkb | I'm not sure how you can add more restrictive rules without pinning | 21:36 |
| clarkb | how do you handle something like nova failing a new rule you want to add? | 21:36 |
| tmcpeak | clarkb: we are doing it with profiles. Basically projects maintain their own "profille" which specifically lists the tests they use | 21:36 |
| *** ivar-lazzaro has joined #openstack-infra | 21:36 | |
| tmcpeak | clarkb: in that case we provide new tests, but it's up to the projects to decide when to add the rules to their profiles. We also provide an updated profile with "all tests" so projects can pick and choose | 21:37 |
| clarkb | I see so similar to using version pins but with more selectivity in functionality | 21:37 |
| *** annegent_ has joined #openstack-infra | 21:38 | |
| clarkb | in that case asking gerrit is going to be the simplest way to get the canonical string I think however we try to remove gerrit from the test pipeline to avoid ddosing it | 21:38 |
| *** FallenPegasus has quit IRC | 21:38 | |
| tmcpeak | clarkb: yeah it ends up being pretty similar | 21:38 |
| tmcpeak | clarkb: what do you think about this? https://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml | 21:39 |
| *** matrohon has quit IRC | 21:39 | |
| clarkb | tmcpeak: some projects in that file are still in stackforge | 21:39 |
| clarkb | as there is lag time to moving them in gerrit as it requires a gerrit downtime | 21:39 |
| tmcpeak | clarkb: I think that's ok, as long as the projects we need are in there | 21:40 |
| *** annegentle has quit IRC | 21:40 | |
| clarkb | tmcpeak: they would be but they won't tell you what the canonical path is | 21:40 |
| *** claudiub has quit IRC | 21:40 | |
| tmcpeak | can I just take what's listed in repo and append to the git URL? | 21:44 |
| tmcpeak | clarkb: ^ | 21:44 |
| *** jcoufal has quit IRC | 21:45 | |
| *** regXboi has joined #openstack-infra | 21:45 | |
| clarkb | tmcpeak: you can if you don't mind the race that exists between us updating gerrit and that repo being updated | 21:46 |
| clarkb | tmcpeak: my point is that it isn't perfect due to that administrative task in gerrit requiring downtime | 21:46 |
| *** yolanda has quit IRC | 21:47 | |
| tmcpeak | clarkb: ok cool, I think we can live with it | 21:47 |
| tmcpeak | I'll start with this approach and if we have headaches I'll rethink :) | 21:47 |
| *** FallenPegasus has joined #openstack-infra | 21:47 | |
| *** regXboi has quit IRC | 21:48 | |
| tmcpeak | clarkb: thanks for the advice | 21:48 |
| *** rlandy has quit IRC | 21:49 | |
| clarkb | tmcpeak: also note that there is a potential race between changes to bandit and the projects if you aren't cogating | 21:49 |
| *** Sukhdev has quit IRC | 21:49 | |
| clarkb | basically bandit could approve a change that doesn't break nova now, but at the same time nova has approved a change that would break | 21:49 |
| *** rmoats has quit IRC | 21:49 | |
| *** cbader has quit IRC | 21:49 | |
| clarkb | then depending on the order of merging you can end up in a state where bandit no longer works against nova | 21:50 |
| *** yolanda has joined #openstack-infra | 21:50 | |
| tmcpeak | clarkb: yeah, I think that could happen regardless. This automation is just to help out with what I'm mostly doing by hand to try to make sure this doesn't happen | 21:50 |
| *** rlandy has joined #openstack-infra | 21:50 | |
| clarkb | tmcpeak: it cannot happen if you cogate | 21:50 |
| clarkb | but there are drawbacks to that | 21:51 |
| tmcpeak | it sounds complicated... | 21:51 |
| clarkb | it just requires you to run the same test against nova too | 21:51 |
| clarkb | then zuul does what it does | 21:51 |
| *** Sukhdev has joined #openstack-infra | 21:51 | |
| *** FallenPegasus has quit IRC | 21:51 | |
| clarkb | the drawbacks there are bandit has to go into the giant gate queue with everything else | 21:52 |
| tmcpeak | then Bandit could literally block Nova updates? | 21:52 |
| clarkb | yes thats what zuul does | 21:52 |
| clarkb | in general we have gone with using releases with linters instead though | 21:52 |
| *** ivar-lazzaro has quit IRC | 21:52 | |
| clarkb | because hodling everyone to a changing set of rules is hard whereas updating every 6 months is consumable and doesn't have these issues | 21:53 |
| *** jtomasek has quit IRC | 21:54 | |
| tmcpeak | yeah, we've wanted to avoid that to this point, in part because we're still making overall Bandit enhancements (not new tests) that we'd like projects to consume faster | 21:54 |
| *** FallenPegasus has joined #openstack-infra | 21:54 | |
| tmcpeak | clarkb: we're doing our security midcycle this week though, I'll kick it around with some of the other folks | 21:55 |
| tmcpeak | maybe somebody wants to set up a cogate PoC :) | 21:55 |
| tmcpeak | sounds like something bknudson might be into | 21:56 |
| bknudson | just run bandit against keystone? that doesn't sound that bad. | 21:57 |
| bknudson | even running it against the other projects doesn't sound bad | 21:57 |
| clarkb | the big drawback i really for bandit itself where you have to entire the giant queue | 21:57 |
| tmcpeak | that is… also not great :) | 21:57 |
| clarkb | s/entire/enter/ | 21:58 |
| openstackgerrit | Adam Harwell proposed openstack-infra/project-config: Update Barbican Gate to use new Devstack config https://review.openstack.org/217330 | 21:58 |
| *** sabeen has joined #openstack-infra | 21:58 | |
| rm_work | So I have this updated gate job, and I have a change in the project repo that will make it actually work -- which one should i set as "depends-on" for which | 21:58 |
| clarkb | rm_work: can you use specific examples? | 21:59 |
| rm_work | err sorry, the review that just posted | 21:59 |
| *** FallenPegasus has quit IRC | 21:59 | |
| rm_work | https://review.openstack.org/217330 | 21:59 |
| *** wenlock has quit IRC | 21:59 | |
| rm_work | and https://review.openstack.org/#/c/167885/ is in the project repo | 21:59 |
| clarkb | rm_work: 167885 should depend on 217330 since there is no way to test them together until 217330 merges | 22:00 |
| openstackgerrit | Merged openstack-infra/shade: Move glanceclient to new common interface https://review.openstack.org/219005 | 22:00 |
| clarkb | mordred: ^ | 22:00 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for sqla-migrate 0.9.8 bug 1490740 https://review.openstack.org/219033 | 22:01 |
| openstack | bug 1490740 in OpenStack-Gate "sqlalchemy-migrate 0.9.8 release breaks stable/kilo because of capped pbr" [Undecided,In progress] https://launchpad.net/bugs/1490740 - Assigned to Matt Riedemann (mriedem) | 22:01 |
| rm_work | clarkb: they are being tested together via an experimental job already | 22:01 |
| rm_work | (the thing we were discussing briefly earlier) | 22:01 |
| clarkb | rm_work: sure, but there is no job that will take the contents of those two changes and run them together dynamically | 22:01 |
| rm_work | err | 22:01 |
| rm_work | not dynamically... but they ARE being run together | 22:01 |
| clarkb | so you should have the project depend on the code that it needs to run | 22:01 |
| rm_work | ok | 22:02 |
| clarkb | in this case 167885 requires 217330 to function properly | 22:02 |
| mriedem | fungi: we should talk in irc next time there is a gate explosion :) | 22:02 |
| *** sharikapongubala has quit IRC | 22:02 | |
| openstackgerrit | Adam Harwell proposed openstack-infra/project-config: Update Barbican Gate to use new Devstack config https://review.openstack.org/217330 | 22:03 |
| *** sharikapongubala has joined #openstack-infra | 22:03 | |
| *** tmcpeak has quit IRC | 22:04 | |
| rm_work | clarkb: so then with 167885 depending on 217330(gate) in the project we can +2/+W the change and it will wait patiently for whenever the project-config change is merged? | 22:04 |
| clarkb | rm_work: yup | 22:04 |
| rm_work | cool, thanks | 22:04 |
| *** sergmelikyan has quit IRC | 22:04 | |
| *** sergmelikyan has joined #openstack-infra | 22:04 | |
| *** thedodd has joined #openstack-infra | 22:05 | |
| *** thiagop has quit IRC | 22:06 | |
| *** sergmelikyan has quit IRC | 22:07 | |
| *** eharney has quit IRC | 22:07 | |
| clarkb | ok so nova clones do take about ~550mb on centos7 | 22:07 |
| *** sergmelikyan has joined #openstack-infra | 22:08 | |
| clarkb | whcih I believe is a slight improvement but not as major as I original thought | 22:08 |
| *** Adri2000 has quit IRC | 22:08 | |
| *** dimtruck is now known as zz_dimtruck | 22:11 | |
| *** markvoelker has joined #openstack-infra | 22:11 | |
| *** ivar-lazzaro has joined #openstack-infra | 22:12 | |
| *** sergmelikyan has quit IRC | 22:12 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:15 | |
| *** markvoelker has quit IRC | 22:16 | |
| *** amuller has quit IRC | 22:16 | |
| *** asettle has joined #openstack-infra | 22:18 | |
| *** sergmelikyan has joined #openstack-infra | 22:18 | |
| *** sergmelikyan has quit IRC | 22:19 | |
| *** ivar-lazzaro has quit IRC | 22:21 | |
| *** sergmelikyan has joined #openstack-infra | 22:21 | |
| *** Piet has quit IRC | 22:22 | |
| *** dguerri` is now known as dguerri | 22:27 | |
| *** zeih has joined #openstack-infra | 22:27 | |
| *** edmondsw has quit IRC | 22:28 | |
| *** tlian2 has joined #openstack-infra | 22:28 | |
| *** spzala has quit IRC | 22:29 | |
| *** tlian has quit IRC | 22:30 | |
| *** dguerri is now known as dguerri` | 22:31 | |
| *** amitgandhinz has quit IRC | 22:31 | |
| *** jaypipes has quit IRC | 22:32 | |
| *** annegent_ has quit IRC | 22:32 | |
| *** zeih has quit IRC | 22:32 | |
| *** annegentle has joined #openstack-infra | 22:32 | |
| *** pradk has quit IRC | 22:33 | |
| *** pcm_ has quit IRC | 22:33 | |
| *** ociuhandu has joined #openstack-infra | 22:33 | |
| *** thorst1 has quit IRC | 22:35 | |
| *** mjturek1 has quit IRC | 22:35 | |
| *** btully has quit IRC | 22:36 | |
| armax | clarkb: ping | 22:38 |
| *** hashar has quit IRC | 22:38 | |
| armax | regarding the keystonemiddleware pbr conundrum | 22:38 |
| armax | you aware of a fix? | 22:38 |
| clarkb | I am not | 22:39 |
| *** sridhar_ram has quit IRC | 22:39 | |
| armax | clarkb: yeah, I am looking but I cannot find anything so far | 22:39 |
| lifeless | armax: pbr conundrum ? | 22:41 |
| armax | lifeless: I was simply referring to this error ContextualVersionConflict: (pbr 0.11.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('pbr<2.0,>=1.3'), set(['sqlalchemy-migrate'])) | 22:41 |
| armax | caused by keystonemiddleware apparently | 22:42 |
| anteaya | I have not gone for a walk all day and will be going now | 22:42 |
| lifeless | armax: in devstack? | 22:42 |
| armax | lifeless: nothing specific to pbr per se | 22:42 |
| clarkb | lifeless on the older branches/grenade I think | 22:42 |
| clarkb | because no constraints | 22:43 |
| lifeless | so grenade would show that if the new side didn't use constraints | 22:43 |
| lifeless | but we made it use constraints | 22:43 |
| lifeless | armax: is this in a log, or a personal machine | 22:43 |
| lifeless | ? | 22:43 |
| armax | no in the gate | 22:43 |
| *** MarkAtwood has joined #openstack-infra | 22:43 | |
| *** resker has joined #openstack-infra | 22:43 | |
| clarkb | its the old sidr | 22:43 |
| lifeless | armax: what job ? Got a log for me ? | 22:43 |
| armax | lifeless: one example here | 22:43 |
| armax | http://logs.openstack.org/87/218987/1/check/gate-tempest-dsvm-large-ops/a32152f/logs/devstacklog.txt.gz#_2015-08-31_20_28_11_159 | 22:43 |
| armax | another one here: | 22:44 |
| armax | http://logs.openstack.org/72/217272/2/check/gate-grenade-dsvm-neutron/fc65317/logs/old/devstacklog.txt.gz#_2015-08-31_20_42_19_978 | 22:44 |
| mtreinish | lifeless: it's kilo | 22:44 |
| mtreinish | armax: it was caused by the sqlalchemy migrate release | 22:44 |
| lifeless | so kilo has <1.0 on pbr nearly universally still | 22:44 |
| mtreinish | armax: mriedem has a g-r cap going through the gate now | 22:44 |
| mtreinish | that'll fix it | 22:44 |
| lifeless | cool | 22:44 |
| lifeless | mtreinish: thanks | 22:44 |
| armax | mtreinish: that’s awesome | 22:44 |
| * lifeless stops poking | 22:44 | |
| armax | ditto? | 22:44 |
| armax | mtreinish: link? | 22:45 |
| mtreinish | armax: https://review.openstack.org/#/c/219027/ | 22:45 |
| mtreinish | armax: there is also a ML thread about it | 22:45 |
| openstackgerrit | Jerry Zhao proposed openstack-infra/project-config: Add networking-fortinet project to neutron https://review.openstack.org/219055 | 22:45 |
| armax | mtreinish: thanks, there’s been too much events going on the last few hours | 22:45 |
| armax | mtreinish: it’s been hard to catch up | 22:45 |
| mtreinish | http://lists.openstack.org/pipermail/openstack-dev/2015-August/073340.html | 22:45 |
| armax | mtreinish: thanks | 22:45 |
| clarkb | is that the same rhing? | 22:46 |
| clarkb | just transitive problem with middleare? | 22:46 |
| *** esker has quit IRC | 22:47 | |
| mriedem | clarkb: it's the same thing | 22:48 |
| *** resker has quit IRC | 22:48 | |
| *** esker has joined #openstack-infra | 22:48 | |
| *** Piet has joined #openstack-infra | 22:52 | |
| *** ihrachys has quit IRC | 22:53 | |
| *** hdd has quit IRC | 23:00 | |
| *** mwagner_lap has joined #openstack-infra | 23:01 | |
| *** gordc has quit IRC | 23:02 | |
| *** angela-s has joined #openstack-infra | 23:03 | |
| jhesketh | Morning | 23:03 |
| *** stevemar has joined #openstack-infra | 23:04 | |
| *** annegentle has quit IRC | 23:04 | |
| Nakato | lifeless: went back through constraints discussions and caught something I missed. I've updated the changes, what do you think? | 23:04 |
| Nakato | https://review.openstack.org/#/c/207262/ | 23:04 |
| Nakato | https://review.openstack.org/#/c/205931/ | 23:04 |
| *** ashleighfarnham has quit IRC | 23:04 | |
| *** wenlock1 has quit IRC | 23:07 | |
| *** tmcpeak has joined #openstack-infra | 23:09 | |
| *** ramashri has quit IRC | 23:11 | |
| *** thedodd has quit IRC | 23:12 | |
| kfox1111 | what distro's apps.openstack.org running on? | 23:13 |
| *** tmcpeak has left #openstack-infra | 23:13 | |
| lifeless | Nakato: whats the change ? | 23:17 |
| Nakato | lifeless: in tox.ini the default is to use upper-constraints.txt from git, meaning nothing special to run tox -epyNN-constraints, and thus could be easily added as a default run option. | 23:17 |
| *** angela-s has left #openstack-infra | 23:17 | |
| kfox1111 | is it ubuntu? | 23:17 |
| Nakato | lifeless: and exporting a new varible in run-tox.sh | 23:17 |
| *** ddieterly has joined #openstack-infra | 23:20 | |
| *** Zhongjun has joined #openstack-infra | 23:21 | |
| pvaneck | kfox1111: seems so https://github.com/openstack-infra/system-config/blob/master/manifests/site.pp#L989 | 23:21 |
| kfox1111 | ok. thanks. :) | 23:23 |
| *** glauco has quit IRC | 23:27 | |
| *** zeih has joined #openstack-infra | 23:28 | |
| lifeless | Nakato: I like. | 23:31 |
| lifeless | sdague: ^ should address most of your concerns I think | 23:31 |
| lifeless | sdague: in that its Just Works for devs | 23:31 |
| *** rlandy has quit IRC | 23:33 | |
| *** zeih has quit IRC | 23:33 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Add option to ignore ref-updated events emitted by branch deletions https://review.openstack.org/178833 | 23:34 |
| *** ZZelle has quit IRC | 23:35 | |
| Nakato | Thanks fungi | 23:35 |
| *** spzala has joined #openstack-infra | 23:36 | |
| *** stevemar has quit IRC | 23:38 | |
| *** ramashri has joined #openstack-infra | 23:40 | |
| gmann | sdake: that was regarding netaddr thing, now all set :) | 23:41 |
| sdake | not sure how i can help with that :) | 23:42 |
| sdake | maybey ou mean sdague? | 23:42 |
| gmann | sdake: ahh, i misread author on https://review.openstack.org/#/c/218720/ :) | 23:44 |
| *** sergmelikyan has quit IRC | 23:44 | |
| *** yushiro has joined #openstack-infra | 23:47 | |
| yushiro | good morning infra-team, | 23:47 |
| clarkb | good morning | 23:48 |
| yushiro | clarkb: morning :) | 23:48 |
| *** teran has quit IRC | 23:48 | |
| yushiro | Would you please review my patch about project-config https://review.openstack.org/#/c/218067/ ? | 23:49 |
| clarkb | armax: mestery dougwig ^ is that something you all want to ACK first? | 23:51 |
| * armax looks | 23:51 | |
| clarkb | not quite sure what the relationship between neutron and networking-fujitsu is but figure I can ask | 23:51 |
| *** Sukhdev has quit IRC | 23:51 | |
| yushiro | sorry. I'll ask them first. | 23:51 |
| yushiro | clarkb: Thank you. | 23:52 |
| armax | looking | 23:52 |
| *** btully has joined #openstack-infra | 23:52 | |
| *** ramashri has quit IRC | 23:53 | |
| *** sridhar_ram has joined #openstack-infra | 23:53 | |
| *** DericHorn-HP has quit IRC | 23:54 | |
| *** dingyichen has joined #openstack-infra | 23:55 | |
| *** spzala has quit IRC | 23:55 | |
| *** ramashri has joined #openstack-infra | 23:56 | |
| *** btully has quit IRC | 23:57 | |
| openstackgerrit | Kevin Fox proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory and gzip https://review.openstack.org/218898 | 23:57 |
| *** achanda_ has joined #openstack-infra | 23:58 | |
| *** mayurig has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
