| *** thorst__ has quit IRC | 00:00 | |
| *** thorst__ has joined #openstack-infra | 00:00 | |
| *** thors____ has quit IRC | 00:01 | |
| *** thorst___ has joined #openstack-infra | 00:01 | |
| JayF | etherpad.openstack.org seems to be unhappy right now | 00:01 |
|---|---|---|
| *** thor_____ has quit IRC | 00:02 | |
| clarkb | JayF: daily db backups should be happy in a minute or two | 00:02 |
| JayF | ah, midnight utc?\ | 00:02 |
| clarkb | yup | 00:02 |
| *** bpokorny_ has joined #openstack-infra | 00:02 | |
| JayF | yep | 00:02 |
| fungi | gerrit gets similarly slow at the same time (and for the same reasons) | 00:02 |
| nibalizer | its a nice whistle at the end of the day for west coasters | 00:03 |
| *** thors____ has joined #openstack-infra | 00:03 | |
| fungi | shift change! | 00:03 |
| *** thorst_ has quit IRC | 00:03 | |
| * fungi swaps his left shift key for the right one | 00:03 | |
| fungi | much better | 00:04 |
| *** thorst_ has joined #openstack-infra | 00:04 | |
| *** thor_____ has joined #openstack-infra | 00:05 | |
| *** thorst__ has quit IRC | 00:05 | |
| *** bpokorny has quit IRC | 00:05 | |
| *** thorst___ has quit IRC | 00:05 | |
| *** thorst__ has joined #openstack-infra | 00:06 | |
| *** thorst___ has joined #openstack-infra | 00:07 | |
| *** bpokorny_ has quit IRC | 00:07 | |
| *** cody-somerville has quit IRC | 00:07 | |
| *** tho______ has joined #openstack-infra | 00:08 | |
| *** thors____ has quit IRC | 00:08 | |
| *** IlyaG has quit IRC | 00:08 | |
| *** thorst_ has quit IRC | 00:08 | |
| *** thorst_ has joined #openstack-infra | 00:08 | |
| *** thors____ has joined #openstack-infra | 00:09 | |
| *** thor_____ has quit IRC | 00:09 | |
| *** thorst__ has quit IRC | 00:10 | |
| *** jgriffith_ has joined #openstack-infra | 00:11 | |
| *** thorst___ has quit IRC | 00:11 | |
| *** signed8bit is now known as signed8bit_Zzz | 00:11 | |
| *** tho______ has quit IRC | 00:12 | |
| *** thorst__ has joined #openstack-infra | 00:12 | |
| *** thorst_ has quit IRC | 00:13 | |
| *** thorst_ has joined #openstack-infra | 00:13 | |
| *** thors____ has quit IRC | 00:14 | |
| *** ddieterly has joined #openstack-infra | 00:14 | |
| *** dimtruck is now known as zz_dimtruck | 00:14 | |
| *** thorst___ has joined #openstack-infra | 00:14 | |
| *** thors____ has joined #openstack-infra | 00:15 | |
| *** thor_____ has joined #openstack-infra | 00:16 | |
| *** thorst__ has quit IRC | 00:17 | |
| *** thorst__ has joined #openstack-infra | 00:17 | |
| *** thorst_ has quit IRC | 00:18 | |
| *** _ari_ has joined #openstack-infra | 00:18 | |
| *** thorst_ has joined #openstack-infra | 00:18 | |
| *** _ari_ has quit IRC | 00:18 | |
| *** thorst___ has quit IRC | 00:18 | |
| *** jgriffith has quit IRC | 00:19 | |
| *** thors____ has quit IRC | 00:19 | |
| *** thorst___ has joined #openstack-infra | 00:19 | |
| *** thors____ has joined #openstack-infra | 00:20 | |
| *** thor_____ has quit IRC | 00:20 | |
| *** thor_____ has joined #openstack-infra | 00:21 | |
| *** tho______ has joined #openstack-infra | 00:22 | |
| *** jsavak has quit IRC | 00:22 | |
| *** thorst__ has quit IRC | 00:22 | |
| *** thorst_ has quit IRC | 00:23 | |
| *** thorst_ has joined #openstack-infra | 00:23 | |
| *** thorst___ has quit IRC | 00:23 | |
| *** thors____ has quit IRC | 00:24 | |
| *** thorst__ has joined #openstack-infra | 00:24 | |
| *** thorst___ has joined #openstack-infra | 00:25 | |
| *** thor_____ has quit IRC | 00:25 | |
| *** tho______ has quit IRC | 00:26 | |
| *** thors____ has joined #openstack-infra | 00:26 | |
| *** kmartin has quit IRC | 00:26 | |
| *** thor_____ has joined #openstack-infra | 00:27 | |
| *** thorst_ has quit IRC | 00:27 | |
| *** thorst_ has joined #openstack-infra | 00:28 | |
| *** thorst__ has quit IRC | 00:29 | |
| *** thorst__ has joined #openstack-infra | 00:29 | |
| *** thorst___ has quit IRC | 00:30 | |
| *** thorst___ has joined #openstack-infra | 00:30 | |
| *** thors____ has quit IRC | 00:31 | |
| *** thors____ has joined #openstack-infra | 00:31 | |
| *** thor_____ has quit IRC | 00:31 | |
| *** thorst_ has quit IRC | 00:32 | |
| *** thorst_ has joined #openstack-infra | 00:32 | |
| *** duobei has joined #openstack-infra | 00:33 | |
| *** thor_____ has joined #openstack-infra | 00:33 | |
| *** tho______ has joined #openstack-infra | 00:34 | |
| *** thorst__ has quit IRC | 00:34 | |
| *** thorst___ has quit IRC | 00:35 | |
| *** thorst__ has joined #openstack-infra | 00:35 | |
| *** zz_dimtruck is now known as dimtruck | 00:35 | |
| *** thors____ has quit IRC | 00:36 | |
| *** hichihara has joined #openstack-infra | 00:36 | |
| *** gyee has quit IRC | 00:36 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix OpenSUSE support https://review.openstack.org/312325 | 00:36 |
| *** thorst___ has joined #openstack-infra | 00:37 | |
| *** thorst_ has quit IRC | 00:37 | |
| *** thor_____ has quit IRC | 00:37 | |
| *** thorst_ has joined #openstack-infra | 00:38 | |
| *** tho______ has quit IRC | 00:38 | |
| *** thors____ has joined #openstack-infra | 00:39 | |
| *** thorst__ has quit IRC | 00:39 | |
| *** thorst__ has joined #openstack-infra | 00:40 | |
| *** amitgandhinz has joined #openstack-infra | 00:40 | |
| *** thor_____ has joined #openstack-infra | 00:41 | |
| *** thorst___ has quit IRC | 00:41 | |
| *** mtanino has quit IRC | 00:41 | |
| *** thorst___ has joined #openstack-infra | 00:42 | |
| *** thorst_ has quit IRC | 00:42 | |
| *** thorst_ has joined #openstack-infra | 00:43 | |
| *** thors____ has quit IRC | 00:43 | |
| *** thors____ has joined #openstack-infra | 00:44 | |
| *** thorst__ has quit IRC | 00:44 | |
| *** amitgandhinz has quit IRC | 00:44 | |
| *** thorst__ has joined #openstack-infra | 00:45 | |
| *** thor_____ has quit IRC | 00:45 | |
| *** baoli has joined #openstack-infra | 00:45 | |
| *** lifeless has quit IRC | 00:46 | |
| *** thorst___ has quit IRC | 00:46 | |
| *** lifeless has joined #openstack-infra | 00:47 | |
| *** thorst___ has joined #openstack-infra | 00:47 | |
| *** derekh has joined #openstack-infra | 00:47 | |
| *** thorst_ has quit IRC | 00:47 | |
| *** thorst_ has joined #openstack-infra | 00:48 | |
| *** thors____ has quit IRC | 00:48 | |
| derekh | fungi: the tripleo cloud isn't coming back tonight, our jump host is refusing to get past the POST process and without it we got no access, we need to wait until tomorrow to find others with appropriate access to figure it out | 00:49 |
| *** thors____ has joined #openstack-infra | 00:49 | |
| *** thorst__ has quit IRC | 00:49 | |
| derekh | fungi: I guess you might be gone, will send a mail to the list | 00:49 |
| *** thorst__ has joined #openstack-infra | 00:50 | |
| *** thor_____ has joined #openstack-infra | 00:51 | |
| *** thorst___ has quit IRC | 00:52 | |
| fungi | derekh: yeah, i'm essentially gone, but thanks for the heads up! | 00:52 |
| derekh | fungi: no prob | 00:52 |
| *** thorst___ has joined #openstack-infra | 00:52 | |
| *** dimtruck is now known as zz_dimtruck | 00:52 | |
| *** zz_dimtruck is now known as dimtruck | 00:52 | |
| *** thorst_ has quit IRC | 00:53 | |
| *** gildub has quit IRC | 00:53 | |
| *** thors____ has quit IRC | 00:53 | |
| *** gildub has joined #openstack-infra | 00:53 | |
| *** rcernin has quit IRC | 00:54 | |
| *** thorst_ has joined #openstack-infra | 00:54 | |
| *** thorst__ has quit IRC | 00:55 | |
| *** thorst__ has joined #openstack-infra | 00:55 | |
| *** thor_____ has quit IRC | 00:56 | |
| *** fitoduarte has quit IRC | 00:56 | |
| *** thors____ has joined #openstack-infra | 00:56 | |
| *** thorst___ has quit IRC | 00:56 | |
| *** gildub_ has joined #openstack-infra | 00:57 | |
| *** thorst___ has joined #openstack-infra | 00:57 | |
| *** thor_____ has joined #openstack-infra | 00:58 | |
| *** thorst_ has quit IRC | 00:58 | |
| *** thorst_ has joined #openstack-infra | 00:59 | |
| *** gildub has quit IRC | 00:59 | |
| *** thorst__ has quit IRC | 01:00 | |
| *** mixos has joined #openstack-infra | 01:00 | |
| *** zhurong has joined #openstack-infra | 01:00 | |
| *** thorst__ has joined #openstack-infra | 01:01 | |
| *** thors____ has quit IRC | 01:01 | |
| *** thorst___ has quit IRC | 01:01 | |
| *** thorst___ has joined #openstack-infra | 01:01 | |
| pabelanger | okay, I think I deleted all the USED nodes we leaked. | 01:02 |
| *** dimtruck is now known as zz_dimtruck | 01:02 | |
| pabelanger | gate looks much better now | 01:02 |
| *** Qiming_ has joined #openstack-infra | 01:02 | |
| *** MarkAtwood has quit IRC | 01:02 | |
| *** thor_____ has quit IRC | 01:02 | |
| *** thors____ has joined #openstack-infra | 01:03 | |
| *** thorst_ has quit IRC | 01:03 | |
| *** thorst_ has joined #openstack-infra | 01:04 | |
| *** markvoelker_ has joined #openstack-infra | 01:05 | |
| *** thorst__ has quit IRC | 01:05 | |
| *** thor_____ has joined #openstack-infra | 01:05 | |
| ianw | jeblair / others : i have the first part of the bandersnatch index change merged into trunk with https://github.com/ianw/bandersnatch/tree/hash-index | 01:05 |
| ianw | i'll do the second part, then figure out getting it back into mercurial and proposing as a pull request | 01:06 |
| *** thorst___ has quit IRC | 01:06 | |
| *** thorst__ has joined #openstack-infra | 01:06 | |
| *** tlian has quit IRC | 01:06 | |
| *** thorst___ has joined #openstack-infra | 01:07 | |
| *** thors____ has quit IRC | 01:08 | |
| *** signed8bit_Zzz is now known as signed8bit | 01:08 | |
| *** thorst_ has quit IRC | 01:09 | |
| *** thors____ has joined #openstack-infra | 01:09 | |
| *** derekh has quit IRC | 01:09 | |
| *** thorst_ has joined #openstack-infra | 01:10 | |
| *** thor_____ has quit IRC | 01:10 | |
| *** thorst__ has quit IRC | 01:10 | |
| *** thorst__ has joined #openstack-infra | 01:10 | |
| *** thorst___ has quit IRC | 01:11 | |
| *** tphummel has quit IRC | 01:11 | |
| *** thorst___ has joined #openstack-infra | 01:11 | |
| *** thor_____ has joined #openstack-infra | 01:12 | |
| *** tho______ has joined #openstack-infra | 01:13 | |
| *** thors____ has quit IRC | 01:13 | |
| *** thors____ has joined #openstack-infra | 01:14 | |
| *** thorst_ has quit IRC | 01:14 | |
| *** mriedem has quit IRC | 01:15 | |
| *** thorst__ has quit IRC | 01:15 | |
| *** thorst_ has joined #openstack-infra | 01:15 | |
| *** thorst___ has quit IRC | 01:16 | |
| openstackgerrit | Merged openstack-infra/release-tools: add extra metadata to the tag messages https://review.openstack.org/314141 | 01:16 |
| *** thorst__ has joined #openstack-infra | 01:16 | |
| *** thorst__ has quit IRC | 01:16 | |
| *** thorst__ has joined #openstack-infra | 01:16 | |
| *** thor_____ has quit IRC | 01:17 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/314878 | 01:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Make sure worker arguments are integers https://review.openstack.org/315304 | 01:18 |
| *** tho______ has quit IRC | 01:18 | |
| pabelanger | clarkb: fungi: Well need to land ^. nodepool-builder was confusing the args as strings | 01:18 |
| *** thorst__ has quit IRC | 01:19 | |
| *** thors____ has quit IRC | 01:19 | |
| *** thorst_ has quit IRC | 01:20 | |
| *** pvaneck has quit IRC | 01:20 | |
| *** smarcet has joined #openstack-infra | 01:27 | |
| *** hongbin has joined #openstack-infra | 01:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add function testing for nodepoold workers https://review.openstack.org/315307 | 01:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add unit testing for nodepoold workers https://review.openstack.org/315307 | 01:35 |
| *** smarcet has quit IRC | 01:36 | |
| *** mixos has quit IRC | 01:36 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/irc-meetings: Change time for FWaaS meeting https://review.openstack.org/315308 | 01:37 |
| *** vhosakot has joined #openstack-infra | 01:39 | |
| *** amitgandhinz has joined #openstack-infra | 01:40 | |
| *** Goneri has quit IRC | 01:41 | |
| *** zhurong has quit IRC | 01:42 | |
| *** zz_dimtruck is now known as dimtruck | 01:42 | |
| *** zhurong has joined #openstack-infra | 01:43 | |
| *** amitgandhinz has quit IRC | 01:45 | |
| *** amotoki has joined #openstack-infra | 01:46 | |
| *** gdubreui has joined #openstack-infra | 01:47 | |
| *** amitgandhinz has joined #openstack-infra | 01:48 | |
| *** tqtran has quit IRC | 01:48 | |
| *** gildub_ has quit IRC | 01:49 | |
| *** salv-orlando has joined #openstack-infra | 01:51 | |
| *** markvoelker_ has quit IRC | 01:51 | |
| *** yuanying has quit IRC | 01:51 | |
| *** yuanying has joined #openstack-infra | 01:52 | |
| *** yuanying has quit IRC | 01:52 | |
| *** yuanying has joined #openstack-infra | 01:53 | |
| *** Goneri has joined #openstack-infra | 01:54 | |
| *** salv-orlando has quit IRC | 01:56 | |
| *** markvoelker has joined #openstack-infra | 01:59 | |
| *** markvoelker_ has joined #openstack-infra | 01:59 | |
| *** kengo_sakai has joined #openstack-infra | 02:02 | |
| *** markvoelker has quit IRC | 02:03 | |
| *** zhurong has quit IRC | 02:04 | |
| *** zhurong has joined #openstack-infra | 02:05 | |
| openstackgerrit | Merged openstack-infra/nodepool: Cap paramiko<2.0.0 https://review.openstack.org/315291 | 02:08 |
| *** zqfan has joined #openstack-infra | 02:14 | |
| *** ccamacho has quit IRC | 02:14 | |
| *** ccamacho has joined #openstack-infra | 02:15 | |
| *** smarcet has joined #openstack-infra | 02:16 | |
| *** apoorvad has quit IRC | 02:19 | |
| *** Goneri has quit IRC | 02:20 | |
| *** tonytan4ever has joined #openstack-infra | 02:21 | |
| *** rfolco has quit IRC | 02:22 | |
| *** ddieterly has quit IRC | 02:23 | |
| *** Daisy has joined #openstack-infra | 02:29 | |
| hongbin | yolanda: clarkb AJaeger : anyone there? | 02:30 |
| *** markvoelker_ has quit IRC | 02:31 | |
| *** vhosakot has quit IRC | 02:34 | |
| *** Swami_ has joined #openstack-infra | 02:34 | |
| *** nadya has joined #openstack-infra | 02:34 | |
| *** Daisy has quit IRC | 02:37 | |
| *** binbincong has quit IRC | 02:37 | |
| *** Swami has quit IRC | 02:38 | |
| *** daemontool has quit IRC | 02:38 | |
| *** markvoelker_ has joined #openstack-infra | 02:38 | |
| *** gdubreui has quit IRC | 02:39 | |
| *** gildub has joined #openstack-infra | 02:39 | |
| *** nadya has quit IRC | 02:39 | |
| *** signed8bit has quit IRC | 02:40 | |
| hongbin | yolanda: clarkb AJaeger Maybe I leave the message. I need your help to add me to the gerrit groups: higgins-core, higgins-release | 02:46 |
| ianw | i'm sure mercurial is great, but I really only have room for one dvcs in my life | 02:47 |
| ianw | https://bitbucket.org/pypa/bandersnatch/pull-requests/21/add-option-to-dir-hash-index-files <- i think this in bandersnatch and then we can go back to using upstream | 02:48 |
| ianw | jeblair / jhesketh : ^ | 02:48 |
| *** nelsnelson has quit IRC | 02:50 | |
| *** mixos has joined #openstack-infra | 02:50 | |
| *** baoli has quit IRC | 02:51 | |
| *** binbincong has joined #openstack-infra | 02:52 | |
| openstackgerrit | Merged openstack/requirements: Add smaug projects to projects.txt https://review.openstack.org/301543 | 02:52 |
| openstackgerrit | Merged openstack/requirements: Update ovs to 2.5.0. https://review.openstack.org/291769 | 02:52 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/313721 | 02:55 |
| openstackgerrit | lvdongbing proposed openstack-infra/project-config: New project: python-bileanclient https://review.openstack.org/315326 | 02:55 |
| *** rfolco has joined #openstack-infra | 02:59 | |
| *** yuanying has quit IRC | 03:00 | |
| *** Sukhdev has quit IRC | 03:02 | |
| *** amitgandhinz has quit IRC | 03:06 | |
| *** tonytan4ever has quit IRC | 03:08 | |
| *** markvoelker_ has quit IRC | 03:13 | |
| *** hongbin has quit IRC | 03:13 | |
| openstackgerrit | Merged openstack/requirements: Block dnspython 1.13.0 https://review.openstack.org/315003 | 03:20 |
| *** yuanying has joined #openstack-infra | 03:24 | |
| *** tlian has joined #openstack-infra | 03:25 | |
| *** tlian has quit IRC | 03:25 | |
| *** dimtruck is now known as zz_dimtruck | 03:29 | |
| lifeless | mordred: whats this 12-factor thing? did it come up again? | 03:29 |
| *** links has joined #openstack-infra | 03:34 | |
| *** jpr has quit IRC | 03:37 | |
| *** gildub has quit IRC | 03:39 | |
| *** julim has joined #openstack-infra | 03:40 | |
| *** timl0101 has quit IRC | 03:41 | |
| *** yuanying has quit IRC | 03:42 | |
| *** yuanying has joined #openstack-infra | 03:44 | |
| *** sree has joined #openstack-infra | 03:49 | |
| *** anteaya has quit IRC | 03:51 | |
| *** rfolco has quit IRC | 03:52 | |
| *** yuanying has quit IRC | 04:01 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Updating Install Guide meeting details https://review.openstack.org/314831 | 04:03 |
| *** yuanying has joined #openstack-infra | 04:07 | |
| *** julim has quit IRC | 04:07 | |
| *** binbincong has quit IRC | 04:07 | |
| *** gildub has joined #openstack-infra | 04:11 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Propose changing Zaqar meeting time https://review.openstack.org/314848 | 04:12 |
| *** Liuqing has joined #openstack-infra | 04:21 | |
| *** binbincong has joined #openstack-infra | 04:23 | |
| *** pcaruana has joined #openstack-infra | 04:25 | |
| *** yuanying has quit IRC | 04:27 | |
| *** yushiro has joined #openstack-infra | 04:28 | |
| *** harlowja_at_home has joined #openstack-infra | 04:28 | |
| *** pcaruana has quit IRC | 04:32 | |
| *** yuanying has joined #openstack-infra | 04:34 | |
| *** nadya has joined #openstack-infra | 04:35 | |
| *** yuanying has quit IRC | 04:36 | |
| *** yuanying has joined #openstack-infra | 04:37 | |
| *** salv-orlando has joined #openstack-infra | 04:39 | |
| *** yuanying has quit IRC | 04:39 | |
| *** psachin has joined #openstack-infra | 04:39 | |
| yushiro | Hi, good evening. | 04:40 |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job https://review.openstack.org/314795 | 04:40 |
| *** nadya has quit IRC | 04:40 | |
| *** yuanying has joined #openstack-infra | 04:41 | |
| yushiro | I'm testing some patches in our vendor repository. Then, 'requirements' | 04:41 |
| yushiro | test has failed with error message. | 04:41 |
| yushiro | The Error message: AttributeError: 'Requirement' object has no attribute 'extras' | 04:41 |
| *** yuanying has quit IRC | 04:42 | |
| yushiro | Would you please tell me how to fix it? | 04:42 |
| *** yuanying has joined #openstack-infra | 04:43 | |
| yushiro | URL: http://logs.openstack.org/43/308143/2/check/gate-networking-fujitsu-requirements/ea229d9/console.html | 04:44 |
| *** salv-orlando has quit IRC | 04:46 | |
| *** yuanying has quit IRC | 04:48 | |
| *** yuanying has joined #openstack-infra | 04:51 | |
| *** yuanying has quit IRC | 04:53 | |
| *** cloudtrainme has quit IRC | 04:54 | |
| *** yuanying has joined #openstack-infra | 04:54 | |
| *** Sukhdev has joined #openstack-infra | 04:56 | |
| *** sdake has quit IRC | 04:57 | |
| *** salv-orlando has joined #openstack-infra | 04:58 | |
| *** pilgrimstack has joined #openstack-infra | 04:59 | |
| *** pilgrimstack1 has quit IRC | 04:59 | |
| *** yuanying has quit IRC | 05:00 | |
| ianw | yushiro: ahh, that's on stable/liberty branch ... | 05:00 |
| yushiro | ianw, Yes. Only stable/liberty branch occurs this error. | 05:01 |
| ianw | yushiro: i'm thinking reno should probably be in global-requirements to match liberty branch | 05:03 |
| ianw | but there might be a bug, and bknudson was the last person to touch that line :) | 05:04 |
| ianw | https://review.openstack.org/#/c/278588/ | 05:04 |
| yushiro | ianw, oh... I see it's probably a bug. | 05:06 |
| *** amitgandhinz has joined #openstack-infra | 05:07 | |
| ianw | yushiro: yes, "extras" not defined in liberty branch of openstack-requirements | 05:07 |
| *** amitgandhinz has quit IRC | 05:12 | |
| *** armax has quit IRC | 05:13 | |
| yushiro | ianw, I see. Hmm.. I'm not sure 'extras' attribute will be supported for stable/liberty branch. However, currently, it hasn't been supported yet. | 05:13 |
| yushiro | ianw, Therefore, I think this commit should be reverted. | 05:14 |
| ianw | yushiro: https://review.openstack.org/315348 <- i think this backport will work ... | 05:15 |
| yushiro | ianw, ah, OK. Then, I hope this patch will be merged ASAP :) | 05:18 |
| *** yuanying has joined #openstack-infra | 05:18 | |
| ianw | yushiro: added a comment on there, probably best to watch that review. i'm sure if it's wrong, someone will do the right thing quickly :) | 05:18 |
| *** woodster_ has quit IRC | 05:18 | |
| ianw | as i always say, fastest way to get something fixed is to propose the wrong way to do it :) | 05:18 |
| yushiro | ianw, Thanks for your kindness and I understand your advice. I'll put "+1" if jenkins check passed :) | 05:21 |
| *** javeriak has joined #openstack-infra | 05:22 | |
| *** yuanying has quit IRC | 05:23 | |
| *** yuanying has joined #openstack-infra | 05:25 | |
| *** javeriak_ has joined #openstack-infra | 05:26 | |
| *** Alekhya has joined #openstack-infra | 05:26 | |
| *** javeriak has quit IRC | 05:27 | |
| Alekhya | failed to create instance in jenkins slave node and getting error in debug.log as ERROR nodepool.NodeLauncher: OpenStackCloudException launching node | 05:28 |
| *** kushal has quit IRC | 05:31 | |
| *** yuanying has quit IRC | 05:31 | |
| *** harlowja_at_home has quit IRC | 05:31 | |
| *** yuanying has joined #openstack-infra | 05:32 | |
| *** yuanying has quit IRC | 05:33 | |
| *** yuanying has joined #openstack-infra | 05:35 | |
| *** amotoki_ has joined #openstack-infra | 05:39 | |
| Alekhya | hello can anyone help me with this issue: | 05:42 |
| Alekhya | failed to create instance in jenkins slave node and getting error in debug.log as ERROR nodepool.NodeLauncher: OpenStackCloudException launching node | 05:42 |
| *** amotoki has quit IRC | 05:43 | |
| *** binbincong has quit IRC | 05:44 | |
| stevemar | looks like a keystone tempest test is cause all sorts of issues | 05:48 |
| *** kushal has joined #openstack-infra | 05:51 | |
| *** mixos has quit IRC | 05:54 | |
| *** rcernin has joined #openstack-infra | 05:54 | |
| *** yuanying has quit IRC | 05:55 | |
| *** yuanying has joined #openstack-infra | 05:57 | |
| *** binbincong has joined #openstack-infra | 06:00 | |
| *** asettle has joined #openstack-infra | 06:02 | |
| *** yuanying has quit IRC | 06:03 | |
| *** yuanying has joined #openstack-infra | 06:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/315360 | 06:05 |
| *** andreas_s has joined #openstack-infra | 06:06 | |
| *** asettle has quit IRC | 06:06 | |
| *** yushiro has quit IRC | 06:07 | |
| *** amitgandhinz has joined #openstack-infra | 06:08 | |
| *** pcaruana has joined #openstack-infra | 06:12 | |
| *** amitgandhinz has quit IRC | 06:13 | |
| AJaeger | hongbin, sorry, don't have permissions to add you to new groups | 06:16 |
| *** khappone has quit IRC | 06:17 | |
| *** SergK has quit IRC | 06:21 | |
| *** SergK has joined #openstack-infra | 06:21 | |
| *** oanson has joined #openstack-infra | 06:22 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/314878 | 06:26 |
| *** jcoufal has joined #openstack-infra | 06:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/315360 | 06:29 |
| *** nadya has joined #openstack-infra | 06:36 | |
| *** salv-orlando has quit IRC | 06:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:39 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Actually use the service domain in V3 jobs https://review.openstack.org/315370 | 06:42 |
| *** smarcet has quit IRC | 06:43 | |
| *** cloudtrainme has joined #openstack-infra | 06:43 | |
| *** smarcet has joined #openstack-infra | 06:43 | |
| *** jiggalator has joined #openstack-infra | 06:44 | |
| *** reed has quit IRC | 06:44 | |
| *** knikolla has quit IRC | 06:45 | |
| *** netsin has quit IRC | 06:46 | |
| *** Hunner has quit IRC | 06:46 | |
| *** Hunner has joined #openstack-infra | 06:46 | |
| *** Hunner has quit IRC | 06:46 | |
| *** Hunner has joined #openstack-infra | 06:46 | |
| *** reed has joined #openstack-infra | 06:46 | |
| *** Swami_ has quit IRC | 06:47 | |
| *** knikolla has joined #openstack-infra | 06:48 | |
| *** nelsnelson has joined #openstack-infra | 06:48 | |
| *** yamamoto has quit IRC | 06:49 | |
| *** Swami_ has joined #openstack-infra | 06:50 | |
| *** yamamoto has joined #openstack-infra | 06:50 | |
| *** thiagop has quit IRC | 06:50 | |
| *** khappone has joined #openstack-infra | 06:52 | |
| *** nelsnelson has quit IRC | 06:52 | |
| *** thiagop has joined #openstack-infra | 06:53 | |
| *** aeng_ has quit IRC | 06:53 | |
| *** oanson has quit IRC | 06:54 | |
| *** cloudtrainme has quit IRC | 06:54 | |
| *** cloudtrainme has joined #openstack-infra | 06:55 | |
| *** tbarron has quit IRC | 06:57 | |
| *** knikolla has quit IRC | 06:58 | |
| *** crinkle_ has joined #openstack-infra | 06:59 | |
| *** Sukhdev has quit IRC | 06:59 | |
| *** crinkle has quit IRC | 07:00 | |
| *** StevenK has quit IRC | 07:00 | |
| *** crinkle_ is now known as crinkle | 07:00 | |
| *** tbarron has joined #openstack-infra | 07:00 | |
| *** mmedvede has quit IRC | 07:00 | |
| *** knikolla has joined #openstack-infra | 07:01 | |
| *** bodepd_ has quit IRC | 07:01 | |
| *** StevenK has joined #openstack-infra | 07:01 | |
| akijak | hi, I'd like to create a new channel: #openstack-pl (general disscussions, support in Polish). I've read https://wiki.openstack.org/wiki/IRC and here I am :) Can you help? | 07:01 |
| *** bodepd_ has joined #openstack-infra | 07:02 | |
| *** lennyb has quit IRC | 07:03 | |
| *** jaosorior has joined #openstack-infra | 07:03 | |
| *** sflanigan has quit IRC | 07:03 | |
| *** sflanigan has joined #openstack-infra | 07:03 | |
| *** sflanigan has joined #openstack-infra | 07:03 | |
| *** tesseract has joined #openstack-infra | 07:05 | |
| AJaeger | akijak: better come back once the US wakes up | 07:06 |
| *** ggnel_t has joined #openstack-infra | 07:07 | |
| akijak | AJaeger: ok, no problem. who can help me? do you know someone? ;) | 07:07 |
| AJaeger | akijak: what kind of help do you need? | 07:08 |
| *** amitgandhinz has joined #openstack-infra | 07:09 | |
| akijak | AJaeger: I need someone to create a channel for me | 07:09 |
| *** tesseract has quit IRC | 07:09 | |
| *** matrohon has joined #openstack-infra | 07:11 | |
| *** jlanoux has joined #openstack-infra | 07:11 | |
| *** rkukura has quit IRC | 07:12 | |
| *** arxcruz has joined #openstack-infra | 07:12 | |
| *** amitgandhinz has quit IRC | 07:13 | |
| *** shardy has joined #openstack-infra | 07:13 | |
| *** dalgaaf has joined #openstack-infra | 07:15 | |
| *** pahuang_ has quit IRC | 07:15 | |
| *** lennyb has joined #openstack-infra | 07:16 | |
| *** cloudtrainme has quit IRC | 07:17 | |
| *** rkukura has joined #openstack-infra | 07:17 | |
| AJaeger | akijak: I think you can do this yourself - but if you have doubts, wait until later (16:00 your time) and ask again here, there should be people around to help. | 07:18 |
| akijak | AJaeger: ok, thanks | 07:18 |
| *** jed56 has joined #openstack-infra | 07:19 | |
| *** esikachev has joined #openstack-infra | 07:19 | |
| *** cloudtrainme has joined #openstack-infra | 07:19 | |
| *** daemontool has joined #openstack-infra | 07:19 | |
| AJaeger | team, project-config reviewers, stevemar: This whole patch series strikes me as odd: https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:v3-only-gates | 07:20 |
| AJaeger | It duplicates functional jobs and adds jobs to check queues that do not test the projects at all. | 07:20 |
| *** lennyb has quit IRC | 07:20 | |
| openstackgerrit | Dirk Mueller proposed openstack/requirements: Add ovs with Python 3 support https://review.openstack.org/291775 | 07:20 |
| AJaeger | I've done some review on them but think this needs some better way - and won't have time to do it myself. If you review any of these changes, try to keep the big picture in mind | 07:21 |
| *** e0ne has joined #openstack-infra | 07:23 | |
| *** vincentll has joined #openstack-infra | 07:25 | |
| *** e0ne has quit IRC | 07:26 | |
| *** BobH_ has quit IRC | 07:26 | |
| *** mmedvede has joined #openstack-infra | 07:26 | |
| *** mikelk has joined #openstack-infra | 07:28 | |
| *** |-paul-| has joined #openstack-infra | 07:28 | |
| *** zeih has joined #openstack-infra | 07:29 | |
| *** kushal has quit IRC | 07:30 | |
| *** kzaitsev_mb has quit IRC | 07:30 | |
| *** yamahata has joined #openstack-infra | 07:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:31 | |
| *** lennyb has joined #openstack-infra | 07:32 | |
| *** salv-orlando has joined #openstack-infra | 07:37 | |
| *** dmk0202 has joined #openstack-infra | 07:40 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Fix apt-sources configuration for debian-minimal https://review.openstack.org/315265 | 07:41 |
| *** jlanoux has quit IRC | 07:42 | |
| *** salv-orlando has quit IRC | 07:42 | |
| *** hichihara has quit IRC | 07:44 | |
| *** kzaitsev_mb has quit IRC | 07:44 | |
| *** yaume has joined #openstack-infra | 07:48 | |
| *** Swami_ has quit IRC | 07:49 | |
| *** nadya has quit IRC | 07:52 | |
| *** cloudtrainme has quit IRC | 07:52 | |
| *** asettle has joined #openstack-infra | 07:52 | |
| daemontool | rcarrillocruz, yolanda at your convenience, +A is needed here https://review.openstack.org/#/c/314285/ new repo creation | 07:53 |
| *** gsilvis has quit IRC | 07:55 | |
| *** gsilvis has joined #openstack-infra | 07:56 | |
| *** dizquierdo_afk is now known as dizquierdo | 07:59 | |
| *** hashar has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** javeriak_ has quit IRC | 08:05 | |
| *** HeOS has joined #openstack-infra | 08:07 | |
| *** snarwade has quit IRC | 08:07 | |
| *** abregman has joined #openstack-infra | 08:07 | |
| *** rossella_s has joined #openstack-infra | 08:08 | |
| *** ifarkas has joined #openstack-infra | 08:08 | |
| *** chem has joined #openstack-infra | 08:09 | |
| *** ociuhandu has quit IRC | 08:09 | |
| *** amitgandhinz has joined #openstack-infra | 08:09 | |
| *** jamielennox is now known as jamielennox|away | 08:10 | |
| *** zhurong has quit IRC | 08:11 | |
| *** zhurong has joined #openstack-infra | 08:12 | |
| *** kushal has joined #openstack-infra | 08:14 | |
| *** amitgandhinz has quit IRC | 08:15 | |
| *** javeriak has joined #openstack-infra | 08:15 | |
| *** dtantsur|afk is now known as dtantsur | 08:17 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Integrate ansible-dev-magnum with real job in shade https://review.openstack.org/315401 | 08:17 |
| *** lucas-dinner is now known as lucasagomes | 08:17 | |
| *** sflanigan has quit IRC | 08:18 | |
| *** mhickey has joined #openstack-infra | 08:20 | |
| *** zhurong_ has joined #openstack-infra | 08:20 | |
| *** Liuqing_ has joined #openstack-infra | 08:20 | |
| *** Jeffrey4l has quit IRC | 08:22 | |
| *** zhurong has quit IRC | 08:22 | |
| *** Liuqing has quit IRC | 08:22 | |
| *** nadya has joined #openstack-infra | 08:23 | |
| *** asettle has quit IRC | 08:25 | |
| *** pt_15 has quit IRC | 08:26 | |
| *** Jeffrey4l has joined #openstack-infra | 08:26 | |
| *** abregman has quit IRC | 08:26 | |
| yolanda | good morning | 08:28 |
| *** dingyichen has quit IRC | 08:28 | |
| *** heyongli has quit IRC | 08:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Actually use the service domain in V3 jobs https://review.openstack.org/315370 | 08:30 |
| *** heyongli has joined #openstack-infra | 08:31 | |
| *** asettle has joined #openstack-infra | 08:33 | |
| *** oanson has joined #openstack-infra | 08:33 | |
| *** mjblack has quit IRC | 08:34 | |
| openstackgerrit | Merged openstack-infra/project-config: New project: python-bileanclient https://review.openstack.org/315326 | 08:34 |
| *** mjblack has joined #openstack-infra | 08:34 | |
| *** jistr has joined #openstack-infra | 08:35 | |
| *** kashyap has joined #openstack-infra | 08:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Migrating liberasurecode and pyeclib projects https://review.openstack.org/314724 | 08:38 |
| *** yamahata has quit IRC | 08:38 | |
| *** e0ne has joined #openstack-infra | 08:39 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Document "bindep test" usage https://review.openstack.org/314574 | 08:39 |
| openstackgerrit | Merged openstack-infra/project-config: Explicitly define if ipxe is enabled or not https://review.openstack.org/313799 | 08:41 |
| ajo | are we still building http://pypi.openstack.org/simple/ somewhre? | 08:41 |
| ajo | somewhere | 08:41 |
| ajo | I was considering to use the pypi-mirror tool to build one locally | 08:41 |
| ajo | but it doesn't work | 08:41 |
| openstackgerrit | Merged openstack-infra/project-config: Run identity v3 only job in keystone https://review.openstack.org/311169 | 08:41 |
| daemontool | yolanda, good morning and thanks | 08:42 |
| *** dtardivel has joined #openstack-infra | 08:42 | |
| kashyap | AJaeger: Heya, got a moment? | 08:43 |
| openstackgerrit | Merged openstack-infra/project-config: Gate tempest on - gate-tempest-dsvm-full-devstack-plugin-ceph https://review.openstack.org/315124 | 08:43 |
| openstackgerrit | Merged openstack-infra/project-config: Add ci-cd-pipeline-app-murano project to openstack https://review.openstack.org/309383 | 08:44 |
| kashyap | AJaeger: When you do: Can I turn this into a cherry-pick without messing up? Or just update the Change-ID similar to that of its corresponding Mitaka/Newton patch? -- https://review.openstack.org/#/c/314012/4 | 08:44 |
| *** esikachev has quit IRC | 08:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Publish debian packages after build https://review.openstack.org/311190 | 08:45 |
| openstackgerrit | Merged openstack-infra/project-config: Added openstack-server-publish-jobs to ironic-ui https://review.openstack.org/315107 | 08:50 |
| openstackgerrit | Merged openstack-infra/project-config: Create a tripleo-ci project in JJB https://review.openstack.org/315062 | 08:50 |
| openstackgerrit | Merged openstack-infra/project-config: api-ref gate for senlin https://review.openstack.org/314832 | 08:50 |
| openstackgerrit | Merged openstack-infra/project-config: Disaster Recovery new repo for the Freezer Project https://review.openstack.org/314285 | 08:52 |
| openstackgerrit | Merged openstack-infra/puppet-zuul: enable customize layout file name in zuul.conf https://review.openstack.org/314921 | 08:52 |
| *** electrofelix has joined #openstack-infra | 08:54 | |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 08:54 |
| *** salv-orlando has joined #openstack-infra | 08:56 | |
| AJaeger | kashyap: what you can do: create a branch off stable/mitaka and then run "git review -X ID" to cherry-pick | 08:57 |
| AJaeger | That's my preferred way to do backports | 08:58 |
| *** oanson has quit IRC | 08:58 | |
| kashyap | AJaeger: Err, I manually updated the change ID -- https://review.openstack.org/#/c/315423/ :-( | 08:58 |
| kashyap | AJaeger: I should've waited to hear you, before doing that | 08:58 |
| openstackgerrit | Dougal Matthews proposed openstack/requirements: Add Mistral to global requirements https://review.openstack.org/314638 | 08:59 |
| openstackgerrit | Merged openstack-infra/project-config: gnocchi: Add grenade jobs https://review.openstack.org/306363 | 08:59 |
| openstackgerrit | Merged openstack-infra/project-config: [Rally] Adding two new non-voiting jobs for `rally verify` https://review.openstack.org/310151 | 08:59 |
| openstackgerrit | Merged openstack-infra/system-config: Update AFS docs to reference afs02.dfw https://review.openstack.org/314677 | 08:59 |
| openstackgerrit | Merged openstack-infra/system-config: Correct new afs fileserver docs https://review.openstack.org/314678 | 09:00 |
| AJaeger | kashyap: note that if you change the change id, it becomes a new request, so abandon the old one. The ChangeID is a key. | 09:00 |
| openstackgerrit | Merged openstack-infra/project-config: Add sub projects to Kuryr https://review.openstack.org/313981 | 09:01 |
| kashyap | AJaeger: Yeah. I realized that | 09:01 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config: gnocchi: use Ceph devstack plugin https://review.openstack.org/314430 | 09:01 |
| *** pcaruana is now known as pcaruana|afk| | 09:01 | |
| kashyap | AJaeger: Can I somehow turn this new change with correct change-ID into a 'cherry-picked' one? | 09:01 |
| *** salv-orlando has quit IRC | 09:02 | |
| AJaeger | Just redo it - and if you send a new patch with the same change-ID, it will override the existing one | 09:02 |
| sileht | AJaeger, Can you +A again this one: https://review.openstack.org/314430, gerrit was complaining about merge conflict, and rebase 'button' fix it | 09:02 |
| *** esikachev has joined #openstack-infra | 09:03 | |
| kashyap | AJaeger: Okay, let me try that. Your "redo it" -- I take it that "Try a proper cherry-pick, resolve conflicts and use the same change ID> | 09:03 |
| kashyap | Thanks | 09:03 |
| yolanda | AJaeger, can you review https://review.openstack.org/315401 ? | 09:03 |
| *** jtomasek_ has joined #openstack-infra | 09:04 | |
| AJaeger | kashyap: yes | 09:04 |
| AJaeger | yolanda: sure | 09:04 |
| *** openstackgerrit has quit IRC | 09:04 | |
| *** openstackgerrit has joined #openstack-infra | 09:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Move gate-grenade-dsvm-manila job to check pipeline https://review.openstack.org/314497 | 09:06 |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting gate-swift-tox-bandit job https://review.openstack.org/314100 | 09:06 |
| openstackgerrit | David TARDIVEL proposed openstack-infra/project-config: Add new puppet-watcher repository https://review.openstack.org/315431 | 09:06 |
| yolanda | thx | 09:07 |
| *** hparekh has quit IRC | 09:11 | |
| *** amitgandhinz has joined #openstack-infra | 09:11 | |
| *** hparekh has joined #openstack-infra | 09:13 | |
| *** sonuk has joined #openstack-infra | 09:13 | |
| *** amitgandhinz has quit IRC | 09:16 | |
| *** sonuk has left #openstack-infra | 09:16 | |
| openstackgerrit | Merged openstack-infra/project-config: Integrate ansible-dev-magnum with real job in shade https://review.openstack.org/315401 | 09:20 |
| openstackgerrit | Merged openstack-infra/project-config: Enable translation jobs for manila-ui https://review.openstack.org/314905 | 09:21 |
| AJaeger | Wow, yolanda. What a review party ;) | 09:22 |
| yolanda | yep, lots of things pending | 09:22 |
| *** jtomasek has quit IRC | 09:22 | |
| yolanda | i'm checking puppetmaster to see if jenkins is processing correctly | 09:23 |
| AJaeger | thanks, yolanda | 09:26 |
| openstackgerrit | Merged openstack-infra/project-config: gnocchi: use Ceph devstack plugin https://review.openstack.org/314430 | 09:28 |
| openstackgerrit | Pranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting. https://review.openstack.org/314998 | 09:29 |
| skraynev | yolanda: thank you for the merge my patch. Could you also add me to these groups as a member https://review.openstack.org/#/admin/groups/1384,members https://review.openstack.org/#/admin/groups/1385,members | 09:34 |
| yolanda | sure | 09:34 |
| *** jtomasek_ is now known as jtomasek | 09:35 | |
| skraynev | thanks yolanda | 09:35 |
| *** mhickey has quit IRC | 09:36 | |
| yolanda | mm, jenkins-dev is failing | 09:36 |
| yolanda | fatal: [jenkins-dev.openstack.org]: FAILED! => {"changed": false, "failed": true, "msg": "/usr/bin/timeout -s 9 30m /usr/bin/puppet apply --detailed-exitcodes --logdest syslog --no-noop /opt/system-config/production/manifests/site.pp failed with return code: 6", "rc": 6, "stderr": "Could not retrieve fact='package_provider', resolution='<anonymous>': uninitialized constant Gem\n", "stdout": "", "stdout_lines": []} | 09:36 |
| yolanda | going to check other jenkins masters as well | 09:37 |
| AJaeger | yolanda: could you add hongbin also to higgins-core and higgins-release, please? https://review.openstack.org/#/c/313935/ - he asked earlier about it. | 09:37 |
| AJaeger | OOps ;( | 09:37 |
| openstackgerrit | Pranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting. https://review.openstack.org/314998 | 09:37 |
| yolanda | ok adding him | 09:37 |
| *** mhickey has joined #openstack-infra | 09:40 | |
| *** dizquierdo has quit IRC | 09:40 | |
| apuimedo | yolanda: thanks for the review! | 09:43 |
| *** hashar has quit IRC | 09:44 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Updating Watcher team meeting time https://review.openstack.org/314613 | 09:45 |
| yolanda | np | 09:45 |
| *** dizquierdo has joined #openstack-infra | 09:45 | |
| *** yaume has quit IRC | 09:47 | |
| *** amotoki has joined #openstack-infra | 09:52 | |
| *** amotoki_ has quit IRC | 09:54 | |
| *** gildub has quit IRC | 09:56 | |
| *** tosky has joined #openstack-infra | 09:57 | |
| *** ociuhandu has joined #openstack-infra | 09:57 | |
| *** kmartin has joined #openstack-infra | 10:00 | |
| *** yamamoto has quit IRC | 10:05 | |
| *** jlanoux has joined #openstack-infra | 10:05 | |
| *** zhurong_ has quit IRC | 10:06 | |
| *** tosky has quit IRC | 10:06 | |
| *** paul-carlton2 has joined #openstack-infra | 10:06 | |
| *** Qiming_ has quit IRC | 10:06 | |
| *** tosky has joined #openstack-infra | 10:06 | |
| *** kmartin has quit IRC | 10:08 | |
| *** vkramskikh_ has joined #openstack-infra | 10:08 | |
| *** esikachev has quit IRC | 10:08 | |
| *** Liuqing_ has quit IRC | 10:08 | |
| *** amitgandhinz has joined #openstack-infra | 10:12 | |
| *** psachin has quit IRC | 10:12 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/requirements: Add global requirement update on python-brick-cinderclient-ext project https://review.openstack.org/309084 | 10:13 |
| *** chem has quit IRC | 10:16 | |
| *** amitgandhinz has quit IRC | 10:17 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/irc-meetings: Change time for FWaaS meeting https://review.openstack.org/315308 | 10:18 |
| *** psachin has joined #openstack-infra | 10:18 | |
| *** sdague has joined #openstack-infra | 10:34 | |
| *** yamamoto_ has joined #openstack-infra | 10:34 | |
| *** duobei has quit IRC | 10:40 | |
| openstackgerrit | Artem Silenkov proposed openstack-infra/puppet-jenkins: Fix incorrect directory permissions in jenkinsuser.pp https://review.openstack.org/315479 | 10:54 |
| *** asilenkov_ has joined #openstack-infra | 10:56 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Enable api-ref jobs in Sahara https://review.openstack.org/315480 | 10:57 |
| asilenkov_ | hello! could you please review https://review.openstack.org/#/c/315479/ - manifest for jenkins | 10:57 |
| *** sree has quit IRC | 11:00 | |
| *** kushal has quit IRC | 11:00 | |
| *** dizquierdo is now known as dizquierdo_afk | 11:00 | |
| *** sree has joined #openstack-infra | 11:01 | |
| *** madhu_ak has quit IRC | 11:01 | |
| *** bastelfreak has quit IRC | 11:05 | |
| *** sree has quit IRC | 11:05 | |
| *** amotoki has quit IRC | 11:06 | |
| *** tellesnobrega is now known as tellesnobrega_af | 11:06 | |
| *** claudiub has joined #openstack-infra | 11:06 | |
| *** amotoki has joined #openstack-infra | 11:10 | |
| *** hamzy has quit IRC | 11:12 | |
| openstackgerrit | Pranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting. https://review.openstack.org/314998 | 11:12 |
| *** amitgandhinz has joined #openstack-infra | 11:13 | |
| *** tosky has quit IRC | 11:14 | |
| *** amotoki has quit IRC | 11:14 | |
| *** dansmith has quit IRC | 11:14 | |
| *** ccamacho is now known as ccamacho|lunch | 11:15 | |
| openstackgerrit | Pranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting. https://review.openstack.org/314998 | 11:16 |
| *** dansmith has joined #openstack-infra | 11:17 | |
| *** dansmith is now known as Guest51926 | 11:17 | |
| *** amitgandhinz has quit IRC | 11:17 | |
| *** julim has joined #openstack-infra | 11:22 | |
| *** deadnull_ has joined #openstack-infra | 11:23 | |
| *** bastelfreak has joined #openstack-infra | 11:24 | |
| *** Liuqing has joined #openstack-infra | 11:25 | |
| *** jaosorior has quit IRC | 11:28 | |
| *** jaosorior has joined #openstack-infra | 11:29 | |
| *** salv-orlando has joined #openstack-infra | 11:30 | |
| *** gordc has joined #openstack-infra | 11:31 | |
| *** salv-orlando has quit IRC | 11:35 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Allow to set BUILD_TIMEOUT from the job https://review.openstack.org/315496 | 11:37 |
| *** ldnunes has joined #openstack-infra | 11:38 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/315499 | 11:38 |
| *** Liuqing has quit IRC | 11:38 | |
| *** vgridnev- has joined #openstack-infra | 11:39 | |
| *** vgridnev- is now known as vgridnev | 11:39 | |
| *** Liuqing has joined #openstack-infra | 11:39 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Allow to set BUILD_TIMEOUT from the job https://review.openstack.org/315496 | 11:41 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/315499 | 11:41 |
| *** ZZelle has joined #openstack-infra | 11:42 | |
| *** weshay has joined #openstack-infra | 11:43 | |
| *** tpsilva has joined #openstack-infra | 11:44 | |
| *** hashar has joined #openstack-infra | 11:45 | |
| *** baoli has joined #openstack-infra | 11:46 | |
| *** baoli has quit IRC | 11:46 | |
| *** baoli has joined #openstack-infra | 11:47 | |
| *** vincentll has quit IRC | 11:48 | |
| *** vincentll has joined #openstack-infra | 11:49 | |
| *** baoli_ has joined #openstack-infra | 11:49 | |
| *** baoli has quit IRC | 11:52 | |
| *** hamzy has joined #openstack-infra | 11:54 | |
| *** javeriak has quit IRC | 11:56 | |
| *** ttx has quit IRC | 11:56 | |
| *** dprince has joined #openstack-infra | 11:59 | |
| *** salv-orlando has joined #openstack-infra | 11:59 | |
| *** vincentll has quit IRC | 12:00 | |
| *** vincentll has joined #openstack-infra | 12:00 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow lists of boards and worklists to be paged https://review.openstack.org/309481 | 12:03 |
| *** ttx has joined #openstack-infra | 12:07 | |
| *** psilvad has joined #openstack-infra | 12:12 | |
| *** amitgandhinz has joined #openstack-infra | 12:14 | |
| *** kengo_sakai has quit IRC | 12:15 | |
| *** rodrigods has quit IRC | 12:15 | |
| *** rodrigods has joined #openstack-infra | 12:15 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Add Higgins project to projects.txt https://review.openstack.org/315249 | 12:16 |
| *** pradk has joined #openstack-infra | 12:17 | |
| *** amitgandhinz has quit IRC | 12:19 | |
| *** ddieterly has joined #openstack-infra | 12:21 | |
| *** javeriak has joined #openstack-infra | 12:23 | |
| *** aysyd has joined #openstack-infra | 12:27 | |
| *** julim has quit IRC | 12:28 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Improve the worklist filtering functionality https://review.openstack.org/310287 | 12:28 |
| *** raildo has joined #openstack-infra | 12:28 | |
| *** kgiusti has joined #openstack-infra | 12:28 | |
| raildo | AJaeger: hey :) | 12:28 |
| AJaeger | hi raildo ! Thanks | 12:29 |
| *** |-paul-| has quit IRC | 12:29 | |
| raildo | AJaeger: I liked your plan, i'll prepare an email to explain this patches and our plan | 12:30 |
| AJaeger | Looking at your changes, I think we should redo them. Let me point out some issues that I see, might be easier to discuss here. | 12:30 |
| AJaeger | Change | 12:30 |
| AJaeger | Argh, trying again... | 12:30 |
| pabelanger | morning | 12:30 |
| AJaeger | Change https://review.openstack.org/#/c/315224/1/zuul/layout.yaml adds a job to magnum that does not even run magnum during the test AFAIK | 12:30 |
| *** ddieterly is now known as ddieterly[away] | 12:30 | |
| AJaeger | And that job is added in a few changes - so that's not testing what you expect at all. | 12:30 |
| AJaeger | morning, pabelanger | 12:31 |
| AJaeger | pabelanger: we discuss https://review.openstack.org/#/q/topic:v3-only-gates | 12:31 |
| AJaeger | raildo: So, those are obviously wrong... | 12:31 |
| AJaeger | Second problem: I think *all* test should only be run on master now and future stable branches (newton,...) | 12:31 |
| AJaeger | So, use *one* regex - see the novaclient review that covers all these jobs | 12:32 |
| raildo | AJaeger: actually this patch runs the tempest tests in a devstack with keystone v3-only. So I'm supposing that the service have tests on tempest | 12:32 |
| AJaeger | But it tests keystone - and is not using magnum at all AFAIK | 12:32 |
| *** pblaho has quit IRC | 12:33 | |
| AJaeger | Did you verify that it does? | 12:33 |
| raildo | AJaeger: second point, i totally agree with you, I'll fix this patches today :) | 12:33 |
| AJaeger | If I'M wrong, please tell me - best to state that in your email and explain why you do it... | 12:33 |
| *** zz_dimtruck is now known as dimtruck | 12:33 | |
| AJaeger | 3rd issue: You duplicate all the functional jobs. | 12:33 |
| AJaeger | I think, it might be better to use a *single* functional template and have an if in there. | 12:34 |
| *** hashar has quit IRC | 12:34 | |
| AJaeger | So, something like - looking at https://review.openstack.org/#/c/313659/4/jenkins/jobs/swift.yaml: | 12:34 |
| AJaeger | name: '{pipeline}-swift-dsvm-functional{special}{job-suffix}' | 12:35 |
| *** javeriak has quit IRC | 12:35 | |
| *** rhallisey has joined #openstack-infra | 12:35 | |
| AJaeger | with if [ "{special} == '-identity-v3-only' ] ; then export DEVSTACK_LOCAL_CONFIG="ENABLE_IDENTITY_V2=False" fi | 12:35 |
| AJaeger | and then instantiate the template twice, once with special: '', once with special: '-identity-v3-only' | 12:36 |
| raildo | AJaeger: got it, this makes sense... I'll refactor all those patches with your suggestions :) | 12:36 |
| AJaeger | Reason: It looks like you duplicate the functional jobs completely with just that single line changed... | 12:36 |
| AJaeger | raildo: Refactor one or two, let'S review them - and then do the rest ;) | 12:37 |
| raildo | AJaeger: I didn't had find a best way to make this happen... | 12:37 |
| raildo | AJaeger: thank you sir, I'll send a email explain my plan and make this changes asap | 12:37 |
| AJaeger | raildo: also, as a FYI: I'm offline starting from saturday for two weeks. I can help today with reviewing and suggestions but afterwards others have to take over - that's why I'm doing the discussio here publically, so that other project-config experts can chime in... | 12:37 |
| *** rlandy has quit IRC | 12:38 | |
| *** pblaho has joined #openstack-infra | 12:38 | |
| *** ccamacho|lunch is now known as ccamacho | 12:38 | |
| AJaeger | raildo: Thanks! Seeing the big picture will help with the reviews... | 12:38 |
| raildo | AJaeger: I appreciate that | 12:38 |
| *** rlandy has joined #openstack-infra | 12:38 | |
| *** nelsnelson has joined #openstack-infra | 12:39 | |
| *** kushal has joined #openstack-infra | 12:39 | |
| *** matt-borland has joined #openstack-infra | 12:39 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Improve the board filtering functionality https://review.openstack.org/310301 | 12:40 |
| *** esker has joined #openstack-infra | 12:42 | |
| *** nelsnelson has quit IRC | 12:43 | |
| *** esker has quit IRC | 12:45 | |
| *** amrith is now known as _amrith_ | 12:46 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Enable api-ref jobs in Sahara https://review.openstack.org/315480 | 12:46 |
| *** sree has joined #openstack-infra | 12:46 | |
| *** esker has joined #openstack-infra | 12:47 | |
| *** andymaier has joined #openstack-infra | 12:47 | |
| *** rfolco has joined #openstack-infra | 12:48 | |
| *** andymaier has quit IRC | 12:49 | |
| *** kushal has quit IRC | 12:49 | |
| *** sree has quit IRC | 12:50 | |
| pabelanger | #success centos AFS rpm-mirror online and being used in the gate! EG: http://mirror.dfw.rax.openstack.org/centos/7/ | 12:52 |
| openstackstatus | pabelanger: Added success to Success page | 12:52 |
| pabelanger | next stop epel | 12:52 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/system-config: Add openstack-requirements irc channel to bots https://review.openstack.org/315528 | 12:53 |
| pabelanger | not a fan of having a 3:1 building / in use ratio for OSIC. Looks to be related to FIPs again | 12:54 |
| *** ddieterly[away] is now known as ddieterly | 12:54 | |
| *** binbincong has quit IRC | 12:56 | |
| AJaeger | pabelanger: great! | 12:56 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Bumps os-win version to 0.4.2 https://review.openstack.org/308506 | 12:56 |
| AJaeger | dims: do you only have a change for system-config? | 12:56 |
| *** julim has joined #openstack-infra | 12:57 | |
| dims | AJaeger : was staring at project-config now :) | 12:57 |
| AJaeger | dims: Add one for project-config - accessbot and gerritbot first - and make the system-config dependent on it ;) | 12:57 |
| AJaeger | dims: Great! | 12:57 |
| dims | AJaeger : ack. appreciate it | 12:57 |
| sdague | dims: wow, all the requirements pushes :) | 12:58 |
| AJaeger | dims: glad to see that the team is forming now... | 12:58 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Allow to set BUILD_TIMEOUT from the job https://review.openstack.org/315496 | 12:58 |
| dims | sdague : making use of the new folks :) | 12:58 |
| AJaeger | dims: keep them busy ;) | 12:59 |
| dims | AJaeger : yay :) | 12:59 |
| sdague | man, the times when the gate was empty in the morning :) | 12:59 |
| sdague | and openstack was all PST startups | 12:59 |
| *** ddieterly has quit IRC | 12:59 | |
| dims | sdague good ol' times :) | 13:00 |
| *** sree has joined #openstack-infra | 13:00 | |
| dims | sdague : i loved the apiref sprint emails, hoping seeing their names in the updates motivated folks :) | 13:01 |
| *** pcaruana|afk| is now known as pcaruana | 13:01 | |
| skraynev | AJaeger: hi. I have a short question about project-config and gerrit behavior. Do I correctly understand, that specified upstream repo will be cloned once during merging patch to project-config? the issue, that I late to realize, that my github repo was outdated, so we have 20 more commits above the head. I updated it, but looks like copy on openstack/* is not synced. How it should be fixed correctly ? | 13:01 |
| *** chem has joined #openstack-infra | 13:01 | |
| sdague | dims: me too :) | 13:01 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/315499 | 13:01 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Allow to set BUILD_TIMEOUT from the job https://review.openstack.org/315496 | 13:01 |
| skraynev | repeat all history manually or anybody can sync it manually ? | 13:01 |
| AJaeger | skraynev: correct, "upstream" means "import once" | 13:01 |
| skraynev | s/all/whole | 13:01 |
| skraynev | ough | 13:02 |
| AJaeger | skraynev: 20 commits? Send them as proper reviews through review.openstack.org. | 13:02 |
| dims | sdague : there's only so many carrots we have :) | 13:03 |
| * AJaeger agrees with dims: Great work, sdague ! | 13:03 | |
| *** nelsnelson has joined #openstack-infra | 13:04 | |
| sdague | dims: I'm actually kind of thinking about how the tooling could be generalized for any sprint. There is conceptually really only 2 things. 1) a way to count what's left and put it into a csv for the burndown chart 2) a gerrit query that returns all the related work | 13:04 |
| *** mriedem has joined #openstack-infra | 13:04 | |
| skraynev | AJaeger: yeah. ok. looks like I create more work for myself ;))) thank you. | 13:04 |
| AJaeger | skraynev: an infra admin might be able to help - but I'm not one of them. | 13:04 |
| *** coreycb has quit IRC | 13:04 | |
| dims | sdague : right | 13:04 |
| *** edmondsw has joined #openstack-infra | 13:04 | |
| *** coreycb has joined #openstack-infra | 13:04 | |
| AJaeger | skraynev: if you have an idea on how to improve http://docs.openstack.org/infra/manual/creators.html#add-the-project-to-the-master-projects-list so that not others hit the same problem: Changes are welcome! | 13:04 |
| *** rkukura has quit IRC | 13:05 | |
| *** sree__ has joined #openstack-infra | 13:05 | |
| *** duobei has joined #openstack-infra | 13:06 | |
| skraynev | AJaeger: :) I suppose, that it's more tricky and rare situation. | 13:06 |
| *** andymaier has joined #openstack-infra | 13:06 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Add new channel #openstack-requirements https://review.openstack.org/315543 | 13:07 |
| *** keedya has joined #openstack-infra | 13:07 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/system-config: Add openstack-requirements irc channel to bots https://review.openstack.org/315528 | 13:08 |
| *** sree has quit IRC | 13:08 | |
| *** binbincong has joined #openstack-infra | 13:08 | |
| *** psilvad has quit IRC | 13:08 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Increase tempest baremetal deploy/unprovision timeouts https://review.openstack.org/315545 | 13:09 |
| *** psilvad has joined #openstack-infra | 13:09 | |
| *** yanyanhu has joined #openstack-infra | 13:09 | |
| *** rossella_s has quit IRC | 13:09 | |
| *** rossella_s has joined #openstack-infra | 13:09 | |
| *** itisha has quit IRC | 13:09 | |
| *** rkukura has joined #openstack-infra | 13:10 | |
| *** amitgandhinz has joined #openstack-infra | 13:11 | |
| *** amitgandhinz has quit IRC | 13:11 | |
| *** jsavak has joined #openstack-infra | 13:11 | |
| *** amitgandhinz has joined #openstack-infra | 13:12 | |
| *** akaszuba has joined #openstack-infra | 13:12 | |
| *** akaszuba_ has joined #openstack-infra | 13:13 | |
| *** akaszuba_ has quit IRC | 13:13 | |
| *** links has quit IRC | 13:14 | |
| *** jsavak has quit IRC | 13:16 | |
| *** duobei has quit IRC | 13:16 | |
| *** jsavak has joined #openstack-infra | 13:16 | |
| *** javeriak has joined #openstack-infra | 13:17 | |
| *** cloudtrainme has joined #openstack-infra | 13:17 | |
| *** xyang1 has joined #openstack-infra | 13:18 | |
| *** chem has quit IRC | 13:19 | |
| *** psilvad has quit IRC | 13:20 | |
| *** chem has joined #openstack-infra | 13:21 | |
| *** jamesmca_ has joined #openstack-infra | 13:21 | |
| *** amotoki has joined #openstack-infra | 13:21 | |
| *** esikachev has joined #openstack-infra | 13:25 | |
| *** Guest51926 is now known as dansmith | 13:26 | |
| *** denisra has quit IRC | 13:26 | |
| *** regXboi has joined #openstack-infra | 13:27 | |
| *** Goneri has joined #openstack-infra | 13:27 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:28 | |
| *** _ari_ has joined #openstack-infra | 13:29 | |
| *** nadya has quit IRC | 13:29 | |
| *** psachin has quit IRC | 13:30 | |
| *** psilvad has joined #openstack-infra | 13:30 | |
| *** zeih has quit IRC | 13:31 | |
| *** psachin has joined #openstack-infra | 13:31 | |
| *** zeih has joined #openstack-infra | 13:31 | |
| *** yanyanhu has quit IRC | 13:33 | |
| *** yamamoto_ has quit IRC | 13:33 | |
| *** psilvad has quit IRC | 13:34 | |
| *** dimtruck is now known as zz_dimtruck | 13:34 | |
| *** zz_dimtruck is now known as dimtruck | 13:34 | |
| *** ramishra has quit IRC | 13:35 | |
| *** psilvad has joined #openstack-infra | 13:35 | |
| *** ddieterly has joined #openstack-infra | 13:36 | |
| *** psilvad has quit IRC | 13:37 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: [WIP] Trying a new target just for u-c sanity check https://review.openstack.org/315569 | 13:37 |
| *** psilvad has joined #openstack-infra | 13:37 | |
| *** ramishra has joined #openstack-infra | 13:38 | |
| *** jamesmca_ has quit IRC | 13:39 | |
| *** psilvad has quit IRC | 13:39 | |
| *** psilvad has joined #openstack-infra | 13:39 | |
| *** kien-ha has joined #openstack-infra | 13:40 | |
| *** kien-ha_ has joined #openstack-infra | 13:41 | |
| *** kien-ha_ has quit IRC | 13:41 | |
| *** kien-ha has quit IRC | 13:41 | |
| *** kien-ha has joined #openstack-infra | 13:41 | |
| *** dtantsur is now known as dtantsur|bbl | 13:42 | |
| *** dizquierdo_afk is now known as dizquierdo | 13:42 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/irc-meetings: Change time for FWaaS meeting https://review.openstack.org/315308 | 13:42 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: [WIP] Trying a new target just for u-c sanity check https://review.openstack.org/315569 | 13:42 |
| *** jamesmca_ has joined #openstack-infra | 13:42 | |
| *** Alekhya has quit IRC | 13:42 | |
| *** cloudtrainme has quit IRC | 13:43 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin https://review.openstack.org/314862 | 13:43 |
| *** dimtruck is now known as zz_dimtruck | 13:44 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin https://review.openstack.org/314862 | 13:44 |
| fungi | heads up, i've got a bunch of guests coming to stay tonight through the weekend so i'll only be around intermittently over the next couple days. disappearing now for an hour or two so i can run some errands but should be back again by 16:00 utc | 13:44 |
| *** wanghua has quit IRC | 13:44 | |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/bindep: Add support for openSUSE and SLES https://review.openstack.org/315575 | 13:45 |
| AJaeger | toabctl: yeah! ^ | 13:46 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: New tox target just for u-c sanity check https://review.openstack.org/315569 | 13:46 |
| toabctl | AJaeger: but still a question - I need to install diffutils for rpm-packaging but looks like it does not work. see http://logs.openstack.org/85/311285/6/check/gate-rpm-packaging-tox-lint/ec9e337/console.html | 13:46 |
| toabctl | AJaeger: any idea why? | 13:46 |
| *** ddieterly is now known as ddieterly[away] | 13:47 | |
| *** eharney has joined #openstack-infra | 13:47 | |
| *** javeriak has quit IRC | 13:47 | |
| *** jamesmca_ has quit IRC | 13:48 | |
| *** tosky has joined #openstack-infra | 13:49 | |
| AJaeger | toabctl: strange ;( | 13:49 |
| pabelanger | fungi: ack | 13:49 |
| pabelanger | toabctl: AJaeger: looks like other-requirements.txt is empty? | 13:49 |
| *** jamesmca_ has joined #openstack-infra | 13:50 | |
| AJaeger | toabctl: see my comment on 315575, a line like "diffutils [pkg:suse linux test]" might not work | 13:50 |
| AJaeger | toabctl: if you add other-requirements, you replace the bindep-fallback.txt file that is normally installed. Perhaps you need some more packages | 13:51 |
| toabctl | AJaeger: ok. but that's independent from the jenkins failure. | 13:51 |
| toabctl | pabelanger: it isn't | 13:51 |
| AJaeger | toabctl: project-config/jenkins/data/bindep-fallback.txt is the complete list, add that to other-requirements to exclude that you're missing files | 13:52 |
| pabelanger | toabctl: is diffutils valid on ubuntu? | 13:52 |
| toabctl | pabelanger: yes. see http://packages.ubuntu.com/trusty/diffutils | 13:53 |
| AJaeger | toabctl: it fails for me locally with the same problem as in the gate ;) | 13:53 |
| toabctl | AJaeger: oh. really? it passes for me | 13:54 |
| *** jamesmca_ has quit IRC | 13:54 | |
| toabctl | AJaeger: what fails? tox -elint ? | 13:54 |
| AJaeger | tox -e lint fails | 13:55 |
| *** ddieterly[away] is now known as ddieterly | 13:55 | |
| AJaeger | open("/usr/share/spec-cleaner/excludes-bracketing.txt", O_RDONLY) = -1 ENOENT (No such file or directory) | 13:55 |
| AJaeger | toabctl: remove spec-cleaner from your install and try again... | 13:55 |
| pabelanger | toabctl: AJaeger: Oh, diffutils likely installed already, that's why bindep is not attempting to install it | 13:55 |
| AJaeger | Looks like spec-cleaner in the venv is not working properly | 13:56 |
| toabctl | oh. so spec-cleaner is missing. not diff | 13:56 |
| *** psilvad has quit IRC | 13:56 | |
| toabctl | ok. thanks a lot! | 13:56 |
| AJaeger | toabctl: spec-cleaner is isntalled - but spec-cleaner looks for excludes-bracketing.txt in wrong location | 13:56 |
| *** andymaier has quit IRC | 13:56 | |
| AJaeger | toabctl: packaging error in spec-cleaner - it does not work in venv! | 13:56 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: New CI job for requirements repo https://review.openstack.org/315584 | 13:57 |
| *** anteaya has joined #openstack-infra | 13:58 | |
| pabelanger | I am seeing: OpenStackCloudTimeout: Timeout waiting for the floating IP to be attached. in osic-cloud1 a fair bit | 14:03 |
| *** mhickey has quit IRC | 14:03 | |
| *** jpr has joined #openstack-infra | 14:04 | |
| *** sdake has joined #openstack-infra | 14:05 | |
| *** esker has quit IRC | 14:06 | |
| *** esker has joined #openstack-infra | 14:06 | |
| pabelanger | Cleaned 83 FIPs from openstackjenkins-osic-cloud1 :( | 14:07 |
| pabelanger | mordred: ^ | 14:07 |
| *** zz_dimtruck is now known as dimtruck | 14:08 | |
| AJaeger | OOps ;( | 14:08 |
| pabelanger | Ya, we appear to be timing out attaching to them | 14:08 |
| *** _amrith_ is now known as amrith | 14:09 | |
| *** jpr has quit IRC | 14:09 | |
| *** roxanaghe has joined #openstack-infra | 14:10 | |
| *** baumann has joined #openstack-infra | 14:11 | |
| *** doug-fish has joined #openstack-infra | 14:11 | |
| cloudnull | pabelanger: regarding the OSIC cloud w/ floats. I can reproduce that issue. | 14:12 |
| *** sree has joined #openstack-infra | 14:12 | |
| cloudnull | I've been testing with building 50+ nodes using heat and all of them are coming online without issue. | 14:12 |
| *** roxanaghe has quit IRC | 14:13 | |
| cloudnull | If you have instance IDs and such I can go log diving again to see if I can find the problem | 14:14 |
| *** d0ugal has quit IRC | 14:14 | |
| *** maishsk has joined #openstack-infra | 14:14 | |
| *** roxanaghe has joined #openstack-infra | 14:14 | |
| cloudnull | https://gist.github.com/cloudnull/36c92e0da5e61b13510560ae15227453 -- super simple heat template to build N instances and give it a float | 14:14 |
| pabelanger | cloudnull: sure, I can get an instance id | 14:15 |
| morgan | pabelanger: that is no good :( | 14:15 |
| *** sree__ has quit IRC | 14:15 | |
| *** d0ugal has joined #openstack-infra | 14:16 | |
| pabelanger | cloudnull: bebeeb5e-9125-417d-92e3-4b88e89e8dd4 is the last instances that timed out | 14:17 |
| *** psilvad has joined #openstack-infra | 14:17 | |
| *** zeih has quit IRC | 14:17 | |
| *** mhickey has joined #openstack-infra | 14:17 | |
| * cloudnull looking now | 14:18 | |
| d0ugal | Anyone got a moment to checkout this requirement addition? It just adds an openstack project. https://review.openstack.org/314638 | 14:18 |
| *** rbrndt has joined #openstack-infra | 14:18 | |
| pabelanger | morgan: cloudnull this is the error from shade: 2016-05-12 14:09:53,411 ERROR shade: Timeout waiting for floating IP to become active. Floating IP 172.99.106.69:463d50d6-bfaf-4ea2-b943-180162c34138 was created for server bebeeb5e-9125-417d-92e3-4b88e89e8dd4 but is being deleted due to activation failure. | 14:19 |
| *** roxanaghe has quit IRC | 14:19 | |
| pabelanger | going to look at code path now | 14:19 |
| *** yaume has joined #openstack-infra | 14:20 | |
| AJaeger | d0ugal: best ask on #openstack-requirements | 14:20 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin https://review.openstack.org/314862 | 14:21 |
| *** AJaeger has quit IRC | 14:21 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin https://review.openstack.org/314862 | 14:21 |
| *** nadya has joined #openstack-infra | 14:21 | |
| *** deadnull_ is now known as _deadnull | 14:22 | |
| morgan | pabelanger: ok, let me know if you need extra eyes. at least we know the FIP cleaner works well. | 14:22 |
| *** d0ugal has quit IRC | 14:23 | |
| *** cbader_ has joined #openstack-infra | 14:23 | |
| *** flaper87 has quit IRC | 14:24 | |
| *** nadya has quit IRC | 14:24 | |
| *** cbader has quit IRC | 14:24 | |
| *** Shrews has quit IRC | 14:24 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Remove check-requirements from Ceilometer and Aodh https://review.openstack.org/315595 | 14:24 |
| dstufft | fungi: do any of the other places support gpg? I can't think of one but I dunno! | 14:24 |
| akijak | hi, I'd like to create a new channel: #openstack-pl (general disscussions, support in Polish). I've read https://wiki.openstack.org/wiki/IRC and here I am :) Who can help? | 14:24 |
| *** Shrews has joined #openstack-infra | 14:24 | |
| *** ggillies has quit IRC | 14:25 | |
| *** ggillies has joined #openstack-infra | 14:26 | |
| clarkb | akijak: I am sure several of the people.here can help. Do you have specific questions? | 14:26 |
| *** baumann1 has joined #openstack-infra | 14:26 | |
| *** flaper87 has joined #openstack-infra | 14:27 | |
| *** flaper87 has quit IRC | 14:27 | |
| *** flaper87 has joined #openstack-infra | 14:27 | |
| *** xwizard has joined #openstack-infra | 14:27 | |
| pabelanger | morgan: sure, I am trying to trace the path to see why we are leaking the FIP now | 14:28 |
| clarkb | akijak: http://docs.openstack.org/infra/system-config/irc.html has details on registering a channel and adding our bots | 14:28 |
| *** sree has quit IRC | 14:29 | |
| morgan | pabelanger: i am on a plane so limited access, will be on the ground in a couple hrs. def can jump in then, if you need eyes sooner i'll have a copy of shade handy but it'll take me a bit to be in sync. | 14:29 |
| morgan | pabelanger: but i'll keep eyes here in case i can help | 14:29 |
| *** baumann has quit IRC | 14:29 | |
| *** _deadnull is now known as deadnull_ | 14:31 | |
| *** zeih has joined #openstack-infra | 14:32 | |
| anteaya | akijak: have you read http://docs.openstack.org/infra/system-config/irc.html | 14:32 |
| anteaya | akijak: it tells you how to register a channel yourself | 14:32 |
| *** jamesmca_ has joined #openstack-infra | 14:32 | |
| *** yamamoto_ has joined #openstack-infra | 14:33 | |
| akijak | clarkb: anteaya thx I'll check that. I've read https://wiki.openstack.org/wiki/IRC which says to come here and ask :) | 14:34 |
| anteaya | akijak: thank you for pointing that out | 14:35 |
| *** sdake has quit IRC | 14:35 | |
| *** pt_15 has joined #openstack-infra | 14:36 | |
| cloudnull | pabelanger: do you have a reference to how shade running the instance create + FIP assosiate + cleanup? maybe i can try and recreate things from that persepective because I'm not able to using the API or via heat template. | 14:36 |
| *** links has joined #openstack-infra | 14:37 | |
| *** jamesmca_ has quit IRC | 14:37 | |
| *** hashar has joined #openstack-infra | 14:38 | |
| *** esker has quit IRC | 14:38 | |
| akijak | anteaya: so the channel (#openstack-pl) already exists but it's not mentioned on https://wiki.openstack.org/wiki/IRC. how to add it to the wiki? | 14:39 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Import ldappool into gerrit and setup project https://review.openstack.org/315267 | 14:39 |
| anteaya | akijak: if you go to the wiki in the top right hand corner thaere is a link to log in | 14:39 |
| nibalizer | go squ | 14:39 |
| anteaya | akijak: what happens when you tryt to log in? | 14:40 |
| nibalizer | haha ohai | 14:40 |
| *** aratus has joined #openstack-infra | 14:40 | |
| anteaya | akijak: also the channel is not registered | 14:40 |
| anteaya | akijak: I suggest you ask the other inhabitant to leave the channel so that you may register it | 14:41 |
| anteaya | or ask them to register the name | 14:41 |
| anteaya | akijak: it might be possible the person created the channel when they saw your earlier question about the channel | 14:42 |
| akijak | anteaya: I get 500 after login | 14:42 |
| anteaya | akijak: I highly recommend registering the channel prior to adding it to the wiki | 14:42 |
| akijak | anteaya: ok, thanks. i'll try to contact him. | 14:42 |
| anteaya | akijak: okay that means you haven't edited our wiki before and you can't edit it now, as we have disabled new account creation due to wiki spam | 14:43 |
| *** yamamoto_ has quit IRC | 14:43 | |
| anteaya | akijak: you work on getting the channel registered | 14:43 |
| anteaya | akijak: if you can get it registered I can add it to the wikipage | 14:43 |
| akijak | anteaya: ok, understood. thank you | 14:44 |
| anteaya | akijak: thank you | 14:44 |
| anteaya | akijak: if you can't contact them or get them to leave come back after about 24 hours of trying and we will see what we can do | 14:44 |
| akijak | anteaya: ok | 14:44 |
| anteaya | thank you | 14:44 |
| clarkb | cloudnull: I believe that shade creates the fip and provides the server port to attach to at the same time. This in theory makes create and attach an atomic operation from api perspective | 14:45 |
| *** psilvad has quit IRC | 14:45 | |
| pabelanger | cloudnull: It looks like we are calling create_server from shade, then wait_for_server. Obviously will all cloud bits for OSIC. Wonder if we could write a quick script to reproduce it | 14:45 |
| pabelanger | clarkb: ah, good to know | 14:46 |
| cloudnull | ++ im looking at that now too. if you have something available already I can pick at i can go from there. | 14:46 |
| pabelanger | seems like we can always get about 11/12 FIPs to attach at once given time. The rest, fail. http://grafana.openstack.org/dashboard/db/nodepool-osic | 14:48 |
| *** jamesmca_ has joined #openstack-infra | 14:48 | |
| anteaya | fungi: happy company | 14:50 |
| *** tonytan4ever has joined #openstack-infra | 14:51 | |
| morgan | clarkb: that is the idea, shade is intended to create/attach fip in one call afair | 14:52 |
| mordred | yes | 14:52 |
| mordred | I have a script - one sec | 14:52 |
| morgan | pabelanger: ick | 14:52 |
| *** jlanoux has quit IRC | 14:53 | |
| mordred | cloudnull, clarkb, morgan: http://paste.openstack.org/show/496923 | 14:54 |
| mordred | that's how we create servers | 14:54 |
| * morgan nods. | 14:54 | |
| pabelanger | mordred: nice, thanks | 14:54 |
| cloudnull | Im monitoring the logs in vivo right now and doing the a similar op w/ heat using a node count of 25 https://gist.github.com/cloudnull/36c92e0da5e61b13510560ae15227453#file-usage-txt and its complementing the op in <2 min https://snag.gy/QYdnAp.jpg | 14:55 |
| cloudnull | mordred: testing that now | 14:55 |
| mordred | cloudnull: lemme go look at heat source and see how they're doing server_floating_ip | 14:56 |
| mordred | looks to be about the same | 14:58 |
| *** zeih has quit IRC | 15:00 | |
| mwhahaha | any chance on getting the ci box back on gerrit today? i noticed http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-05-11.log.html#t2016-05-11T15:41:59 from the logs yesterday | 15:01 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/315499 | 15:01 |
| *** esker has joined #openstack-infra | 15:01 | |
| *** mhickey has quit IRC | 15:01 | |
| mordred | mwhahaha: I have now run that command | 15:02 |
| *** nadya has joined #openstack-infra | 15:02 | |
| *** thorst_ has joined #openstack-infra | 15:02 | |
| mwhahaha | thanks | 15:02 |
| *** esker has quit IRC | 15:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add EPEL mirror https://review.openstack.org/315620 | 15:02 |
| *** Liuqing has quit IRC | 15:03 | |
| *** esker has joined #openstack-infra | 15:04 | |
| *** ddieterly is now known as ddieterly[away] | 15:05 | |
| *** ddieterly[away] is now known as ddieterly | 15:05 | |
| *** ddieterly is now known as ddieterly[away] | 15:05 | |
| *** ddieterly[away] is now known as ddieterly | 15:06 | |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/bindep: Add support for openSUSE and SLES https://review.openstack.org/315575 | 15:06 |
| mordred | pabelanger: I'm going to start calling you mr. mirror creator man person | 15:06 |
| pabelanger | mordred: love me some AFS | 15:07 |
| *** mixos has joined #openstack-infra | 15:09 | |
| anteaya | mordred: that's a lot to type | 15:11 |
| anteaya | mr. silver lining | 15:11 |
| mordred | anteaya: tab complete also doesn't work very well for it | 15:14 |
| *** r-mibu has quit IRC | 15:14 | |
| anteaya | mordred: true | 15:14 |
| *** r-mibu has joined #openstack-infra | 15:14 | |
| *** mixos has quit IRC | 15:14 | |
| *** mhickey has joined #openstack-infra | 15:15 | |
| *** yamahata has joined #openstack-infra | 15:16 | |
| *** kmartin has joined #openstack-infra | 15:17 | |
| *** sdake has joined #openstack-infra | 15:17 | |
| *** d0ugal has joined #openstack-infra | 15:17 | |
| *** aratus has quit IRC | 15:20 | |
| *** aratus has joined #openstack-infra | 15:20 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Revert "Actually use the service domain in V3 jobs" https://review.openstack.org/315631 | 15:21 |
| sdague | can I get a quick review of ^^^ | 15:22 |
| *** vhosakot has joined #openstack-infra | 15:22 | |
| *** esikachev has quit IRC | 15:22 | |
| sdague | there is a devstack fix out there, but everything is so backed up it would be better to just revert, let the fix land, then reapply | 15:22 |
| *** harlowja_at_home has joined #openstack-infra | 15:22 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Clean floating IPs on OSIC and BlueBox https://review.openstack.org/315632 | 15:23 |
| mordred | sdague: looking | 15:23 |
| mordred | sdague: lgtm | 15:23 |
| sdague | mordred: thanks | 15:24 |
| *** jlanoux has joined #openstack-infra | 15:24 | |
| *** |-paul-| has joined #openstack-infra | 15:24 | |
| mordred | pabelanger: when we restart nodepool today to pick up the fip cleaning patch, we should also land https://review.openstack.org/315632 | 15:24 |
| mordred | pabelanger: so that we use it :) | 15:24 |
| sdague | there are some devstack neutron fixes in gate (outside the sliding window) that would be good to let land. Hopefully the project config gets applied before they get out of the window | 15:25 |
| *** aratus has left #openstack-infra | 15:25 | |
| openstackgerrit | Merged openstack-infra/nodepool: Cleanup leaked floating ips periodically https://review.openstack.org/307755 | 15:25 |
| clarkb | mordred: I would really like to have owrking fips in bluebox and osic before we do that | 15:25 |
| clarkb | mordred: reduce the total number of variables at play | 15:25 |
| mordred | clarkb: k | 15:27 |
| nibalizer | so many leaked ips in osic | 15:27 |
| nibalizer | and a ton of instances without ip addresses | 15:27 |
| mordred | clarkb: is there something else we think is breaking now other than the delete code not deleting them? | 15:27 |
| * mordred feels he may be missing something | 15:27 | |
| clarkb | mordred: yes, the attach isn't attaching which leads to the need for a delete which then fails | 15:27 |
| *** asilenkov_ has quit IRC | 15:27 | |
| clarkb | mordred: ideally we would figure out hwy it takes an hour for that fip attach to eventually time out | 15:28 |
| nibalizer | clarkb: should i go ahead and clean up the leaks? | 15:28 |
| mordred | oh! I was not aware it was taking an hour | 15:28 |
| clarkb | mordred: if you look at the server age they basically hang around for an hour then go away | 15:28 |
| mordred | clarkb: is that taking an hour on our side? | 15:28 |
| clarkb | at least that is what it looked like on the small sample I grepped logs on | 15:28 |
| pabelanger | mordred: sure | 15:29 |
| mordred | clarkb: sorry for being dense - but could you say that again with less pronouns? | 15:29 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Remove voting gate v3 only job https://review.openstack.org/315636 | 15:29 |
| pabelanger | just read clarkb response | 15:29 |
| mtreinish | anteaya, yolanda, fungi: ^^^ if you get a sec | 15:30 |
| mordred | pabelanger, clarkb: also, I think we can still restart nodepool to pick up the patch and just leave it unenabled in the project-config | 15:30 |
| clarkb | mordred: nodepool boots an instance in osic, creates/attaches fip to it, this create/attach times out after an hour, this timeout triggers the delete fip code which fails, at this point we have leaked the fip | 15:30 |
| mordred | clarkb: awesome. thank you | 15:30 |
| *** ningy__ has joined #openstack-infra | 15:31 | |
| clarkb | mordred: my suggestion is that we don't worry about the auto leaked fip cleanup yet because it won't address the root issue which is fip create/attach time outs | 15:31 |
| *** nadya has quit IRC | 15:31 | |
| clarkb | mordred: that would also be fine | 15:32 |
| clarkb | mordred: where that == leave it disabled for now | 15:32 |
| pabelanger | ya, it takes exactly 60mins for the FIP timeout to happen. What are the downsides to exposing that to nodepool and lowering it? | 15:32 |
| *** links has quit IRC | 15:32 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Revert "Changing gate on devstack identity v3 only voting" https://review.openstack.org/315638 | 15:32 |
| clarkb | pabelanger: I don't think lowering it is the right behavior either, we need to figure out why a floating ip attach owuld take that long and not succeed | 15:32 |
| mordred | clarkb: well, we're passing in a timeout of 3600 | 15:32 |
| mordred | to wait for server | 15:33 |
| clarkb | nodepool passes that in? I thought defeault boot timeout was something around 300 seconds | 15:33 |
| mordred | def waitForServer(self, server, timeout=3600): | 15:33 |
| mordred | it's possible we should change that | 15:33 |
| clarkb | in any case the actual timeout value isn't super important, the why is there such a high rate of timeouts is what we need to determine | 15:33 |
| pabelanger | clarkb: agreed, was just suggesting maybe a more conservative moving forward | 15:33 |
| mordred | yah. agree | 15:33 |
| anteaya | mtreinish: I want to see how a gate only job got added as well as how a job got added to devstack and tempest without qa knowing about it | 15:34 |
| *** armax has joined #openstack-infra | 15:34 | |
| mordred | clarkb: so - I don't think there is an actual mystery | 15:34 |
| mordred | the floating ip attach is just not working | 15:35 |
| mordred | it's broken | 15:35 |
| mordred | but we've told nodepool to wait for an hour | 15:35 |
| openstackgerrit | Yaroslav Lobankov proposed openstack-infra/project-config: [Rally] Fixing sahara installation for `rally verify` jobs https://review.openstack.org/315640 | 15:35 |
| mordred | to see if it'll work | 15:35 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Revert "Changing gate on devstack identity v3 only voting" https://review.openstack.org/315638 | 15:35 |
| clarkb | mordred: but it only just recently started to break | 15:35 |
| mordred | so I'm pretty sure the underlying problem is "sometimes floating ips don't attach" | 15:35 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin https://review.openstack.org/312271 | 15:35 |
| clarkb | mordred: like this worked fine for a couple months | 15:35 |
| mtreinish | anteaya: I repushed it as a revert: https://review.openstack.org/#/c/315638/ | 15:35 |
| mtreinish | so the history is there | 15:35 |
| anteaya | mtreinish: thanks that helps | 15:35 |
| *** dmk0202 has quit IRC | 15:35 | |
| mtreinish | anteaya: it looks like sdague -1'd it and then it was approved on the next rev (without sdague's vote) | 15:36 |
| openstackgerrit | Yaroslav Lobankov proposed openstack-infra/project-config: [Rally] Fixing sahara installation for `rally verify` jobs https://review.openstack.org/315640 | 15:36 |
| *** dmk0202 has joined #openstack-infra | 15:36 | |
| mordred | clarkb: we only recently added the code to do a wait loop looking for the active state on the floating ips | 15:36 |
| *** e0ne has quit IRC | 15:36 | |
| mordred | so the hour-wait is a result of that change | 15:37 |
| clarkb | mordred: so previously we would fail on the ssh timeout? | 15:37 |
| anteaya | mtreinish: yeah I see that | 15:37 |
| clarkb | which is shorter? | 15:37 |
| mordred | clarkb: yah | 15:37 |
| Shrews | mordred: i think we weren't passing the timeout down to the attach before, iirc | 15:37 |
| mordred | or we'd fail on the server not having a FIP in its addresses record actually | 15:37 |
| *** arxcruz has quit IRC | 15:37 | |
| *** mikelk has quit IRC | 15:37 | |
| mordred | Shrews: yah. that's right | 15:37 |
| anteaya | yolanda jhesketh can you share your thoughts on why 315638 was +2'd and +A'd without further consultation with sdague who was -1 on the patch? | 15:37 |
| clarkb | mordred: ok, I think the correct value to pass into wait for server would be the boot timeout in the config or some derivative of that number | 15:37 |
| mordred | clarkb: I agree | 15:38 |
| mordred | clarkb: oh - we do pass in self.launch_timeout | 15:38 |
| anteaya | mtreinish: I'm +2 on your revert | 15:39 |
| * mordred needs to go board plane - will look again once he's on board | 15:39 | |
| anteaya | mordred: happy flying | 15:39 |
| *** psachin has quit IRC | 15:39 | |
| mtreinish | anteaya: thanks | 15:40 |
| anteaya | mtreinish: thanks for posting it as a revert | 15:40 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add virtualbmc from source non-voting test https://review.openstack.org/315646 | 15:40 |
| pabelanger | clarkb: a follow up patch to nodepool-builder from yesterday: https://review.openstack.org/#/c/315304/ Making sure our worker args are integers | 15:40 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update Plot plugin https://review.openstack.org/315647 | 15:40 |
| *** deadnull_ is now known as _deadnull | 15:41 | |
| pabelanger | clarkb: since puppet is still disabled, I manually applied the patch and restarted nodepool-builder last night | 15:41 |
| clarkb | ok will review | 15:41 |
| sdake | pabelanger there may be a problem with the centos bindep images - possibly related to selinux | 15:41 |
| *** mtanino has joined #openstack-infra | 15:41 | |
| sdake | pabelanger we are capturing the audit log now, and i'll let you know | 15:41 |
| sdake | if there is something wrong with them, could you fix it? | 15:41 |
| sdake | if we identify somethign labeled wrong? | 15:41 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update Plot plugin https://review.openstack.org/315647 | 15:42 |
| sdake | rabbitmq consistently crashess in the gate | 15:42 |
| clarkb | pabelanger: todays uploads will run with 12 threads then? | 15:42 |
| pabelanger | sdake: send a log once you have it. We landed a patch to DIB to fix selinux security context a while back | 15:43 |
| pabelanger | clarkb: yup, it should | 15:43 |
| pabelanger | clarkb: I tested it last night with centos-7 | 15:43 |
| *** madhu_ak has joined #openstack-infra | 15:43 | |
| clarkb | pabelanger: sdake and rabbitmq is installed by the jobs which should confoigure selinux at job run time not at image build | 15:43 |
| sdake | clarkb we install rabbitmq in a container | 15:44 |
| sdake | clarkb i dont think this affects other gate jobs outside of kolla | 15:44 |
| ningy__ | dhellmann: could you take a look at this patch https://review.openstack.org/#/c/251674/? | 15:44 |
| *** hashar has quit IRC | 15:44 | |
| ningy__ | or, anyone could take a look at https://review.openstack.org/#/c/251674/? it's blocking rally jenkins ci job | 15:45 |
| pabelanger | clarkb: I would think that too | 15:45 |
| *** andreas_s has quit IRC | 15:46 | |
| ningy__ | it blocks rally monasca gate job | 15:46 |
| ningy__ | https://review.openstack.org/#/c/251674/ | 15:46 |
| ningy__ | I can't get rally monasca gate job pass without this one merged | 15:46 |
| pabelanger | clarkb: uploads not as fast as last night, since we are iterating over providers. Currently uploading to osic-cloud1, ovh-bhs1 and ovh-gra1 | 15:46 |
| morgan | wq! | 15:46 |
| clarkb | ningy__: there is a push to treat the requirements as its own project team and they have a new irc channel now #openstack-requirements | 15:47 |
| clarkb | ningy__: probably best to start there as you should get attention from the people workong on that stuff | 15:47 |
| ningy__ | ok. thanks clarkb | 15:47 |
| *** tphummel has joined #openstack-infra | 15:47 | |
| *** matrohon has quit IRC | 15:49 | |
| *** yaume has quit IRC | 15:50 | |
| *** vincentll has quit IRC | 15:51 | |
| *** harlowja_at_home has quit IRC | 15:52 | |
| *** timl0101 has joined #openstack-infra | 15:53 | |
| *** claudiub has quit IRC | 15:53 | |
| *** Swami has joined #openstack-infra | 15:53 | |
| *** madhu_ak has quit IRC | 15:54 | |
| *** |-paul-| has quit IRC | 15:56 | |
| *** kmartin has quit IRC | 15:57 | |
| *** MarkAtwood has joined #openstack-infra | 15:57 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new repo for fuel-plugin-datera-cinder https://review.openstack.org/315651 | 15:58 |
| *** pfallenop has quit IRC | 15:58 | |
| *** ffrank has joined #openstack-infra | 15:59 | |
| *** jcoufal has quit IRC | 16:00 | |
| *** ddieterly is now known as ddieterly[away] | 16:00 | |
| fungi | dstufft: yeah, i doubt maven, npm, goget, et cetera pay any attention to gpg either. the only place i know we would end up being able to leverage it is for deb repos we publish (and then it would be signatures on the indices not the packages since that's what secure apt pays attention to anyway) | 16:01 |
| *** doug-fis_ has joined #openstack-infra | 16:01 | |
| jeblair | dstufft: ianw updated my bandersnatch PR as requested by christian here: https://bitbucket.org/pypa/bandersnatch/pull-requests/21/add-option-to-dir-hash-index-files/diff | 16:03 |
| *** ddieterly[away] is now known as ddieterly | 16:03 | |
| jeblair | (i abandoned (declined?) my own pr in favor of that) | 16:03 |
| *** jtomasek has quit IRC | 16:03 | |
| *** doug-fi__ has joined #openstack-infra | 16:03 | |
| *** rossella_s has quit IRC | 16:04 | |
| *** pfallenop has joined #openstack-infra | 16:04 | |
| *** doug-fish has quit IRC | 16:05 | |
| *** AJaeger has joined #openstack-infra | 16:06 | |
| *** doug-fis_ has quit IRC | 16:06 | |
| *** jaosorior has quit IRC | 16:06 | |
| *** doug-fi__ has quit IRC | 16:08 | |
| *** ri0 has joined #openstack-infra | 16:13 | |
| *** sdake has quit IRC | 16:13 | |
| * fungi suddenly remenbers that there is no rebase in hg | 16:14 | |
| *** mkoderer__ has quit IRC | 16:14 | |
| openstackgerrit | David Lyle proposed openstack/requirements: Remove XStatic-QUnit https://review.openstack.org/313795 | 16:14 |
| *** dmk0202 has quit IRC | 16:15 | |
| anteaya | fungi: :/ | 16:15 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Changing gate on devstack identity v3 only voting" https://review.openstack.org/315638 | 16:16 |
| anteaya | thanks AJaeger | 16:16 |
| anteaya | mtreinish: ^^ | 16:16 |
| *** kien-ha has quit IRC | 16:17 | |
| fungi | oh, heh, now i see 315636 was abandoned | 16:17 |
| clarkb | fungi: there is sort of but I never get it to work right (hg rebase is a plugin) | 16:17 |
| *** apoorvad has joined #openstack-infra | 16:19 | |
| *** akaszuba has quit IRC | 16:19 | |
| fungi | weird that https://review.openstack.org/292894 got approved when the only qa reviewer to provide feedback left a -1 on it | 16:19 |
| pabelanger | clarkb: jeblair: we appear to be pushing 400Mbps uploads from nodepool.o.o ATM. Do you know what the limit of our pipe is? | 16:20 |
| fungi | especially as it was reverted for entirely the same reason sdague pointed out in his review | 16:20 |
| fungi | raildo: ^ what was the longer term plan there? | 16:20 |
| *** d0ugal has quit IRC | 16:20 | |
| clarkb | pabelanger: I want to say its 800mbps total, but would have to check the rax flavor details | 16:21 |
| *** d0ugal has joined #openstack-infra | 16:21 | |
| raildo | fungi: we had this patch nv for a while on tempest and devstack ( almost one year) and looking the job history, we thought it could start be voting, after that, I start to add this same job in every service | 16:22 |
| *** andymaier has joined #openstack-infra | 16:22 | |
| pabelanger | clarkb: that gives me something to work with | 16:23 |
| raildo | fungi: for now, I'm adding it as non-voting | 16:23 |
| cloudnull | clarkb mordred pabelanger: so i can recreate the FIP timeout issue using shade but not w/ other methods and it seems it only happens when I multi-thread the jobs. | 16:23 |
| morgan | AJaeger: i expect to have the last bits needed for ldappool import ready today. :) | 16:23 |
| morgan | AJaeger: thanks for the quick review yesterday | 16:23 |
| fungi | raildo: yeah, it needs to be running on other services changes too, and passing, then switched to voting on all of them at the same time if that's the desire | 16:23 |
| cloudnull | doing it in serial doesn't seem to cause the issue | 16:23 |
| fungi | raildo: but having it voting only on qa repo changes is just going to create deadlocks | 16:24 |
| zigo | fungi: The build now always fails, and I have no explanation: https://review.openstack.org/#/c/315603/ | 16:24 |
| zigo | What am I missing? | 16:24 |
| raildo | fungi: got it... sorry about that, I'll take a note on it | 16:24 |
| raildo | fungi: thanks for the help | 16:24 |
| clarkb | cloudnull: huh, we serialize the api calls to handle rate limiting sanely but we do run a different shade cloud object for each cloud region in different threads | 16:24 |
| cloudnull | ive been trying this https://gist.github.com/cloudnull/2f3931e51062dc7b325faf9a24974006 | 16:24 |
| zigo | It's looking like it never can fetch packages... | 16:24 |
| cloudnull | when i do it in a serial loop it seems to work fine | 16:25 |
| fungi | raildo: also, our project-config reviewers shouldn't have approved the change without positive feedback from qa representatives (qa ptl or qa-infra liaison) | 16:25 |
| cloudnull | then again im only testing w/ 25 nodes | 16:25 |
| *** xiangxinyong has quit IRC | 16:25 | |
| morgan | AJaeger: ok ldappool should now pass pep8 and py27 tests and we should good for importing. | 16:25 |
| fungi | did we recently update the hideci.js again to try and solve the requirements failure message handling? looks like the results table is broken again | 16:26 |
| cloudnull | I also noticed that in some cases I've ended up with duplicate instances. | 16:26 |
| clarkb | fungi: yes mordred indicated he touched the file earlier today as per your directions yesterday | 16:26 |
| mordred | fungi: I did the touch earlier | 16:26 |
| *** x00350071 has joined #openstack-infra | 16:26 | |
| cloudnull | well instance names that is . | 16:26 |
| fungi | clarkb: mordred: oh, so the change we merged yesterday hadn't taken effect? anyone have a revert going yet? | 16:26 |
| cloudnull | like in my last round i had two nodes w/ the name "ADMIN_TEST_NODE-bea07735-63f2-4b26-9052-d2682733f9c6" | 16:26 |
| *** tellesnobrega_af is now known as tellesnobrega | 16:27 | |
| raildo | fungi: right now, AJaeger is help me with those new patches, do you recommend someone else that I can ask for reviews? | 16:27 |
| *** d0ugal has quit IRC | 16:27 | |
| mordred | cloudnull: lovely | 16:27 |
| * zigo tried to add an apt-get update | 16:27 | |
| clarkb | fungi: no revert that I have seen | 16:27 |
| mordred | cloudnull: that's against a devstack? | 16:27 |
| fungi | clarkb: mordred: in the past we saw hideci.js cease getting bundled entirely ifg it updated without updating the timestamp on the site header, so this (continuing working but not updating) is new behavior in 2.11 i guess | 16:27 |
| *** baumann1 has quit IRC | 16:27 | |
| cloudnull | and idk if thats my cloud problem or if shade is retrying an op and creating a new node | 16:27 |
| *** nadya has joined #openstack-infra | 16:27 | |
| cloudnull | mordred: no thats aginst the osic | 16:28 |
| mriedem | is it just me or are all CI comments toggled by default now? | 16:28 |
| fungi | raildo: sdague reviewed (and left a -1 on) the original patch which was subsequently reverted by mtreinish, so either of them would be good candidates | 16:28 |
| *** mkoderer__ has joined #openstack-infra | 16:28 | |
| mordred | cloudnull: kk. admin/SuperSecrete threw me :) | 16:28 |
| fungi | mriedem: it's broken, we're in the process of reverting | 16:28 |
| mriedem | fungi: sweet | 16:28 |
| mriedem | thanks | 16:28 |
| anteaya | fungi raildo also voting jobs are in both check and gate, not just on gate | 16:28 |
| *** bpokorny has joined #openstack-infra | 16:28 | |
| mordred | Shrews: ^^ cloudnull has a script that reproduces the problem | 16:28 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Actually use the service domain in V3 jobs" https://review.openstack.org/315631 | 16:29 |
| raildo | anteaya: yeap, I learned this in the wrong way :( | 16:29 |
| anteaya | fungi raildo the patch had voting jobs only on the gate | 16:29 |
| anteaya | raildo fungi well as long as you learned it | 16:29 |
| anteaya | raildo fungi the hard way can also be effective | 16:29 |
| cloudnull | this is the stack i see http://cdn.pasteraw.com/dvfmr9i3gpq9j83rbn39iji6u0ui21u | 16:29 |
| raildo | anteaya: ++ | 16:29 |
| pabelanger | zigo: Ya, refreshing apt caches is a good start. never know how older they'll be when we launch a new node | 16:29 |
| morgan | anteaya: if you have a moment https://review.openstack.org/#/c/315267/ your eyes (to make sure I didn't do something silly) would be welcome | 16:30 |
| mordred | cloudnull: yah. that's the error we're getting too | 16:30 |
| zigo | pabelanger: Thanks for your review btw! :) | 16:30 |
| anteaya | raildo: great thank you | 16:30 |
| anteaya | morgan: I will look | 16:31 |
| morgan | anteaya: much appreciated | 16:31 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "Allow 3rd party CI's to have same pipeline names" https://review.openstack.org/315662 | 16:31 |
| *** asettle has quit IRC | 16:31 | |
| zigo | mordred: I'm trying to get the publish job to run with a new patch (since the publish macro got merged), then we'll have to work on the pick-up job. Will you be able to contribute it? | 16:31 |
| raildo | anteaya: fungi thank you guys :) | 16:31 |
| fungi | clarkb: mordred: sdague: ^ | 16:31 |
| mordred | zigo: yup! | 16:31 |
| anteaya | raildo: you are welcome and I'm not a guy | 16:31 |
| raildo | anteaya: ops... sorry | 16:32 |
| anteaya | morgan: is this the lib that has the question up to legal discuss | 16:32 |
| *** bpokorny has quit IRC | 16:32 | |
| anteaya | raildo: if you care, I like folks people team group infra and occasionally cats | 16:32 |
| anteaya | if you use cats do so sparingly | 16:32 |
| morgan | anteaya: the only legal concern was that someone was moving it in-tree of keystone and causing licensing issues. we're adopting it so no conflict now | 16:32 |
| * mordred hands anteaya a cat | 16:32 | |
| *** bpokorny has joined #openstack-infra | 16:32 | |
| clarkb | mordred: cloudnull setting auto_ip to False results in floating ip creation with shade? | 16:33 |
| morgan | anteaya: the previous maintainer is transferring to me/openstack-keystone team to forward, as it is a dead project otherwise | 16:33 |
| anteaya | mordred: thank you, my cat is hunting again at night, warning to all mice in the neighbourhood | 16:33 |
| morgan | anteaya: we will need to keep MPL license though (meh, could be worse) | 16:33 |
| raildo | anteaya: I'll start to use cats :) | 16:33 |
| mordred | anteaya: mine has been bringing us half-eaten lizards | 16:33 |
| anteaya | morgan: ah wonderful | 16:33 |
| clarkb | oh setting the pool is like setting auto ip | 16:33 |
| clarkb | but more explicit | 16:33 |
| anteaya | raildo: ha ha ha | 16:33 |
| pabelanger | clarkb: is the reason we upload raw images to ovh because of a performance issue with qcow2 in the cloud? | 16:33 |
| clarkb | pabelanger: I think it is because they don't accept qcow2 | 16:34 |
| anteaya | mordred: nice | 16:34 |
| morgan | anteaya: yeah he was super happy his code was helping someone. | 16:34 |
| anteaya | mordred: can you keep the lizard skin? | 16:34 |
| *** jlanoux has quit IRC | 16:34 | |
| * anteaya has several tails from red squirrels in the freezer | 16:34 | |
| anteaya | morgan: oh awesome | 16:34 |
| pabelanger | clarkb: okay, that make sense | 16:35 |
| clarkb | pabelanger: it is probably worth double checking that assumption though | 16:35 |
| *** slaweq has joined #openstack-infra | 16:35 | |
| mordred | agree | 16:35 |
| *** sridhar_ram has joined #openstack-infra | 16:36 | |
| sdague | fungi: gah, ok. | 16:37 |
| pabelanger | clarkb: sure, I can ask | 16:37 |
| clarkb | pabelanger: or just try an upload of qcow2 and boot it :) | 16:38 |
| *** slaweq has quit IRC | 16:38 | |
| mordred | cloudnull: looking at your code, since you're using multiprocess it's not an internal shade needs-a-mutex issue or anything ... this makes me wonder if the issue is that neutron gets screwed up somewhere with a certain level of parallel actions | 16:38 |
| anteaya | morgan: did you mean to +1 your own patch? | 16:38 |
| mordred | s/neutron/some combination of nova and neutron/ | 16:38 |
| morgan | anteaya: eh no, it just happened when i commented the upstream was ready for import | 16:38 |
| mordred | the instance name thing is weird too | 16:38 |
| clarkb | mordred: cloudnull or if a particular sequence of actions is problematic | 16:38 |
| *** slaweq has joined #openstack-infra | 16:39 | |
| morgan | anteaya: it doesn't really matter, unfortunately on plane wifi so hard to load gerrit. | 16:39 |
| mordred | clarkb: yah - like a sequence that happens in some combinatoin due to the parallel actions | 16:39 |
| openstackgerrit | Carl Baldwin proposed openstack-infra/irc-meetings: Moving the Neutron L3 meeting agenda to etherpad https://review.openstack.org/315664 | 16:39 |
| clarkb | mordred: ya jus thinking that nodepool won't do parallel actions against a single cloud region | 16:39 |
| mordred | thread one does one step then thread two does another one or something | 16:39 |
| mordred | righ | 16:39 |
| mordred | but | 16:39 |
| sdague | fungi: do you have an instance where things were broken by this? | 16:39 |
| sdague | to see if I can find a quick hotfix? | 16:39 |
| anteaya | morgan: yup, figured it was something like that, just looked unusual to me for you | 16:39 |
| *** baumann has joined #openstack-infra | 16:39 | |
| mordred | clarkb: if it's like "thread one create, thread two get, thread one get, thread three delete" | 16:40 |
| mordred | we won't do more than one API call at a time | 16:40 |
| mordred | but we can have multiple actions in flight at the cloud that we're polling on status for | 16:40 |
| openstackgerrit | Alistair Coles proposed openstack-infra/project-config: Make gate-swift-tox-func-in-process-fast-post voting https://review.openstack.org/315665 | 16:40 |
| clarkb | ya | 16:40 |
| fungi | sdague: sure, https://review.openstack.org/315304 for example | 16:40 |
| mordred | this is gonna be FUN | 16:40 |
| clarkb | we can interleave the api calls for multiple "tasks" | 16:40 |
| mordred | yah | 16:40 |
| fungi | sdague: i'm seeing the results table entirely missing under firefox at least | 16:41 |
| AJaeger | mordred: so, for your ldappool repo: That is a complete independent repo, correct? Not part of Big Tent via keystone? | 16:41 |
| sdague | fungi: actually, it looks like it might be an easy fix | 16:41 |
| stevemar | morgan: ^ | 16:41 |
| sdague | it's just a var rename miss | 16:41 |
| mordred | when I get airborne I'm going to try the cloudnull script with a ton of logging | 16:41 |
| fungi | sdague: oh, cool. i'm happy to iterate on that real quick if you can push it up | 16:41 |
| morgan | AJaeger: correct | 16:41 |
| sdague | yeh, give me 10 minutes to locally test this fix | 16:42 |
| *** d0ugal has joined #openstack-infra | 16:42 | |
| fungi | sdague: i'll hand-patch it in and confirm so we can get stuff working again | 16:42 |
| jeblair | clarkb, mordred: would it be helpful to add more logging of tasks (so we know exactly which tasks (ie, which ips/servers/etc) we are executing? | 16:42 |
| AJaeger | morgan: ok. Please make openstackci Package Index Owner... | 16:42 |
| *** esker has quit IRC | 16:42 | |
| morgan | AJaeger: the coreteam will be made up of -keystone (and likely will inherit openstack-keystone core) but it is not intended to be bigtent | 16:42 |
| clarkb | jeblair: maybe? its also possible that using cloudnull's reproduction case is a better place for that | 16:42 |
| pabelanger | clarkb: that will work too | 16:42 |
| AJaeger | morgan: as soon as pypi is fine, I can +2... | 16:43 |
| jeblair | clarkb: k, just throwing out ideas | 16:43 |
| morgan | AJaeger: maintainer is not enough? i'll up it to owner :) | 16:43 |
| clarkb | jeblair: just because it is so much simpler than nodepools mass of logging | 16:43 |
| AJaeger | morgan: As documented: Owner, please ;) | 16:43 |
| *** doug-fish has joined #openstack-infra | 16:43 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Puppetboard renamed the module https://review.openstack.org/313876 | 16:44 |
| morgan | AJaeger: done | 16:44 |
| *** andymaier has quit IRC | 16:44 | |
| *** signed8bit has joined #openstack-infra | 16:45 | |
| AJaeger | morgan: +2ed | 16:45 |
| *** jamesmcarthur has quit IRC | 16:45 | |
| *** Jeffrey4l has quit IRC | 16:45 | |
| morgan | AJaeger: woot. | 16:45 |
| morgan | AJaeger: are new projects processed on fridays still or just as folks get around to it? | 16:45 |
| AJaeger | morgan: as time permits... | 16:45 |
| anteaya | morgan: so http://lists.openstack.org/pipermail/legal-discuss/2016-May/000404.html didn't get any traffic on it | 16:45 |
| morgan | i only ask since i'm corrdinating with the former maintainer :) | 16:45 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: fix typo in hideci.js https://review.openstack.org/315668 | 16:46 |
| sdague | fungi: that should be it | 16:46 |
| anteaya | does anyone have any opinions on importing a project with an MPL licence? | 16:46 |
| anteaya | it is about to become an openstack repo? | 16:46 |
| morgan | anteaya: MPL is apache2 compat. | 16:46 |
| anteaya | morgan: awesome | 16:46 |
| morgan | atafaik | 16:46 |
| sdague | there is also a regression about jenkins no longer always being in the top of the list, let me see if I can sort out that one. | 16:46 |
| morgan | anteaya: the only real issue was moving itnto tree with keystone and causing odd licencing issues with keystone having MPL code in it's tree | 16:47 |
| anteaya | morgan: I know very little about licensing and since the thread on legal discuss got no traffic asking once more for objections | 16:47 |
| AJaeger | fungi, pabelanger: for the bindep changes: As I'm going now on vacation, I'm not continuing on the bindep changes. Feel free to move it forward as you see fit and change the plan as needed ;) | 16:47 |
| morgan | anteaya: this is not big tent, so it just uses the ci-infra and publishes to pypi | 16:47 |
| *** gyee has joined #openstack-infra | 16:47 | |
| anteaya | morgan: ah okay, so since this is it's own project, no licensing clashes with keystone now | 16:47 |
| morgan | stevemar: ^ can you update that leagal thread to indicate this is resolved ? | 16:47 |
| pabelanger | AJaeger: sure, great work driving this | 16:47 |
| anteaya | morgan: perfect thank you | 16:48 |
| fungi | sdague: giving it a shot now | 16:48 |
| *** doug-fish has quit IRC | 16:48 | |
| morgan | anteaya: exactly | 16:48 |
| *** jyuso has joined #openstack-infra | 16:48 | |
| fungi | sdague: i hand-patched it in and a force reload in my browser shows a previously broken display is working again | 16:49 |
| *** sdake has joined #openstack-infra | 16:49 | |
| fungi | sdague: i'll approve this and abandon my revert--thanks! | 16:49 |
| sdague | fungi: cool | 16:49 |
| *** alex_xu has quit IRC | 16:49 | |
| *** thorst_ has quit IRC | 16:50 | |
| *** heyongli has quit IRC | 16:50 | |
| *** eliqiao has quit IRC | 16:50 | |
| *** jyuso1 has quit IRC | 16:51 | |
| *** eliqiao has joined #openstack-infra | 16:51 | |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/314878 | 16:52 |
| *** heyongli has joined #openstack-infra | 16:52 | |
| *** woodster_ has joined #openstack-infra | 16:52 | |
| openstackgerrit | Merged openstack/requirements: Add global requirement update on python-brick-cinderclient-ext project https://review.openstack.org/309084 | 16:52 |
| *** alex_xu has joined #openstack-infra | 16:52 | |
| cloudnull | sorry in osa meeting. | 16:54 |
| *** javeriak has joined #openstack-infra | 16:54 | |
| cloudnull | mordred: it may be a parallel action problem and neutron becoming un-happy. | 16:54 |
| *** lucasagomes is now known as lucas-dinner | 16:54 | |
| *** sambetts is now known as sambetts|afk | 16:55 | |
| cloudnull | however I cant recreate it in heat. | 16:55 |
| *** matt-borland has quit IRC | 16:55 | |
| cloudnull | which may be due to some retry shinanigans they have to make that go | 16:55 |
| nibalizer | now i just touch 15 things a day and get no real progress done | 16:55 |
| *** thorst_ has joined #openstack-infra | 16:56 | |
| stevemar | morgan: anteaya i'll reply to my own post on legal :P | 16:56 |
| *** mkoderer__ has quit IRC | 16:57 | |
| cloudnull | also i running and rerunning tests i reliably get nodes with the same names and i cant see where thats coming from. | 16:57 |
| anteaya | stevemar: thank you, you seem to be the only one concerned with the topic, and I thank you for that! | 16:58 |
| fungi | nibalizer: sounds like my day! | 16:58 |
| stevemar | anteaya: :P | 16:58 |
| anteaya | stevemar: :) | 16:58 |
| anteaya | nibalizer: the infra piano | 16:59 |
| nibalizer | fungi: i think you get more done than me :) | 16:59 |
| fungi | tickle those ivories | 16:59 |
| nibalizer | that was acutally a ww for a differen conversation, but it applies here ofc | 16:59 |
| fungi | nibalizer: i hardly see how. so... much... interruption | 16:59 |
| * nibalizer looks at puppetdb01.openstack.org, weeps | 16:59 | |
| mordred | cloudnull: so I wonder ... in shade we're not considering the fip actually attached until it shows up in the nova server dict | 16:59 |
| mordred | cloudnull: but heat is tracking the fip directly, so probably does not care whether nova "knows" about the fip at all | 17:00 |
| mordred | clarkb, Shrews ^^ | 17:00 |
| * anteaya hands nibalizer some tissue and a cat | 17:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support ftp publisher https://review.openstack.org/309095 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: run shell builders defined in JJB https://review.openstack.org/308099 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support zuul stop commands https://review.openstack.org/309148 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: register jobs from JJB https://review.openstack.org/307515 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: send ZMQ start/complete events https://review.openstack.org/308053 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add zuul_runner module https://review.openstack.org/309221 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: run ansible https://review.openstack.org/308082 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support job timeouts https://review.openstack.org/308488 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support scp publisher https://review.openstack.org/308656 | 17:01 |
| *** thorst_ has quit IRC | 17:01 | |
| openstackgerrit | Carl Baldwin proposed openstack-infra/irc-meetings: Move the agenda for routed networks meeting to etherpad https://review.openstack.org/315674 | 17:01 |
| *** mhickey has quit IRC | 17:01 | |
| cloudnull | mordred: that may be | 17:02 |
| mordred | also - heat is not checking for the fip to go into an active state - it's just issuing the create call | 17:02 |
| *** abregman has joined #openstack-infra | 17:02 | |
| morgan | mordred: i think this is a case where shade needs to learn to track the fip =/ | 17:02 |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: Create stable order for CI results stable https://review.openstack.org/315677 | 17:02 |
| morgan | mordred: or pointing towards that | 17:02 |
| mordred | morgan: it's starting to sound like that | 17:02 |
| morgan | mordred: unfortunate :( | 17:03 |
| sdague | fungi: one more niceity there https://review.openstack.org/315677 | 17:03 |
| *** e0ne has joined #openstack-infra | 17:03 | |
| *** pvaneck has joined #openstack-infra | 17:03 | |
| *** thorst_ has joined #openstack-infra | 17:03 | |
| mordred | morgan: I will try a patch that does that inefficiently and see if the script still fails | 17:03 |
| morgan | mordred: ok. i will happily look over said code when you have it | 17:04 |
| *** mkoderer__ has joined #openstack-infra | 17:04 | |
| *** jerryz has joined #openstack-infra | 17:05 | |
| cloudnull | morgan mordred so maybe https://github.com/openstack-infra/shade/blob/master/shade/openstackcloud.py#L3625-L3628 needs to be adjusted to track the FIP instead of looking for the update in server info | 17:05 |
| fungi | sdague: is sort_by_name only sorting by the first character in each string, or is there some magic iteration hidden in the sort() method? | 17:05 |
| *** lakshmiS has joined #openstack-infra | 17:06 | |
| sdague | fungi: this is javascript | 17:06 |
| mordred | cloudnull: yah. thats' exactly that I'm thinking | 17:06 |
| morgan | cloudnull: that is the thought | 17:06 |
| sdague | that's not a string index | 17:06 |
| sdague | fungi: https://github.com/openstack-infra/system-config/blob/376582d98b2d3843fc33858aa3a15c35e4124ab0/modules/openstack_project/files/gerrit/hideci.js#L177 | 17:06 |
| sdague | there is a tuple | 17:06 |
| sdague | well, array | 17:06 |
| fungi | sdague: yeah, i should remember with js to check all assumptions at the door | 17:06 |
| sdague | we've built up 2 arrays, names + results, which then a bigger datastructure is created from | 17:07 |
| fungi | so nonfirstparty is an array of arrays. now it makes more sense | 17:07 |
| sdague | yep | 17:07 |
| sdague | [0] is the ci+pipeline name | 17:07 |
| sdague | [1] is the result list | 17:07 |
| sdague | as an array | 17:07 |
| *** thorst_ has quit IRC | 17:07 | |
| fungi | yep, lgtm | 17:07 |
| * fungi still gets cross-eyed looking at javascript... at least it's not java! | 17:08 | |
| sdague | heh | 17:08 |
| fungi | soon i can say "at least it's not go!" (except, of course, when it _is_ go i guess) | 17:08 |
| sdague | at least it's not haskell | 17:09 |
| * Clint squints. | 17:09 | |
| * fungi still codes with a lisp | 17:09 | |
| * clarkb imagines fungi would enjoy haskell, that math background would be helpful | 17:09 | |
| fungi | "haskell: at least it's not fortran!" | 17:10 |
| AJaeger | cobol ;) | 17:10 |
| fungi | oh yeah, there are definitely some unfortunate cobol experiences i'd like to forget | 17:10 |
| fungi | on sco osr, no less | 17:11 |
| AJaeger | at least it gave me some money at that time... | 17:11 |
| *** savihou has joined #openstack-infra | 17:11 | |
| fungi | at least i got to work with some awesome (for the day) 4-way smp i486 servers | 17:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add private_key_file configuration for launcher https://review.openstack.org/311639 | 17:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add usernname configuration option for launcher https://review.openstack.org/311725 | 17:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add streaming console support https://review.openstack.org/309501 | 17:12 |
| fungi | crammed full of 32-line serial terminal multiplexer cards | 17:12 |
| AJaeger | the good old days ;) | 17:13 |
| fungi | er, concentrator cards. the multiplexers were in separate boxes | 17:13 |
| *** davideagnello has joined #openstack-infra | 17:14 | |
| *** signed8bit is now known as signed8bit_Zzz | 17:16 | |
| pabelanger | clarkb: ssh 158.69.85.189 that is our qcow2 image in ovh-bhs1 | 17:16 |
| clarkb | pabelanger: neat | 17:17 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Send ansible-puppet timing information to graphite https://review.openstack.org/285740 | 17:17 |
| mordred | awesome! | 17:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch to qcow2 for OVH clouds https://review.openstack.org/315680 | 17:18 |
| pabelanger | clarkb: ^think just that | 17:18 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add ssl upgrade playbook https://review.openstack.org/312261 | 17:20 |
| clarkb | pabelanger: can you share an image show on that image? | 17:21 |
| greghaynes | ooo more qcow2 clouds | 17:21 |
| nibalizer | pabelanger: so i can't log in to that | 17:21 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add BHS1 region to openstackjenkins-ovh all-clouds https://review.openstack.org/315686 | 17:21 |
| clarkb | I just ran into ^ when trying to find the image myself | 17:22 |
| pabelanger | nibalizer: I just deleted it | 17:22 |
| pabelanger | let me launch another | 17:22 |
| openstackgerrit | Merged openstack-infra/nodepool: Make sure worker arguments are integers https://review.openstack.org/315304 | 17:22 |
| nibalizer | pabelanger: no need | 17:22 |
| *** doug-fish has joined #openstack-infra | 17:22 | |
| clarkb | pabelanger: what user and region is the image? I want to look at the details | 17:22 |
| nibalizer | just wondering if that was expected | 17:22 |
| pabelanger | clarkb: http://paste.openstack.org/show/496951/ | 17:22 |
| clarkb | that works too :) | 17:22 |
| clarkb | ok disk format is still qcow2 so they aren't doing a conversion or anything, cool | 17:23 |
| pabelanger | clarkb: it was ovh-bhs1 | 17:23 |
| *** doug-fish has quit IRC | 17:23 | |
| clarkb | but it has an rbd path | 17:23 |
| pabelanger | I've deleted it already, but can upload again | 17:23 |
| clarkb | I thought that qcow2 and ceph didn't play nice but it seems to work | 17:23 |
| *** doug-fish has joined #openstack-infra | 17:24 | |
| *** dtantsur|bbl is now known as dtantsur | 17:24 | |
| *** jistr has quit IRC | 17:24 | |
| *** abregman has quit IRC | 17:24 | |
| pabelanger | I didn't know that, will have to google to learn more | 17:24 |
| *** vkramskikh_ has quit IRC | 17:24 | |
| *** e0ne has quit IRC | 17:24 | |
| *** salv-orlando has quit IRC | 17:24 | |
| *** DevBox has joined #openstack-infra | 17:24 | |
| *** jamesmca_ has quit IRC | 17:25 | |
| clarkb | pabelanger: it has to do with copy on write in ceph with a shared image. Basically that image is shared by all 100 of our current test instances and only the bits unique to each get written | 17:25 |
| *** ddieterly is now known as ddieterly[away] | 17:25 | |
| clarkb | pabelanger: it is possible they just don't do that when we qcow2 | 17:25 |
| openstackgerrit | Merged openstack-infra/system-config: fix typo in hideci.js https://review.openstack.org/315668 | 17:26 |
| pabelanger | clarkb: okay, good to know | 17:26 |
| *** signed8bit_Zzz is now known as signed8bit | 17:26 | |
| *** tosky has quit IRC | 17:26 | |
| *** shardy has quit IRC | 17:27 | |
| clarkb | pabelanger: maybe we can get jean daniel to ack that change? | 17:28 |
| *** doug-fish has quit IRC | 17:28 | |
| pabelanger | clarkb: WFM | 17:28 |
| AJaeger | anteaya, sdague, mtreinish please see http://lists.openstack.org/pipermail/openstack-dev/2016-May/094829.html for raildo 's v3 identity plan... | 17:32 |
| *** IlyaG has joined #openstack-infra | 17:33 | |
| *** IlyaG has quit IRC | 17:34 | |
| *** d0ugal has quit IRC | 17:34 | |
| *** IlyaG has joined #openstack-infra | 17:34 | |
| *** julim has quit IRC | 17:35 | |
| *** rfolco has quit IRC | 17:35 | |
| raildo | thanks AJaeger, I already got some feedback from anteaya and fungi, but I appreciate any other feedback related to this :) | 17:35 |
| *** rfolco has joined #openstack-infra | 17:35 | |
| anteaya | nibalizer: what is the line in https://review.openstack.org/#/c/312124/2/manifests/site.pp that realizes Zara on the dev server? I'd like to learn so I can spot it next time, I looked at that patch and obviously missed it | 17:35 |
| *** gyee has quit IRC | 17:35 | |
| *** pt_15 has quit IRC | 17:36 | |
| anteaya | stevemar: thank you for replying to yourself on legal-discuss | 17:37 |
| *** ldnunes has quit IRC | 17:37 | |
| *** _deadnull is now known as deadnull_ | 17:37 | |
| clarkb | jesusaur: ok, when is a good time today to work on getting the logstash indexer rule stuff merged and applied to the workers? | 17:38 |
| *** ldnunes has joined #openstack-infra | 17:38 | |
| *** alex_xu has quit IRC | 17:38 | |
| clarkb | hrm /me notices we need to upgrade a puppet module https://review.openstack.org/#/c/311224/ | 17:40 |
| clarkb | pleia2: ^ the zanata manifests also use the archive module, any concern doing that upgrade from zanata's perspective? | 17:41 |
| AJaeger | raildo: I suggest to mark all your changes as WIP for now, and go forward with one or two and iterate on them before finalizing the others... | 17:41 |
| *** markvoelker has joined #openstack-infra | 17:41 | |
| *** Sukhdev has joined #openstack-infra | 17:42 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:42 | |
| raildo | AJaeger: I'm applying the suggestions on the nova patch, when I finish it, I'll mark all the other patches as WIP | 17:42 |
| AJaeger | ok | 17:42 |
| notmyname | clarkb: mordred: fungi: FYI the current 3rd party CI for the golang code in swift is using a Makefile (https://github.com/openstack/swift/blob/feature/hummingbird/go/Makefile) and simply runs `make test` in jenkins | 17:43 |
| notmyname | I added that info to the etherpad | 17:43 |
| mordred | notmyname: awesome. thakns | 17:43 |
| sdague | raildo: my primary concern is the stuff we think might cause breaks, like that project-config patch, we verify manually | 17:43 |
| fungi | notmyname: excellent! | 17:43 |
| sdague | a part of me longs for 'make test' in the python projects | 17:43 |
| mordred | sdague: clarkb has been advocating for that for ages | 17:44 |
| nibalizer | anteaya: there is no line | 17:44 |
| fungi | sdague: you and clarkb could collaborate on getting his make-based tox replacement going again | 17:44 |
| notmyname | sdague: `./configure && make && sudo make install` FTW ;-) | 17:44 |
| sdague | notmyname: ++ exactly | 17:44 |
| nibalizer | i gad hoped for the patch i -1'd to be parented on my patch | 17:44 |
| AJaeger | sdague: is https://review.openstack.org/#/c/304263/1/zuul/layout.yaml correct? I doubt that gate-tempest-dsvm-neutron-identity-v3-only-full-nv tests swift - can you enlighten me, please? | 17:44 |
| mordred | notmyname: ++ | 17:44 |
| *** e0ne has joined #openstack-infra | 17:44 | |
| nibalizer | im realizing now this was a secret plan i did not communicate, sorry | 17:45 |
| mordred | notmyname: you know I was doing a ton of autoconf/automake stuff in drizzle before we spun up this openstack thing | 17:45 |
| notmyname | mordred: cool. or brain-warping. I've heard all that is rather crazy ;-) | 17:45 |
| notmyname | mordred: m4 or something like that? | 17:45 |
| mordred | it is | 17:45 |
| clarkb | sdague: https://review.openstack.org/#/c/88698/ | 17:45 |
| mordred | notmyname: if you think pbr is bad, you should see the m4 version of it | 17:45 |
| fungi | m4 is what makes it _less_ brain-warping actually | 17:45 |
| notmyname | mordred: lol | 17:45 |
| *** ffrank has quit IRC | 17:46 | |
| fungi | you don't even want to look at the non-macro versions of that stuff | 17:46 |
| mordred | notmyname, fungi http://bazaar.launchpad.net/~mordred/pandora-build/devel/files | 17:47 |
| raildo | sdague: do you have any suggestion to improve the plan that I explained on the email? because this is my concern too, I don't want to send patches to cause breaks | 17:47 |
| notmyname | mordred: of course it's called "pandora". you don't want to open it | 17:47 |
| anteaya | nibalizer: that would be why I can't find the line, there also is no spoon | 17:47 |
| clarkb | mordred: you are bringing back memories of swig and python and sadness | 17:48 |
| anteaya | nibalizer: well you are communicating it now | 17:48 |
| mordred | http://bazaar.launchpad.net/~mordred/pandora-build/devel/view/head:/m4/pandora_vc_build.m4 is especially fun | 17:48 |
| anteaya | nibalizer: so that you for that | 17:48 |
| mordred | notmyname: :) | 17:48 |
| *** doug-fis_ has joined #openstack-infra | 17:48 | |
| jeblair | even supports git | 17:49 |
| *** mjblack has quit IRC | 17:49 | |
| mordred | yup | 17:49 |
| sdague | raildo: mostly, we just need to make sure the patches get some manual verification, or test patches against projects that might break | 17:49 |
| mordred | jeblair: although I think we've learned more git now than at the time we wrote that - so I think those git things could likely be done better | 17:50 |
| *** agentleweb has joined #openstack-infra | 17:50 | |
| openstackgerrit | Merged openstack-infra/project-config: Import ldappool into gerrit and setup project https://review.openstack.org/315267 | 17:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Batch calls to list_floating_ips https://review.openstack.org/315697 | 17:50 |
| anteaya | nibalizer: is there anything I need to do to help your secret plan to realize Zara to be realized? | 17:50 |
| mordred | anteaya: nicely phrased | 17:51 |
| anteaya | mordred: thanks | 17:51 |
| *** tosky has joined #openstack-infra | 17:51 | |
| anteaya | nibalizer: I did the best I could by submitted a patch I knew folks wouldn't like | 17:51 |
| clarkb | jesusaur: see comment on https://review.openstack.org/#/c/310046/1 re making sure our rules are up to date | 17:51 |
| openstackgerrit | Merged openstack/requirements: Add openstack/designate-tempest-plugin to projects list https://review.openstack.org/309127 | 17:51 |
| anteaya | I can't language anymore | 17:52 |
| anteaya | I try to language | 17:52 |
| anteaya | have tenses match | 17:52 |
| *** d0ugal has joined #openstack-infra | 17:52 | |
| anteaya | I fail consistently | 17:52 |
| JayF | Where are the mirror configs at for you all? I see a puppet-bandersnatch module in openstack-infra ord but can't find the repo that calls it | 17:52 |
| anteaya | I have no language anymore | 17:52 |
| JayF | helping mat128 do some research on adding Tiny Core Linux (tinyipa) mirrors in | 17:53 |
| anteaya | JayF: pabelanger is our resident many mirrors man person | 17:53 |
| *** amrith has quit IRC | 17:53 | |
| anteaya | JayF: he is a wealth of information | 17:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Also mirror centos 7 cloud repo https://review.openstack.org/315698 | 17:53 |
| JayF | pabelanger: is a good human being, and helpful and kind too :) | 17:53 |
| pabelanger | EmilienM: ^ | 17:53 |
| anteaya | mordred: wasn't that your name for him earlier today? | 17:53 |
| anteaya | JayF: indeed he is | 17:53 |
| raildo | sdague: ++ I'll be aware of this manual verification on the next patches | 17:54 |
| mordred | mr. many mirrors guy person? | 17:54 |
| EmilienM | pabelanger: I love it | 17:54 |
| JayF | pabelanger: Where are mirrors configured? i.e. where would I go to find out what I'd have to do to add an additional mirror (tinycorelinux) to it? | 17:54 |
| anteaya | mordred: that might have been it | 17:54 |
| *** doug-fis_ has quit IRC | 17:54 | |
| JayF | uh is that patch the asnwer to my question? lol | 17:54 |
| anteaya | mordred: I mis-typed, see my earlier mini rant on no longer being able to language | 17:54 |
| mat128 | JayF: looks close to it! | 17:54 |
| pabelanger | JayF: look at mirror_update.pp in openstack-infra/system-config that will get you started | 17:54 |
| *** wajdi has joined #openstack-infra | 17:54 | |
| pabelanger | JayF: mat128: you might be able to just rsync the mirror, but it depends on if said mirror purges files. | 17:55 |
| pabelanger | JayF: mat128: other wise, we need to do more creative things | 17:55 |
| mat128 | pabelanger: I'll check with the TinyCore guys how they expect us to clone/mirror their repo, because there are no docs on it | 17:55 |
| nibalizer | anteaya: realize( Plan) yes | 17:56 |
| mat128 | pabelanger: but otherwise, simply adding it to mirror_update.pp should get us started, right? | 17:56 |
| anteaya | wordclouds-r-us | 17:56 |
| mat128 | pabelanger: then it's all plumbing | 17:56 |
| anteaya | nibalizer: awesome | 17:56 |
| nibalizer | anteaya: yes basically what you do is add that realize statement to the node definition i added in my patch | 17:56 |
| nibalizer | or, even better | 17:56 |
| *** agentleweb has quit IRC | 17:56 | |
| pabelanger | mat128: Yup, mirror_update.pp is the entry point. We'd likely need to discussion what the AFS structure looks like with jeblair too | 17:57 |
| nibalizer | create a class called openstack_project::storyboard::dev next to the existing storyboard class and all it does is has htat realize statemen then calls the real storyboard class | 17:57 |
| anteaya | create a class as a child to your patch? | 17:57 |
| clarkb | jesusaur: I think if we can get pleia2 and maybe nibalizer to ack the puppet-archive update we are good to approve these chages | 17:57 |
| clarkb | jesusaur: then double check the rules in the reconsumable repo match what we have in production then switch to them | 17:58 |
| *** tqtran has joined #openstack-infra | 17:58 | |
| anteaya | clarkb: isn't crinkle +2 on that repo yet? | 17:58 |
| crinkle | anteaya: not system-config | 17:58 |
| *** pcaruana has quit IRC | 17:58 | |
| *** asselin has quit IRC | 17:58 | |
| anteaya | ah system-config, sorry | 17:59 |
| anteaya | had modules in my head | 17:59 |
| *** dtardivel has quit IRC | 17:59 | |
| *** asettle has joined #openstack-infra | 17:59 | |
| *** esikache1 has joined #openstack-infra | 18:00 | |
| nibalizer | yep | 18:00 |
| anteaya | okey dokey, thank you | 18:01 |
| nibalizer | clarkb: jeblair link? | 18:01 |
| clarkb | nibalizer: https://review.openstack.org/311224 that one | 18:01 |
| nibalizer | er jesusaur sorry jeblair | 18:01 |
| stevemar | morgan: AJaeger is there some manual intervention needed for creating the ldappool repo in git.openstack.org? | 18:01 |
| jeblair | whew | 18:01 |
| morgan | stevemar: i'm guessing puppet/tasks have not run yet. | 18:02 |
| morgan | stevemar: i don't see the stuff in gerrit yet. | 18:02 |
| anteaya | stevemar: shouldn't be needed | 18:02 |
| anteaya | stevemar morgan I'm going with what morgan is saying | 18:02 |
| nibalizer | +2a | 18:02 |
| anteaya | stevemar: might I suggest a cup of tea? | 18:02 |
| morgan | stevemar: i'm keeping watch on it | 18:02 |
| clarkb | nibalizer: yay ty | 18:02 |
| stevemar | morgan: anteaya ah yeah, it's still going through the post job | 18:03 |
| anteaya | stevemar: ah yeah that would have an effect | 18:03 |
| stevemar | anteaya: i will take your suggestion, but go with coffee and girl guide cookies instead | 18:03 |
| *** mriedem has quit IRC | 18:03 | |
| anteaya | mmmm, girl guide cookies | 18:03 |
| *** mriedem has joined #openstack-infra | 18:03 | |
| anteaya | stevemar: do you have the chocolate covered ones? | 18:03 |
| tdasilva | hello, can some help me adding me to liberasurecode-core and liberasurecode-release groups in gerrit | 18:03 |
| anteaya | stevemar: and hey did you see how much the red cross raised? | 18:04 |
| anteaya | stevemar: record numbers | 18:04 |
| stevemar | anteaya: yep, not the mint kind though | 18:04 |
| anteaya | stevemar: cool, they sound good | 18:04 |
| *** andymaier has joined #openstack-infra | 18:04 | |
| stevemar | anteaya: yah, gov matched all donations right | 18:05 |
| anteaya | will match, yes, not sure when that will happen though | 18:05 |
| *** doug-fish has joined #openstack-infra | 18:05 | |
| anteaya | money is still pouring in as far as I know | 18:05 |
| AJaeger | stevemar: wait an hour after the change merges... | 18:05 |
| *** hongbin has joined #openstack-infra | 18:05 | |
| stevemar | AJaeger: but i want it *now* :P | 18:05 |
| AJaeger | stevemar: then timetravel in the future ;) | 18:06 |
| pleia2 | clarkb, jesusaur: ugh, quite the delta between what we're using and where we are, I'll have a look at the changelog to see if anything pops out at me as a potential issue | 18:06 |
| anteaya | stevemar: plus alberta gov is matching all alberta donations, so any alberta donations is * 3 | 18:06 |
| stevemar | AJaeger: :) | 18:06 |
| stevemar | anteaya: nice | 18:06 |
| clarkb | pleia2: should we remove nibalizer's +A? | 18:06 |
| anteaya | stevemar: yeah, they were handing out money and cash cards yesterday | 18:07 |
| pleia2 | clarkb: ah, no, he's more familiar with the module than I am | 18:07 |
| *** esikache1 has quit IRC | 18:07 | |
| *** dtantsur is now known as dtantsur|afk | 18:07 | |
| pleia2 | clarkb: and I just did look at our use of it, it's pretty simple, should be fine | 18:08 |
| *** d0ugal has quit IRC | 18:08 | |
| *** doug-fis_ has joined #openstack-infra | 18:08 | |
| *** _amrith_ has joined #openstack-infra | 18:08 | |
| *** _amrith_ is now known as amrith | 18:08 | |
| openstackgerrit | Merged openstack/requirements: Add magnum-ui to projects.txt https://review.openstack.org/289584 | 18:09 |
| *** electrofelix has quit IRC | 18:09 | |
| *** rcernin has quit IRC | 18:09 | |
| jesusaur | clarkb: im currently working on a bandersnatch upgrade, but that should be done by lunch | 18:09 |
| openstackgerrit | Merged openstack/requirements: Add horizon-cisco-ui to projects.txt https://review.openstack.org/300062 | 18:09 |
| clarkb | jesusaur: cool, I will try to get the prep work of merging the cleanup changes done | 18:09 |
| *** asilenkov_ has joined #openstack-infra | 18:10 | |
| *** doug-fish has quit IRC | 18:10 | |
| stevemar | anteaya: costco had it set up so you could donate as your paid for groceries | 18:11 |
| tdasilva | AJaeger, yolanda: can you help configuring liberasurecode-core and liberasurecode-release groups in gerrit? | 18:11 |
| *** andymaier has quit IRC | 18:11 | |
| *** jpr has joined #openstack-infra | 18:11 | |
| AJaeger | better ask soomebody else... | 18:11 |
| clarkb | tdasilva: you need the initial group member set? | 18:12 |
| *** deadnull_ has quit IRC | 18:12 | |
| tdasilva | clarkb: correct | 18:12 |
| *** tphummel has quit IRC | 18:12 | |
| clarkb | tdasilva: what gerrit user should be added? | 18:12 |
| anteaya | stevemar: awesome! | 18:13 |
| *** doug-fis_ has quit IRC | 18:13 | |
| tdasilva | clarkb: thiago | 18:13 |
| nibalizer | yah archive didn't break the basics | 18:13 |
| pleia2 | nibalizer: thanks | 18:13 |
| nibalizer | and if it did we can fix archive :P | 18:13 |
| * nibalizer would love so much to have infra voting on voxpupuli | 18:13 | |
| clarkb | tdasilva: done | 18:14 |
| * persia finally understands why so many folk were dressed in red yesterday | 18:14 | |
| tdasilva | clarkb: thanks!!! | 18:14 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config: Realize Zara on the newly created storyboard-dev server https://review.openstack.org/315706 | 18:14 |
| pleia2 | we shouldn't actually notice any problems immediately anyway, not until we update something (the xml file or a source), and we'd likely do that on dev first anyway | 18:14 |
| *** javeriak has quit IRC | 18:14 | |
| anteaya | persia: why were so many folks dressed in red yesterday? | 18:14 |
| * anteaya was not dressed in red yesterday | 18:14 | |
| *** jistr has joined #openstack-infra | 18:15 | |
| *** jistr is now known as jistr|afk | 18:15 | |
| persia | anteaya: Red Cross drive | 18:15 |
| anteaya | persia: where you are? | 18:15 |
| *** amitgandhinz has quit IRC | 18:15 | |
| persia | Vancouver | 18:16 |
| anteaya | persia: ah, yeah that would be it | 18:16 |
| anteaya | nibalizer: 315706 | 18:16 |
| *** jpr has quit IRC | 18:16 | |
| *** javeriak has joined #openstack-infra | 18:16 | |
| *** amitgandhinz has joined #openstack-infra | 18:16 | |
| *** julim has joined #openstack-infra | 18:17 | |
| *** openstackgerrit has quit IRC | 18:18 | |
| *** openstackgerrit has joined #openstack-infra | 18:18 | |
| *** markvoelker_ has joined #openstack-infra | 18:19 | |
| *** andymaier has joined #openstack-infra | 18:20 | |
| *** hashar has joined #openstack-infra | 18:20 | |
| *** javeriak has quit IRC | 18:21 | |
| *** markvoelker has quit IRC | 18:22 | |
| anteaya | nibalizer: off line for a while, will check back later, thanks for your help on this | 18:22 |
| *** jtomasek has joined #openstack-infra | 18:26 | |
| *** ddieterly[away] has quit IRC | 18:26 | |
| *** ri0 has quit IRC | 18:27 | |
| *** onovy has quit IRC | 18:28 | |
| *** onovy has joined #openstack-infra | 18:28 | |
| openstackgerrit | Merged openstack-infra/system-config: Upgrade the archive puppet module version https://review.openstack.org/311224 | 18:28 |
| *** ri0 has joined #openstack-infra | 18:29 | |
| *** njohnston__ has left #openstack-infra | 18:29 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/irc-meetings: Change time for FWaaS meeting https://review.openstack.org/315308 | 18:29 |
| *** njohnston__ has joined #openstack-infra | 18:29 | |
| *** johnny_ has joined #openstack-infra | 18:29 | |
| openstackgerrit | Merged openstack-infra/puppet-logstash: Fix beaker tests https://review.openstack.org/310291 | 18:29 |
| *** chem has quit IRC | 18:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add floating IPs to server dict ourselves https://review.openstack.org/315713 | 18:30 |
| mordred | Shrews: ^^ | 18:31 |
| *** claudiub has joined #openstack-infra | 18:31 | |
| *** jpr has joined #openstack-infra | 18:31 | |
| mordred | cloudnull: ^^ that is not tested at all - but is a first-sketch at pulling the info from neutron ourselves | 18:31 |
| mordred | it needs a config flag | 18:31 |
| *** ri0 has quit IRC | 18:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Properly shutdown apscheduler on reconfigure https://review.openstack.org/315715 | 18:34 |
| jeblair | clarkb, Shrews, morgan, pabelanger, jhesketh: ^ that should address the flaky test_timer that has been sniping all of the v25 changes | 18:35 |
| cloudnull | nice! | 18:35 |
| cloudnull | i love the commit message. | 18:35 |
| cloudnull | much sadness | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support ftp publisher https://review.openstack.org/309095 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: run shell builders defined in JJB https://review.openstack.org/308099 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support zuul stop commands https://review.openstack.org/309148 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: register jobs from JJB https://review.openstack.org/307515 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: send ZMQ start/complete events https://review.openstack.org/308053 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add zuul_runner module https://review.openstack.org/309221 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add streaming console support https://review.openstack.org/309501 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: run ansible https://review.openstack.org/308082 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support job timeouts https://review.openstack.org/308488 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support scp publisher https://review.openstack.org/308656 | 18:35 |
| pabelanger | jeblair: ack | 18:35 |
| *** mjblack has joined #openstack-infra | 18:36 | |
| *** ddieterly has joined #openstack-infra | 18:36 | |
| *** kmartin has joined #openstack-infra | 18:36 | |
| mordred | cloudnull: :) | 18:37 |
| *** esker has joined #openstack-infra | 18:38 | |
| *** esker has quit IRC | 18:39 | |
| *** esker has joined #openstack-infra | 18:39 | |
| *** tphummel has joined #openstack-infra | 18:39 | |
| *** tphummel has quit IRC | 18:39 | |
| jeblair | mordred: why continue on ipv6? | 18:41 |
| clarkb | jeblair: neutron doens't support ipv6 floating IPs | 18:41 |
| jeblair | clarkb: good, i hoped that was the answer :) | 18:41 |
| *** e0ne has quit IRC | 18:41 | |
| mordred | clarkb, jeblair: details notwithstanding - does that general idea seem somewhat un vomit inducing? | 18:42 |
| openstackgerrit | Merged openstack-infra/system-config: Add BHS1 region to openstackjenkins-ovh all-clouds https://review.openstack.org/315686 | 18:44 |
| *** sdake has quit IRC | 18:44 | |
| clarkb | mordred: it is unfortunate that we have to list all floating IPs to check | 18:45 |
| clarkb | but that is a result of not having a shared key here other than the ip addr | 18:45 |
| clarkb | *reliable shared key | 18:45 |
| jeblair | mordred: i think it could probably use a code comment explaining why we have to do that... i honestly think most of that explanation is in irc atm -- so that's my feedback as someone who isn't deeply involved in this issue. | 18:45 |
| mordred | clarkb: yah. that is, btw, why the previous patch adds batched listing | 18:45 |
| jeblair | mordred: and i left a nit on the change. | 18:45 |
| mordred | jeblair: cool. I can do that | 18:45 |
| *** nadya has quit IRC | 18:46 | |
| *** dmk0202 has joined #openstack-infra | 18:47 | |
| mordred | cloudnull: any chance I could get an account on osic with enough quota to do your 25-server test? (and to test shade things on it from time to time?) I will most of the time not use it | 18:47 |
| *** oanson has joined #openstack-infra | 18:48 | |
| *** IlyaG has quit IRC | 18:49 | |
| *** jamesmcarthur has joined #openstack-infra | 18:49 | |
| openstackgerrit | Merged openstack/requirements: Add pointer for SUSE python packaging devel project https://review.openstack.org/311564 | 18:52 |
| cloudnull | mordred: i want to say yes but so let me ask the right people. :) | 18:52 |
| openstackgerrit | Merged openstack/requirements: Update global-requirements.txt for python-kafka version https://review.openstack.org/297995 | 18:52 |
| * cloudnull is a follower of production controls these days. | 18:52 | |
| clarkb | mordred: maybe we can suggest that neutron api grow a reverse lookup? | 18:53 |
| *** tellesnobrega has left #openstack-infra | 18:53 | |
| clarkb | mordred: seems more efficient ot have eg mysql do this join than python doing multiple api queries | 18:53 |
| *** ddieterly is now known as ddieterly[away] | 18:54 | |
| *** dizquierdo is now known as dizquierdo_afk | 18:55 | |
| cloudnull | also mordred on the FIP topic when i do a create of my 25 nodes I see 25 returns for a FIP however not all are unique | 18:55 |
| cloudnull | https://gist.github.com/cloudnull/d90eff7efc524844d8d3504fea8fa419 | 18:55 |
| cloudnull | the number differs from run to run | 18:55 |
| pabelanger | clarkb: jeblair: we're making good progress on the DIB uploads today, 23 left. That is with 12 upload workers. I haven't see our TX peak 400Mb, so we could likely bump up a few more upload-workers, assuming 800Mb TX | 18:56 |
| cloudnull | but in general there is some number of non-uniqe ids and that number is the amount of servers missing a fip | 18:56 |
| *** awaugama has joined #openstack-infra | 18:56 | |
| jeblair | pabelanger: yeah, i'm still thinking 16 :) | 18:56 |
| clarkb | pabelanger: we should double check on that, rax does some weird thing where you get half your bandwidth on public interface iirc | 18:56 |
| jeblair | clarkb: i thought that was old? | 18:56 |
| openstackgerrit | Merged openstack/requirements: Add pifpaf to requirements https://review.openstack.org/301565 | 18:57 |
| clarkb | jeblair: if they changed that behavior then I didn't realize they did | 18:57 |
| pabelanger | bluebox we are pushing 227Mb currently, definitely the fastest one | 18:57 |
| pabelanger | clarkb: suggests who to confirm with? | 18:57 |
| clarkb | pabelanger: osic and bluebox are quick and use qcow2 so its double win | 18:57 |
| pabelanger | clarkb: yar | 18:58 |
| clarkb | pabelanger: I think you can flavor list to get the rx/tx factor the ndouble check rax docs for what those numbers mean | 18:58 |
| pabelanger | we're only getting 20Mb from rax | 18:58 |
| pabelanger | clarkb: ack | 18:58 |
| fungi | pabelanger: clarkb: yep, it does look like we're hitting the 400m cap probably http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=982&rra_id=all | 18:58 |
| pabelanger | fungi: Ah, good idea checking cacti | 18:58 |
| *** sambetts|afk has quit IRC | 18:59 | |
| awaugama | may be in the wrong place, I'm new to openstack and trying to sigh the ICLA on review.openstack.org and I'm getting a "Code Review - Error, server error, cannot store information" every time I try to submit the agreement. any ideas? | 18:59 |
| mordred | cloudnull: WOW | 18:59 |
| fungi | so i wouldn't expect to be able to push it any faster than we already are (and we may already be overdriving that to the point where we're hampering throughput slightly, depending on how graceful their rate limiting implementation is) | 19:00 |
| mordred | cloudnull: that seems like a large bug | 19:00 |
| fungi | awaugama: have you been following the instructions at http://docs.openstack.org/infra/manual/developers.html#account-setup ? | 19:00 |
| fungi | awaugama: the most likely causes are covered in the second paragraph (about setting up your foundation membership correctly first) | 19:01 |
| mordred | armax: cloudnull may have found a quite spectacular bug in neutron | 19:01 |
| armax | mordred: only one? :) | 19:01 |
| openstackgerrit | Merged openstack/requirements: Upgrade PyKMIP to 0.5.0 https://review.openstack.org/308095 | 19:01 |
| openstackgerrit | Merged openstack/requirements: Drop influxdb https://review.openstack.org/313824 | 19:01 |
| mordred | armax: :) | 19:02 |
| fungi | armax: well, it has to do with floating ips, so i suppose you could class that as an entire family of bugs | 19:02 |
| *** d0ugal has joined #openstack-infra | 19:02 | |
| cloudnull | to get that data I added a print here https://github.com/openstack-infra/shade/blob/master/shade/openstackcloud.py#L3605 took the stdout and did my gistthiname's | 19:02 |
| mordred | armax: we're trying to track down why sometimes creating a floating ip on a server's port does not result in a floating ip on that server | 19:02 |
| *** sambetts has joined #openstack-infra | 19:03 | |
| armax | mordred: wasn’t that we talked about last time? | 19:03 |
| *** jerryz has quit IRC | 19:03 | |
| *** HeOS has quit IRC | 19:03 | |
| cloudnull | mordred: this may be our clouds fault too. we're still running kilo | 19:04 |
| mordred | armax: yes - but we've got a friendly cloud operator helping us find more issues | 19:04 |
| cloudnull | so maybe fixed in !kilo | 19:04 |
| cloudnull | we're going to upgrade tomorrow though | 19:04 |
| *** mjblack has quit IRC | 19:04 | |
| fungi | osic has deployed openstack's shiny new kilo release | 19:04 |
| mordred | armax: f1169479-7d88-4526-a60a-82f18b617077 is an id for 2 different fip entries | 19:04 |
| fungi | cloudnull: ooh! mitaka? or liberty? | 19:04 |
| cloudnull | liberty | 19:04 |
| cloudnull | mitaka fast follow | 19:05 |
| mordred | armax: in https://gist.github.com/cloudnull/d90eff7efc524844d8d3504fea8fa419 | 19:05 |
| fungi | baby steps. better than not upgrading! | 19:05 |
| cloudnull | indeed | 19:05 |
| AJaeger | fungi, speaking about kilo: openstack-manuals has frozen stable/kilo - could you put this on your list of branches to retire, when you go through them, please? | 19:05 |
| cloudnull | i wanted to do the upgrade ages ago , it was hard to convince folks | 19:05 |
| awaugama | fungi: yeah, i followed the instructions, just tried to recreate the account to verify the email was in use. | 19:05 |
| cloudnull | but i think we're getting on the right track | 19:06 |
| awaugama | fungi: actually, just saw, I think I did community. Let me fix that | 19:06 |
| fungi | awaugama: yep, we tried to call that out explicitly in the directions since it's a common mistake some people make | 19:06 |
| dansmith | do we know that zuul queue is at 4600 items? | 19:06 |
| fungi | let me know if that doesn't help, and we can take troubleshooting further | 19:06 |
| fungi | dansmith: we do now i guess? ;) | 19:07 |
| AJaeger | dansmith: continue watching - it should go down again eventually ;) | 19:07 |
| dansmith | it's been climbing for like 30m | 19:07 |
| awaugama | fungi: thanks. Guess I was going too fast. Do I need to register again or is there a way to upgrade accounts? | 19:07 |
| mordred | cloudnull: I find it curious that it was hard to convince people ... seems like running mitaka would be more useful for osic's mission | 19:07 |
| dansmith | which is usually a bad sign I thought | 19:07 |
| *** armax has quit IRC | 19:07 | |
| mordred | cloudnull: but I'm glad you have had success! | 19:07 |
| cloudnull | anyway. armax, this may be a kilo specific bug and maybe infra is seeing that problem in OSIC and BlueBox because we're still on the Kilo release? | 19:07 |
| fungi | armax just lost his floating ip | 19:07 |
| awaugama | fungi: found it, nevermind | 19:08 |
| cloudnull | mordred: yes i would agree. but I have to deal with people and people dont make sense. :) | 19:09 |
| fungi | dansmith: most common cause for the events/results queues spiking is reconfiguration taking a while. i'm looking now to see if the debug logs indicate any signs of actual distress | 19:09 |
| *** matt-borland has joined #openstack-infra | 19:09 | |
| *** asettle has quit IRC | 19:09 | |
| mordred | cloudnull: people are often the bug | 19:09 |
| cloudnull | +1 | 19:09 |
| mordred | cloudnull: I think our bluebox cloud is actually juno | 19:10 |
| mordred | jlk: yah? | 19:10 |
| dansmith | fungi: okay, I just have it in red right above my head all the time, so I notice when it starts to grow.. usually gives me a feeling of what is normal and not | 19:10 |
| jlk | correct | 19:10 |
| AJaeger | zuul queue just went from 5500 to 0... | 19:10 |
| dansmith | ah just flushed | 19:10 |
| * fungi ceases investigating | 19:10 | |
| mordred | jlk: btw - see the scrollback about what cloudnull found wrt duplicate FIP entries | 19:10 |
| dtroyer | Anyone know what builds the ICS files from the irc-meetings repo? | 19:11 |
| fungi | dansmith: looks like maybe a bunch of changes merged for multiple high-volume projects and zuul was generating the merge-check work requests for all their remaining open changes | 19:11 |
| dtroyer | the real question is what decides the odd/even week start... | 19:11 |
| clarkb | dtroyer: yaml2ical iirc | 19:11 |
| mordred | infra-root: I'm inclined to wait until osic upgrades to see if that fixes it before we land the "build the network info ourselves" patch | 19:12 |
| dansmith | fungi: yeah, and fake-merge-conflict'd several of mine :) | 19:12 |
| clarkb | dtroyer: its some iso standard and date prints them out with the right flag | 19:12 |
| jlk | oh good times | 19:12 |
| dansmith | fungi: anyway, just seemed really high and long, so.. sorry :) | 19:12 |
| mordred | jlk: right? | 19:12 |
| jlk | cloudnull: if you have a bug filed on that, can you link me? I want to make sure we get any fix into our Mitaka update | 19:12 |
| fungi | dansmith: which ones got flagged as merge-conflicting? | 19:12 |
| jeblair | fungi: saw another ord connection error | 19:12 |
| dansmith | fungi: for example : https://review.openstack.org/#/c/315681/2 | 19:12 |
| *** asettle has joined #openstack-infra | 19:13 | |
| clarkb | dtroyer: date +%U | 19:13 |
| fungi | dansmith: oh, that's marked as merge conflicting because it has a non-current parent. gerrit won't actually let that merge | 19:13 |
| dtroyer | clarkb: nice. Too bad $EMPLOYER, and Mac Calendar both tell me this is w20 | 19:14 |
| dansmith | fungi: it does? it's the bottom of the stack | 19:14 |
| fungi | dansmith: orange dot next to the "parents" | 19:14 |
| dtroyer | clarkb: thanks | 19:14 |
| dansmith | fungi: and I just rechecked and it started running | 19:14 |
| pabelanger | clarkb: apparently we have 1600.00/mbps: https://developer.rackspace.com/docs/cloud-servers/v2/developer-guide/#listing-flavors-nova-client | 19:14 |
| fungi | dansmith: oh, nevermind, that's just saying that the parent is no longer the branch tip i guess, not that the change depended on a change which has a newer patchset | 19:15 |
| pabelanger | clarkb: but according to http://serverfault.com/questions/656582/what-does-rxtx-factor-mean-what-is-it-and-how-does-it-affect-my-server?answertab=active#tab-top Outbound public Internet bandwidth can be up to 40% of the aggregate limit | 19:15 |
| pabelanger | so, 640mbps? | 19:16 |
| AJaeger | fungi, shall I send an email to openstack-infra for kilo branch retirement of openstack-manual, or do you have a list somewhere? | 19:16 |
| fungi | jeblair: what sort of ord network interruption? | 19:16 |
| *** oanson has quit IRC | 19:16 | |
| *** salv-orlando has joined #openstack-infra | 19:16 | |
| fungi | AJaeger: the stable branch team has been assembling a list of repos for me to delete branches on, i haven't started yet (2015.1.4 was only tagged a day or two ago) | 19:17 |
| *** ddieterly[away] is now known as ddieterly | 19:17 | |
| AJaeger | it's not critical, just wanted to know what's the best way is to get added to that list - since the stable team does not take care of openstack-manuals in general. Shall I ask them to add it? | 19:18 |
| *** bpokorny has quit IRC | 19:18 | |
| *** timl0101 has quit IRC | 19:18 | |
| fungi | AJaeger: not sure, i've been letting them handle more of that work this time around for the projects under release management, but an e-mail to the infra ml is probably fine as a placeholder | 19:18 |
| *** timl0101 has joined #openstack-infra | 19:18 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Bump nodepool-builder to 16 upload-workers https://review.openstack.org/315727 | 19:19 |
| AJaeger | will do, thanks | 19:19 |
| pabelanger | jeblair: clarkb: fungi: ^ lets see if we can break 400mbps | 19:19 |
| fungi | pabelanger: i'm not immediately opposed, but judging from the graph i have a feeling we're already seeing some minor packet loss slowing down the current transfers and will just end up making that worse if we try to add more | 19:20 |
| *** jang has joined #openstack-infra | 19:21 | |
| AJaeger | team, I'll sign off in a few minutes - and then it's vacation time. Might do some reviews tomorrow but don't count on me for the next two weeks... Have fun! | 19:21 |
| mordred | AJaeger: enjoy! | 19:21 |
| fungi | AJaeger: enjoy your time off! | 19:22 |
| AJaeger | thanks, will try ;) | 19:22 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only https://review.openstack.org/313588 | 19:23 |
| *** kfarr has joined #openstack-infra | 19:23 | |
| clarkb | mordred: ok finally left comments on the fip change, its just questions no -1 | 19:24 |
| mordred | clarkb: cool - did you catch my thing above about wanting to see if the osic upgrade fixes it before landing that? | 19:25 |
| clarkb | mordred: I didn't but that seems reaosnably, note bluebox also suffers this behavior | 19:25 |
| cloudnull | jlk: i've not filled a bug yet. however I will when I can rule out its not just a problem on my end | 19:26 |
| mordred | clarkb: yah. they're also going to upgrade our bluebox cloud | 19:27 |
| mordred | clarkb: I think we'll likely wind up landing the patch anyway, so that shade works well with older clouds - but if we've got info on whether it's fixed or not it'll help us track when/how we need to enable the setting | 19:27 |
| *** AJaeger has quit IRC | 19:28 | |
| mordred | clarkb: since it does have a cost to enable | 19:28 |
| *** r-daneel has joined #openstack-infra | 19:28 | |
| cloudnull | I'm hoping (figers crossed) that this is some neutron thing that was resolved in liberty. | 19:30 |
| cloudnull | im going to keep beating on it, it may be something else | 19:30 |
| jeblair | fungi: oh the usual couldn't contact git.o.o from a job running in ord | 19:30 |
| jeblair | fungi: just mentioning it so we know it's still occasionally happening | 19:31 |
| fungi | jeblair: oh yeah, it's definitely still happening http://status.openstack.org//elastic-recheck/#1298006 | 19:31 |
| *** Sukhdev has quit IRC | 19:32 | |
| fungi | if you go to the logstash link there and add the node_provider column, you'll see it's (almost) all rax-ord | 19:32 |
| *** Sukhdev has joined #openstack-infra | 19:32 | |
| fungi | ocassionally bluebox-sjc1 | 19:32 |
| jeblair | ya | 19:32 |
| *** slaweq_ has joined #openstack-infra | 19:32 | |
| *** esikache1 has joined #openstack-infra | 19:32 | |
| *** slaweq has quit IRC | 19:32 | |
| fungi | going back a couple pages, i can find one hit for rax-iad too | 19:32 |
| *** esikache1 has quit IRC | 19:33 | |
| *** nadya has joined #openstack-infra | 19:35 | |
| *** asettle has quit IRC | 19:35 | |
| *** apoorvad has quit IRC | 19:36 | |
| *** slaweq_ has quit IRC | 19:37 | |
| cloudnull | mordred: i've slammed this together in shade to track a FIP and return server data when attached http://paste.openstack.org/show/496967/ -- its quite a bit faster but the duplicate FIP ID issues are still present. | 19:38 |
| * cloudnull goes off to eat | 19:38 | |
| *** bswartz has quit IRC | 19:39 | |
| *** slaweq_ has joined #openstack-infra | 19:41 | |
| pabelanger | fungi: ya, if people want to try, I am game. | 19:42 |
| *** ociuhandu has quit IRC | 19:43 | |
| pabelanger | we're also down to 6 images, only rax-iad | 19:43 |
| pabelanger | and 00:04:28:52 into uploads, so we've made progress | 19:43 |
| *** bpokorny has joined #openstack-infra | 19:45 | |
| pabelanger | fungi: clarkb: any good documentation for creating a kilo-eol tag? | 19:47 |
| pabelanger | looking in the release team docs now | 19:47 |
| fungi | pabelanger: it's something i've done manually in the past, though i'm trying to entice te stable branch managers and/or release team to take on more of that this time around | 19:48 |
| fungi | rather, they've expressed interest in doing more of it | 19:48 |
| fungi | they're still in the process of putting together the list of repos that need it done | 19:49 |
| *** tphummel has joined #openstack-infra | 19:49 | |
| *** signed8bit has quit IRC | 19:49 | |
| pabelanger | fungi: okay, so maybe I can just add openstack-manuals to that list | 19:49 |
| mtreinish | clarkb: I'm working on adding elastic-recheck support to openstack-health: https://review.openstack.org/311919 and right now I'm trying to get around how slow it can be | 19:51 |
| *** matrohon has joined #openstack-infra | 19:51 | |
| fungi | pabelanger: yep, i just asked about it over in #openstack-stable | 19:51 |
| *** jistr|afk has quit IRC | 19:52 | |
| clarkb | mtreinish: the es queries being slow? write better queries :) | 19:52 |
| clarkb | mtreinish: last I saw many of them are search every file for this one string that only shows up in ome place ever | 19:52 |
| mtreinish | clarkb: right now the fastest I've managed to get it is about 20sec but it often takes several minutes to run all the queries all the 10 most recent changes (it's doing it in parallel with a thread per change) | 19:52 |
| *** yamahata has quit IRC | 19:52 | |
| pabelanger | fungi: sure, connecting in | 19:53 |
| clarkb | mtreinish: the trick is to filter as much as possible so build_name specifics and filename specifics and so on | 19:53 |
| mtreinish | clarkb: heh, I'm treating the e-r queries as a black box right now | 19:53 |
| mtreinish | clarkb: it's using whatever the e-r classifier.classify() filters too | 19:53 |
| mtreinish | which I think is zuul build uuid, change num, and rev | 19:54 |
| mtreinish | and I also have it filter to within the last hour | 19:54 |
| fungi | pabelanger: conversation there is a lot more intermittent than here, so probably safe to wait until http://eavesdrop.openstack.org/irclogs/%23openstack-stable/latest.log.html refreshes and that way i won't need to reiterate | 19:54 |
| *** Swami has quit IRC | 19:54 | |
| mtreinish | but it's still dependent on whatever the actually query in e-r is | 19:54 |
| clarkb | mtreinish: oh you are doing it per change? maybe I dont understand what you are trying to do | 19:54 |
| pabelanger | fungi: Yup, I'm good with that | 19:55 |
| clarkb | er per build | 19:55 |
| *** apoorvad has joined #openstack-infra | 19:55 | |
| mtreinish | clarkb: on o-h's home page there is a list of 10 recent failures | 19:55 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Adding library Delimiter to the CI system https://review.openstack.org/315232 | 19:55 |
| mtreinish | I'm trying to add a list of identified bugs to that list | 19:55 |
| pabelanger | Wouldn't mind a quick review on https://review.openstack.org/#/c/315698/ adds the cloud repo to our centos mirror. About 1.7G of new data | 19:56 |
| clarkb | gotcha that should be reasonably quick, definitely double chwck the queries though bad queries will be slow | 19:56 |
| asselin_ | dstufft, does anything else need to be done after upgrading to 1.10 bandersnatch? or should it just 'work'? | 19:56 |
| jeblair | fungi, pabelanger: http://eavesdrop.openstack.org/irclogs/%23openstack-stable/%23openstack-stable.2016-05-12.log | 19:56 |
| mtreinish | so I'm using the classifier from e-r bot which, loops over all the queries for a single build | 19:56 |
| dstufft | asselin_: should just work | 19:56 |
| dstufft | I mean | 19:57 |
| clarkb | mtreinish: we also have a ton of queries that can be deleted | 19:57 |
| dstufft | upgrading will trigger a full sync | 19:57 |
| mtreinish | clarkb: yeah, I guess I"ll need to instrument e-r a bit better to see where it's spending the most time | 19:57 |
| dstufft | so you'll have to wait for that | 19:57 |
| mtreinish | clarkb: yeah, there is definitely some pruning we can do on the e-r | 19:57 |
| asselin_ | dstufft, oh...when? how? | 19:57 |
| *** Swami has joined #openstack-infra | 19:57 | |
| fungi | jeblair: alternate url for the same page... what's the relevance? | 19:58 |
| dstufft | asselin_: I'm not sure the question, the next time you run `bandersnatch mirror` it should automatically do a full sync | 19:58 |
| dstufft | but ti'll take longer than normal | 19:58 |
| jeblair | fungi: it updates immediately | 19:58 |
| *** yamahata has joined #openstack-infra | 19:58 | |
| fungi | jeblair: neither of them are updating immediately for me | 19:58 |
| pabelanger | jeblair: danke | 19:58 |
| fungi | jeblair: still has 19:39 as the most recent content | 19:59 |
| *** asettle has joined #openstack-infra | 19:59 | |
| asselin_ | dstufft, ok....seems only one package updated. will check again in next 5 minutes.... | 19:59 |
| *** asettle has quit IRC | 19:59 | |
| jeblair | fungi: 2016-05-12T19:53:23 <pabelanger> ohai! | 19:59 |
| pabelanger | yup, I see that | 19:59 |
| fungi | jeblair: weird. forced refreshes in my browser are still at 19:39 on both those pages | 19:59 |
| *** asettle has joined #openstack-infra | 20:00 | |
| * fungi wonders if there are some weird cache headers set on there | 20:00 | |
| jeblair | fungi: that is weird. i see 19:39 only on the latest.log.html; date.log shows 19:53 | 20:00 |
| *** asettle has quit IRC | 20:00 | |
| fungi | jeblair: refreshed again just now and they both updated for me | 20:00 |
| jeblair | fungi: they are bothe at 19:53 now for me as well | 20:01 |
| *** jamesmcarthur has quit IRC | 20:03 | |
| *** slaweq_ is now known as slaweq | 20:05 | |
| openstackgerrit | David Lyle proposed openstack/requirements: Remove django-bootstrap-form https://review.openstack.org/314203 | 20:06 |
| *** ddieterly is now known as ddieterly[away] | 20:07 | |
| fungi | jeblair: having trouble tracking down why dansmith's 315681,2 change consistently failed to merge according to zuul. even with merge -s resolve it didn't conflict with the master branch tip for me, and the times zuul reported merge failures for it didn't seem to coincide with events on its depends-on change either. the zuul debug log just reports that it failed to merge | 20:07 |
| *** awaugama has quit IRC | 20:07 | |
| dansmith | fungi: it failed after recheck too.. I just pushed up another | 20:08 |
| fungi | dansmith: yeah, i saw that. seems strange, but it's likely i'm missing something | 20:08 |
| dansmith | I don't get it either.. it failed immediately when I first pushed it up, hardly had time to be in conflict with anything :) | 20:09 |
| fungi | well, actually the last merge failure for it happened when zuul reported job failures on 312755,13 | 20:09 |
| fungi | (its depends-on) | 20:09 |
| *** yamahata has quit IRC | 20:09 | |
| fungi | but previous merge failures didn't seem to coorelate to anything there | 20:09 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Consul has now merged (and it works) https://review.openstack.org/315751 | 20:10 |
| jeblair | fungi: ack, will look | 20:10 |
| fungi | dansmith: also on 312755,13 what did you mean by "jobs didn't start"? | 20:11 |
| fungi | looks like you got check results reported on that change 15 minutes before you added that recheck comment | 20:11 |
| *** ddieterly[away] is now known as ddieterly | 20:11 | |
| *** amrith is now known as _amrith_ | 20:12 | |
| dansmith | fungi: I meant one of the jobs did this: http://logs.openstack.org/55/312755/13/check/gate-tempest-dsvm-full/ad73268/ | 20:13 |
| *** njohnston__ is now known as njohnston_afk | 20:13 | |
| fungi | dansmith: oh, joy. that's yet another network failure in rackspace ord | 20:14 |
| dansmith | fungi: \o/ | 20:14 |
| * fungi checks up on the long-standing ticket about this | 20:14 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: add check-requirements to networking projects https://review.openstack.org/315753 | 20:15 |
| fungi | sigh. they're confused _still_ | 20:15 |
| *** maishsk has quit IRC | 20:17 | |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: add a validation command to ensure projects use check-requirements https://review.openstack.org/315755 | 20:17 |
| *** maishsk has joined #openstack-infra | 20:18 | |
| *** ayoung has quit IRC | 20:20 | |
| *** wajdi has quit IRC | 20:20 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Consul has now merged (and it works) https://review.openstack.org/315751 | 20:21 |
| *** dizquierdo_afk has quit IRC | 20:22 | |
| *** keedya has quit IRC | 20:22 | |
| jeblair | zaro: what's the status on the git gc investigation? | 20:23 |
| *** Sukhdev has quit IRC | 20:23 | |
| asselin_ | dstufft, seems to be running now, though for some reason we don't see the outpu in syslog. Might be an issue on our system. Thanks for the confirmation. | 20:23 |
| *** Sukhdev has joined #openstack-infra | 20:24 | |
| *** kgiusti has left #openstack-infra | 20:24 | |
| fungi | rackspace's network engineering team wants us to avoid deleting instances which exhibit this failure mode. anybody have ideas on how we might make that dream a reality? throwaway d-g change to bysy-wait the setup script if a ping check to git.o.o fails but short-circuits and ends immediately if not, then recheck it a bunch and hold one that takes too long? | 20:24 |
| mtreinish | clarkb: it looks like each individual es query is taking like 1-2 secs., but that adds up when we have to run 115 of them | 20:24 |
| *** rcernin has joined #openstack-infra | 20:25 | |
| jeblair | fungi: ooh, in zuulv2.5 or zuulv3, we could have a special exit code or something that we use to tell nodepool to hold the node | 20:26 |
| jesusaur | clarkb: so where are we with the logstash-filter repo? | 20:26 |
| mordred | jeblair: oooh | 20:26 |
| openstackgerrit | Nikhil Komawar proposed openstack-infra/project-config: Add #openstack-quota to the IRC channels https://review.openstack.org/315759 | 20:27 |
| openstackgerrit | David Lyle proposed openstack/requirements: Remove XStatic-JQuery.Bootstrap.Wizard https://review.openstack.org/313803 | 20:27 |
| fungi | jeblair: speaking of gerrit git gc, not sure if you saw while you were away last week but i ended up doubling the ~gerrit2 volume because we've seen constant growth in the git repos since the 2.11 upgrade | 20:27 |
| *** abregman has joined #openstack-infra | 20:27 | |
| jeblair | fungi: i missed that, thanks | 20:28 |
| fungi | jeblair: the yearly graph is pretty obvious http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2839&rra_id=all | 20:28 |
| fungi | mid-december it starts growing linearly | 20:28 |
| fungi | jeblair: unrelated to git but still in a similar vein, we also nearly ran out of space in the gerrit trove instance, so i grew its backing volume from 10gb to 50gb | 20:29 |
| *** wznoinsk has quit IRC | 20:32 | |
| *** yamahata has joined #openstack-infra | 20:33 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: ensure all projects syncing global requirements use check-requirements https://review.openstack.org/315761 | 20:33 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/devstack-gate: DO NOT MERGE - attempt to find a network failure https://review.openstack.org/315762 | 20:34 |
| clarkb | jesusaur: the test fixes have merged, need to make sure those rules are up to date then merge the rule changes and merge switch over to them | 20:35 |
| clarkb | jesusaur: I am finishing up lunch give me aboht 20 minutes | 20:35 |
| *** ericksonsantos has quit IRC | 20:35 | |
| mordred | jeblair, fungi: related to that, I have started (eversoslightly) poking at trove clustering and replication and how that might or might not help yus | 20:35 |
| mordred | us | 20:35 |
| jesusaur | clarkb: ok | 20:35 |
| *** abregman has quit IRC | 20:36 | |
| *** hamzy has quit IRC | 20:36 | |
| mordred | I have not accomplished as much investigation as I'd like | 20:36 |
| clarkb | jesusaur: maybe you can diff new rules and production? | 20:36 |
| jesusaur | clarkb: yeah, was just about to make sure our filters are up-to-date | 20:36 |
| openstackgerrit | Nikhil Komawar proposed openstack-infra/project-config: Adding library Delimiter to the CI system https://review.openstack.org/315232 | 20:36 |
| fungi | mordred: that would be an awesome outcome if it accomplishes what we need | 20:37 |
| fungi | i'm going to go find some grub, back in a while | 20:37 |
| *** aeng has joined #openstack-infra | 20:39 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Remove queries with no hits https://review.openstack.org/315765 | 20:39 |
| *** raildo is now known as raildo-afk | 20:41 | |
| *** eharney has quit IRC | 20:42 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] make grenade job temporary non-voting https://review.openstack.org/315767 | 20:42 |
| timothyb89 | fungi, pabelanger, clarkb, anteaya, mtreinish: if you have a moment, could you take a look at this revised stackviz patch? https://review.openstack.org/#/c/311815/ | 20:42 |
| *** thorst_ has joined #openstack-infra | 20:42 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] return acls for sahara-tests https://review.openstack.org/312503 | 20:43 |
| jeblair | dansmith, fungi: 315681 Depends-On 312755 has parent 312751; If i check out master and cherry-pick 312751, 312755, and 315681, i get a file-level merge conflict | 20:43 |
| pabelanger | timothyb89: Ya, I meant to leave a comment on that. It looks good to be, but I'll likely drop a -1 workflow on it, so I can help monitor it once we land it | 20:43 |
| pabelanger | timothyb89: if we get enough votes, we can likely build images this weekend | 20:43 |
| timothyb89 | pabelanger: good to hear, thank you! | 20:43 |
| jeblair | dansmith, fungi: which zuul did as well: http://paste.openstack.org/show/496977/ | 20:44 |
| *** gyee has joined #openstack-infra | 20:44 | |
| jesusaur | clarkb: I've confirmed that the filter section is identical, we can either change the example output to use elasticsaerch instead of elasticsearch_http in a new patchset or in a follow-up change | 20:44 |
| *** _ari_ has quit IRC | 20:44 | |
| *** wznoinsk has joined #openstack-infra | 20:45 | |
| pabelanger | timothyb89: Ya, I'm happy now that we don't sudo npm install npm | 20:45 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Increase default OS_TEST_TIMEOUT to 1200 https://review.openstack.org/305270 | 20:45 |
| dansmith | jeblair: ah | 20:45 |
| *** Sukhdev has quit IRC | 20:46 | |
| *** ddieterly is now known as ddieterly[away] | 20:46 | |
| openstackgerrit | Merged openstack/requirements: Add review guideline about projects.txt updates https://review.openstack.org/315139 | 20:46 |
| openstackgerrit | Merged openstack/requirements: Update to recent release of tosca-parser 0.5.0 https://review.openstack.org/312739 | 20:47 |
| *** amitgandhinz has quit IRC | 20:49 | |
| *** amitgandhinz has joined #openstack-infra | 20:49 | |
| *** armax has joined #openstack-infra | 20:50 | |
| *** Goneri has quit IRC | 20:50 | |
| armax | mordred, fungi: sorry for the abrupt disconnection | 20:50 |
| clarkb | jesusaur: lets do a followup change | 20:50 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Realize Zara on the newly created storyboard-dev server https://review.openstack.org/315706 | 20:50 |
| clarkb | jesusaur: now to find the change that uses that new list of rules | 20:50 |
| nibalizer | anteaya: ^^ is what I'm thinking | 20:51 |
| *** nadya has quit IRC | 20:51 | |
| *** andymaier has quit IRC | 20:51 | |
| jesusaur | clarkb: 310052 | 20:51 |
| *** nadya has joined #openstack-infra | 20:51 | |
| *** DevBox has quit IRC | 20:52 | |
| anteaya | nibalizer: thanks, yeah I'm glad you did the patch yourself I didn't get that vision from the words I read and re-read earlier | 20:52 |
| openstackgerrit | Edgar Magana proposed openstack-infra/irc-meetings: Moving meetings from odd weeks to even https://review.openstack.org/315771 | 20:52 |
| anteaya | nibalizer: thank you | 20:52 |
| sdague | jhesketh: when you get up - https://review.openstack.org/#/c/315677/ would be awesome to put Jenkins back on top | 20:52 |
| jesusaur | clarkb: which depends on 310051 | 20:52 |
| clarkb | ty ty | 20:53 |
| armax | mordred: so it looks two concurrent requests end up getting the same FIP | 20:53 |
| clarkb | jesusaur: I have approved the filters project change | 20:53 |
| armax | at least from what I gather reading https://gist.github.com/cloudnull/d90eff7efc524844d8d3504fea8fa419 | 20:53 |
| openstackgerrit | Merged openstack-infra/logstash-filters: Move the input and output sections to example files https://review.openstack.org/310046 | 20:54 |
| mordred | armax: yah. that's our take | 20:54 |
| mordred | armax: they're currently running kilo but are upgrading soon to liberty and then to mitaka soon after | 20:54 |
| mordred | armax: so I think we'd like to see if it's fixed in newer | 20:54 |
| *** kzaitsev_mb has joined #openstack-infra | 20:54 | |
| armax | knowing how well we deal with concurrency, it’s entirely possible that unless this was found out before | 20:54 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config: Realize Zara on the newly created storyboard-dev server https://review.openstack.org/315706 | 20:54 |
| *** dprince has quit IRC | 20:54 | |
| armax | it’s still in master | 20:54 |
| clarkb | jesusaur: I am going to disable puppet on all but one of the logstash workers then we can get 31005[12] in | 20:55 |
| armax | mordred: let me see what my fishing expedition is gonna yield | 20:55 |
| anteaya | nibalizer: did I get your email address correct for the Co-Authored By: bit ? | 20:55 |
| jesusaur | clarkb: sounds good | 20:55 |
| nibalizer | anteaya: i prefer nibz@spencerkrum.com but I don't actually care :) | 20:55 |
| mordred | armax: cool - thanks - and let me know if I can be helpful | 20:56 |
| anteaya | nibalizer: I care, I'll change it, thanks | 20:56 |
| cloudnull | armax: i might be able to get a newton env stood up today and see if its still bothersome their using the same shade bits i'm attacking the OSIC with. | 20:56 |
| armax | mordred: will do, I’ll file a bug nonethless just to err on the side of caution | 20:56 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config: Realize Zara on the newly created storyboard-dev server https://review.openstack.org/315706 | 20:56 |
| armax | cloudnull: how often does that occur? | 20:56 |
| *** nadya has quit IRC | 20:56 | |
| cloudnull | depends on how lazy im feeling that day | 20:57 |
| clarkb | jesusaur: puppet disabling is done, now to finish up code reviews | 20:57 |
| armax | I don’t recall a race like this ever being tackled but I am getting old fast so my memory is starting to fail me | 20:57 |
| cloudnull | ++ ^ I know how you feel . :) | 20:58 |
| * cloudnull goes to yell at some kids | 20:58 | |
| anteaya | clarkb: 31005 looks like it is missing a digit if it is a patch id | 20:58 |
| armax | cloudnull: do you happen to have the sequence of steps you take to trigger the issue? | 20:58 |
| anteaya | cloudnull: make sure they get off your lawn | 20:58 |
| mordred | armax: https://gist.github.com/cloudnull/2f3931e51062dc7b325faf9a24974006 | 20:58 |
| cloudnull | ^ that | 20:58 |
| clarkb | anteaya: 310051 and 310052 | 20:58 |
| armax | super | 20:59 |
| clarkb | anteaya: my regex would match both | 20:59 |
| *** rbrndt has quit IRC | 20:59 | |
| cloudnull | anteaya: indeed. :) | 20:59 |
| clarkb | anyone else want to review 310051 and 310052 before jesusaur and I switch the logstash workers over to use the logstash filters project? | 20:59 |
| *** kzaitsev_mb has quit IRC | 20:59 | |
| armax | cloudnull: so the create_server does the whole lot: boot and assign fip? | 20:59 |
| *** maishsk has quit IRC | 21:00 | |
| anteaya | clarkb: thanks | 21:00 |
| clarkb | armax: correct, and it assigns the fip using the server's port id | 21:00 |
| anteaya | clarkb: ah I see what you are saying, sorry | 21:00 |
| cloudnull | armax: in shade yes. | 21:00 |
| cloudnull | though it may be able to be broken down. | 21:00 |
| * cloudnull today is my first day looking at the shade internals | 21:00 | |
| armax | cloudnull: that’s ok | 21:01 |
| *** puiterwijk is now known as puiterwijk|cld | 21:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:01 | |
| mordred | the steps in create server are essentially "nova create server ; port list (find server port id) ; neutron floating ip create (port_id)" | 21:01 |
| mordred | then poll waiting for the fip to go active, then poll the nova server until the fip shows up in the server addresses list | 21:01 |
| armax | mordred: yup got it | 21:01 |
| mordred | woot | 21:02 |
| mordred | cloudnull: looking at shade internals is like looking straight into a pit of evil | 21:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add private_key_file configuration for launcher https://review.openstack.org/311639 | 21:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add usernname configuration option for launcher https://review.openstack.org/311725 | 21:02 |
| * clarkb doesn't hear any screaming and will move forward on approving these changes | 21:03 | |
| cloudnull | mordred: its not _SO_ bad | 21:03 |
| mordred | clarkb: I support your decisions wrt ES | 21:03 |
| armax | I am pulling kevinbenton in, as he’s younger and his memory is better than mine | 21:03 |
| *** sdake has joined #openstack-infra | 21:03 | |
| mordred | ah youth | 21:03 |
| *** yolanda has quit IRC | 21:04 | |
| armax | mordred: I know, right? | 21:04 |
| armax | mordred: I think this race sounds mildy familiar and relatively low hanging fruit | 21:04 |
| *** yolanda has joined #openstack-infra | 21:04 | |
| clarkb | jesusaur: logstash-worker20 will be the chosen host, if you want ot watch puppet board once things merge | 21:04 |
| kevinbenton | so what are the API requests being made that race? | 21:04 |
| cloudnull | ***he's going to change the world*** | 21:04 |
| kevinbenton | the association call right? | 21:05 |
| mordred | kevinbenton: create-floating-ip(port_id) | 21:05 |
| anteaya | cloudnull: he's like that | 21:05 |
| jesusaur | clarkb: thanks | 21:05 |
| kevinbenton | mordred: ah, so a create with a specified port id up front | 21:05 |
| mordred | yup | 21:05 |
| armax | kevinbenton: so it’s association | 21:06 |
| *** maishsk has joined #openstack-infra | 21:06 | |
| kevinbenton | mordred, armax: perhaps my memory isn't as good. i thought i saw a patch about this floating around somewhere | 21:08 |
| kevinbenton | (pun not intended) | 21:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Add support for additional launcher config options https://review.openstack.org/315774 | 21:08 |
| armax | kevinbenton: it wouldn’t be difficult to find out whether this has been fixed by looking at the code and logical model | 21:08 |
| openstackgerrit | Matthew Thode proposed openstack/requirements: Add Gentoo to distro status https://review.openstack.org/311490 | 21:09 |
| armax | kevinbenton: I am filing a bug so that we don’t forget | 21:09 |
| kevinbenton | armax: yeah, it doens't look fixed | 21:09 |
| openstackgerrit | Jason Dunsmore proposed openstack/requirements: Update oslo.service to 1.9.0 https://review.openstack.org/315775 | 21:09 |
| *** baoli_ has quit IRC | 21:10 | |
| kevinbenton | mordred: do they even have to be concurrent? | 21:10 |
| cloudnull | So maybe its not my cloud is old and crusty :) | 21:11 |
| mordred | kevinbenton: we've only ever seen the problem occur when doing it concurrently | 21:11 |
| mordred | doing the same thing serially does not seem to break (or at least not frequently enough for us to catch it) | 21:11 |
| armax | cloudnull: you happen to see any trace in the L3 agent for this? | 21:11 |
| cloudnull | no i've not. | 21:12 |
| *** ddieterly[away] is now known as ddieterly | 21:12 | |
| armax | mordred: I would be surprised if we were that broken | 21:12 |
| kevinbenton | mordred: but if you request a floating IP for a port that already has a floating IP, don't you get an error? | 21:12 |
| pabelanger | mordred: clarkb: quick rax question, do we know on average how long ImageTaskCreate takes? First time I have looked into it | 21:12 |
| clarkb | pabelanger: when I did initial investigation forever ago a single one run without other uploads happening took 30-45 minutes | 21:12 |
| clarkb | pabelanger: but I have seen it take many hours | 21:13 |
| mordred | kevinbenton: the opposite thing is what's happening - we're requesting floating ips on two different ports and they're each getting the same floating ip | 21:13 |
| *** lakshmiS has quit IRC | 21:13 | |
| clarkb | pabelanger: its also tricky because shade will silently retry due to known errors that aren't really actionable by the user | 21:13 |
| kevinbenton | mordred: ooooh | 21:13 |
| armax | kevinbenton, mordred: bug 1581220 | 21:13 |
| openstack | bug 1581220 in neutron "same floating ip can be associated to separate ports" [Undecided,New] https://launchpad.net/bugs/1581220 | 21:13 |
| mordred | kevinbenton: look at https://gist.github.com/cloudnull/d90eff7efc524844d8d3504fea8fa419 and look for f1169479-7d88-4526-a60a-82f18b617077 | 21:13 |
| pabelanger | clarkb: Ya, we upload surprisingly fast in DFW, 9mins for 15GB | 21:13 |
| pabelanger | clarkb: but ya, the ImageTaskCreate is now waiting | 21:14 |
| clarkb | pabelanger: oh my numbers were for the total thing, but the task for image create is the one that gets retried | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| kevinbenton | armax, mordred: this doesn't seem possible. the floating IP datamodel has a single column for which port it's associated to | 21:15 |
| *** HeOS has joined #openstack-infra | 21:15 | |
| kevinbenton | armax, mordred: https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L126 | 21:15 |
| kevinbenton | and it's been that way basically since floating IPs were introduced | 21:16 |
| mordred | kevinbenton: yup. each entry has a distinct value there | 21:16 |
| mordred | kevinbenton: but there are two fips with the same id | 21:16 |
| mordred | so I think it IS two different objects - but somehow two objects are getting the same uuid | 21:16 |
| kevinbenton | mordred: oh, sounds like DB retry logic leaked | 21:17 |
| clarkb | mordred: do they have the same ip addr too? | 21:17 |
| mordred | clarkb: no, they don't | 21:17 |
| *** mixos has joined #openstack-infra | 21:18 | |
| mordred | they look like properly formed things in all ways except for sharing the same id | 21:18 |
| openstackgerrit | Merged openstack-infra/puppet-logstash: Properly support logstash conf.d https://review.openstack.org/310051 | 21:18 |
| *** Sukhdev has joined #openstack-infra | 21:18 | |
| armax | kevinbenton: what’s possible is that the last FIP association wins | 21:18 |
| kevinbenton | armax: based on the output in the bug, they looks like quite separate floating IPs | 21:19 |
| *** regXboi has quit IRC | 21:19 | |
| openstackgerrit | Merged openstack-infra/system-config: Use logstash filters from filters repo https://review.openstack.org/310052 | 21:20 |
| kevinbenton | mordred: is that listing a result of a standard floatingip-list call to Neutron directly? | 21:20 |
| armax | kevinbenton: https://gist.github.com/cloudnull/d90eff7efc524844d8d3504fea8fa419#file-fips-data-py-L10 | 21:20 |
| armax | kevinbenton: you looking at the same gist? | 21:21 |
| mordred | kevinbenton: not quite - it's the output of that but then run through a normalization function we have | 21:21 |
| kevinbenton | mordred: ack | 21:21 |
| kevinbenton | armax: yeah, i'm just wondering why the messed up one would even come back from the API | 21:22 |
| mordred | kevinbenton: http://git.openstack.org/cgit/openstack-infra/shade/tree/shade/_utils.py#n287 | 21:22 |
| *** hashar has quit IRC | 21:22 | |
| armax | mordred: I think that from Neutron you’d only see the last FIP association | 21:22 |
| armax | nova may as well get confused and see two | 21:22 |
| mordred | that function takes the output of the standard neutron list floating ips call | 21:23 |
| kevinbenton | mordred: yeah, just ensuring it's not the proxied one through nova | 21:23 |
| openstackgerrit | Jef Mallal proposed openstack-infra/jenkins-job-builder: XUnitDotNetTestType xUnit publisher support https://review.openstack.org/315779 | 21:23 |
| *** sdake_ has joined #openstack-infra | 21:24 | |
| mordred | kevinbenton: yah - shouldn't be that one | 21:24 |
| pabelanger | clarkb: Ya, so increasing upload-workers will have an impact for us, because nodepool blocks that worker waiting for ImageTaskCreate to finish | 21:25 |
| kevinbenton | mordred: because i there were cases where that could become stale because it was based on some cache IIRC | 21:25 |
| pabelanger | clarkb: which is good, because we are done uploading all images about 45mins ago | 21:25 |
| armax | mordred: so just to be clear you’d the the same FIP associated to two separate servers using the Nova API | 21:26 |
| clarkb | jesusaur: we shoul dsee if this is drop in replacement in the next 25 minutes or so | 21:26 |
| mordred | kevinbenton: oh yeah - we have tons of fun with stale info | 21:26 |
| clarkb | er 20 minutes | 21:26 |
| pabelanger | with that in mind, I'm happy to bump our image-workers to 16: https://review.openstack.org/#/c/315727/ | 21:26 |
| mordred | armax: nope. we have not used the Nova API for any of this | 21:26 |
| pabelanger | s/image-/upload- | 21:26 |
| mordred | it's all neutron | 21:26 |
| *** baumann has quit IRC | 21:26 | |
| *** thorst_ has quit IRC | 21:27 | |
| jesusaur | clarkb: yeah, I just noticed that it merged, but wasn't picked up by the puppet run a couple minutes ago | 21:27 |
| armax | mordred: ok, but say after you booted the servers and went to see the port associated to the offending FIP | 21:27 |
| *** thorst_ has joined #openstack-infra | 21:27 | |
| mordred | yah - sorry - was unclear ... | 21:27 |
| armax | mordred: any trace of one association would be gone | 21:28 |
| armax | mordred: if not in the server object that has the same FIP as another server | 21:28 |
| mordred | we boot the server with nova. then we do the fip stuff all with neutron. then once the fip reports active we go backto nova and do a nova server get to get a refreshed copy of the server dict from it | 21:28 |
| mordred | armax: yah. which would then explain why nova does not see a fip assocated with the server | 21:28 |
| *** sdake has quit IRC | 21:28 | |
| mordred | and the overall process fails, due to nova not knowing about it | 21:29 |
| armax | mordred: because connection to it fails, right? | 21:29 |
| *** jerryz has joined #openstack-infra | 21:29 | |
| mordred | no - we actively report it as an error if we can't get a nova server record with the ip information attached | 21:29 |
| openstackgerrit | Jef Mallal proposed openstack-infra/jenkins-job-builder: XUnitDotNetTestType xUnit publisher support https://review.openstack.org/315779 | 21:29 |
| mordred | we've been polling nova for an hour, btw, and it never shows up in these cases | 21:30 |
| mordred | which I think is consistent with what you're saying | 21:30 |
| kevinbenton | mordred: but it always shows up from neutron, right? | 21:30 |
| armax | mordred: ok, I sense that it’s plausible under the existing code that two FIP association requests interfere, the last one wins and everything goes out the window | 21:30 |
| *** sdake has joined #openstack-infra | 21:31 | |
| anteaya | so has anyone ever seen folks introduce a question with 'long complicated question' or something similar? I ask because I see people use the phrase 'quick question' before they ask a question and it usually isn't | 21:31 |
| anteaya | this isn't related to anything, just wondering | 21:31 |
| *** thorst_ has quit IRC | 21:31 | |
| mordred | kevinbenton: yes | 21:32 |
| *** sdake_ has quit IRC | 21:32 | |
| cloudnull | anteaya: Real quick, I have a question :) | 21:32 |
| anteaya | cloudnull: I bet you do | 21:32 |
| anteaya | how fast can you say it | 21:32 |
| armax | kevinbenton: it’s possible that the two requests interleave when it’s time to check if the FIP is associated already | 21:32 |
| *** thorst_ has joined #openstack-infra | 21:33 | |
| kevinbenton | armax: a FIP ID is a primary key so it's not dupable, so def same IP. and it's associated fixed_port_id is a single column entry | 21:33 |
| armax | kevinbenton: anyway I need more time to think about this, I am rambling | 21:33 |
| *** dtantsur|afk has quit IRC | 21:33 | |
| kevinbenton | armax: i mean it must be one FIP record | 21:34 |
| *** csaba has quit IRC | 21:35 | |
| *** matt-borland has quit IRC | 21:35 | |
| *** csaba has joined #openstack-infra | 21:35 | |
| *** dtantsur has joined #openstack-infra | 21:35 | |
| *** thorst_ has quit IRC | 21:36 | |
| *** thorst_ has joined #openstack-infra | 21:36 | |
| armax | kevinbenton: right, something else must be happen where two requests get hold of the same FIP record | 21:36 |
| armax | kevinbenton: I would need to take the time to go through shade to see the exact sequence of steps involved to see what’s truly happening | 21:37 |
| *** ffrank has joined #openstack-infra | 21:37 | |
| mordred | armax, kevinbenton but it looks like two different records, no? | 21:37 |
| mordred | becuae there are two different floating_ip_address values | 21:37 |
| armax | the FIP uuid is prmary key on the FIP table | 21:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add zuul launcher hosts https://review.openstack.org/315781 | 21:38 |
| kevinbenton | mordred: https://gist.github.com/cloudnull/d90eff7efc524844d8d3504fea8fa419#file-fips-data-py-L10-L11 | 21:38 |
| mordred | armax: wow | 21:38 |
| kevinbenton | mordred: same floating address | 21:38 |
| armax | mordred: the floating_ip_address is the same | 21:38 |
| mordred | OH | 21:38 |
| kevinbenton | mordred: only difference is which port is associated | 21:38 |
| mordred | it is the same address | 21:38 |
| kevinbenton | mordred: so it smells like stale data someshow | 21:38 |
| *** ddieterly is now known as ddieterly[away] | 21:38 | |
| kevinbenton | mordred: and we don't cache queries or anything like that on neutron | 21:39 |
| mordred | we do cache some things - but we cache the entire list output, not individual bits | 21:40 |
| mordred | but I'll also go through our stuff and make sure we're not double-writing something | 21:40 |
| clarkb | jesusaur: ok logstash worker20 was updated but the logstash daemon wasn't running when I logged in | 21:40 |
| clarkb | jesusaur: and if I restart it dies, investigating but hunch is we need to delete the old conf.d file | 21:41 |
| *** ddieterly[away] has quit IRC | 21:41 | |
| kevinbenton | mordred: you don't happen to have the neutron server debug logs for one of these runs do you? | 21:41 |
| clarkb | jesusaur: complaining about addr already in use possibly because we have two identical inputs now? I am going to remove that file manually and try restarting | 21:41 |
| jesusaur | clarkb: oh, hm | 21:42 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Add oslo.tools repo/project https://review.openstack.org/314342 | 21:42 |
| clarkb | jesusaur: that seemed to be it, so in your if else to handle backward compat I think we can add an ensure absent for the old file in the else block | 21:43 |
| mordred | kevinbenton: I do not - cloudnull might be able to help (this is running aginst the OSIC cloud) | 21:43 |
| jesusaur | clarkb: ya, i'll write that up now | 21:43 |
| *** doug-fish has joined #openstack-infra | 21:43 | |
| kevinbenton | cloudnull: only if it's really convenient | 21:43 |
| cloudnull | kevinbenton: i dont have debug logs for it. I could go turn debug on and capture some things. | 21:44 |
| cloudnull | its kinda a pain, but i can make it happen if you think itll help | 21:44 |
| *** kjorgensen has joined #openstack-infra | 21:45 | |
| *** brucet has joined #openstack-infra | 21:45 | |
| *** gordc has quit IRC | 21:45 | |
| kevinbenton | cloudnull: ah, don't worry about it yet | 21:45 |
| cloudnull | ok. you let me know though | 21:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: Support configurable site roots for ftp/scp https://review.openstack.org/315784 | 21:46 |
| *** earlephilhower has quit IRC | 21:46 | |
| openstackgerrit | Merged openstack/requirements: Add networking-generic-switch to project list https://review.openstack.org/307619 | 21:47 |
| openstackgerrit | Merged openstack/requirements: Add python-monascaclient to the global requirements https://review.openstack.org/251674 | 21:47 |
| *** bswartz has joined #openstack-infra | 21:48 | |
| *** doug-fish has quit IRC | 21:48 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-logstash: Remove old config when switching to split configs https://review.openstack.org/315785 | 21:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Exercise the launcher in beaker tests https://review.openstack.org/310228 | 21:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Add support for additional launcher config options https://review.openstack.org/315774 | 21:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Add support for zuul launcher https://review.openstack.org/310205 | 21:48 |
| jesusaur | clarkb: https://review.openstack.org/315785 | 21:48 |
| clarkb | jesusaur: yup reviewing | 21:48 |
| clarkb | jesusaur: it is what I expected, I have approved it and we can have puppet run on logstash-worker19, but that should be it and we can deploy it across the board | 21:49 |
| *** dprince has joined #openstack-infra | 21:49 | |
| *** matrohon has quit IRC | 21:49 | |
| *** smarcet has quit IRC | 21:49 | |
| *** maishsk has quit IRC | 21:50 | |
| jesusaur | clarkb: ++ | 21:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Install extra packages for ansible launcher https://review.openstack.org/315787 | 21:50 |
| kevinbenton | mordred: is this following the _neutron_create_floating_ip path in shade? | 21:51 |
| openstackgerrit | Jef Mallal proposed openstack-infra/jenkins-job-builder: XUnitDotNetTestType xUnit publisher support https://review.openstack.org/315779 | 21:52 |
| kevinbenton | mordred: or is it _neutron_attach_ip_to_server ? | 21:52 |
| jeblair | mordred, clarkb, pabelanger, fungi, Shrews, morgan, jhesketh: i think "topic:v25" is ready, and i'd like to try deploying a zuul server tomorrow [we said we would just use a jenkins server during the summit, but i think now that we are not under that pressure, a new server makes sense] | 21:53 |
| *** jang has quit IRC | 21:53 | |
| morgan | Makes sense. | 21:54 |
| *** earlephilhower has joined #openstack-infra | 21:55 | |
| pabelanger | jeblair: exciting | 21:57 |
| pabelanger | jeblair: we should be able to use the current jenkins credentials too, that's what I did when I was testing locally. | 21:58 |
| *** rkukura has quit IRC | 21:58 | |
| pabelanger | need to finish up the zuul-slave DIB element too | 21:58 |
| jeblair | pabelanger: yeah, that's the way i have staged everything; we can change them once all of jenkins is gone | 21:58 |
| jeblair | pabelanger: ooh, that'll be handy, thanks | 21:59 |
| *** thorst_ has quit IRC | 21:59 | |
| *** asilenkov_ has quit IRC | 21:59 | |
| pabelanger | jeblair: we might need libselinux-python for our centos-7 / fedora images. However, I was running into that issue when ansible tried to do yum stuff | 22:00 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:00 | |
| *** dmk0202 has quit IRC | 22:00 | |
| pabelanger | or we limit our initial testing to ubuntu | 22:01 |
| pabelanger | either way, exciting! | 22:01 |
| jeblair | pabelanger: i don't follow the selinux train of thought. | 22:01 |
| jeblair | pabelanger: and we have no choice about what we run this on -- it's all or nothing | 22:01 |
| pabelanger | jeblair: let me see if I can find a log file explaining the issue | 22:02 |
| *** thorst_ has joined #openstack-infra | 22:05 | |
| *** thorst_ has quit IRC | 22:05 | |
| *** thorst_ has joined #openstack-infra | 22:06 | |
| *** ccamacho has quit IRC | 22:06 | |
| *** brucet has quit IRC | 22:06 | |
| clarkb | did kolla ever get back to us on their potential selinux issue? | 22:06 |
| *** brucet has joined #openstack-infra | 22:07 | |
| *** Sukhdev has quit IRC | 22:07 | |
| lbragstad | does anyone know off-hand what version of mysql we're running in the gate? | 22:07 |
| clarkb | lbragstad: its captured in the logs, | 22:08 |
| clarkb | lbragstad: it is whatever version is available for that particular distro release | 22:08 |
| *** ccamacho has joined #openstack-infra | 22:09 | |
| *** jsavak has quit IRC | 22:09 | |
| clarkb | I lbragstad in the case of ubuntu trusty it is 5.5 based on a quick seach on my local machine | 22:09 |
| lbragstad | clarkb awesome - i assume it's in the mysql logs? | 22:10 |
| lbragstad | clarkb specifically looking here - http://logs.openstack.org/30/314330/3/check/gate-tempest-dsvm-neutron-dvr/3d9272f/logs/mysql.txt.gz | 22:10 |
| lbragstad | and here - http://logs.openstack.org/30/314330/3/check/gate-tempest-dsvm-neutron-dvr/3d9272f/logs/mysql_err.txt.gz | 22:10 |
| clarkb | lbragstad: no, for unittests it is in the console log when we install the packages, and for devstack type things it iwll be in the devstakc log and in the dpkg.log file | 22:10 |
| clarkb | lbragstad: http://logs.openstack.org/30/314330/3/check/gate-tempest-dsvm-neutron-dvr/3d9272f/logs/dpkg-l.txt.gz | 22:11 |
| lbragstad | clarkb perfect - looks like 5.5.49-0ubuntu0.14.04.1 | 22:11 |
| lbragstad | clarkb thanks! | 22:11 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-bandersnatch: Ignore "bandersnatch mirror" non-zero return status https://review.openstack.org/315790 | 22:12 |
| *** dimtruck is now known as zz_dimtruck | 22:12 | |
| *** mixos has quit IRC | 22:13 | |
| *** aysyd has quit IRC | 22:13 | |
| *** tosky has left #openstack-infra | 22:14 | |
| *** vhosakot has quit IRC | 22:15 | |
| pabelanger | jeblair: Oh, I think I fixed it by installing it into my DIB. | 22:15 |
| pabelanger | jeblair: but I can test it again | 22:15 |
| pabelanger | jeblair: this was my last error, that I have debugged yet: http://paste.openstack.org/show/496992/ | 22:15 |
| *** thorst_ has quit IRC | 22:16 | |
| pabelanger | jeblair: failure in zuul_runner task | 22:16 |
| jeblair | pabelanger: for selinux, i don't understand how this will be any different than what jenkins is doing today | 22:16 |
| *** brucet has quit IRC | 22:16 | |
| jeblair | pabelanger: that paste looks like a bug in the callback module | 22:17 |
| pabelanger | jeblair: ansible will try to setup security contexts on some files, which it need python-libselinux for | 22:17 |
| pabelanger | jeblair: without it, ansible bombs and says install the missing package | 22:17 |
| kevinbenton | cloudnull, mordred: added a couple of comments to the bug report. can you reply there? https://bugs.launchpad.net/neutron/+bug/1581220 | 22:17 |
| openstack | Launchpad bug 1581220 in neutron "race leads same floating ip to be associated to separate ports" [High,Confirmed] | 22:17 |
| pabelanger | err libselinux-python | 22:18 |
| jeblair | pabelanger: interesting; it seems like ansible ought to take care of copying that over itself | 22:18 |
| pabelanger | jeblair: http://docs.ansible.com/ansible/intro_installation.html#managed-node-requirements | 22:18 |
| jeblair | pabelanger: well, i guess that covers it. :) | 22:19 |
| jeblair | "at least it's documented" | 22:19 |
| cloudnull | kevinbenton: will do | 22:19 |
| pabelanger | jeblair: Ya, so maybe we need a bootstrap playbook for zuul to add dependencies like this | 22:19 |
| pabelanger | or a way to call a ready script like we do in nodepool | 22:19 |
| jeblair | pabelanger: i think putting it in the dib makes sense; sounds like we should go ahead and put that in now. | 22:19 |
| pabelanger | jeblair: sorry, I have to run (family) but interested in continue the discussion | 22:19 |
| pabelanger | jeblair: Ya, that's what I did to fix it | 22:19 |
| clarkb | odd that ansible doesn't copy that over and add it to its run path | 22:20 |
| *** brucet_ has joined #openstack-infra | 22:22 | |
| *** thorst_ has joined #openstack-infra | 22:22 | |
| *** vhosakot has joined #openstack-infra | 22:22 | |
| *** jpr has quit IRC | 22:22 | |
| *** andymaier has joined #openstack-infra | 22:22 | |
| openstackgerrit | Morgan Fainberg proposed openstack/requirements: Add pyldap to g-r https://review.openstack.org/315793 | 22:23 |
| *** kzaitsev_mb has quit IRC | 22:24 | |
| *** lakshmiS has joined #openstack-infra | 22:24 | |
| *** jamielennox|away is now known as jamielennox | 22:24 | |
| *** ningy__ has quit IRC | 22:24 | |
| *** ayoung has joined #openstack-infra | 22:25 | |
| openstackgerrit | Morgan Fainberg proposed openstack/requirements: Add ldappool to projects.txt for g-r updates/checking https://review.openstack.org/315794 | 22:25 |
| *** markvoelker_ has quit IRC | 22:25 | |
| *** thorst_ has quit IRC | 22:27 | |
| *** tpsilva has quit IRC | 22:27 | |
| openstackgerrit | Morgan Fainberg proposed openstack/requirements: Add pyldap to g-r https://review.openstack.org/315793 | 22:28 |
| *** brucet_ has quit IRC | 22:30 | |
| clarkb | jesusaur: we are just going to miss this puppet run, will have to be the next one | 22:30 |
| jesusaur | clarkb: :( | 22:30 |
| openstackgerrit | Merged openstack-infra/puppet-logstash: Remove old config when switching to split configs https://review.openstack.org/315785 | 22:30 |
| *** brucet_ has joined #openstack-infra | 22:30 | |
| jesusaur | clarkb: did you start puppet back up already? | 22:31 |
| clarkb | jesusaur: not on 19, will get it once the run is properly started | 22:31 |
| clarkb | otherwise there is racyness between modules being updated and when the hosts lists for ansible is generated (I think we still do those as separate steps) | 22:32 |
| *** weshay has quit IRC | 22:33 | |
| jesusaur | oh right, I forgot that puppet is triggered by ansible | 22:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Remove openstack/requirements from #-infra https://review.openstack.org/315798 | 22:34 |
| jeblair | for discussion ^ :) | 22:34 |
| *** mriedem has quit IRC | 22:34 | |
| clarkb | discussion we should just approve :) | 22:34 |
| *** mriedem has joined #openstack-infra | 22:34 | |
| *** mriedem has quit IRC | 22:35 | |
| jeblair | good discussion | 22:35 |
| *** jerryz has quit IRC | 22:37 | |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job https://review.openstack.org/314795 | 22:38 |
| *** MarkAtwood has quit IRC | 22:38 | |
| cloudnull | kevinbenton: i've updated the LP issue with a bit more data. | 22:38 |
| clarkb | we can continue to help with emergencies but I don't think we need to alert this channel of every requirements update | 22:38 |
| cloudnull | let me know if you need anything else or want me to poke something with a stick | 22:38 |
| clarkb | jesusaur: 19 is removed fro mthe exclusion list and should get updated at the 45 minute run | 22:39 |
| *** Qiming_ has joined #openstack-infra | 22:39 | |
| *** brucet_ has quit IRC | 22:39 | |
| *** kfarr has quit IRC | 22:41 | |
| *** amitgandhinz has quit IRC | 22:44 | |
| *** asselin has joined #openstack-infra | 22:44 | |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job https://review.openstack.org/314795 | 22:45 |
| *** pahuang has joined #openstack-infra | 22:47 | |
| *** bpokorny_ has joined #openstack-infra | 22:47 | |
| *** hongbin has quit IRC | 22:47 | |
| *** vhosakot has quit IRC | 22:49 | |
| pabelanger | jeblair: does this look correct for zuul-launcher playbook? http://paste.openstack.org/show/496995/ | 22:49 |
| *** bpokorny has quit IRC | 22:51 | |
| zaro | jeblair: sorry, i got held up a bit this week with gerrit gc investigation | 22:52 |
| kevinbenton | cloudnull: but once it's in a broken state can you actually see if neutron floatingip-list returns duplicates? | 22:52 |
| jeblair | pabelanger: heh, wow, i've never actually looked at the output :) | 22:52 |
| kevinbenton | cloudnull: i want to rule out some race condition interally in shade or the neutron client as shade is using it | 22:52 |
| zaro | jeblair: i just imported the projects to review-dev for testing. running into a bit of an issue with the disk space. seems like it's really low on space now. | 22:53 |
| pabelanger | jeblair: Ya I am thinking the !!python/unicode is not correct | 22:53 |
| clarkb | jesusaur: that worked, I am going to let puppet run on al of them shortly | 22:54 |
| jeblair | pabelanger: yeah, probably not necessary. | 22:54 |
| jesusaur | clarkb: \o/ | 22:55 |
| zaro | clarkb, jeblair :what is the process to add more disk space to review-dev? | 22:57 |
| *** dprince has quit IRC | 22:57 | |
| *** dprince has joined #openstack-infra | 22:57 | |
| *** jcooley has joined #openstack-infra | 22:58 | |
| *** thiagop has quit IRC | 22:59 | |
| zaro | or maybe clean up files to clear up space? | 22:59 |
| clarkb | zaro: we don't seem to be space constrained on that host | 23:00 |
| nikhil | fungi: clarkb : hi, we are stuck on these patches https://review.openstack.org/#/c/315232/ as the #openstack-quota channel has lost the only op it had. Is there a way to get back an op for this channel or somehow fix the chanserv issue? (I did not create/op that channel) | 23:00 |
| *** lakshmiS has quit IRC | 23:01 | |
| clarkb | nikhil: if you get everyone to exit the channel then you can register it properly (assuming it isn't already registered) | 23:01 |
| clarkb | if it is already registered then you should talk to the person that has registered it | 23:01 |
| nikhil | clarkb: it's not registered. so that's the only way? | 23:01 |
| clarkb | if you don't have a founder present I think so | 23:01 |
| zaro | clarkb: i am trying to test performance of gerrit gc on nova project on review-dev. i've got the project there and am trying to clone it locally. when I clone i get "No space left on device" error | 23:01 |
| *** sdague has quit IRC | 23:01 | |
| nikhil | clarkb: the founder is present but he lost the op (possibly after reconnect) :) | 23:02 |
| clarkb | zaro: which file path, / has 13GB and /opt has 33GB | 23:02 |
| zaro | i'm cloning kdtest/nova-no-gc project into my home folder. | 23:03 |
| zaro | i blieve that's / | 23:03 |
| zaro | here's the error: Cloning into 'nova-no-gc'... fatal: failed to copy file to 'nova-no-gc/.git/objects/85/d6ae81b95f6baa56f41c5a13d57c4e44463d0d': No space left on device | 23:03 |
| *** ayoung has quit IRC | 23:03 | |
| clarkb | what command I can try to reproduce | 23:04 |
| clarkb | is kdtest/nova-no-gc hosted by gerrit? | 23:04 |
| zaro | clarkb: git clone /home/gerrit2/review_site/git/kdtest/nova-no-gc.git | 23:04 |
| zaro | clarkb: yes, i imported that project | 23:05 |
| *** sridhar_ram has quit IRC | 23:05 | |
| *** sridhar_ram has joined #openstack-infra | 23:06 | |
| zaro | i was able to clone it onto my local machine | 23:06 |
| *** xyang1 has quit IRC | 23:06 | |
| *** Sukhdev has joined #openstack-infra | 23:09 | |
| *** tonytan4ever has quit IRC | 23:09 | |
| *** |-paul-| has joined #openstack-infra | 23:11 | |
| clarkb | zaro: it is cloning for me so far but down to les than 7GB disk now, lets see if it will complete | 23:11 |
| clarkb | zaro: in any case you can probably use /opt if / doesn't have enough. We can't really expand / anyways and can only attach volumes that are mounted elsewhere | 23:11 |
| clarkb | zaro: it failed for me, how big is that repo? | 23:12 |
| zaro | clarkb: i believe each clone is 13G | 23:12 |
| * clarkb runs some du | 23:12 | |
| clarkb | du says 7GB | 23:12 |
| *** gildub has joined #openstack-infra | 23:12 | |
| *** asselin has quit IRC | 23:13 | |
| *** mjblack has joined #openstack-infra | 23:13 | |
| *** andymaier has quit IRC | 23:16 | |
| zaro | hmm, after clone onto my local machine du says only 672M for me. not sure why the error happens. | 23:17 |
| clarkb | it may need extra tmp space to verify files or something | 23:17 |
| *** HeOS has quit IRC | 23:18 | |
| openstackgerrit | Merged openstack/requirements: Bump pyghmi version to 1.0.3 https://review.openstack.org/315161 | 23:19 |
| openstackgerrit | Merged openstack/requirements: Only install unicodecsv on Python < 3.0 https://review.openstack.org/310650 | 23:19 |
| zaro | clarkb: I guess it'll probably still fail if i attempt to clone to /opt? since /tmp is the thing that clone is having issues with? | 23:20 |
| clarkb | zaro: it may not use /tmp. I would try using /opt as it may use the local dir | 23:21 |
| clarkb | if it is /tmp being used then ya would be the same | 23:21 |
| zaro | ok. giving it a try now | 23:21 |
| *** yuanying_ has joined #openstack-infra | 23:22 | |
| *** r-mibu has quit IRC | 23:23 | |
| jlvillal | Why does Gerrit warn about messages over 70 characters when the docs says 72 characters is the width to use. | 23:23 |
| *** yuanying has quit IRC | 23:23 | |
| jlvillal | https://wiki.openstack.org/wiki/GitCommitMessages#Summary_of_Git_commit_message_structure | 23:24 |
| *** Swami has quit IRC | 23:24 | |
| jlvillal | remote: (W) 86186d1: too many commit message lines longer than 70 characters; manually wrap lines | 23:24 |
| zaro | jlvillal: we probably just need to adjust that. it's just a warning correct? | 23:25 |
| jlvillal | zaro: Correct. It did not reject | 23:25 |
| *** dingyichen has joined #openstack-infra | 23:25 | |
| *** markvoelker has joined #openstack-infra | 23:26 | |
| zaro | jlvillal: thanks for letting us know. i'll make a note to either adjust the gerrit param (or the docs :) | 23:26 |
| jlvillal | zaro: I think 72 is a well known value. The default that VIM uses. | 23:26 |
| jlvillal | 80 - 8 (tab) = 72 | 23:27 |
| *** r-daneel has quit IRC | 23:27 | |
| zaro | jvrbanac: sounds good to me. | 23:28 |
| clarkb | https://code.google.com/p/gerrit/issues/detail?id=841 | 23:28 |
| clarkb | glhf | 23:28 |
| *** bpokorny_ has quit IRC | 23:28 | |
| zaro | oops that was meant for jlvillal | 23:28 |
| *** j_king has quit IRC | 23:29 | |
| jlvillal | zaro: :) | 23:29 |
| *** bpokorny has joined #openstack-infra | 23:29 | |
| clarkb | that bug is more than 5 years old | 23:29 |
| *** j_king has joined #openstack-infra | 23:29 | |
| jlvillal | clarkb: Thanks :) | 23:30 |
| zaro | haha, i asume the bug wanted 72 to be the default | 23:30 |
| *** mahatic has quit IRC | 23:31 | |
| *** markvoelker has quit IRC | 23:31 | |
| *** claudiub has quit IRC | 23:31 | |
| *** mahatic has joined #openstack-infra | 23:32 | |
| clarkb | jesusaur: at this point it may make sense sending mail to jj and or the ops list and say hey its a thing please to use it | 23:32 |
| zaro | clarkb: ok, it worked in /opt thanks | 23:33 |
| *** chlong has quit IRC | 23:35 | |
| *** liusheng has quit IRC | 23:36 | |
| *** liusheng has joined #openstack-infra | 23:36 | |
| jlvillal | clarkb: zaro http://paste.openstack.org/show/496999/ | 23:37 |
| jlvillal | https://gerrit-review.googlesource.com/Documentation/config-plugins.html#commit-message-length-validator | 23:37 |
| jlvillal | I'm guessing OpenStack Gerrit is running that plugin | 23:37 |
| *** johnny_ has quit IRC | 23:38 | |
| *** cbader_ has quit IRC | 23:39 | |
| *** amitgandhinz has joined #openstack-infra | 23:45 | |
| *** r-mibu has joined #openstack-infra | 23:45 | |
| *** mriedem has joined #openstack-infra | 23:46 | |
| *** brucet_ has joined #openstack-infra | 23:47 | |
| *** Sukhdev has quit IRC | 23:48 | |
| *** amitgandhinz has quit IRC | 23:49 | |
| *** asettle has joined #openstack-infra | 23:56 | |
| *** SumitNaiksatam has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
