| *** salv-orlando has quit IRC | 00:01 | |
| *** yamahata has joined #openstack-infra | 00:04 | |
| *** kzaitsev_mb has joined #openstack-infra | 00:06 | |
| *** gardlt has quit IRC | 00:07 | |
| *** esikache1 has joined #openstack-infra | 00:08 | |
| *** marst_ has quit IRC | 00:09 | |
| *** masber has joined #openstack-infra | 00:09 | |
| *** kzaitsev_mb has quit IRC | 00:11 | |
| *** SumitNaiksatam has quit IRC | 00:12 | |
| *** esikache1 has quit IRC | 00:12 | |
| *** sflanigan has joined #openstack-infra | 00:12 | |
| *** felipemonteiro has quit IRC | 00:13 | |
| *** geekg_000_ has joined #openstack-infra | 00:14 | |
| mordred | clarkb: cool. I have submitted patch to release repo and given release team a headsup | 00:16 |
|---|---|---|
| *** sree has joined #openstack-infra | 00:16 | |
| *** sree has quit IRC | 00:20 | |
| *** esikache1 has joined #openstack-infra | 00:21 | |
| *** gyee has quit IRC | 00:22 | |
| *** hongbin has quit IRC | 00:22 | |
| *** s-shiono has joined #openstack-infra | 00:25 | |
| *** gcb has quit IRC | 00:26 | |
| *** esikache1 has quit IRC | 00:26 | |
| *** numans has quit IRC | 00:27 | |
| *** ijw has quit IRC | 00:28 | |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: WIP: Add pagination for the list_volumes call https://review.openstack.org/466927 | 00:29 |
| *** rlandy has quit IRC | 00:30 | |
| *** esberglu has quit IRC | 00:30 | |
| *** esberglu has joined #openstack-infra | 00:30 | |
| *** esberglu has quit IRC | 00:31 | |
| *** numans has joined #openstack-infra | 00:32 | |
| *** yamahata has quit IRC | 00:33 | |
| *** esikache1 has joined #openstack-infra | 00:35 | |
| *** jamesmca_ has joined #openstack-infra | 00:38 | |
| *** esikache1 has quit IRC | 00:40 | |
| *** feisky has joined #openstack-infra | 00:42 | |
| *** esikache1 has joined #openstack-infra | 00:51 | |
| *** vhosakot_ has quit IRC | 00:55 | |
| *** esikache1 has quit IRC | 00:56 | |
| *** nicolasbock has joined #openstack-infra | 00:56 | |
| *** gildub has joined #openstack-infra | 00:57 | |
| *** Julien-zte has joined #openstack-infra | 00:57 | |
| *** esikache1 has joined #openstack-infra | 00:57 | |
| *** esikache1 has quit IRC | 01:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Gather facts for run-bindep role https://review.openstack.org/466918 | 01:02 |
| *** ijw has joined #openstack-infra | 01:02 | |
| *** esikache1 has joined #openstack-infra | 01:05 | |
| *** dimtruck is now known as zz_dimtruck | 01:09 | |
| *** esikache1 has quit IRC | 01:10 | |
| *** vhosakot has joined #openstack-infra | 01:13 | |
| *** jamesmca_ has quit IRC | 01:14 | |
| *** dprince has joined #openstack-infra | 01:14 | |
| *** jamesmca_ has joined #openstack-infra | 01:20 | |
| *** jamesmca_ has quit IRC | 01:20 | |
| *** jamesmca_ has joined #openstack-infra | 01:20 | |
| *** gildub has quit IRC | 01:22 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Adding unit testing for configuration https://review.openstack.org/466959 | 01:25 |
| *** gildub has joined #openstack-infra | 01:25 | |
| *** thorst_afk has joined #openstack-infra | 01:29 | |
| *** thorst_afk has quit IRC | 01:30 | |
| *** yolkfull has joined #openstack-infra | 01:31 | |
| *** dave-mccowan has quit IRC | 01:36 | |
| *** vhosakot has quit IRC | 01:37 | |
| *** dprince has quit IRC | 01:39 | |
| *** ijw has quit IRC | 01:41 | |
| sbezverk | sdake: ping | 01:41 |
| *** sree has joined #openstack-infra | 01:53 | |
| *** jamielennox is now known as jamielennox|away | 01:54 | |
| *** jamesmca_ has quit IRC | 01:55 | |
| *** zz_dimtruck is now known as dimtruck | 01:56 | |
| *** lamt has joined #openstack-infra | 01:57 | |
| *** sree has quit IRC | 01:57 | |
| *** links has joined #openstack-infra | 01:58 | |
| *** mriedem_afk is now known as mriedem | 01:59 | |
| *** bobh has joined #openstack-infra | 02:01 | |
| *** esikache1 has joined #openstack-infra | 02:02 | |
| *** gcb has joined #openstack-infra | 02:04 | |
| *** gongysh has joined #openstack-infra | 02:05 | |
| *** esikache1 has quit IRC | 02:06 | |
| *** dprince has joined #openstack-infra | 02:06 | |
| *** hongbin has joined #openstack-infra | 02:06 | |
| *** kzaitsev_mb has joined #openstack-infra | 02:07 | |
| *** rfolco has joined #openstack-infra | 02:08 | |
| *** rfolco has quit IRC | 02:08 | |
| *** dprince has quit IRC | 02:09 | |
| *** kzaitsev_mb has quit IRC | 02:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Adding unit testing for configuration https://review.openstack.org/466959 | 02:15 |
| *** bobh has quit IRC | 02:21 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:25 | |
| *** mdrabe has quit IRC | 02:32 | |
| *** yamahata has joined #openstack-infra | 02:34 | |
| *** esikache1 has joined #openstack-infra | 02:38 | |
| *** xinliang has quit IRC | 02:40 | |
| *** esikache1 has quit IRC | 02:43 | |
| *** ihrachys has quit IRC | 02:44 | |
| *** hongbin_ has joined #openstack-infra | 02:48 | |
| *** lamt has quit IRC | 02:49 | |
| *** esikache1 has joined #openstack-infra | 02:50 | |
| *** xinliang has joined #openstack-infra | 02:52 | |
| *** xinliang has quit IRC | 02:52 | |
| *** xinliang has joined #openstack-infra | 02:52 | |
| *** bobh has joined #openstack-infra | 02:52 | |
| *** geekg_000_ has quit IRC | 02:54 | |
| *** esikache1 has quit IRC | 02:54 | |
| *** hongbin has quit IRC | 02:55 | |
| *** jamesmca_ has joined #openstack-infra | 02:55 | |
| *** hongbin_ has quit IRC | 02:56 | |
| *** hongbin has joined #openstack-infra | 02:56 | |
| *** lamt has joined #openstack-infra | 02:56 | |
| *** esikache1 has joined #openstack-infra | 03:00 | |
| *** esikache1 has quit IRC | 03:06 | |
| *** baoli has joined #openstack-infra | 03:06 | |
| *** adam_g has quit IRC | 03:07 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:10 | |
| *** bobh has quit IRC | 03:10 | |
| *** adam_g has joined #openstack-infra | 03:11 | |
| *** baoli has quit IRC | 03:11 | |
| *** ijw has joined #openstack-infra | 03:12 | |
| *** kzaitsev_mb has quit IRC | 03:16 | |
| *** ijw has quit IRC | 03:17 | |
| *** Apoorva has joined #openstack-infra | 03:18 | |
| *** jamesmca_ has quit IRC | 03:18 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-dev/cookiecutter master: Usage: install cookiecutter firstly https://review.openstack.org/466996 | 03:23 |
| adriant | Anyway around who can help me with a tempest failure due to this patch: https://github.com/openstack/oslo.i18n/commit/0658992b0b9bb8a9ad71ddf962d89d3ede27b95e | 03:24 |
| adriant | this is the error we're getting: http://paste.openstack.org/show/610397/ | 03:24 |
| adriant | something in the tempest plugin discovery ends up triggering that but because that oslo lib no longer has babal as a runtime requirement it falls over as it was never installed. | 03:25 |
| adriant | this import right here: https://github.com/openstack/oslo.i18n/blob/master/oslo_i18n/_gettextutils.py#L24 | 03:26 |
| *** esikache1 has joined #openstack-infra | 03:27 | |
| *** adam_g has quit IRC | 03:28 | |
| *** adam_g has joined #openstack-infra | 03:31 | |
| *** esikache1 has quit IRC | 03:32 | |
| *** vikrant has joined #openstack-infra | 03:34 | |
| *** csomerville has quit IRC | 03:36 | |
| *** sflanigan is now known as sflanigan_afk | 03:37 | |
| *** cody-somerville has joined #openstack-infra | 03:37 | |
| *** lamt has quit IRC | 03:41 | |
| *** esikache1 has joined #openstack-infra | 03:41 | |
| *** adam_g has quit IRC | 03:43 | |
| *** adam_g has joined #openstack-infra | 03:44 | |
| adriant | after a bit more digging it looks like the older version of tempest we are running executes a code path that calls that module. | 03:44 |
| adriant | and since it doesn't have a cap on the requirements file it pulls in the newest osli.i18n package and breaks... | 03:45 |
| *** esikache1 has quit IRC | 03:45 | |
| *** lamt has joined #openstack-infra | 03:46 | |
| *** ykarel has joined #openstack-infra | 03:46 | |
| *** psachin has joined #openstack-infra | 03:48 | |
| *** esikache1 has joined #openstack-infra | 03:49 | |
| *** sflanigan_afk is now known as sflanigan | 03:49 | |
| *** mriedem has quit IRC | 03:50 | |
| *** esikache1 has quit IRC | 03:53 | |
| *** geekg_000_ has joined #openstack-infra | 04:01 | |
| *** geekg_000_ has quit IRC | 04:01 | |
| *** esikache1 has joined #openstack-infra | 04:01 | |
| *** esikache1 has quit IRC | 04:06 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:13 | |
| *** nicolasbock has quit IRC | 04:13 | |
| *** MatthewLi has joined #openstack-infra | 04:14 | |
| *** david-lyle has quit IRC | 04:14 | |
| *** esikache1 has joined #openstack-infra | 04:14 | |
| *** jamesmca_ has joined #openstack-infra | 04:15 | |
| *** bobh has joined #openstack-infra | 04:15 | |
| *** bobh has quit IRC | 04:18 | |
| *** esikache1 has quit IRC | 04:19 | |
| *** hongbin has quit IRC | 04:20 | |
| *** Apoorva has quit IRC | 04:22 | |
| *** jamielennox|away is now known as jamielennox | 04:23 | |
| *** kzaitsev_mb has quit IRC | 04:25 | |
| *** ihrachys has joined #openstack-infra | 04:25 | |
| *** gyee has joined #openstack-infra | 04:31 | |
| *** winggundamth has joined #openstack-infra | 04:35 | |
| *** ihrachys has quit IRC | 04:35 | |
| *** adam_g has quit IRC | 04:38 | |
| *** jamesmca_ has quit IRC | 04:38 | |
| *** sree has joined #openstack-infra | 04:39 | |
| *** armax has quit IRC | 04:40 | |
| *** ramishra_ has quit IRC | 04:40 | |
| *** adam_g has joined #openstack-infra | 04:41 | |
| *** ramishra has joined #openstack-infra | 04:42 | |
| *** marst_ has joined #openstack-infra | 04:49 | |
| dirk | adriant: this is in process of being reverted | 04:56 |
| *** dpawlik has joined #openstack-infra | 04:56 | |
| *** pgadiya has joined #openstack-infra | 04:56 | |
| *** danpawlik has quit IRC | 04:56 | |
| dirk | adriant: see lp bug 1692773 | 04:57 |
| openstack | Launchpad bug 1692773 in oslo.i18n "Removal of Babel in 3.15.2 requirements.txt breaks package" [Undecided,New] https://launchpad.net/bugs/1692773 - Assigned to Akihiro Motoki (amotoki) | 04:57 |
| *** dpawlik1 has joined #openstack-infra | 05:02 | |
| *** sree has quit IRC | 05:04 | |
| *** dpawlik has quit IRC | 05:04 | |
| *** eranrom has quit IRC | 05:06 | |
| *** gildub has quit IRC | 05:08 | |
| *** prateek has joined #openstack-infra | 05:08 | |
| *** jamielennox is now known as jamielennox|away | 05:11 | |
| *** pgadiya has left #openstack-infra | 05:12 | |
| *** esikache1 has joined #openstack-infra | 05:13 | |
| *** cody-somerville has quit IRC | 05:14 | |
| *** dpawlik1 has quit IRC | 05:15 | |
| adriant | dirk, ty, good to know I'm not just doing something silly :) | 05:16 |
| *** esikache1 has quit IRC | 05:17 | |
| *** claudiub has joined #openstack-infra | 05:20 | |
| *** dpawlik1 has joined #openstack-infra | 05:21 | |
| *** jaosorior_away is now known as jaosorior | 05:24 | |
| *** dpawlik1 has quit IRC | 05:26 | |
| *** dpawlik2 has joined #openstack-infra | 05:26 | |
| *** ltomasbo|away is now known as ltomasbo | 05:32 | |
| *** gyee has quit IRC | 05:32 | |
| *** gongysh has quit IRC | 05:34 | |
| *** mtreinish has quit IRC | 05:38 | |
| *** cshastri has joined #openstack-infra | 05:41 | |
| *** ramishra has quit IRC | 05:42 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Add a more generic tree->graph parser https://review.openstack.org/465417 | 05:42 |
| *** gildub has joined #openstack-infra | 05:42 | |
| *** ramishra has joined #openstack-infra | 05:43 | |
| *** esikache1 has joined #openstack-infra | 05:45 | |
| *** jamielennox|away is now known as jamielennox | 05:51 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Add a more generic tree->graph parser https://review.openstack.org/465417 | 05:54 |
| *** esikache1 has quit IRC | 05:54 | |
| *** isaacb has joined #openstack-infra | 06:01 | |
| *** rcernin has quit IRC | 06:01 | |
| *** esikache1 has joined #openstack-infra | 06:08 | |
| *** esikache1 has quit IRC | 06:13 | |
| *** rcernin has joined #openstack-infra | 06:18 | |
| *** jamielennox is now known as jamielennox|away | 06:23 | |
| *** ijw has joined #openstack-infra | 06:24 | |
| *** ijw has quit IRC | 06:28 | |
| *** namnh has joined #openstack-infra | 06:30 | |
| *** gildub has quit IRC | 06:31 | |
| *** gildub has joined #openstack-infra | 06:33 | |
| *** sbezverk has quit IRC | 06:33 | |
| *** mtreinish has joined #openstack-infra | 06:33 | |
| *** sbezverk has joined #openstack-infra | 06:33 | |
| *** pcaruana has joined #openstack-infra | 06:39 | |
| *** dhajare has joined #openstack-infra | 06:41 | |
| *** lamt has quit IRC | 06:48 | |
| *** esikache1 has joined #openstack-infra | 06:54 | |
| *** flepied has quit IRC | 06:54 | |
| *** jascott1 has joined #openstack-infra | 06:55 | |
| *** matrohon has joined #openstack-infra | 06:57 | |
| *** dizquierdo has joined #openstack-infra | 06:59 | |
| *** dizquierdo has quit IRC | 06:59 | |
| *** dizquierdo has joined #openstack-infra | 07:00 | |
| *** isaacb has quit IRC | 07:03 | |
| *** camunoz has quit IRC | 07:03 | |
| *** adam_g has quit IRC | 07:04 | |
| *** esikache1 has quit IRC | 07:04 | |
| *** adam_g has joined #openstack-infra | 07:04 | |
| *** esikache1 has joined #openstack-infra | 07:06 | |
| *** sree has joined #openstack-infra | 07:09 | |
| *** salv-orlando has joined #openstack-infra | 07:10 | |
| *** esikache1 has quit IRC | 07:10 | |
| *** psachin has quit IRC | 07:12 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-dev/cookiecutter master: Only testr allows parameter --slowest https://review.openstack.org/467042 | 07:13 |
| *** rcernin has quit IRC | 07:16 | |
| *** rcernin has joined #openstack-infra | 07:17 | |
| *** hashar has joined #openstack-infra | 07:19 | |
| *** DangerousDaren has joined #openstack-infra | 07:21 | |
| *** jpich has joined #openstack-infra | 07:21 | |
| *** salv-orlando has quit IRC | 07:25 | |
| *** andreas_s has joined #openstack-infra | 07:26 | |
| *** esikache1 has joined #openstack-infra | 07:27 | |
| *** Nilesh has joined #openstack-infra | 07:27 | |
| jianghuaw | hi, will someone help check if we can merge this Akihiro Motoki (amotoki)'s fix earlier: https://review.openstack.org/#/c/467001/. - project: openstack/requirements | 07:27 |
| jianghuaw | At the moment, all tempest CI jobs failed due to this issue. | 07:28 |
| jianghuaw | thanks. | 07:28 |
| Nilesh | Hello, most of the Cinder CI started failing | 07:28 |
| Nilesh | http://52.35.51.88/30/463230/5/check/dsvm-veritas-hyperscale-cinder-driver/49a580e/console.html | 07:28 |
| *** priteau has joined #openstack-infra | 07:28 | |
| *** nunchuck has joined #openstack-infra | 07:29 | |
| *** psachin has joined #openstack-infra | 07:29 | |
| jianghuaw | Nilesh, yes. that's the same issue. | 07:30 |
| *** clarkb has quit IRC | 07:30 | |
| jianghuaw | Should be fixed after the above fix got merged. | 07:30 |
| *** clarkb has joined #openstack-infra | 07:30 | |
| Nilesh | okay. thanks | 07:30 |
| *** flepied has joined #openstack-infra | 07:31 | |
| *** Hal has joined #openstack-infra | 07:32 | |
| jianghuaw | Nilesh, you are welcome. Hope someone will +w on that fix soon. I'm suffering from this issue too. | 07:33 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add dracut-regenerate elements https://review.openstack.org/447047 | 07:35 |
| openstackgerrit | Merged openstack-infra/infra-manual master: clarify the expected policy for managing releases https://review.openstack.org/466739 | 07:36 |
| *** jpena|off is now known as jpena | 07:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Revert "puppet: disable voting on ubuntu/integration jobs""" https://review.openstack.org/466726 | 07:42 |
| *** dpawlik2 is now known as danpawlik | 07:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove karbor team direct tagging ACLs https://review.openstack.org/466736 | 07:42 |
| *** psachin has quit IRC | 07:43 | |
| *** ltomasbo is now known as ltomasbo|away | 07:43 | |
| *** psachin has joined #openstack-infra | 07:45 | |
| *** prateek_ has joined #openstack-infra | 07:46 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Cinder block device driver job https://review.openstack.org/466503 | 07:47 |
| *** prateek has quit IRC | 07:48 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: New subsection on voting third-party CI permission https://review.openstack.org/462176 | 07:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch osc functional-tips job to py35 https://review.openstack.org/466382 | 07:50 |
| openstackgerrit | Merged openstack-infra/project-config master: Add functional-python35 to neutron experimental queue https://review.openstack.org/466122 | 07:51 |
| *** ralonsoh has joined #openstack-infra | 07:53 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add Debian nv check for osa-security https://review.openstack.org/466387 | 07:54 |
| openstackgerrit | Merged openstack-infra/project-config master: new project: tc-as-a-service https://review.openstack.org/457469 | 07:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Disable the requirements job for networking-arista stable/mitaka-hpb https://review.openstack.org/466901 | 07:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Add Almanach client project https://review.openstack.org/465152 | 07:56 |
| *** prateek__ has joined #openstack-infra | 07:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "puppet: Disable voting on ubuntu beaker jobs" https://review.openstack.org/465070 | 07:57 |
| openstackgerrit | Merged openstack-infra/project-config master: TripleO: container upgrade non-voting, not experimental https://review.openstack.org/465088 | 07:57 |
| *** prateek_ has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** shardy has joined #openstack-infra | 08:07 | |
| *** rossella__ has joined #openstack-infra | 08:07 | |
| *** dizquierdo has quit IRC | 08:16 | |
| *** lucas-afk is now known as lucasagomes | 08:16 | |
| gcb | jianghuaw, Nilesh : I just ping ttx, and he has approved it | 08:17 |
| *** openstackgerrit has quit IRC | 08:18 | |
| jianghuaw | gcb, cool. Thanks. | 08:20 |
| *** rossella_ has joined #openstack-infra | 08:21 | |
| *** openstackgerrit has joined #openstack-infra | 08:21 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] add containerized deployment on multinode https://review.openstack.org/454152 | 08:21 |
| gcb | jianghuaw: np | 08:22 |
| *** jascott1 has quit IRC | 08:22 | |
| *** jascott1 has joined #openstack-infra | 08:23 | |
| *** rossella__ has quit IRC | 08:23 | |
| *** mandre is now known as mandre|away | 08:23 | |
| *** gongysh has joined #openstack-infra | 08:26 | |
| *** jascott1 has quit IRC | 08:27 | |
| *** pbourke has joined #openstack-infra | 08:29 | |
| *** rossella_ has quit IRC | 08:32 | |
| *** Julien-zte has quit IRC | 08:35 | |
| *** Julien-zte has joined #openstack-infra | 08:36 | |
| *** derekh has joined #openstack-infra | 08:36 | |
| *** baoli has joined #openstack-infra | 08:37 | |
| *** dingyichen has quit IRC | 08:38 | |
| *** s-shiono has quit IRC | 08:40 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/irc-meetings master: Remove the defunct SR-IOV meeting https://review.openstack.org/467081 | 08:41 |
| *** Julien-zte has quit IRC | 08:41 | |
| *** baoli has quit IRC | 08:42 | |
| *** Julien-zte has joined #openstack-infra | 08:42 | |
| *** rossella_ has joined #openstack-infra | 08:44 | |
| *** ltomasbo|away is now known as ltomasbo | 08:44 | |
| *** ccamacho|brb is now known as ccamacho | 08:45 | |
| *** dtantsur|afk is now known as dtantsur | 08:45 | |
| *** ilpianista_ has quit IRC | 08:46 | |
| *** masayukig[m] has quit IRC | 08:46 | |
| *** bhavik1 has joined #openstack-infra | 08:46 | |
| *** Julien-zte has quit IRC | 08:47 | |
| *** loquacity has quit IRC | 08:48 | |
| *** jasondotstar has quit IRC | 08:48 | |
| *** masayukig[m] has joined #openstack-infra | 08:48 | |
| *** sambetts|afk has quit IRC | 08:48 | |
| *** NikitaKonovalov has quit IRC | 08:48 | |
| *** amotoki has quit IRC | 08:49 | |
| *** Julien-zte has joined #openstack-infra | 08:50 | |
| *** jamesmca_ has joined #openstack-infra | 08:50 | |
| *** loquacities has joined #openstack-infra | 08:51 | |
| *** amotoki has joined #openstack-infra | 08:51 | |
| *** NikitaKonovalov has joined #openstack-infra | 08:51 | |
| *** jasondotstar has joined #openstack-infra | 08:51 | |
| *** sambetts_ has joined #openstack-infra | 08:52 | |
| *** ashishb_ has joined #openstack-infra | 08:52 | |
| *** dtantsur is now known as dtantsur|brb | 08:54 | |
| *** jamesmca_ has quit IRC | 08:55 | |
| ashishb_ | Hello everyone! I am trying to change the company for my details in openstack/stackalytics: etc/default_data.json | 08:55 |
| *** jistr is now known as jistr|call | 08:56 | |
| *** Kevin_Zheng has quit IRC | 08:56 | |
| ashishb_ | however, it fails to pass Jenkins and the error is not very helpful to understand what's going wrong | 08:56 |
| ashishb_ | any help is welcome! | 08:57 |
| *** aviau has quit IRC | 08:57 | |
| *** aviau has joined #openstack-infra | 08:57 | |
| *** electrofelix has joined #openstack-infra | 09:04 | |
| *** ilpianista_ has joined #openstack-infra | 09:04 | |
| *** aarefiev_afk is now known as aarefiev | 09:04 | |
| *** shardy has quit IRC | 09:05 | |
| sshnaidm | what is Xiexianbin? that started gate jobs: https://review.openstack.org/#/c/451523/ | 09:06 |
| *** bhavik1 has quit IRC | 09:08 | |
| *** shardy has joined #openstack-infra | 09:15 | |
| *** neiljerram has joined #openstack-infra | 09:17 | |
| *** jistr|call is now known as jistr | 09:19 | |
| *** tosky has joined #openstack-infra | 09:20 | |
| *** masber has quit IRC | 09:24 | |
| *** sambetts_ is now known as sambetts | 09:24 | |
| *** e0ne has joined #openstack-infra | 09:25 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a more generic tree->graph parser https://review.openstack.org/465417 | 09:25 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/467098 | 09:29 |
| *** Julien-zte has quit IRC | 09:33 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: [NO MERGE] test multinode containers https://review.openstack.org/467100 | 09:33 |
| *** Julien-zte has joined #openstack-infra | 09:34 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: [NO MERGE] test multinode containers https://review.openstack.org/467100 | 09:35 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a more generic tree->graph parser https://review.openstack.org/465417 | 09:36 |
| *** stakeda has quit IRC | 09:36 | |
| *** Julien-zte has quit IRC | 09:36 | |
| *** Julien-zte has joined #openstack-infra | 09:37 | |
| *** bulat has joined #openstack-infra | 09:39 | |
| bulat | Folks, please review https://review.openstack.org/#/c/463554/ | 09:40 |
| *** Julien-zte has quit IRC | 09:41 | |
| *** hashar is now known as hasharAway | 09:42 | |
| *** yolkfull has quit IRC | 09:42 | |
| *** yolkfull has joined #openstack-infra | 09:43 | |
| *** Julien-zte has joined #openstack-infra | 09:43 | |
| *** ivyyy has quit IRC | 09:45 | |
| *** ivyyy has joined #openstack-infra | 09:45 | |
| *** esikache1 has quit IRC | 09:45 | |
| *** Julien-zte has quit IRC | 09:45 | |
| *** Julien-zte has joined #openstack-infra | 09:46 | |
| *** panda is now known as panda|bbl | 09:54 | |
| *** Julien-zte has quit IRC | 09:55 | |
| *** Julien-zte has joined #openstack-infra | 09:56 | |
| *** adriant has quit IRC | 09:57 | |
| *** faizy has joined #openstack-infra | 09:59 | |
| *** esikache1 has joined #openstack-infra | 10:01 | |
| *** sdague has joined #openstack-infra | 10:03 | |
| *** danpawlik has quit IRC | 10:04 | |
| *** rossella_ has quit IRC | 10:05 | |
| *** esikache1 has quit IRC | 10:05 | |
| *** esikache1 has joined #openstack-infra | 10:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add 'propose' job for openstack-ansible-tests https://review.openstack.org/453130 | 10:07 |
| *** faizy has quit IRC | 10:07 | |
| gmann | gcb: have we got opt bot patch on oslo.log for i18n requirement fix ? | 10:09 |
| *** Julien-zte has quit IRC | 10:10 | |
| *** esikache1 has quit IRC | 10:11 | |
| *** adarazs is now known as adarazs_lunch | 10:12 | |
| *** dtantsur|brb is now known as dtantsur | 10:13 | |
| *** yee379 has quit IRC | 10:13 | |
| *** yee379 has joined #openstack-infra | 10:14 | |
| *** e0ne has quit IRC | 10:14 | |
| *** e0ne has joined #openstack-infra | 10:14 | |
| *** sc` has quit IRC | 10:17 | |
| *** adam_g has quit IRC | 10:17 | |
| *** sc` has joined #openstack-infra | 10:17 | |
| gmann | gcb: or if you can approve andreaf patch - https://review.openstack.org/#/c/467113/1 | 10:17 |
| *** adam_g has joined #openstack-infra | 10:18 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Adapt getthelogs UX for more use cases https://review.openstack.org/449552 | 10:18 |
| gcb | gmann: just approve https://review.openstack.org/#/c/467113 | 10:19 |
| gmann | gcb: thanks. | 10:20 |
| gcb | gmann: np | 10:21 |
| *** lamt has joined #openstack-infra | 10:25 | |
| *** lamt has quit IRC | 10:25 | |
| *** rossella_ has joined #openstack-infra | 10:25 | |
| *** mpaolino has joined #openstack-infra | 10:26 | |
| *** esikache1 has joined #openstack-infra | 10:28 | |
| *** gongysh has quit IRC | 10:36 | |
| *** esikache1 has quit IRC | 10:36 | |
| *** lamt has joined #openstack-infra | 10:40 | |
| *** jkilpatr has quit IRC | 10:40 | |
| *** gildub has quit IRC | 10:44 | |
| *** askb has quit IRC | 10:48 | |
| *** salv-orlando has joined #openstack-infra | 10:49 | |
| *** nicolasbock has joined #openstack-infra | 10:51 | |
| *** salv-orlando has quit IRC | 10:53 | |
| *** Hal has quit IRC | 10:55 | |
| *** namnh has quit IRC | 10:56 | |
| *** apevec has joined #openstack-infra | 10:56 | |
| *** jkilpatr has joined #openstack-infra | 10:57 | |
| *** adam_g has quit IRC | 11:01 | |
| *** adam_g has joined #openstack-infra | 11:03 | |
| *** esikache1 has joined #openstack-infra | 11:03 | |
| *** wolverineav has joined #openstack-infra | 11:04 | |
| *** thorst_afk has joined #openstack-infra | 11:05 | |
| *** Nilesh has quit IRC | 11:05 | |
| *** yolkfull has quit IRC | 11:05 | |
| *** panda|bbl is now known as panda | 11:10 | |
| *** adam_g has quit IRC | 11:11 | |
| apevec | ttx, hi, I'm trying to trace down why puppet-nova 9.5.1 tarball is missing http://tarballs.openstack.org/puppet-nova/?C=M;O=D Release request was https://review.openstack.org/465791 | 11:11 |
| apevec | how to find the logs from the corresponding "tag" job? | 11:12 |
| *** adam_g has joined #openstack-infra | 11:12 | |
| *** wolverineav has quit IRC | 11:14 | |
| *** adarazs_lunch is now known as adarazs | 11:14 | |
| *** yamamoto has quit IRC | 11:17 | |
| *** wolverineav has joined #openstack-infra | 11:24 | |
| *** gorba has joined #openstack-infra | 11:25 | |
| *** wolverineav has quit IRC | 11:29 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove all centos-release-* repos https://review.openstack.org/461497 | 11:31 |
| *** ralonsoh_ has joined #openstack-infra | 11:32 | |
| *** ralonsoh_ has quit IRC | 11:32 | |
| AJaeger | jianghuaw: better ask on #openstack-requirements next time | 11:32 |
| AJaeger | apevec: did you see the discussion on openstack-dev mailing list? | 11:33 |
| AJaeger | apevec: see also https://docs.openstack.org/infra/manual/developers.html#post-processing - and use "git show-ref 9.5.1" for the sha1 | 11:34 |
| apevec | ah thanks for the link! | 11:34 |
| apevec | ok, I see thread now | 11:35 |
| *** gouthamr_ has joined #openstack-infra | 11:35 | |
| *** wolverineav has joined #openstack-infra | 11:37 | |
| *** lucasagomes is now known as lucas-afk | 11:38 | |
| *** lucas-afk is now known as lucas-hungry | 11:38 | |
| *** ldnunes has joined #openstack-infra | 11:41 | |
| *** wolverineav has quit IRC | 11:41 | |
| *** salv-orlando has joined #openstack-infra | 11:42 | |
| *** gildub has joined #openstack-infra | 11:43 | |
| *** Julien-zte has joined #openstack-infra | 11:47 | |
| *** salv-orlando has quit IRC | 11:52 | |
| AJaeger | infra-root, another github set up problem - is something broken on our tooling? This happens too often recently ;( | 11:53 |
| AJaeger | infra-root, https://github.com/openstack/tc-as-a-service and https://github.com/openstack/python-almanachclient are not setup - could you check, please? | 11:53 |
| AJaeger | http://git.openstack.org/cgit/openstack/tc-as-a-service and http://git.openstack.org/cgit/openstack/python-almanachclient look fine | 11:54 |
| *** yamamoto has joined #openstack-infra | 11:58 | |
| *** sree has quit IRC | 11:58 | |
| *** gordc has joined #openstack-infra | 11:58 | |
| *** thorst_afk has quit IRC | 11:59 | |
| *** yamamoto_ has joined #openstack-infra | 12:00 | |
| *** Kevin_Zheng has joined #openstack-infra | 12:01 | |
| *** gouthamr_ has quit IRC | 12:02 | |
| *** Julien-zte has quit IRC | 12:02 | |
| *** rossella_ has quit IRC | 12:02 | |
| *** yamamoto has quit IRC | 12:04 | |
| *** rhallisey has joined #openstack-infra | 12:04 | |
| *** hasharAway is now known as hashar | 12:06 | |
| *** masayukig[m] has quit IRC | 12:09 | |
| *** ilpianista_ has quit IRC | 12:09 | |
| *** esikache1 has quit IRC | 12:09 | |
| *** charz_ has quit IRC | 12:11 | |
| *** mrunge has quit IRC | 12:11 | |
| *** wolverineav has joined #openstack-infra | 12:12 | |
| *** dgonzalez has quit IRC | 12:12 | |
| *** masayukig[m] has joined #openstack-infra | 12:13 | |
| *** mrunge has joined #openstack-infra | 12:14 | |
| *** dgonzalez has joined #openstack-infra | 12:14 | |
| *** charz has joined #openstack-infra | 12:14 | |
| *** dave-mccowan has joined #openstack-infra | 12:14 | |
| *** esikache1 has joined #openstack-infra | 12:15 | |
| mordred | AJaeger: gross. sorry - I will check in just a sec | 12:17 |
| mordred | dirk: vhd-util now built for xenial! | 12:17 |
| cmurphy | \o/ | 12:17 |
| *** lamt has quit IRC | 12:18 | |
| *** esikache1 has quit IRC | 12:19 | |
| *** ociuhandu has joined #openstack-infra | 12:21 | |
| *** Julien-zte has joined #openstack-infra | 12:23 | |
| *** ilpianista_ has joined #openstack-infra | 12:25 | |
| *** Julien-zte has quit IRC | 12:27 | |
| *** Julien-zte has joined #openstack-infra | 12:28 | |
| *** dprince has joined #openstack-infra | 12:30 | |
| *** rossella_ has joined #openstack-infra | 12:34 | |
| *** lucas-hungry is now known as lucasagomes | 12:36 | |
| *** jcoufal has joined #openstack-infra | 12:38 | |
| *** esikache1 has joined #openstack-infra | 12:38 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| *** sflanigan has quit IRC | 12:42 | |
| *** baoli has joined #openstack-infra | 12:43 | |
| *** esikache1 has quit IRC | 12:46 | |
| *** salv-orlando has joined #openstack-infra | 12:48 | |
| *** jamielennox|away is now known as jamielennox | 12:49 | |
| *** jpena is now known as jpena|lunch | 12:49 | |
| *** esikache1 has joined #openstack-infra | 12:49 | |
| *** kgiusti has joined #openstack-infra | 12:50 | |
| *** ricardoas has joined #openstack-infra | 12:51 | |
| *** jamesmca_ has joined #openstack-infra | 12:51 | |
| *** salv-orlando has quit IRC | 12:53 | |
| *** esikache1 has quit IRC | 12:54 | |
| *** rvba` is now known as rvba | 12:54 | |
| *** jamesmca_ has quit IRC | 12:55 | |
| *** jamesmca_ has joined #openstack-infra | 12:55 | |
| *** gcb has quit IRC | 12:55 | |
| *** esberglu has joined #openstack-infra | 12:56 | |
| *** jaosorior has quit IRC | 12:57 | |
| *** Goneri has joined #openstack-infra | 12:57 | |
| *** jaosorior has joined #openstack-infra | 12:57 | |
| *** lewo has quit IRC | 12:58 | |
| *** lewo has joined #openstack-infra | 12:58 | |
| *** lamt has joined #openstack-infra | 12:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix get_compute_usage normalization problem https://review.openstack.org/467226 | 13:01 |
| *** esikache1 has joined #openstack-infra | 13:01 | |
| mordred | cmurphy: while I'm on it, I'm going to go ahead and upload for zesty | 13:03 |
| AJaeger | mordred: are the github repos fixed? do you have any idea what fails? | 13:03 |
| *** mdrabe has joined #openstack-infra | 13:04 | |
| *** lamt has quit IRC | 13:04 | |
| pabelanger | mordred: cool, I'll start work on nodepool-builders today | 13:04 |
| *** lamt has joined #openstack-infra | 13:04 | |
| *** esikache1 has quit IRC | 13:05 | |
| *** esikache1 has joined #openstack-infra | 13:06 | |
| *** Julien-zte has quit IRC | 13:07 | |
| *** Julien-zte has joined #openstack-infra | 13:08 | |
| *** winggundamth has quit IRC | 13:08 | |
| cmurphy | mordred: cool | 13:08 |
| *** thorst_afk has joined #openstack-infra | 13:09 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use ssh for git-upload-pack https://review.openstack.org/436802 | 13:10 |
| *** esikache1 has quit IRC | 13:10 | |
| *** mriedem has joined #openstack-infra | 13:11 | |
| *** kgiusti has left #openstack-infra | 13:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix get_compute_limits error message https://review.openstack.org/467231 | 13:13 |
| mordred | AJaeger: just looking at that one now | 13:14 |
| *** kgiusti has joined #openstack-infra | 13:14 | |
| *** salv-orlando has joined #openstack-infra | 13:15 | |
| pabelanger | mordred: clarkb: fungi: jeblair: I'd like stop nb01.o.o (trusty) to prepare for nb01.o.o (xenial) replacement. Otherwise, I am okay with standing up nb03.o.o, thoughts? | 13:16 |
| mordred | AJaeger: ok - I see the error message for tc-as-a-service ... looking at code real quick before I fix the problem | 13:16 |
| pabelanger | maybe just using nb03.o.o is better, then we don't have to delete existing uploads / diskimages for nb01.o.o | 13:17 |
| fungi | pabelanger: also the inventory gets funky if you don't completely delete the old instance before booting another with the exact same name | 13:18 |
| pabelanger | fungi: that is true, okay. I'll start with nb03 this morning | 13:18 |
| *** esikache1 has joined #openstack-infra | 13:20 | |
| *** salv-orlando has quit IRC | 13:20 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Do not merge - Testing featureset020 https://review.openstack.org/465003 | 13:20 |
| *** Julien-zte has quit IRC | 13:22 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Do not merge: Test Featureset 021 https://review.openstack.org/465476 | 13:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Don't mark as created in github until we're done https://review.openstack.org/467235 | 13:22 |
| mordred | AJaeger: ^^ that should fix the issues in general - and the two you listed are now fixed by hand | 13:24 |
| AJaeger | thanks, mordred | 13:24 |
| mordred | fungi, pabelanger: ^^ if you have a sec, that jeepyb patch should fix half-created projects | 13:24 |
| *** sree_ has joined #openstack-infra | 13:24 | |
| *** gildub has quit IRC | 13:24 | |
| *** esikache1 has quit IRC | 13:25 | |
| *** sree_ is now known as Guest85309 | 13:25 | |
| *** shardy has quit IRC | 13:25 | |
| *** ihrachys has joined #openstack-infra | 13:26 | |
| dirk | pabelanger: excellent, let me know if anything is failing | 13:27 |
| pabelanger | mordred: I'll trade you: https://review.openstack.org/#/c/460740/ | 13:27 |
| mordred | pabelanger: looks great to me! | 13:27 |
| pabelanger | also +2 | 13:28 |
| *** shardy has joined #openstack-infra | 13:29 | |
| *** Guest85309 has quit IRC | 13:29 | |
| pabelanger | dirk: yup, just bringing server online now. | 13:30 |
| pabelanger | will be awesome in zuulv3 when we have a pipeline to do this | 13:30 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/467098 | 13:32 |
| *** esikache1 has joined #openstack-infra | 13:32 | |
| *** fguillot has joined #openstack-infra | 13:33 | |
| *** vikrant has quit IRC | 13:35 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use ssh for git-upload-pack https://review.openstack.org/436802 | 13:36 |
| *** esikache1 has quit IRC | 13:37 | |
| *** bobh has joined #openstack-infra | 13:37 | |
| *** dimtruck is now known as zz_dimtruck | 13:39 | |
| mtreinish | SpamapS: so I think I figured out the log capture on your stestr patch. The capture is triggered by an env variable which isn't being set anymore | 13:40 |
| mtreinish | it was done in the .testr.conf before, but stestr doesn't execute an arbitrary shell command for running tests so we can't set it in the .stestr.conf like that | 13:41 |
| *** esikache1 has joined #openstack-infra | 13:41 | |
| mtreinish | we'll either need to add it to the env variable list in the tox.ini or just remove the conditionals around setting up the capture (which is what I would probably choose, who doesn't want the capture turned on) | 13:41 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use ssh for git-upload-pack https://review.openstack.org/436802 | 13:43 |
| *** gcb has joined #openstack-infra | 13:45 | |
| *** prateek__ has quit IRC | 13:45 | |
| dirk | pabelanger: you see me excited | 13:45 |
| *** jamesdenton has joined #openstack-infra | 13:46 | |
| fungi | mordred: pep8 failure on 467235 | 13:46 |
| *** ralonsoh has quit IRC | 13:47 | |
| *** dave-mcc_ has joined #openstack-infra | 13:47 | |
| *** armax has joined #openstack-infra | 13:48 | |
| *** hamzy has quit IRC | 13:48 | |
| jeblair | no idea what that failure is | 13:48 |
| jeblair | http://logs.openstack.org/35/467235/1/check/gate-jeepyb-pep8-ubuntu-xenial/f475b7f/console.html | 13:49 |
| *** dave-mccowan has quit IRC | 13:49 | |
| *** esikache1 has quit IRC | 13:49 | |
| *** bh526r has joined #openstack-infra | 13:50 | |
| *** dave-mccowan has joined #openstack-infra | 13:50 | |
| mtreinish | jeblair: hmm, running pep8 on that patch locally I got: http://paste.openstack.org/show/610458/ | 13:50 |
| mtreinish | not sure why it doesn't show in the log though | 13:51 |
| *** hongbin has joined #openstack-infra | 13:51 | |
| jeblair | weird | 13:51 |
| jeblair | i rechecked | 13:51 |
| *** thorst_afk has quit IRC | 13:52 | |
| *** eharney has joined #openstack-infra | 13:53 | |
| *** dave-mcc_ has quit IRC | 13:53 | |
| jeblair | thanks to logs from mtreinishCI i pointed out the fix too. | 13:54 |
| *** jpena|lunch is now known as jpena | 13:54 | |
| *** rcarrillocruz has quit IRC | 13:55 | |
| *** chlong has joined #openstack-infra | 13:55 | |
| *** links has quit IRC | 13:57 | |
| *** esikache1 has joined #openstack-infra | 13:57 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/zuul feature/zuulv3: Switch from testrepository to stestr https://review.openstack.org/466071 | 13:58 |
| *** Julien-zte has joined #openstack-infra | 13:58 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove the defunct SR-IOV meeting https://review.openstack.org/467081 | 13:59 |
| *** rpi has quit IRC | 13:59 | |
| *** rhallisey has quit IRC | 13:59 | |
| mtreinish | SpamapS: ^^^ let's see if that fixes it | 13:59 |
| *** jamesmca_ has quit IRC | 13:59 | |
| *** rpi has joined #openstack-infra | 14:00 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/jeepyb master: Add bindep.txt file https://review.openstack.org/467246 | 14:00 |
| *** shardy_ has joined #openstack-infra | 14:00 | |
| *** andreas_s has quit IRC | 14:01 | |
| *** ralonsoh has joined #openstack-infra | 14:01 | |
| odyssey4me | we've been seeing what appears to be a CI setup under a personal account over the last 6 hours or so - "Xiexianbin has posted comments on this change. Starting gate jobs. http://192.168.1.50" - anyone else seeing this? | 14:01 |
| *** spzala has joined #openstack-infra | 14:02 | |
| fungi | nope, but we can totally shut down that account | 14:02 |
| fungi | working on it now | 14:02 |
| *** Julien-zte has quit IRC | 14:02 | |
| odyssey4me | 'Gerrit-Reviewer: Xiexianbin <me@xiexianbin.cn>' if that helps | 14:02 |
| odyssey4me | thanks fungi | 14:02 |
| *** Julien-zte has joined #openstack-infra | 14:02 | |
| *** rhallisey has joined #openstack-infra | 14:04 | |
| mtreinish | jeblair: it's too bad my closet cloud isn't powered on 24/7 or I could really do mtreinishCI :) | 14:04 |
| fungi | #status log Disabled Gerrit account 10842 (Xiexianbin) for posting unrequested third-party CI results on changes | 14:04 |
| openstackstatus | fungi: finished logging | 14:04 |
| jeblair | mtreinish: i like how the logs only contain the relevant error messages. that's a real value-add. | 14:04 |
| fungi | mtreinish as a service | 14:05 |
| *** rbrndt has joined #openstack-infra | 14:06 | |
| *** esikache1 has quit IRC | 14:06 | |
| *** ricardoas has left #openstack-infra | 14:06 | |
| *** Julien-zte has quit IRC | 14:08 | |
| *** Julien-zte has joined #openstack-infra | 14:08 | |
| *** rossella_ has quit IRC | 14:09 | |
| *** salv-orlando has joined #openstack-infra | 14:10 | |
| *** gardlt has joined #openstack-infra | 14:10 | |
| *** rossella_ has joined #openstack-infra | 14:10 | |
| *** salv-orlando has quit IRC | 14:14 | |
| *** marst_ has quit IRC | 14:17 | |
| *** zz_dimtruck is now known as dimtruck | 14:17 | |
| *** rcarrillocruz has joined #openstack-infra | 14:19 | |
| _ari_ | electrofelix: any chance you can give me a quick review of this today https://review.openstack.org/#/c/463905/ | 14:21 |
| *** jamesmca_ has joined #openstack-infra | 14:22 | |
| *** thorst_afk has joined #openstack-infra | 14:22 | |
| *** marst_ has joined #openstack-infra | 14:23 | |
| *** eranrom has joined #openstack-infra | 14:25 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/zuul feature/zuulv3: Switch from testrepository to stestr https://review.openstack.org/466071 | 14:26 |
| openstackgerrit | Merged openstack-infra/shade master: Fix get_compute_usage normalization problem https://review.openstack.org/467226 | 14:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Allow a user to submit start and end time as strings https://review.openstack.org/467257 | 14:28 |
| openstackgerrit | Dima Kuznetsov proposed openstack-infra/project-config master: Add networking-sfc to Dragonflow's gate jobs https://review.openstack.org/467259 | 14:32 |
| electrofelix | _ari_: thanks for the poke, had gotten side tracked at work! | 14:33 |
| _ari_ | electrofelix: np thank you | 14:33 |
| *** bulat has quit IRC | 14:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add nb03.openstack.org to zookeeper https://review.openstack.org/467261 | 14:35 |
| *** mattmceuen has joined #openstack-infra | 14:35 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Add support for jms_messaging builder and publisher https://review.openstack.org/463905 | 14:37 |
| *** hamzy has joined #openstack-infra | 14:37 | |
| pabelanger | mnaser: We have a new DNS entry that needs to be setup, what is the procedure again? | 14:37 |
| *** yamahata has quit IRC | 14:38 | |
| *** eharney has quit IRC | 14:38 | |
| fungi | reverse dns entry specifically | 14:40 |
| fungi | (preferably two, for both its v4 and v6 addresses) | 14:40 |
| clarkb | mordred: ok I'm around if you want to make that shade release | 14:40 |
| *** cshastri has quit IRC | 14:42 | |
| openstackgerrit | Frédéric Guillot proposed openstack-infra/project-config master: Disable Python 2.7 checks for Almanach Client https://review.openstack.org/467265 | 14:42 |
| pabelanger | fungi: so, we hold 1/2 dns in rackspace, then reverse in vexxhost? | 14:45 |
| mordred | clarkb: k - three more patches I want to land first from a bug reported this morning (they're already winding through the gate) | 14:45 |
| mordred | clarkb: they will not affect nodepool, but they're a clear broken thing | 14:45 |
| mnaser | hi fungi / pabelanger | 14:45 |
| clarkb | pabelanger: yes forward is associated with the domain name and can be hosted in a single place (possibly by multiple redundant servers). Reverse is typically managed by the entity that owns the ip address range(s) | 14:46 |
| clarkb | pabelanger: since these are vexxhost IP addrs we rely on them to configure the reverse entries | 14:46 |
| fungi | pabelanger: yes, the way dns works, all records within the openstack.org domain are hosted in a zone at one provider (rackspace). reverse dns records are hosted by the holders of the ip address reverse zones (so each network provider is in control of their own) | 14:46 |
| mnaser | i can do a reverse dns if need be :) | 14:47 |
| mnaser | (if you have the ip and new addr handy) | 14:47 |
| *** nicolasbock has quit IRC | 14:47 | |
| pabelanger | okay, I cannot remember which if forward / reverse dns http://paste.openstack.org/show/610466/ | 14:48 |
| pabelanger | I think 2nd set is forward? | 14:48 |
| clarkb | pabelanger: yup 2nd set (the A and AAAA records) are forward | 14:48 |
| clarkb | the first two are reverse | 14:48 |
| pabelanger | k, I'll do that now | 14:48 |
| *** srobert has joined #openstack-infra | 14:48 | |
| mnaser | yep, ill do the first 2 | 14:48 |
| mnaser | we dont have reverse dns for ipv6 right now.. ill do it for the ipv4 and ill get back to you about ipv6 soon! | 14:49 |
| *** bulat has joined #openstack-infra | 14:49 | |
| *** shardy has quit IRC | 14:49 | |
| fungi | thanks mnaser! | 14:50 |
| *** shardy_ has quit IRC | 14:50 | |
| *** psachin has quit IRC | 14:50 | |
| pabelanger | fungi: clarkb: if you don't mind: https://review.openstack.org/#/c/467261/ | 14:50 |
| *** esikache1 has joined #openstack-infra | 14:50 | |
| openstackgerrit | Merged openstack-infra/shade master: Fix get_compute_limits error message https://review.openstack.org/467231 | 14:51 |
| *** shardy has joined #openstack-infra | 14:51 | |
| fungi | mnaser: and yes, ip6.arpa reverses are pretty high on our list of needs, if you're keeping track of a wishlist of some kind | 14:51 |
| clarkb | pabelanger: was systemd on xenial happy with our sys v scripts? | 14:52 |
| *** shardy_ has joined #openstack-infra | 14:52 | |
| clarkb | pabelanger: sometimes you have to edit puppet to do a systemctl config reload because puppet doesn't do that implicitly | 14:52 |
| clarkb | oh I bet the reboot we do actually solved that for us | 14:52 |
| pabelanger | clarkb: yes, nodepool-builder started | 14:52 |
| clarkb | neat | 14:52 |
| pabelanger | but failed to connect to zookeeper | 14:52 |
| pabelanger | see #zuul for that issue | 14:52 |
| pabelanger | we might want to have nodepool keep trying forever | 14:52 |
| pabelanger | otherwise, I'll have to start the service again | 14:53 |
| *** eharney has joined #openstack-infra | 14:53 | |
| clarkb | unrelated I'm trying to make http://logs.openstack.org/27/466927/1/check/gate-shade-dsvm-ansible/296035a/console.html#_2017-05-22_22_34_30_903450 failures like that less common with https://review.openstack.org/#/c/466935/ its hard to reproduce because easy install and global afs mirrors but thinking pip may be more reliable based on that failure log where pip downloads everything just fine and only | 14:55 |
| clarkb | easy_intall fails | 14:55 |
| clarkb | if you have time for a quick review | 14:55 |
| *** adarazs has left #openstack-infra | 14:55 | |
| *** yamahata has joined #openstack-infra | 14:56 | |
| mnaser | pabelanger / fungi: 9.215.19.199.in-addr.arpa domain name pointer nb03.openstack.org. | 14:56 |
| fungi | thanks mnaser! looking forward to when 6.3.3.9.8.9.e.f.f.f.e.3.6.1.8.f.0.0.0.0.1.0.0.0.0.0.1.e.4.0.6.2.ip6.arpa returns a similar ptr | 14:57 |
| pabelanger | mnaser: thank you | 14:58 |
| mnaser | fungi hopefully soon once we find a way not to ddos dns, ahahah | 14:59 |
| fungi | ip6.arpa hosting ddos's your nameservers? | 15:00 |
| *** sshnaidm is now known as sshnaidm|mtg | 15:01 | |
| *** bulat has quit IRC | 15:02 | |
| *** esikache1 has quit IRC | 15:06 | |
| *** funzo_ has joined #openstack-infra | 15:06 | |
| *** mat128 has joined #openstack-infra | 15:06 | |
| mnaser | fungi no but the concept of how we set up records | 15:06 |
| *** esikache1 has joined #openstack-infra | 15:06 | |
| mnaser | doesnt scale up to ipv6 | 15:06 |
| mnaser | :p | 15:06 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Bump citycloud-la1 to max-servers 50 https://review.openstack.org/467279 | 15:07 |
| clarkb | pabelanger: fungi mordred ^ we have that flavor now according to openstackclient | 15:07 |
| *** funzo has quit IRC | 15:08 | |
| fungi | mnaser: got it. good luck on the scaling work! | 15:08 |
| pabelanger | +2 | 15:08 |
| clarkb | (no response from citycloud directly so not sure about network issue, but shade workaround is in the pipeline) | 15:09 |
| *** mat128 has quit IRC | 15:10 | |
| *** shardy_ has quit IRC | 15:13 | |
| *** shardy_ has joined #openstack-infra | 15:13 | |
| *** thorst_afk has quit IRC | 15:14 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Bump diskimage-builder dependency to 2.0.0 https://review.openstack.org/467282 | 15:15 |
| *** hieulq_ has joined #openstack-infra | 15:15 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Bump diskimage-builder dependency to 2.0.0 https://review.openstack.org/467282 | 15:15 |
| *** thorst_afk has joined #openstack-infra | 15:16 | |
| *** cody-somerville has joined #openstack-infra | 15:16 | |
| *** cody-somerville has joined #openstack-infra | 15:16 | |
| *** csomerville has joined #openstack-infra | 15:17 | |
| *** thorst_afk has quit IRC | 15:20 | |
| *** cody-somerville has quit IRC | 15:21 | |
| *** esikache1 has quit IRC | 15:21 | |
| *** rcernin has quit IRC | 15:22 | |
| tushar__ | clarkb : sudo -H -E worked for me ...to set proxy | 15:23 |
| pabelanger | fungi: could I bug you for a +3 on https://review.openstack.org/#/c/467261/ for nb03.o.o | 15:23 |
| tushar__ | clarkb : now DIB is creation failed ...because it is not getting zuul ssh public key | 15:24 |
| *** nicolasbock has joined #openstack-infra | 15:24 | |
| tushar__ | clarkb : error log - https://pastebin.com/JCGQcx8A | 15:24 |
| tushar__ | any idea how can I fix this | 15:25 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Retire Gnocchi https://review.openstack.org/466320 | 15:26 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Update Gnocchi devstack URL https://review.openstack.org/466321 | 15:26 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Stop gating Gnocchi https://review.openstack.org/466318 | 15:26 |
| *** yamamoto_ has quit IRC | 15:26 | |
| *** armax has quit IRC | 15:27 | |
| *** yamamoto has joined #openstack-infra | 15:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Bump citycloud-la1 to max-servers 50 https://review.openstack.org/467279 | 15:27 |
| fungi | pabelanger: yep, sorry, in a meeting so wasn't paying close attention | 15:28 |
| pabelanger | fungi: np, thanks | 15:29 |
| *** chlong has quit IRC | 15:31 | |
| *** mdrabe_ has joined #openstack-infra | 15:31 | |
| *** gyee has joined #openstack-infra | 15:32 | |
| *** yamamoto has quit IRC | 15:32 | |
| *** sshnaidm|mtg is now known as sshnaidm | 15:33 | |
| *** Julien-zte has quit IRC | 15:33 | |
| *** Julien-zte has joined #openstack-infra | 15:33 | |
| *** Julien-zte has quit IRC | 15:33 | |
| *** mdrabe has quit IRC | 15:34 | |
| *** ociuhandu has quit IRC | 15:35 | |
| *** hongbin has quit IRC | 15:36 | |
| clarkb | dansmith: http://status.openstack.org/elastic-recheck/index.html#1660747 is a cells related bug/failure. Not sure if that is on your radar | 15:38 |
| openstackgerrit | Merged openstack-infra/system-config master: Add nb03.openstack.org to zookeeper https://review.openstack.org/467261 | 15:39 |
| *** ldnunes has quit IRC | 15:39 | |
| dansmith | clarkb: I haven't looked, but why do you say that? | 15:40 |
| clarkb | dansmith: it only fails on the cells job | 15:40 |
| clarkb | gate-tempest-dsvm-cells-ubuntu-xenial | 15:40 |
| *** ldnunes has joined #openstack-infra | 15:40 | |
| dansmith | oh, cellsv1 | 15:40 |
| dansmith | okay | 15:40 |
| *** DangerousDaren has quit IRC | 15:41 | |
| clarkb | oh the query is restricted to that job too so that explains why it only shows up with that job in kibana | 15:41 |
| dansmith | clarkb: are you going to widen that to see if it's actually only that job? | 15:41 |
| clarkb | dansmith: ya let me rerun without the job name restriction | 15:42 |
| dansmith | thanks | 15:42 |
| *** chlong has joined #openstack-infra | 15:43 | |
| *** rossella_ has quit IRC | 15:45 | |
| clarkb | dansmith: this query is all kinds of weird, it matches a lot of stuff because listing servers is apparently something we do a lot | 15:45 |
| clarkb | mriedem: melwitt ^ any chance you have the background there to know what the intent was? | 15:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs master: Add nodepool drivers spec https://review.openstack.org/461509 | 15:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs master: Block sphinx 1.6.1 https://review.openstack.org/467291 | 15:46 |
| jeblair | fungi: ^ administrivia and a spec for today's meeting (adding to wiki now) | 15:46 |
| fungi | thanks! | 15:47 |
| mriedem | clarkb: it was fixed in february, so that query is likely hitting on something else now | 15:47 |
| clarkb | mriedem: or it wasn't actually fixed :) | 15:47 |
| *** pcaruana has quit IRC | 15:47 | |
| mriedem | oh it was fixed already | 15:48 |
| mriedem | *alright | 15:48 |
| mriedem | but, idk | 15:49 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Retire Gnocchi https://review.openstack.org/466320 | 15:49 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Update Gnocchi devstack URL https://review.openstack.org/466321 | 15:49 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Stop gating Gnocchi https://review.openstack.org/466318 | 15:49 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql master: Fix dailycount command https://review.openstack.org/467293 | 15:50 |
| melwitt | the first hit is a patch that's failing lots of jobs https://review.openstack.org/#/c/460455/ | 15:51 |
| *** eranrom has quit IRC | 15:51 | |
| melwitt | same with the second one https://review.openstack.org/#/c/466051/ | 15:52 |
| *** jamesmca_ has quit IRC | 15:52 | |
| *** e0ne has quit IRC | 15:52 | |
| mtreinish | is there anyway to get the email address for someone who filed a story in storyboard? | 15:53 |
| clarkb | melwitt: ok maybe the query is too broad then (which is what I am running into trying to see if anything else is affected) | 15:53 |
| mriedem | clarkb: melwitt: i think we should probably delete the query from e-r | 15:54 |
| mriedem | and if there is a new failure spiking, we need a new bug | 15:54 |
| melwitt | yeah, I was thinking the same. usually when we close out a bug we delete the related e-r query | 15:55 |
| melwitt | otherwise in theory all you catch are patches with bugs in them or WIP patches | 15:55 |
| clarkb | melwitt: well the idea is that e-r will only match if the same underlying bug happens again | 15:56 |
| clarkb | if not the query is buggy and too broad | 15:56 |
| *** jcoufal has quit IRC | 15:57 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Separate Puppet OpenStack tarball jobs from Python jobs https://review.openstack.org/467294 | 15:58 |
| *** jamesmca_ has joined #openstack-infra | 15:58 | |
| *** Apoorva has joined #openstack-infra | 15:59 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Add non-voting osa-security job for Fedora 25 https://review.openstack.org/467297 | 15:59 |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Make Debian job voting for osa-security https://review.openstack.org/467298 | 15:59 |
| *** Apoorva has quit IRC | 15:59 | |
| *** Apoorva has joined #openstack-infra | 16:00 | |
| fungi | mtreinish: looking through the sb db, it appears that a made-up e-mail address is substituted if the user didn't grant access to theirs on launchpad, so it depends on which user to a great extent | 16:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create zuul-base-jobs and zuul-jobs https://review.openstack.org/467300 | 16:00 |
| *** esikache1 has joined #openstack-infra | 16:01 | |
| *** dizquierdo has joined #openstack-infra | 16:01 | |
| fungi | mtreinish: looks like ~40% of the users in sb have invalid e-mail addresses on record | 16:01 |
| mtreinish | fungi: https://storyboard.openstack.org/#!/story/2001034 | 16:01 |
| mriedem | clarkb: in this case it's too broad, i'll delete it | 16:02 |
| fungi | mtreinish: that' | 16:02 |
| *** jpich has quit IRC | 16:02 | |
| fungi | s gavin.brebner (at) hpe.com | 16:02 |
| pabelanger | okay, nb03.o.o is online and building centos-7 DIB now | 16:03 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Remove stale query for bug 1660747 https://review.openstack.org/467302 | 16:03 |
| openstack | bug 1660747 in OpenStack Compute (nova) "test_list_servers_filter_by_error_status intermittently fails with MismatchError on no servers in response" [Medium,Fix released] https://launchpad.net/bugs/1660747 - Assigned to melanie witt (melwitt) | 16:03 |
| mriedem | mtreinish: ^ | 16:03 |
| mtreinish | fungi: thanks | 16:03 |
| *** rhallisey has quit IRC | 16:03 | |
| pabelanger | once centos-7 DIB comes online I'll decommission nb01.o.o | 16:03 |
| mtreinish | fungi: I'm kinda dissapointed, I was hoping there was a new subunit2sql user :/ | 16:04 |
| mtreinish | but I knew hpe was using it | 16:04 |
| *** rhallisey has joined #openstack-infra | 16:04 | |
| tushar__ | clarkb : DIB image creation is failing in our CI ...error log - https://pastebin.com/JCGQcx8A | 16:04 |
| mtreinish | mriedem: +A | 16:04 |
| *** jkilpatr has quit IRC | 16:04 | |
| pabelanger | tushar__: you can create a 2nd SSH key for the zuul user, which is not used ATM: http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh#n43 | 16:05 |
| clarkb | tushar__: see https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker/extra-data.d/60-zuul-user | 16:05 |
| pabelanger | tushar__: eventually jenkins user will be removed | 16:05 |
| *** hashar is now known as hasharAway | 16:08 | |
| smcginnis | Could we get an account disabled? | 16:08 |
| smcginnis | Xiexianbin appears to be a user account posting CI message. | 16:08 |
| tushar__ | Pabelanger : you mean I have to create the public key for zuul user manually before image creation | 16:08 |
| tushar__ | ? | 16:08 |
| smcginnis | And useless ones like "Merge failed" and "Starting gate jobs" | 16:08 |
| smcginnis | https://review.openstack.org/#/q/reviewer:Xiexianbin | 16:08 |
| *** esikache1 has quit IRC | 16:09 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql master: Fix dailycount command https://review.openstack.org/467293 | 16:10 |
| clarkb | smcginnis: yes doing so now | 16:11 |
| smcginnis | clarkb: Thank you! | 16:11 |
| *** esikache1 has joined #openstack-infra | 16:11 | |
| clarkb | 10842 was the account id | 16:12 |
| fungi | clarkb: smcginnis: see the status log from earlier where i already did | 16:13 |
| clarkb | oh yup inactive is Y in db | 16:13 |
| smcginnis | fungi: Oh, great. | 16:13 |
| fungi | at 14:04 utc in scrollback | 16:13 |
| *** matrohon has quit IRC | 16:15 | |
| *** kornica has quit IRC | 16:16 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Remove stale query for bug 1660747 https://review.openstack.org/467302 | 16:18 |
| openstack | bug 1660747 in OpenStack Compute (nova) "test_list_servers_filter_by_error_status intermittently fails with MismatchError on no servers in response" [Medium,Fix released] https://launchpad.net/bugs/1660747 - Assigned to melanie witt (melwitt) | 16:18 |
| *** apevec has left #openstack-infra | 16:18 | |
| *** vhosakot has joined #openstack-infra | 16:19 | |
| *** shardy has quit IRC | 16:19 | |
| *** bobh has quit IRC | 16:19 | |
| pabelanger | clarkb: Ah, I see an issue with nb03.o.o, I didn't attach a cinder volume | 16:20 |
| *** jkilpatr has joined #openstack-infra | 16:20 | |
| pabelanger | let me do that now | 16:20 |
| *** trown is now known as trown|lunch | 16:21 | |
| *** ykarel has quit IRC | 16:23 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql master: Fix dailycount command https://review.openstack.org/467293 | 16:24 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql master: Fix unit using incorrect type in fake default https://review.openstack.org/467308 | 16:24 |
| pabelanger | mnaser: Oh, I think we hit our volume quota for vexxhost. I was looking at creating a 1TB volume for nb03.o.o, type HDD | 16:26 |
| *** dtantsur is now known as dtantsur|afk | 16:26 | |
| *** dizquierdo has quit IRC | 16:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Strip unneeded cmd, changed, stdout and stderr from results https://review.openstack.org/467310 | 16:29 |
| *** chlong has quit IRC | 16:30 | |
| *** yamamoto has joined #openstack-infra | 16:30 | |
| *** salv-orlando has joined #openstack-infra | 16:31 | |
| *** mariusv is now known as lunarlamp | 16:33 | |
| *** sshnaidm is now known as sshnaidm|off | 16:34 | |
| *** gouthamr has joined #openstack-infra | 16:35 | |
| *** mdrabe_ has quit IRC | 16:37 | |
| *** tosky has quit IRC | 16:39 | |
| *** armax has joined #openstack-infra | 16:39 | |
| *** yamamoto has quit IRC | 16:39 | |
| clarkb | digging into the citycloud multiple private ip address thing a bit more, only the Sto2 region exhibits the behavior on a spot check. The other three regions are fine | 16:39 |
| clarkb | so I don't think its anything wrong iwth how we configured networkign via the cloud launcher stuff | 16:40 |
| clarkb | and seems to happen in about 1/10 servers in sto2 | 16:40 |
| pabelanger | clarkb: is that nodepool failure to SSH to a node? | 16:42 |
| clarkb | pabelanger: where I've seen it is node to node ssh in multinode tests (so we stopped providing the multinode labels until shade fix gets in) | 16:43 |
| clarkb | pabelanger: its possible that there are nodepool boot fails due to this too, I haven't been able to confirm as nodepool doesn't log the private ip situation | 16:43 |
| *** ltomasbo is now known as ltomasbo|away | 16:44 | |
| *** tphummel has joined #openstack-infra | 16:44 | |
| *** mdrabe_ has joined #openstack-infra | 16:45 | |
| *** aarefiev is now known as aarefiev_afk | 16:45 | |
| pabelanger | clarkb: Hmm, if online we had console-log support landed | 16:46 |
| *** sree has joined #openstack-infra | 16:47 | |
| *** bhavik1 has joined #openstack-infra | 16:47 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Bump diskimage-builder dependency to 2.0.0 https://review.openstack.org/467282 | 16:48 |
| *** derekh has quit IRC | 16:48 | |
| *** dhajare has quit IRC | 16:49 | |
| bkero | Is anybody reviewing jeepyb patches anymore? https://review.openstack.org/#/c/463846/ | 16:49 |
| *** sree has quit IRC | 16:51 | |
| clarkb | bkero: done | 16:52 |
| *** tphummel has quit IRC | 16:52 | |
| *** mpaolino has quit IRC | 16:52 | |
| clarkb | bkero: its less of no one reviewing it anymore and having a review backlog 1k items long and prioritizing within that | 16:52 |
| bkero | clarkb: Ah, i see | 16:53 |
| clarkb | (I don't actually know how big the backlog is anymore I stopped looking at it when it was around 800-900 as it was too depressing) | 16:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Fetch server console log if ssh connection fails https://review.openstack.org/452494 | 16:54 |
| clarkb | *stopped looking at the count | 16:54 |
| openstackgerrit | Ben Kero proposed openstack-infra/jeepyb master: Add optional project to expire_old_reviews https://review.openstack.org/463846 | 16:54 |
| bkero | Wow that's big | 16:55 |
| *** thorst_afk has joined #openstack-infra | 16:55 | |
| *** harlowja has joined #openstack-infra | 16:56 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Separate Puppet OpenStack tarball jobs from Python jobs https://review.openstack.org/467294 | 16:56 |
| *** lucasagomes is now known as lucas-afk | 16:57 | |
| *** thorst_afk has quit IRC | 16:57 | |
| *** thorst_afk has joined #openstack-infra | 16:58 | |
| pabelanger | mgagne: internap question, is AS2.2xE5-2620v1.256GB.8x1TB.HDD.1GbE baremetal hosts? | 16:59 |
| *** eharney has quit IRC | 16:59 | |
| *** baoli has quit IRC | 17:00 | |
| *** jamesmca_ has quit IRC | 17:01 | |
| *** baoli has joined #openstack-infra | 17:01 | |
| *** ralonsoh has quit IRC | 17:01 | |
| *** flepied has quit IRC | 17:01 | |
| pabelanger | clarkb: going to see if I can hit SSH issue with nodepool debug script I have | 17:01 |
| clarkb | pabelanger: against sto2? | 17:01 |
| *** thorst_afk has quit IRC | 17:02 | |
| pabelanger | clarkb: Ya, I could do lon1 also, it had a failure recently | 17:02 |
| *** ijw has joined #openstack-infra | 17:03 | |
| pabelanger | but sto2 has more SSH failures | 17:03 |
| clarkb | ya sto2 exhibits the weirdness more | 17:03 |
| *** salv-orlando has quit IRC | 17:03 | |
| pabelanger | let me start with sto2 | 17:03 |
| *** thorst_afk has joined #openstack-infra | 17:03 | |
| clarkb | what you will eb looking for is mulitple private IPs and in the case of nodepool ssh failure the floating ip is liekly associated to the wrong ip (eg not the private ip configured on the VM() | 17:03 |
| *** jamesmca_ has joined #openstack-infra | 17:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Temporarily(?) add some debug statements around git repo creation https://review.openstack.org/466810 | 17:04 |
| *** Apoorva_ has joined #openstack-infra | 17:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467324 | 17:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467325 | 17:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467326 | 17:05 |
| lunarlamp | heya, how do you guys do capacity monitoring ? | 17:08 |
| *** Apoorva has quit IRC | 17:08 | |
| clarkb | lunarlamp: the zuul status page http://status.openstack.org/zuul/ has simplified versions of what we look at | 17:09 |
| clarkb | lunarlamp: jobs running vs waiting, along with total test node capacity | 17:09 |
| clarkb | grafana.openstack.org has a lot more details you can use to dig in | 17:09 |
| *** bobh has joined #openstack-infra | 17:09 | |
| pabelanger | okay, script looping. Lets see if I hit the SSH failures | 17:10 |
| lunarlamp | I'm referring more to knowing where are you at with the usage | 17:10 |
| lunarlamp | sorry | 17:10 |
| lunarlamp | let's say you have 100 hypervisors without using ceilometer how do you know when you are close to full ? | 17:11 |
| clarkb | lunarlamp: so we don't actually have that problem today, we are largely consumers of cloud not running clouds. (we do run a couple cloud regions but we are the only users of that cloud and we have fixed the quota to be max utilization, so we can then just track what we use on the zuul status page) | 17:12 |
| clarkb | lunarlamp: so we track what we use against cloud quota, not what users are using within a cloud | 17:12 |
| lunarlamp | ah, cool...thanks | 17:12 |
| *** bobh has quit IRC | 17:14 | |
| *** feisky has quit IRC | 17:14 | |
| *** eharney has joined #openstack-infra | 17:15 | |
| *** baoli has quit IRC | 17:15 | |
| *** bhavik1 has quit IRC | 17:16 | |
| *** jaosorior is now known as jaosorior_away | 17:17 | |
| *** SumitNaiksatam has quit IRC | 17:17 | |
| *** tphummel has joined #openstack-infra | 17:17 | |
| *** jamesmca_ has quit IRC | 17:17 | |
| *** salv-orlando has joined #openstack-infra | 17:19 | |
| *** Hal has joined #openstack-infra | 17:19 | |
| *** baoli has joined #openstack-infra | 17:19 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add default-branch property to projects https://review.openstack.org/467334 | 17:20 |
| *** esikache1 has quit IRC | 17:21 | |
| *** adam_g has quit IRC | 17:21 | |
| *** adam_g has joined #openstack-infra | 17:22 | |
| clarkb | lunarlamp: generally though I believe the nova scheduler has some idea of what its available capacity is and can likely be queried. Though I think that is what ceilometer tries to make simple? In any case we don't really do things like that so not sure we are best to help | 17:22 |
| *** thorst_afk has quit IRC | 17:23 | |
| *** salv-orlando has quit IRC | 17:23 | |
| *** thorst_afk has joined #openstack-infra | 17:23 | |
| *** tphummel has quit IRC | 17:24 | |
| *** esikache1 has joined #openstack-infra | 17:24 | |
| *** rbrndt has quit IRC | 17:24 | |
| lunarlamp | clarkb: thanks a lot, getting that data with a python script and push it to collectd but I thought you guys are doing it different | 17:24 |
| *** thorst_afk has quit IRC | 17:27 | |
| *** thorst_afk has joined #openstack-infra | 17:29 | |
| *** thorst_afk has quit IRC | 17:29 | |
| mgagne | pabelanger: they are hosts from our baremetal offering. tbh, they shouldn't be visible in mtl01 | 17:29 |
| *** kzaitsev_mb has joined #openstack-infra | 17:30 | |
| *** jpena is now known as jpena|off | 17:30 | |
| pabelanger | mgagne: I actually ran into a project manager at the airport from boston summit, he mentioned he was going to expose some for us to possible test with | 17:30 |
| fungi | lunarlamp: note that this channel is where we maintain community services (mailing list server, wiki server, code review system, continuous integration testing for proposed development, et cetera). you may be looking for a channel about running openstack, like #openstack | 17:30 |
| *** esikache1 has quit IRC | 17:30 | |
| *** adam_g has quit IRC | 17:30 | |
| mgagne | pabelanger: yes, was my previous boss =) | 17:30 |
| lunarlamp | fungi: yep, thanks! | 17:31 |
| pabelanger | mgagne: Ya, that is what he mentioned | 17:31 |
| mgagne | pabelanger: we can talk about it a bit later today, in a meeting | 17:31 |
| *** adam_g has joined #openstack-infra | 17:31 | |
| pabelanger | mgagne: sure, no rush. | 17:31 |
| openstackgerrit | David Medberry proposed openstack-infra/irc-meetings master: Initial commit for the Operators Meetup Planning https://review.openstack.org/467336 | 17:33 |
| *** Apoorva_ has quit IRC | 17:33 | |
| *** trown|lunch is now known as trown | 17:33 | |
| *** claudiub has quit IRC | 17:33 | |
| *** claudiub has joined #openstack-infra | 17:34 | |
| *** bobh has joined #openstack-infra | 17:35 | |
| *** dimtruck has quit IRC | 17:36 | |
| pabelanger | clarkb: just reproduced SSH failure | 17:36 |
| pabelanger | looking at logs now | 17:36 |
| *** electrofelix has quit IRC | 17:37 | |
| *** Anticimex has quit IRC | 17:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Temporarily(?) add some debug statements around git repo creation https://review.openstack.org/466810 | 17:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467324 | 17:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467325 | 17:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467326 | 17:38 |
| *** dimtruck has joined #openstack-infra | 17:38 | |
| *** electrofelix has joined #openstack-infra | 17:38 | |
| *** Anticimex has joined #openstack-infra | 17:39 | |
| *** mdurrant has quit IRC | 17:40 | |
| pabelanger | clarkb: interesting, we got 2 private IPs | 17:41 |
| pabelanger | | 8e443f62-4a5e-4571-bd53-b8b15a7e96ba | pabelanger-test | ACTIVE | openstackzuul-network1=10.0.1.166, 10.0.1.167, 188.240.223.45 | ubuntu-xenial-1495438147 | | 17:41 |
| pabelanger | I would have expected only 1 | 17:41 |
| *** dhinesh has joined #openstack-infra | 17:41 | |
| *** shardy_ has quit IRC | 17:42 | |
| fungi | pabelanger: also in sto2? | 17:43 |
| pabelanger | fungi: yes | 17:43 |
| *** kzaitsev_mb has quit IRC | 17:43 | |
| *** jamesmca_ has joined #openstack-infra | 17:45 | |
| *** eranrom has joined #openstack-infra | 17:45 | |
| *** thorst_afk has joined #openstack-infra | 17:47 | |
| *** thorst_afk has quit IRC | 17:47 | |
| clarkb | yes thats the problem | 17:48 |
| clarkb | is fip attached to the private ip that is not configured on host? | 17:48 |
| pabelanger | going to detach the 2nd IP and see if FIP works | 17:48 |
| clarkb | console log should tell you which private ip is real | 17:48 |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Add ansible-hardening repository https://review.openstack.org/467339 | 17:51 |
| *** salv-orlando has joined #openstack-infra | 17:51 | |
| pabelanger | I don't see IPs in console log | 17:51 |
| pabelanger | | 188.240.223.45 | 10.0.1.166 | | 17:52 |
| pabelanger | is our mapping | 17:52 |
| pabelanger | addresses | openstackzuul-network1=10.0.1.166, 10.0.1.167, 188.240.223.45 | | 17:52 |
| pabelanger | if server show | 17:52 |
| *** SumitNaiksatam has joined #openstack-infra | 17:52 | |
| pabelanger | I am guessing .166 is dead | 17:52 |
| clarkb | console log should show you which IP dhcpd | 17:53 |
| clarkb | guessing 167 dhcpd | 17:53 |
| clarkb | not 166 hence the fail | 17:53 |
| *** spzala has quit IRC | 17:54 | |
| pabelanger | http://paste.openstack.org/show/610481/ | 17:54 |
| pabelanger | I don't see dhcp | 17:54 |
| pabelanger | let me check working server | 17:55 |
| *** hongbin has joined #openstack-infra | 17:55 | |
| *** Apoorva has joined #openstack-infra | 17:56 | |
| *** jamesmca_ has quit IRC | 17:56 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Add ansible-hardening repository https://review.openstack.org/467339 | 17:56 |
| *** jamesmca_ has joined #openstack-infra | 17:57 | |
| *** thorst_afk has joined #openstack-infra | 17:59 | |
| *** baoli has quit IRC | 17:59 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:59 | |
| pabelanger | clarkb: we should update /etc/issue to include IP address for login prompt | 18:00 |
| pabelanger | eth0: 192.168.1.100 | 18:00 |
| *** baoli has joined #openstack-infra | 18:00 | |
| pabelanger | myserver login: | 18:00 |
| pabelanger | eg | 18:00 |
| *** jamesmca_ has quit IRC | 18:01 | |
| clarkb | pabelanger: rather than run ip addr? | 18:02 |
| *** thorst_afk has quit IRC | 18:03 | |
| pabelanger | clarkb: as long as we output to console log, I am okay with any command | 18:04 |
| *** spzala has joined #openstack-infra | 18:04 | |
| *** jamesmca_ has joined #openstack-infra | 18:04 | |
| clarkb | oh that should already be in the console log | 18:04 |
| pabelanger | clarkb: see paste above | 18:04 |
| *** bobh has quit IRC | 18:04 | |
| clarkb | hrm I wonder if glean is hard setting it rather than dhcping it | 18:05 |
| *** jkilpatr has quit IRC | 18:06 | |
| *** jkilpatr has joined #openstack-infra | 18:07 | |
| pabelanger | Hmm | 18:07 |
| clarkb | pabelanger: what you can do is ssh into another node in that region then ping both addresses | 18:08 |
| pabelanger | good idea | 18:08 |
| clarkb | pabelanger: the one attached to the fip should not respond if fip is unworking and the other one should respond if the theory is the wrong one is configured | 18:09 |
| clarkb | then from there ssh to the one that pings | 18:09 |
| pabelanger | Ya, 167 is up | 18:09 |
| pabelanger | 166 down | 18:09 |
| pabelanger | and ssh works on 167 | 18:09 |
| *** jamesmca_ has quit IRC | 18:10 | |
| *** bobh has joined #openstack-infra | 18:10 | |
| clarkb | ya so this issue is likely behind the nodepool ssh errors, however we don't fail jobs due to that we just delete and try again | 18:10 |
| *** esikache1 has joined #openstack-infra | 18:10 | |
| clarkb | also the shade workaround won't fix this instance of the problem because it relies on assuming the fip is working (which is fine when the test is actually running as fip had to work there) | 18:10 |
| pabelanger | clarkb: so, how is the server getting 2 IPs? | 18:11 |
| rlandy | pabelanger: hi - we are looking to move some upstream ci jobs to rdocloud. We are deciding on the name for the user to execute jobs. It was suggested that we check with you/upstream infra to see of there are requirements for the name of the user. | 18:11 |
| clarkb | pabelanger: we don't know something in the cloud | 18:11 |
| clarkb | pabelanger: I sent email to citycloud about it, but haven't heard back from them on that (I think they got the emails though beacuse the missing flavor in la1 was added) | 18:11 |
| *** bobh has quit IRC | 18:12 | |
| pabelanger | rlandy: nope, just what we layout at https://docs.openstack.org/infra/system-config/third_party.html | 18:12 |
| pabelanger | clarkb: okay, that is what I expected. I couldn't see anything in shade what was requesting multiple IPs | 18:13 |
| *** jamesmca_ has joined #openstack-infra | 18:13 | |
| fungi | rlandy: not planning to use the https://wiki.openstack.org/wiki/ThirdPartySystems/RedHat_RDO_CI account i take it? | 18:14 |
| clarkb | pabelanger: it almost looks like the vm is getting two ports assigned to it in openstack land, each of which has an associated ip aconfigured in the dhcp server. But then when the server is booted only one port is actually attached to the server so only that one is dhcp'd and working | 18:15 |
| *** pvaneck has joined #openstack-infra | 18:15 | |
| rlandy | fungi: it's up for debate ... we need to settle on a user and and an email account. we will likely stick with rdo-ci-admins email account but the user name is in question | 18:15 |
| clarkb | pabelanger: to make things more confusing it doesn't happen 100% of the time, and when it happens we only break nodepool ssh if the floating IP is attached to the wrong one and only break multiode if the floating ip is attached to the second one and that works | 18:16 |
| rlandy | fungi: currently, upstream the name is openstack-nodepool - which is the name we initially requested. the rdocloud suggested something tripleo specific - as there may be other ci users added | 18:16 |
| rlandy | also they have a naming convention | 18:16 |
| clarkb | pabelanger: I'm hoping citycloud will be able to determine what is goign on, until then I think we accept the ssh failures and workaround the issue with multinode (which new release of shade should do) | 18:16 |
| rlandy | so that is why I am asking for input before requesting the exact name | 18:17 |
| fungi | rlandy: just to confirm i'm not misunderstanding, you're asking what username you should select for a new account on review.openstack.org, right? | 18:18 |
| *** hieulq_ has quit IRC | 18:19 | |
| rlandy | fungi: yes and the name that will deploy the test stack in rdocloud - two different entities | 18:19 |
| *** dizquierdo has joined #openstack-infra | 18:20 | |
| clarkb | pabelanger: if you can leave pabelanger-test up I can send citycloud a followup with this failure case (fip attached to wrong ip) and cc you | 18:20 |
| clarkb | pabelanger: since my original email only handled the case of fip attached to correct private ip but nodepool confused and multinode jobs failing | 18:20 |
| *** jamesmca_ has quit IRC | 18:21 | |
| *** jamesmca_ has joined #openstack-infra | 18:22 | |
| *** rbrndt has joined #openstack-infra | 18:24 | |
| pvaneck | hi infra, i have a patch to fix lint errors in puppet-refstack that would be great to have merged if someone could take a look: https://review.openstack.org/#/c/460787/ | 18:25 |
| clarkb | did we just lose osic? lets of node queued in delete /me digs in | 18:26 |
| pabelanger | clarkb: yes, I do see us booting servers with 2 IPs assigned and SSH works, like you said 50/50 chance on wrong one attaching to FIP | 18:27 |
| pabelanger | clarkb: sure, I keep server online | 18:27 |
| *** felipemonteiro has joined #openstack-infra | 18:27 | |
| clarkb | ya looks like osic identity service is gone | 18:28 |
| clarkb | Failed to discover available identity versions when contacting https://cloud1.osic.org:5000. Attempting to parse version from URL. | 18:28 |
| clarkb | on catalog list | 18:28 |
| cloudnull | clarkb: pabelanger: can we disable for now | 18:28 |
| cloudnull | I just got wind of this | 18:28 |
| *** felipemonteiro_ has joined #openstack-infra | 18:28 | |
| cloudnull | looks like our partners nuked the domain nam e | 18:28 |
| pabelanger | clarkb: yes, seeing that too | 18:28 |
| pabelanger | cloudnull: ouch | 18:29 |
| clarkb | cloudnull: ya, how far should we disable? | 18:29 |
| cloudnull | we're looking to restore things. | 18:29 |
| *** electrofelix has quit IRC | 18:29 | |
| clarkb | ok | 18:29 |
| cloudnull | I can give you the IP | 18:29 |
| cloudnull | but ssl will be broken | 18:29 |
| cloudnull | if we can remove cert verification that should still work | 18:29 |
| * clarkb writes change to set max servers to 0 for now | 18:29 | |
| cloudnull | ++ | 18:29 |
| pabelanger | wfm | 18:30 |
| *** baoli has quit IRC | 18:30 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Set osic max servers to 0 https://review.openstack.org/467355 | 18:31 |
| clarkb | lets start there | 18:31 |
| clarkb | cloudnull: we can ignore ssl verification, I'm not sure how comfortable people are doing that especially if things are changing domains/moving (may possibly disclose creds somewhere we shouldn't?) | 18:31 |
| *** baoli has joined #openstack-infra | 18:32 | |
| *** ijw has quit IRC | 18:32 | |
| clarkb | pvaneck: done | 18:32 |
| cloudnull | I'm going to pop a new domain for the time being. It'll take a little while to get things back online. | 18:32 |
| clarkb | cloudnull: no worries | 18:33 |
| *** felipemonteiro has quit IRC | 18:33 | |
| pabelanger | max-server 0 is good, we can get more load on citycloud for a while | 18:33 |
| pabelanger | see how it holds up | 18:33 |
| *** annegentle has joined #openstack-infra | 18:33 | |
| clarkb | pabelanger: ha. Ok going to wriet email to them about your test server and how it can cause floating ip attachements to fail | 18:33 |
| pabelanger | clarkb: ack | 18:33 |
| cloudnull | if we change the domain name what all needs to change on the infra side to make that go ? | 18:34 |
| cloudnull | looks like osic.org is dead jim | 18:34 |
| pvaneck | clarkb: thanks! | 18:34 |
| clarkb | cloudnull: we will need to update your clouds.yaml files to point at the new authurl | 18:34 |
| cloudnull | so I suspect we'll need to sed quite a few things ? | 18:34 |
| clarkb | s/your/our/ | 18:34 |
| clarkb | cloudnull: its 3 template files in openstack-infra/system-config | 18:35 |
| cloudnull | cool . | 18:35 |
| clarkb | cloudnull: let me get you links | 18:35 |
| cloudnull | ok, few... was thinking it was going to be terrible | 18:35 |
| cloudnull | :) | 18:35 |
| *** krtaylor has quit IRC | 18:35 | |
| pabelanger | cloudnull: osic.xyz | 18:36 |
| pabelanger | :D | 18:36 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/clouds.yaml.erb https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/puppetmaster/all-clouds.yaml.erb https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/puppetmaster/ansible-clouds.yaml.erb | 18:36 |
| clarkb | cloudnull: ^ those three files | 18:36 |
| cloudnull | excellent | 18:36 |
| cloudnull | I was thinking osic.null | 18:36 |
| cloudnull | :p | 18:36 |
| clarkb | osic.solutions | 18:36 |
| cloudnull | I like it | 18:37 |
| cloudnull | or osic.not | 18:37 |
| fungi | openstack rennovation center | 18:37 |
| pabelanger | osic.ng | 18:37 |
| clarkb | pabelanger: your test server is in sto2 right? | 18:39 |
| pabelanger | clarkb: yes | 18:40 |
| *** thorst_afk has joined #openstack-infra | 18:43 | |
| clarkb | ok that email is all sent. Meeting in 15 minutes, then mordred I hope we can do shade things after the meeting? | 18:44 |
| *** ihrachys is now known as ihrachys|afk | 18:46 | |
| *** salv-orlando has quit IRC | 18:46 | |
| mhayden | fungi: thanks for the comments! :) | 18:46 |
| fungi | mhayden: my pleasure | 18:47 |
| *** thorst_afk has quit IRC | 18:48 | |
| *** rockyg has joined #openstack-infra | 18:48 | |
| *** kzaitsev_mb has quit IRC | 18:48 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:49 | |
| *** thorst_afk has joined #openstack-infra | 18:50 | |
| *** bobh has joined #openstack-infra | 18:50 | |
| mordred | clarkb: yes. sorry - today has been eaten much more strongly by meetings than I was hoping :( | 18:53 |
| fungi | the meetings are strong with this one | 18:54 |
| *** salv-orlando has joined #openstack-infra | 18:56 | |
| clarkb | mordred: is shade master currently good? i can push new releases ps | 18:56 |
| *** adam_g has quit IRC | 18:57 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Add ansible-hardening repository https://review.openstack.org/467339 | 18:58 |
| *** adam_g has joined #openstack-infra | 18:59 | |
| *** kzaitsev_mb has quit IRC | 18:59 | |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| *** jcoufal has joined #openstack-infra | 19:02 | |
| *** salv-orlando has quit IRC | 19:02 | |
| *** jcoufal has quit IRC | 19:03 | |
| *** funzo_ is now known as funzo | 19:04 | |
| *** jcoufal has joined #openstack-infra | 19:04 | |
| *** Apoorva has quit IRC | 19:04 | |
| *** Apoorva has joined #openstack-infra | 19:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set osic max servers to 0 https://review.openstack.org/467355 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Allow a user to submit start and end time as strings https://review.openstack.org/467257 | 19:06 |
| *** thorst_afk has quit IRC | 19:06 | |
| mordred | clarkb: sorry. that ^^ is the last thing I want in - passing through the datetime's unmodified violates the Nova API | 19:07 |
| *** salv-orlando has joined #openstack-infra | 19:10 | |
| *** sree has joined #openstack-infra | 19:11 | |
| *** salv-orlando has quit IRC | 19:11 | |
| pabelanger | mnaser: did you see my comments about volume quota for vexxhost? | 19:14 |
| *** sshnaidm|off is now known as sshnaidm | 19:15 | |
| *** sree has quit IRC | 19:15 | |
| *** xyang1 has joined #openstack-infra | 19:16 | |
| *** thorst_afk has joined #openstack-infra | 19:17 | |
| *** bh526r has quit IRC | 19:20 | |
| *** thorst_afk has quit IRC | 19:22 | |
| *** krtaylor has joined #openstack-infra | 19:28 | |
| pabelanger | clarkb: Ya, if we attach the FIP to the first private IP, it seems to always fail. If shade picks the 2nd, nodepool seems to ssh | 19:28 |
| clarkb | pabelanger: interesting | 19:28 |
| clarkb | pabelanger: maybe you can respond to the email I cc'd you on with that info? | 19:28 |
| pabelanger | sure | 19:28 |
| mordred | pabelanger: that doesn't sound like logic we could actually put in shade ... I'll try to think of a way to be able to express it in config though ... "if this cloud gives more than one private ip and you're trying to attach a FIP, please use ip [-1]" | 19:29 |
| pabelanger | mordred: thanks | 19:29 |
| clarkb | mordred: ya I'm not sure its useful to shade, but may help citycloud diagnose what is going on | 19:29 |
| mordred | ++ | 19:29 |
| mordred | well - the question is "is this a bug in a release of openstack - or specific to citycloud" | 19:30 |
| mordred | and if it'sa bug, the next question is "is there a use case for getting more than one private ip from the same network" | 19:30 |
| mordred | but maybe we _could_ make the logic be "if a user requests a FIP and the server has two private ips from the same network and the user hasn't otherwise picked one, don't pick the first one, pick the last one" | 19:31 |
| clarkb | mordred: thats why I think its not something we should worry to much about | 19:31 |
| clarkb | I can't figure a reason to have multiple IPs in the same subnet | 19:31 |
| mordred | me either | 19:31 |
| mordred | clarkb: do we have shade debug logging logging to anywhere for nodepool currently? | 19:32 |
| mordred | oh - nevermind- we'd need http tracing to learn what I'm interested in for hose servers | 19:32 |
| clarkb | mordred: we don't | 19:33 |
| pabelanger | mordred: which logger are you looking for? I can add it to my reproduce script | 19:33 |
| mordred | in our case, shade WILL be picking one of them - because we create fips on the port of the fixed ip of the server | 19:33 |
| mordred | pabelanger: nevermind - I just realized logically that there is only one answer to my question :) | 19:33 |
| pabelanger | k | 19:33 |
| mordred | if len(ports) == 1: | 19:34 |
| mordred | port = ports[0] | 19:34 |
| mordred | we're doing that right now | 19:34 |
| mordred | so it's arbitrary | 19:34 |
| mordred | gah. I can't read | 19:36 |
| *** dizquierdo is now known as dizquierdo_afk | 19:36 | |
| *** priteau has quit IRC | 19:39 | |
| *** jamesmca_ has quit IRC | 19:39 | |
| *** jcoufal has quit IRC | 19:45 | |
| *** eharney has quit IRC | 19:46 | |
| *** clenimar has quit IRC | 19:47 | |
| *** thorst_afk has joined #openstack-infra | 19:50 | |
| *** mdrabe_ is now known as mdrabe | 19:51 | |
| *** e0ne has joined #openstack-infra | 19:51 | |
| mordred | clarkb, pabelanger: it is worth communicating to citycloud, since they would likely not know this otherwise... the addresses dict on the server that WE are responding to is the one we build from the neutron port list | 19:52 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/system-config master: Changed the domain name for cloud1.osic.org https://review.openstack.org/467371 | 19:53 |
| clarkb | mordred: it shows up using openstackclient too, so I don't think its shade specific | 19:53 |
| mordred | clarkb, pabelanger: so I think on our side we should capture http logs in the reproduce script so that we can see if nova returned us an empty or full addresses list - and if it returned an addresses list if it had one or or two fixed ips | 19:53 |
| mordred | clarkb: ah - ok. so openstackclient does also show the two ips? | 19:53 |
| clarkb | mordred: yes | 19:53 |
| cloudnull | clarkb: pabelanger: that's done, however it'll take a couple hours to get it going. | 19:53 |
| mordred | clarkb: ok. that's good to know | 19:54 |
| mordred | clarkb: any chance you know if the order of the ips in from openstackclient consistently matches the order in the server addresses list? | 19:54 |
| clarkb | mordred: I do not know that | 19:54 |
| mordred | ok | 19:54 |
| clarkb | cloudnull: thanks | 19:55 |
| clarkb | cloudnull: can you give us a heads up when you think we should merge that and reincrease max-servers? | 19:55 |
| mordred | clarkb: can you remind me the id of the held server? | 19:55 |
| clarkb | mordred: d46ce7f4-8691-43f3-8bf3-cdd0b77ddb8a | 19:56 |
| *** ijw has joined #openstack-infra | 19:57 | |
| mordred | clarkb: ok. so - fwiw, the 'good' one is the most recent one if sorted by created_at | 19:57 |
| mordred | (which is the natural order neutron is returning these in, fwiw) | 19:58 |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Add ansible-hardening repository https://review.openstack.org/467339 | 19:59 |
| pabelanger | okay, volume setup on nb03.o.o, rebooting server to confirm fstab is setup properly | 20:00 |
| *** eharney has joined #openstack-infra | 20:01 | |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Add pagination for the list_volumes call https://review.openstack.org/466927 | 20:08 |
| *** sambetts is now known as sambetts|afk | 20:08 | |
| *** numans has quit IRC | 20:10 | |
| *** dave-mccowan has quit IRC | 20:10 | |
| *** boris-42_ has joined #openstack-infra | 20:11 | |
| *** numans has joined #openstack-infra | 20:12 | |
| pabelanger | okay, nb03.o.o back online | 20:13 |
| pabelanger | will stop nb01.o.o first | 20:13 |
| pabelanger | clarkb: Oh, we are still trying to upload images to osic | 20:14 |
| pabelanger | and failing | 20:14 |
| clarkb | ya, sounds like we should be fixed in the not too distant future | 20:15 |
| openstackgerrit | Hari proposed openstack-infra/project-config master: Add ranger and openstack/ranger-agent project https://review.openstack.org/467374 | 20:16 |
| *** e0ne has quit IRC | 20:18 | |
| *** spzala has quit IRC | 20:19 | |
| *** askb has joined #openstack-infra | 20:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to job repos https://review.openstack.org/467375 | 20:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 20:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 20:21 |
| *** jkilpatr has quit IRC | 20:22 | |
| *** rcernin has joined #openstack-infra | 20:24 | |
| openstackgerrit | Hari proposed openstack-infra/project-config master: Add ranger and openstack/ranger-agent project https://review.openstack.org/467374 | 20:25 |
| *** numans has quit IRC | 20:26 | |
| fungi | pabelanger: i guess you could locally s/0/-1/ on those while you're testing, if you need | 20:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to job repos https://review.openstack.org/467375 | 20:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 20:30 |
| *** kgiusti has quit IRC | 20:30 | |
| *** Apoorva_ has joined #openstack-infra | 20:32 | |
| *** numans has joined #openstack-infra | 20:32 | |
| *** harlowja_ has joined #openstack-infra | 20:32 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/reviewstats master: Adds docs and api-ref stat measures https://review.openstack.org/461280 | 20:33 |
| *** harlowja has quit IRC | 20:33 | |
| *** mattmceuen has quit IRC | 20:34 | |
| *** Apoorva has quit IRC | 20:36 | |
| *** gardlt has quit IRC | 20:36 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add driver-specific pipeline requirements https://review.openstack.org/466105 | 20:37 |
| mordred | clarkb: ok. I have a patch almost done that I think will make the other side of this coin better too ... fixing up tests now | 20:37 |
| *** rockyg has quit IRC | 20:37 | |
| cloudnull | clarkb: so https://cloud1.osic.xyz/ now works however I need to do some letsencrypt tomfoolery w/ our f5 to get that going. If we can disable ssl verification using the new domain for the time being we can flip the max-servers bits now, otherwise we will have to likely wait until tomorrow as I dont have direct access to the f5 | 20:39 |
| *** jamesmca_ has joined #openstack-infra | 20:40 | |
| openstackgerrit | Hari proposed openstack-infra/project-config master: Add ranger and openstack/ranger-agent project https://review.openstack.org/467374 | 20:41 |
| clarkb | infra-root ^ thoughts on turning off ssl verification temporarily? | 20:41 |
| mordred | clarkb: doesn't bother me in this scenario | 20:43 |
| clarkb | mordred: and re shade we will release with that change too? | 20:44 |
| fungi | clarkb: i'd be inclined to just wait... we don't seem to be falling behind in zuul anyway | 20:44 |
| *** jamesmca_ has quit IRC | 20:45 | |
| *** yamahata has quit IRC | 20:45 | |
| openstackgerrit | Hari proposed openstack-infra/project-config master: Add ranger and openstack/ranger-agent project https://review.openstack.org/467374 | 20:46 |
| pabelanger | Ya, I'd be okay with disabling image upload too | 20:47 |
| clarkb | if we are going to wait lets do ^ | 20:47 |
| pabelanger | is -1 max-server what does uploads? | 20:48 |
| pabelanger | can never remember | 20:48 |
| clarkb | ya | 20:48 |
| *** spzala has joined #openstack-infra | 20:49 | |
| pabelanger | k, let me patch | 20:49 |
| *** jamesmca_ has joined #openstack-infra | 20:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Set max-servers -1 for osic https://review.openstack.org/467381 | 20:50 |
| pabelanger | fungi: clarkb mordred ^ | 20:50 |
| *** annegentle has quit IRC | 20:50 | |
| clarkb | approved | 20:52 |
| *** flepied has joined #openstack-infra | 20:52 | |
| *** fguillot has quit IRC | 20:53 | |
| fungi | as an aside, i find it rather surprising that intel ripped out hosting of the domain (a domain they'd already paid for, and presumably the dns hosting for it was costing them basically nothing?). i mean, i'm sure it wasn't a malicious choice, but i can certainly see how it might seem like a spiteful move | 20:54 |
| fungi | by contrast, hosting the hardware and network connectivity is likely costing rackspace a great deal | 20:54 |
| *** david-lyle has joined #openstack-infra | 20:54 | |
| cloudnull | it was a surprise to us | 20:55 |
| cloudnull | sadly | 20:55 |
| cloudnull | we still have the cloud online for community use | 20:55 |
| fungi | yep, and i can't begin to express how grateful i am for that | 20:55 |
| *** thorst_afk has quit IRC | 20:56 | |
| clarkb | ++ | 20:56 |
| cloudnull | it is being decom'd but it was not supposed to happen today :) | 20:56 |
| *** yamahata has joined #openstack-infra | 20:56 | |
| fungi | right, figured that was coming from the day osic ended, but we've been thrilled to be able to continue using it while it's still up | 20:56 |
| fungi | so given that the only hosting burden on intel was a domain they'd already registered (until 2024!!!) it's sad to see they decomissioned that before rackspace got around to decommissioning the gear | 20:58 |
| *** vhosakot_ has joined #openstack-infra | 20:59 | |
| cloudnull | ++ | 20:59 |
| *** vhosakot has quit IRC | 21:00 | |
| *** bobh has quit IRC | 21:01 | |
| *** thorst_afk has joined #openstack-infra | 21:02 | |
| *** trown is now known as trown|outtypewww | 21:04 | |
| *** dprince has quit IRC | 21:04 | |
| *** harlowja_ has quit IRC | 21:05 | |
| *** gouthamr has quit IRC | 21:05 | |
| *** rhallisey has quit IRC | 21:06 | |
| clarkb | we are post failuring | 21:06 |
| *** Apoorva_ has quit IRC | 21:06 | |
| clarkb | log fs is full again | 21:06 |
| *** thorst_afk has quit IRC | 21:06 | |
| *** Apoorva has joined #openstack-infra | 21:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Pick most recent rather than first fixed address https://review.openstack.org/467385 | 21:07 |
| pabelanger | great | 21:07 |
| mordred | clarkb: ^^ tell me what you think of that | 21:07 |
| clarkb | maybe we need to reduce the log retention for the journald files to something really short like a week | 21:07 |
| mordred | clarkb: also, I removed the warning in there becauseI'm pretty sure it's impossible to get to that case :) | 21:07 |
| clarkb | sdague: ^ fyi | 21:07 |
| fungi | grumble | 21:07 |
| *** jamesdenton has quit IRC | 21:08 | |
| pabelanger | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 21:08 |
| pabelanger | wow, something loves logging | 21:08 |
| clarkb | that is what my vote for addressing this would be | 21:08 |
| *** harlowja has joined #openstack-infra | 21:08 | |
| fungi | clarkb: what's the name pattern? | 21:08 |
| pabelanger | about 2 TB in a week | 21:08 |
| clarkb | pabelanger: I'm almost positive its going to be the journald files | 21:08 |
| fungi | i'll happily start a find/delete older than a week | 21:08 |
| fungi | of those specifically | 21:09 |
| clarkb | fungi: http://logs.openstack.org/57/467257/2/check/gate-shade-dsvm-functional-libs-nv/3795f9a/logs/devstack@54c1932a02054eea9b68c63bbf08894e-0000000000000001-00055035c195690d.journal.gz so devstac@.*.journal.gz | 21:09 |
| clarkb | er devstack@.*.journal.gz | 21:09 |
| fungi | devstack@*.journal.gz looks like | 21:09 |
| fungi | (shell glob, not regex) | 21:10 |
| clarkb | wait there is also a devstack.journal.gz | 21:10 |
| clarkb | so drop the @ to get them all | 21:10 |
| fungi | k | 21:10 |
| *** david-lyle has quit IRC | 21:10 | |
| fungi | find /srv/static/logs/ -type f -name "devstack*.journal.gz" -mtime +7 | xargs sudo rm | 21:11 |
| fungi | that lgty? | 21:11 |
| clarkb | ya though you may want to use find to delete rather than xargs | 21:12 |
| clarkb | (six one way half dozen the other | 21:12 |
| fungi | using find invokes a shell out to rm for every matching file, vs a pipe to xargs which should batch them up into far fewer... i guess it doesn't matter from a performance perspective in this case? | 21:12 |
| *** Goneri has quit IRC | 21:12 | |
| fungi | using find -exec does i mean | 21:13 |
| clarkb | fungi: there is a -delete iirc that just unlinks | 21:13 |
| fungi | oh, nice. must be post-posix | 21:13 |
| fungi | i'll do that | 21:13 |
| *** armax has quit IRC | 21:14 | |
| clarkb | and there is a nice warning in the man page about how find evaluates its series of args | 21:14 |
| clarkb | eg don't put -delete first it will delete everything | 21:14 |
| fungi | yep, i read the manpage ;_ | 21:14 |
| fungi | ;) | 21:14 |
| fungi | sudo find /srv/static/logs/ -type f -name "devstack*.journal.gz" -mtime +7 -delete | 21:14 |
| fungi | like that, right? | 21:14 |
| clarkb | ya that looks right to me | 21:15 |
| fungi | actually, gonna run this under a screen session as root | 21:15 |
| *** annegentle has joined #openstack-infra | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| fungi | it's going now. feel free to attach if desired | 21:15 |
| *** eharney has quit IRC | 21:16 | |
| fungi | assuming this helps, we can add it to the script in puppet-openstack_ci | 21:17 |
| fungi | #status alert The logserver has filled up, so jobs are currently aborting with POST_FAILURE results; remediation is underway. | 21:18 |
| openstackstatus | fungi: sending alert | 21:18 |
| clarkb | mordred: ip = ipaddress.ip_address(address['ip_address']) <- that line makes me both giggle and sad at the same time | 21:18 |
| clarkb | ITS REALY AN IP ADDRESS! | 21:18 |
| fungi | yo dawg, i heard you liked ip addresses so i put ip addresses in your ip addresses and now you can route while you route | 21:19 |
| *** thorst_afk has joined #openstack-infra | 21:20 | |
| * fungi regurgitates memes from before the dawn of time | 21:20 | |
| mtreinish | clarkb: I think it needs 3 more things called ip_address just to be sure | 21:21 |
| fungi | all your ip address are belong to shade | 21:21 |
| -openstackstatus- NOTICE: The logserver has filled up, so jobs are currently aborting with POST_FAILURE results; remediation is underway. | 21:21 | |
| *** ChanServ changes topic to "The logserver has filled up, so jobs are currently aborting with POST_FAILURE results; remediation is underway." | 21:21 | |
| clarkb | mordred: left a comment on the change, I think its worth a respin simply because everything is going to fail ci for a bit until we get the fs pressure fixed | 21:21 |
| *** david-lyle has joined #openstack-infra | 21:22 | |
| *** flepied has quit IRC | 21:22 | |
| fungi | we're 3.2gb free at the moment | 21:22 |
| clarkb | up to 4.5GB now | 21:22 |
| clarkb | so I think its helping | 21:22 |
| fungi | seems that way | 21:22 |
| clarkb | I've got to pay slightly less attention to things now because larissa had to go to a doctor appointment | 21:23 |
| clarkb | but I'll watch irc to make sure things are happy | 21:23 |
| fungi | i'm semi-paying-attention and semi-cooking-dinner-before-chris-gets-home | 21:24 |
| openstackstatus | fungi: finished sending alert | 21:25 |
| *** gomarivera has joined #openstack-infra | 21:26 | |
| *** jamesmca_ has quit IRC | 21:27 | |
| *** gomarivera has quit IRC | 21:27 | |
| openstackgerrit | Hari proposed openstack-infra/project-config master: Add openstack/ranger and openstack/ranger-agent project https://review.openstack.org/467374 | 21:28 |
| *** jamesmca_ has joined #openstack-infra | 21:30 | |
| fungi | 14gb free now | 21:30 |
| fungi | we'll #status ok once we get up over, say, 100gb free? | 21:31 |
| clarkb | that seems reasonable | 21:32 |
| *** dimtruck is now known as zz_dimtruck | 21:33 | |
| *** srobert has quit IRC | 21:34 | |
| *** spzala has quit IRC | 21:34 | |
| *** dizquierdo_afk has quit IRC | 21:39 | |
| *** thorst_afk has quit IRC | 21:40 | |
| *** priteau has joined #openstack-infra | 21:40 | |
| *** krtaylor has quit IRC | 21:41 | |
| *** spzala has joined #openstack-infra | 21:41 | |
| mordred | clarkb: kk - thanks! | 21:41 |
| openstackgerrit | Merged openstack-infra/subunit2sql master: Fix unit using incorrect type in fake default https://review.openstack.org/467308 | 21:42 |
| *** jkilpatr has joined #openstack-infra | 21:44 | |
| fungi | up to 21gb free now | 21:45 |
| *** priteau has quit IRC | 21:45 | |
| *** spzala has quit IRC | 21:46 | |
| *** jroll has quit IRC | 21:47 | |
| *** jroll has joined #openstack-infra | 21:47 | |
| mordred | clarkb: I also improved it from an assumption to an actual thing | 21:48 |
| *** jroll has quit IRC | 21:49 | |
| *** xyang1 has quit IRC | 21:49 | |
| *** yamahata has quit IRC | 21:50 | |
| *** salv-orlando has joined #openstack-infra | 21:50 | |
| *** lamt has quit IRC | 21:51 | |
| openstackgerrit | Hari proposed openstack-infra/project-config master: Add openstack/ranger and openstack/ranger-agent project https://review.openstack.org/467374 | 21:52 |
| *** funzo has quit IRC | 21:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set max-servers -1 for osic https://review.openstack.org/467381 | 21:52 |
| *** mdrabe has quit IRC | 21:53 | |
| clarkb | mordred: youll also want to recheck the datetime fix | 21:53 |
| *** zz_dimtruck is now known as dimtruck | 21:53 | |
| *** jroll has joined #openstack-infra | 21:53 | |
| *** jamesmca_ has quit IRC | 21:54 | |
| *** esikache1 has quit IRC | 21:54 | |
| *** esikache1 has joined #openstack-infra | 21:57 | |
| *** vhosakot_ has quit IRC | 21:58 | |
| *** esberglu has quit IRC | 21:59 | |
| *** thorst_afk has joined #openstack-infra | 21:59 | |
| *** esberglu has joined #openstack-infra | 22:00 | |
| *** funzo has joined #openstack-infra | 22:00 | |
| *** david-lyle has quit IRC | 22:01 | |
| mordred | kk | 22:02 |
| *** spzala has joined #openstack-infra | 22:02 | |
| *** esikache1 has quit IRC | 22:03 | |
| *** esberglu has quit IRC | 22:04 | |
| mtreinish | fungi: so I was thinking about our websocket problem on firehose, and I'm wondering should we just run something to act as a websocket proxy in front of mosquitto? | 22:06 |
| mtreinish | instead of waiting for the libwebsockets bugs to get fixed | 22:06 |
| *** spzala has quit IRC | 22:06 | |
| mtreinish | not sure what the best way to do that is, but I'm sure there is something out there to do it | 22:09 |
| * fungi plugs "websocket proxy" into a search engine | 22:10 | |
| *** hasharAway has quit IRC | 22:11 | |
| fungi | hrm... websockify ;) | 22:12 |
| fungi | from our friends at novnc | 22:12 |
| clarkb | mordred us there a new ps yet? I dont see it? | 22:12 |
| *** esikache1 has joined #openstack-infra | 22:12 | |
| mtreinish | fungi: oh, yeah that looks like it'll do the trick | 22:12 |
| *** gardlt has joined #openstack-infra | 22:12 | |
| *** baoli has quit IRC | 22:13 | |
| mtreinish | I'll give it a try locally in a bit | 22:13 |
| *** esberglu has joined #openstack-infra | 22:13 | |
| *** armax has joined #openstack-infra | 22:16 | |
| fungi | formerly the wsproxy component of novnc itself | 22:16 |
| mordred | clarkb: my fix of the sorting broke tets | 22:18 |
| *** esberglu has quit IRC | 22:18 | |
| *** jascott1 has joined #openstack-infra | 22:18 | |
| *** esikache1 has quit IRC | 22:20 | |
| *** ramishra has quit IRC | 22:20 | |
| *** gouthamr has joined #openstack-infra | 22:21 | |
| *** felipemonteiro_ has quit IRC | 22:22 | |
| *** jascott1 has quit IRC | 22:23 | |
| *** sdague has quit IRC | 22:24 | |
| *** sc` has quit IRC | 22:24 | |
| *** sc` has joined #openstack-infra | 22:26 | |
| *** thorst_afk has quit IRC | 22:29 | |
| *** spzala has joined #openstack-infra | 22:33 | |
| *** dasanind has quit IRC | 22:34 | |
| *** dasanind has joined #openstack-infra | 22:34 | |
| *** rackertom has quit IRC | 22:34 | |
| *** rackertom has joined #openstack-infra | 22:34 | |
| *** Julien-zte has joined #openstack-infra | 22:35 | |
| *** NobodyCam has quit IRC | 22:35 | |
| *** NobodyCam has joined #openstack-infra | 22:35 | |
| *** baoli has joined #openstack-infra | 22:36 | |
| *** igorbelikov has quit IRC | 22:36 | |
| *** coreycb has quit IRC | 22:37 | |
| cloudnull | clarkb: mordred: pabelanger: ok our cert nonsense is squared away | 22:37 |
| *** spzala has quit IRC | 22:38 | |
| clarkb | cloudnull: thanks omce I'm not watching kids I can test things and hopefully get change to update url in | 22:38 |
| *** gardlt has quit IRC | 22:38 | |
| cloudnull | I can submit a revert for the -1 | 22:38 |
| *** marst_ has quit IRC | 22:38 | |
| cloudnull | and I have a PR in for the url change. | 22:39 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config master: Revert "Set max-servers -1 for osic" https://review.openstack.org/467401 | 22:39 |
| *** igorbelikov has joined #openstack-infra | 22:39 | |
| clarkb | cloudnull: where is url change? | 22:39 |
| *** coreycb has joined #openstack-infra | 22:40 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config master: Revert "Set max-servers -1 for osic" https://review.openstack.org/467401 | 22:40 |
| cloudnull | clarkb: https://review.openstack.org/467371 | 22:41 |
| *** baoli has quit IRC | 22:41 | |
| cloudnull | https://cloud1.osic.xyz won, ++ pabelanger :) | 22:41 |
| cloudnull | it was $1 to register | 22:42 |
| pabelanger | yay | 22:42 |
| clarkb | nice | 22:43 |
| *** dave-mccowan has joined #openstack-infra | 22:44 | |
| pabelanger | clarkb: I'll have to check nodepool code, -1 didn't stop uploads. I think we wanted paused: true | 22:46 |
| pabelanger | either way, non issue now that domain is back | 22:47 |
| *** salv-orlando has quit IRC | 22:48 | |
| clarkb | oh did we remove that with the v3ificqtion? | 22:48 |
| clarkb | for skme reaso I thought we kept it | 22:48 |
| pabelanger | I think we opted for pause | 22:49 |
| pabelanger | but I forgot that | 22:49 |
| openstackgerrit | Merged openstack-infra/system-config master: Changed the domain name for cloud1.osic.org https://review.openstack.org/467371 | 22:51 |
| *** yamahata has joined #openstack-infra | 22:54 | |
| *** Swami has joined #openstack-infra | 22:55 | |
| *** rbrndt has quit IRC | 22:56 | |
| *** rbrndt has joined #openstack-infra | 22:56 | |
| *** rbrndt has quit IRC | 22:56 | |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: ovb: Remove bin/ and usr/ from working_dir tarball https://review.openstack.org/447174 | 22:58 |
| *** annegentle has quit IRC | 22:58 | |
| *** bobh has joined #openstack-infra | 23:03 | |
| clarkb | cloudnull: pabelanger with ^ merged we should see the large chunk of servers in a delete state go away, then we can merge the revert of the max-servers | 23:05 |
| clarkb | I'm back now and will try to get that done before EOD | 23:05 |
| clarkb | mordred: so for shade you are fixing tests ? | 23:06 |
| *** wolverin_ has joined #openstack-infra | 23:06 | |
| mordred | clarkb: yes. there is one that's weird and I'm trying to decide if the test is testing someting real | 23:06 |
| *** bobh has quit IRC | 23:07 | |
| *** esberglu has joined #openstack-infra | 23:07 | |
| *** esberglu has quit IRC | 23:07 | |
| *** bobh has joined #openstack-infra | 23:07 | |
| *** rlandy has quit IRC | 23:08 | |
| *** wolverineav has quit IRC | 23:08 | |
| *** esikache1 has joined #openstack-infra | 23:08 | |
| pabelanger | clarkb: ya, we just started uploading images again | 23:09 |
| pabelanger | actually... | 23:09 |
| pabelanger | ya, still looping | 23:10 |
| pabelanger | checking puppet | 23:10 |
| pabelanger | k, uploading now | 23:12 |
| *** esikache1 has quit IRC | 23:12 | |
| *** adriant has joined #openstack-infra | 23:16 | |
| pabelanger | clarkb: okay, I think we are ready for the revert. I kicked nodepool from puppetmaster | 23:17 |
| pabelanger | it has updated clouds.yaml file | 23:17 |
| clarkb | pabelanger: are we seeing the deleted nodes actually get deleted? | 23:17 |
| pabelanger | will that happen with -1? | 23:17 |
| clarkb | yes | 23:17 |
| *** tristanC has quit IRC | 23:17 | |
| pabelanger | k, let me check log | 23:18 |
| clarkb | it will be on the cleanup timer though which is every X minutes | 23:18 |
| clarkb | so may have to wait a few | 23:18 |
| pabelanger | there we go | 23:18 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-osic | 23:18 |
| clarkb | ya just confirmed with direct lisint too | 23:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Pick most recent rather than first fixed address https://review.openstack.org/467385 | 23:19 |
| clarkb | was about 200 now 130 ish | 23:19 |
| mordred | clarkb: ok. I have decided it was a bogus test | 23:19 |
| clarkb | so ya good on revert /me reviews | 23:19 |
| mordred | clarkb: and was testing that the test was injecting bad data into its mock | 23:19 |
| clarkb | pabelanger: you want to approve? | 23:19 |
| pabelanger | done | 23:20 |
| mordred | clarkb: sorry that we lurked another day on this one - although I think the insight into the port creation time should also help us fail less citycloud nodes ... I HOPE | 23:20 |
| clarkb | ya | 23:20 |
| pabelanger | EOD for me | 23:20 |
| mordred | clarkb: I promise I will not fix any new bugs in the morning between me waking up and you waking up | 23:21 |
| *** s-shiono has joined #openstack-infra | 23:22 | |
| *** gordc has quit IRC | 23:22 | |
| clarkb | mordred: shade change looks good | 23:22 |
| clarkb | we have 28GB free on logs fs | 23:23 |
| mordred | that should last a minute or two right? | 23:23 |
| mordred | clarkb: I got squashed today - was the cause found? | 23:23 |
| clarkb | I think the change to collect journald files is at least partially related | 23:25 |
| clarkb | since it added quite a bit of extra logging to devstack-gate runs | 23:25 |
| mordred | ah. yah. I'll bet so | 23:26 |
| *** esikache1 has joined #openstack-infra | 23:26 | |
| *** wolverin_ has quit IRC | 23:28 | |
| *** gildub has joined #openstack-infra | 23:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Set max-servers -1 for osic" https://review.openstack.org/467401 | 23:30 |
| *** wolverineav has joined #openstack-infra | 23:30 | |
| *** adam_g has quit IRC | 23:32 | |
| *** esikache1 has quit IRC | 23:33 | |
| *** adam_g has joined #openstack-infra | 23:33 | |
| *** wolverineav has quit IRC | 23:34 | |
| *** Swami has quit IRC | 23:34 | |
| clarkb | mordred: shade ip sorting change failed pep8 | 23:35 |
| mordred | BLASTIT | 23:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Pick most recent rather than first fixed address https://review.openstack.org/467385 | 23:37 |
| *** priteau has joined #openstack-infra | 23:41 | |
| cloudnull | Thanks clarkb mordred pabelanger! | 23:44 |
| cloudnull | sorry everyone for the outage | 23:44 |
| *** hongbin has quit IRC | 23:45 | |
| *** priteau has quit IRC | 23:45 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: DNM: testing global_request_id https://review.openstack.org/467417 | 23:47 |
| *** esikache1 has joined #openstack-infra | 23:47 | |
| *** spzala has joined #openstack-infra | 23:47 | |
| *** dimtruck is now known as zz_dimtruck | 23:48 | |
| mordred | cloudnull: bah - you're always amazing - thanks for being so responsive | 23:48 |
| *** Julien-zte has quit IRC | 23:49 | |
| *** yamahata has quit IRC | 23:50 | |
| *** spzala has quit IRC | 23:50 | |
| *** spzala has joined #openstack-infra | 23:51 | |
| *** zz_dimtruck is now known as dimtruck | 23:51 | |
| *** esikache1 has quit IRC | 23:51 | |
| *** yamahata has joined #openstack-infra | 23:54 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Adding unit testing for configuration https://review.openstack.org/466959 | 23:58 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a more generic tree->graph parser https://review.openstack.org/465417 | 23:58 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] use networkx https://review.openstack.org/467419 | 23:58 |
| *** aeng has quit IRC | 23:59 | |
| *** salv-orlando has joined #openstack-infra | 23:59 | |
| *** SumitNaiksatam has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!