| *** salv-orlando has quit IRC | 00:00 | |
| *** yamamoto has joined #openstack-infra | 00:09 | |
| *** EricGonczer_ has joined #openstack-infra | 00:12 | |
| *** deduped has quit IRC | 00:18 | |
| *** EricGonczer_ has quit IRC | 00:27 | |
| *** salv-orlando has joined #openstack-infra | 00:28 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Revert "Set puppet-cinder xenial gates to non-voting" https://review.openstack.org/500338 | 00:32 |
|---|---|---|
| *** yamamoto has quit IRC | 00:35 | |
| *** markvoelker has joined #openstack-infra | 00:41 | |
| *** thorst_afk has joined #openstack-infra | 00:42 | |
| *** thorst_afk has quit IRC | 00:49 | |
| *** sree has joined #openstack-infra | 01:10 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: upload-logs: fix indentation https://review.openstack.org/500339 | 01:10 |
| *** yamahata has quit IRC | 01:11 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: upload-logs/bindep: fix indentation https://review.openstack.org/500339 | 01:12 |
| *** sree has quit IRC | 01:14 | |
| *** markvoelker has quit IRC | 01:16 | |
| *** slaweq has joined #openstack-infra | 01:21 | |
| *** yamamoto has joined #openstack-infra | 01:25 | |
| *** slaweq has quit IRC | 01:26 | |
| *** ijw has joined #openstack-infra | 01:29 | |
| *** ijw_ has joined #openstack-infra | 01:30 | |
| *** ijw has quit IRC | 01:33 | |
| *** salv-orlando has quit IRC | 01:40 | |
| *** salv-orlando has joined #openstack-infra | 01:40 | |
| *** jascott1 has quit IRC | 01:45 | |
| *** salv-orlando has quit IRC | 01:45 | |
| *** ramishra has joined #openstack-infra | 02:11 | |
| *** psachin has joined #openstack-infra | 02:18 | |
| *** EricGonczer_ has joined #openstack-infra | 02:23 | |
| *** yamamoto has quit IRC | 02:44 | |
| *** xarses_ has joined #openstack-infra | 02:48 | |
| *** andreww has quit IRC | 02:51 | |
| *** EricGonczer_ has quit IRC | 02:51 | |
| *** EricGonczer_ has joined #openstack-infra | 02:55 | |
| *** andreww has joined #openstack-infra | 02:56 | |
| *** claudiub has quit IRC | 02:57 | |
| *** xarses_ has quit IRC | 02:59 | |
| *** salv-orlando has joined #openstack-infra | 02:59 | |
| *** EricGonczer_ has quit IRC | 02:59 | |
| *** yamamoto has joined #openstack-infra | 03:01 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: WIP: web: add jobs controller https://review.openstack.org/466561 | 03:01 |
| *** salv-orlando has quit IRC | 03:03 | |
| *** bobh has joined #openstack-infra | 03:06 | |
| *** sree has joined #openstack-infra | 03:11 | |
| *** bobh has quit IRC | 03:12 | |
| *** markvoelker has joined #openstack-infra | 03:13 | |
| *** sree has quit IRC | 03:15 | |
| *** slaweq has joined #openstack-infra | 03:23 | |
| *** slaweq has quit IRC | 03:27 | |
| *** jascott1 has joined #openstack-infra | 03:43 | |
| *** markvoelker has quit IRC | 03:45 | |
| *** yamamoto has quit IRC | 03:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 04:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: Add devstack_local_conf ansible module https://review.openstack.org/496301 | 04:05 |
| *** yamamoto has joined #openstack-infra | 04:10 | |
| *** eranrom_ has quit IRC | 04:11 | |
| *** eranrom has joined #openstack-infra | 04:11 | |
| *** thingee has quit IRC | 04:14 | |
| *** pabelanger has quit IRC | 04:14 | |
| *** pabelanger has joined #openstack-infra | 04:16 | |
| *** thingee has joined #openstack-infra | 04:16 | |
| *** markvoelker has joined #openstack-infra | 04:43 | |
| *** ramishra has quit IRC | 04:50 | |
| *** ykarel has joined #openstack-infra | 04:50 | |
| *** salv-orlando has joined #openstack-infra | 05:02 | |
| *** salv-orlando has quit IRC | 05:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 05:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: Add devstack_local_conf ansible module https://review.openstack.org/496301 | 05:07 |
| *** ramishra has joined #openstack-infra | 05:08 | |
| *** sree_ has joined #openstack-infra | 05:11 | |
| *** sree_ is now known as Guest37794 | 05:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 05:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: Add devstack_local_conf ansible module https://review.openstack.org/496301 | 05:12 |
| *** Guest37794 has quit IRC | 05:16 | |
| *** markvoelker has quit IRC | 05:16 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 05:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: Add devstack_local_conf ansible module https://review.openstack.org/496301 | 05:20 |
| *** slaweq has joined #openstack-infra | 05:25 | |
| *** ramishra has quit IRC | 05:25 | |
| *** slaweq has quit IRC | 05:30 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 05:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: Add devstack_local_conf ansible module https://review.openstack.org/496301 | 05:37 |
| *** sree_ has joined #openstack-infra | 05:38 | |
| *** sree_ is now known as Guest67667 | 05:38 | |
| *** Guest67667 has quit IRC | 05:42 | |
| *** salv-orlando has joined #openstack-infra | 06:03 | |
| *** salv-orlando has quit IRC | 06:07 | |
| *** markvoelker has joined #openstack-infra | 06:13 | |
| *** markvoelker has quit IRC | 06:47 | |
| *** rcernin has joined #openstack-infra | 06:48 | |
| *** jascott1 has quit IRC | 06:55 | |
| *** lennyb has quit IRC | 06:56 | |
| *** sree_ has joined #openstack-infra | 06:58 | |
| *** sree_ is now known as Guest49585 | 06:59 | |
| *** Guest49585 has quit IRC | 07:02 | |
| *** salv-orlando has joined #openstack-infra | 07:03 | |
| *** slaweq has joined #openstack-infra | 07:07 | |
| *** slaweq has quit IRC | 07:08 | |
| *** salv-orlando has quit IRC | 07:08 | |
| *** jtomasek has joined #openstack-infra | 07:11 | |
| *** links has joined #openstack-infra | 07:16 | |
| *** ykarel has quit IRC | 07:41 | |
| *** slaweq has joined #openstack-infra | 07:46 | |
| *** sree_ has joined #openstack-infra | 07:47 | |
| *** sree_ is now known as Guest60236 | 07:47 | |
| *** rossella_s has joined #openstack-infra | 07:49 | |
| *** Guest60236 has quit IRC | 07:58 | |
| *** links has quit IRC | 08:00 | |
| *** salv-orlando has joined #openstack-infra | 08:04 | |
| *** salv-orlando has quit IRC | 08:09 | |
| *** links has joined #openstack-infra | 08:13 | |
| *** gouthamr has quit IRC | 08:22 | |
| *** yamamoto has quit IRC | 08:27 | |
| *** jascott1 has joined #openstack-infra | 08:28 | |
| *** links has quit IRC | 08:29 | |
| *** lathiat has quit IRC | 08:37 | |
| *** lathiat has joined #openstack-infra | 08:37 | |
| *** markvoelker has joined #openstack-infra | 08:44 | |
| *** yamamoto has joined #openstack-infra | 08:48 | |
| *** slaweq has quit IRC | 08:54 | |
| *** slaweq has joined #openstack-infra | 08:54 | |
| *** yamamoto has quit IRC | 09:04 | |
| *** salv-orlando has joined #openstack-infra | 09:05 | |
| *** yamamoto has joined #openstack-infra | 09:06 | |
| *** yamamoto has quit IRC | 09:07 | |
| *** yamamoto has joined #openstack-infra | 09:07 | |
| *** yamamoto has quit IRC | 09:08 | |
| *** salv-orlando has quit IRC | 09:10 | |
| *** yamamoto has joined #openstack-infra | 09:10 | |
| *** yamamoto has quit IRC | 09:10 | |
| *** slaweq has quit IRC | 09:12 | |
| *** yamamoto has joined #openstack-infra | 09:14 | |
| *** markvoelker has quit IRC | 09:18 | |
| *** yamamoto has quit IRC | 09:26 | |
| *** yamamoto has joined #openstack-infra | 09:26 | |
| *** rossella_s has quit IRC | 09:28 | |
| *** rossella_s has joined #openstack-infra | 09:29 | |
| *** slaweq has joined #openstack-infra | 09:33 | |
| *** rossella_s has quit IRC | 09:39 | |
| *** rossella_s has joined #openstack-infra | 09:39 | |
| *** slaweq has quit IRC | 09:39 | |
| *** tinwood has quit IRC | 09:43 | |
| *** tinwood has joined #openstack-infra | 09:45 | |
| *** claudiub has joined #openstack-infra | 09:58 | |
| *** salv-orlando has joined #openstack-infra | 10:06 | |
| *** salv-orlando has quit IRC | 10:10 | |
| *** markvoelker has joined #openstack-infra | 10:15 | |
| *** jtomasek has quit IRC | 10:20 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 10:32 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 10:32 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: oci: replace paramiko with ansible playbooks https://review.openstack.org/500357 | 10:32 |
| *** zhurong has joined #openstack-infra | 10:42 | |
| *** salv-orlando has joined #openstack-infra | 10:45 | |
| *** markvoelker has quit IRC | 10:47 | |
| *** salv-orlando has quit IRC | 10:50 | |
| *** slaweq has joined #openstack-infra | 10:52 | |
| *** slaweq has quit IRC | 10:53 | |
| *** zhurong has quit IRC | 10:56 | |
| *** salv-orlando has joined #openstack-infra | 10:58 | |
| *** salv-orl_ has joined #openstack-infra | 10:59 | |
| *** salv-orlando has quit IRC | 11:02 | |
| *** ykarel has joined #openstack-infra | 11:04 | |
| *** jokke_ has quit IRC | 11:10 | |
| *** tosky has joined #openstack-infra | 11:11 | |
| *** pbourke has quit IRC | 11:27 | |
| *** pbourke has joined #openstack-infra | 11:29 | |
| *** wolverineav has joined #openstack-infra | 11:31 | |
| *** slaweq has joined #openstack-infra | 11:33 | |
| *** slaweq has quit IRC | 11:37 | |
| *** markvoelker has joined #openstack-infra | 11:44 | |
| *** markvoelker has quit IRC | 12:18 | |
| *** EricGonczer_ has joined #openstack-infra | 12:30 | |
| *** slaweq has joined #openstack-infra | 12:35 | |
| *** slaweq has quit IRC | 12:42 | |
| *** thorst_afk has joined #openstack-infra | 12:57 | |
| *** EricGonczer_ has quit IRC | 12:57 | |
| *** EricGonczer_ has joined #openstack-infra | 12:58 | |
| *** slaweq has joined #openstack-infra | 12:58 | |
| *** bobh has joined #openstack-infra | 13:00 | |
| *** jascott1 has quit IRC | 13:01 | |
| *** nicolasbock has joined #openstack-infra | 13:02 | |
| *** thorst_afk has quit IRC | 13:03 | |
| *** bobh has quit IRC | 13:10 | |
| *** markvoelker has joined #openstack-infra | 13:15 | |
| *** yolanda has quit IRC | 13:17 | |
| *** salv-orl_ has quit IRC | 13:25 | |
| *** salv-orlando has joined #openstack-infra | 13:26 | |
| *** salv-orlando has quit IRC | 13:31 | |
| *** mriedem has joined #openstack-infra | 13:36 | |
| *** salv-orlando has joined #openstack-infra | 13:36 | |
| *** sree has joined #openstack-infra | 13:43 | |
| *** sree has quit IRC | 13:47 | |
| *** markvoelker has quit IRC | 13:48 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 13:48 |
| *** logan- has quit IRC | 13:54 | |
| dmsimard | No xvde failures in recent history, I wonder if it's simply because there's no load.. I guess we'll know for sure tomorrow. | 13:55 |
| *** jascott1 has joined #openstack-infra | 14:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 14:04 |
| mordred | dmsimard: ++ | 14:04 |
| *** ianychoi has quit IRC | 14:06 | |
| dmsimard | Just realized that OpenShift ships a tuned profile for their stuff, maybe we could do the same for OpenStack | 14:10 |
| mordred | dmsimard: I'm not sure what those words mean? | 14:10 |
| dmsimard | mordred: tuned, the daemon that monitors tweaks sysctl things (amongst other things) | 14:11 |
| *** salv-orlando has quit IRC | 14:12 | |
| dmsimard | https://github.com/openshift/origin/blob/master/contrib/tuned/origin-node-guest/tuned.conf | 14:12 |
| dmsimard | https://github.com/openshift/origin/blob/master/contrib/tuned/origin-node-host/tuned.conf | 14:12 |
| dmsimard | It was completely unrelated to the xvde thing, just something I randomly stumbled upon while troubleshooting our openshift instance :D | 14:14 |
| mordred | dmsimard: yah- I think that's harder for us, because of the relative complexity levels | 14:14 |
| mordred | but also - tuned seems to be a rhel thing - I can't seem to find it anywhere else | 14:14 |
| mordred | in any case, I think that might be a good idea for times when "we" == "red hat" - as a thing to ship in rdo or whatnot - but I think for openstack you need to know too much about what services and drivers and network config and hypervisor you're running | 14:16 |
| dmsimard | it's available on ubuntu too afaik | 14:16 |
| dmsimard | it's default in rhel but it's available in ubuntu | 14:16 |
| *** lennyb has joined #openstack-infra | 14:17 | |
| dmsimard | hrm, wait, maybe not ? | 14:17 |
| dmsimard | I saw https://launchpad.net/ubuntu/+source/tuned but packages.ubuntu.com doesn't seem to show it, I guess it's not packaged yet up until zesty. | 14:18 |
| *** links has joined #openstack-infra | 14:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 14:26 |
| *** EricGonczer_ has quit IRC | 14:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 14:27 |
| mordred | dmsimard: yah - I'd never heard of it until today | 14:31 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Add ansible-lint and ansible-playbook syntax as a non-voting job https://review.openstack.org/498140 | 14:31 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Add ansible-lint and ansible-playbook syntax as a non-voting job https://review.openstack.org/498140 | 14:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Add ansible-lint and ansible-playbook syntax as a non-voting job https://review.openstack.org/498140 | 14:34 |
| pabelanger | infra-root: I know the weekend, but do we need any new servers for PTG and zuulv3 migration? EG: we have nl01.o.o now, do we want another launcher? | 14:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 14:41 |
| *** thorst_afk has joined #openstack-infra | 14:42 | |
| *** logan- has joined #openstack-infra | 14:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 14:43 |
| fungi | pabelanger: maybe? i suppose when we cut over we'll cease needing the old mergers and start needing roughly an equivalent number of new launchers... but i don't know whether we have sufficient quota to build them all ahead of time vs a few at a time while deleting the mergers | 14:44 |
| jeblair | pabelanger, fungi: yeah, the zuul executors and mergers are probably more urgent than another nodepool launcher, but if we can, we should. | 14:44 |
| *** salv-orlando has joined #openstack-infra | 14:45 | |
| fungi | er, right i meant executor not launcher ;) | 14:45 |
| *** thorst_afk has quit IRC | 14:45 | |
| *** markvoelker has joined #openstack-infra | 14:45 | |
| pabelanger | okay, did we get a final number of new executors and mergers we wanted? 7 of each? | 14:46 |
| pabelanger | doesn't have to be today either, just thinking if had time between patches, see what needed to bring them online | 14:46 |
| jeblair | maybe we can reduce the number of logstash-workers? | 14:47 |
| jeblair | i don't think we've contracted them since our peak | 14:47 |
| jeblair | (and of course we could add some back later if we need) | 14:47 |
| fungi | is the plan to run a merger and an executor daemon together on the same machines, or have merger-only and executor-only servers? | 14:48 |
| jeblair | fungi: separate | 14:48 |
| pabelanger | jeblair: Ya, good idea, lets see what clarkb thinks | 14:48 |
| jeblair | pabelanger: definitely 7 executors, we may be able get by with fewer mergers | 14:49 |
| jeblair | let's say 5 minimum, but 7 if we can. | 14:49 |
| fungi | ahh, right, there's probably no affinity mechansim to make the launcher get its refs from the merger running on the same machine | 14:49 |
| pabelanger | k | 14:49 |
| jeblair | mergers can be smaller than executors too | 14:49 |
| jeblair | our mergers are 2G | 14:52 |
| jeblair | might even be able to get away with 1g | 14:52 |
| pabelanger | what about volumes for them? what size do you suggest? | 14:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 14:59 |
| jeblair | pabelanger: we just use the ephemeral volume on /opt. no cinder. | 14:59 |
| pabelanger | k | 15:00 |
| *** ykarel has quit IRC | 15:03 | |
| *** armax has joined #openstack-infra | 15:10 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Add ansible-lint and ansible-playbook syntax as a non-voting job https://review.openstack.org/498140 | 15:11 |
| *** ianychoi has joined #openstack-infra | 15:11 | |
| *** markvoelker has quit IRC | 15:18 | |
| *** gmann has quit IRC | 15:23 | |
| *** gmann has joined #openstack-infra | 15:23 | |
| *** pcaruana has joined #openstack-infra | 15:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 15:26 |
| *** psachin has quit IRC | 15:29 | |
| *** thorst_afk has joined #openstack-infra | 15:30 | |
| *** ykarel has joined #openstack-infra | 15:30 | |
| *** thorst_afk has quit IRC | 15:34 | |
| *** armax has quit IRC | 15:37 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Add ansible-lint and ansible-playbook syntax as a non-voting job https://review.openstack.org/498140 | 15:50 |
| *** wolverineav has quit IRC | 15:53 | |
| *** tinwood has quit IRC | 16:02 | |
| *** tinwood has joined #openstack-infra | 16:03 | |
| *** ykarel has quit IRC | 16:05 | |
| *** markvoelker has joined #openstack-infra | 16:15 | |
| *** EricGonczer_ has joined #openstack-infra | 16:20 | |
| *** salv-orlando has quit IRC | 16:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Stab at fixing yaml syntax error https://review.openstack.org/500374 | 16:41 |
| *** thorre has quit IRC | 16:43 | |
| *** thorre has joined #openstack-infra | 16:44 | |
| *** markvoelker has quit IRC | 16:49 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Stab at fixing yaml syntax error https://review.openstack.org/500374 | 16:51 |
| *** ykarel has joined #openstack-infra | 16:55 | |
| *** ykarel has quit IRC | 17:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Stab at fixing yaml syntax error https://review.openstack.org/500374 | 17:05 |
| mordred | clarkb: I have just noticed that we are downloading etcd3 tarball from github during devstack runs and are not pre-caching it in our images | 17:07 |
| mordred | clarkb: it's 8.9M, so it shouldn't be a large difference in image size | 17:07 |
| pabelanger | I remember commenting on an etcd3 patch about that | 17:09 |
| *** jkilpatr has quit IRC | 17:09 | |
| mordred | pabelanger: remote: https://review.openstack.org/500375 Make list_images.sh emit the etcd3 tarball | 17:15 |
| *** rcernin has quit IRC | 17:18 | |
| *** links has quit IRC | 17:18 | |
| pabelanger | mordred: left question | 17:18 |
| mordred | pabelanger: cool - followup patch coming | 17:18 |
| *** gouthamr has joined #openstack-infra | 17:22 | |
| mordred | pabelanger: updated. how's that now? | 17:24 |
| mordred | pabelanger, jeblair: http://zuulv3.openstack.org/static/stream.html?uuid=e2b2146548c94353a74cfe5d7cd55257&logfile=console.log the devstack job is taking a VERY long time to download from infracloud nodes | 17:26 |
| pabelanger | mordred: +1 | 17:27 |
| pabelanger | mordred: jeblair: ya, github.com on infracloud is terrible, I've seen a lot of tripleo jobs fail because of that. Networking seems to be more of an issue now there | 17:28 |
| *** salv-orlando has joined #openstack-infra | 17:30 | |
| pabelanger | mordred: jeblair: we could also carve out another 5 / 10 nodes from remaining clouds and add them to nodepoolv3.yaml files. I mean, we need to do it at PTG, but can front load some of that work sooner | 17:31 |
| *** yland has joined #openstack-infra | 17:32 | |
| mordred | pabelanger: seems sensible to me | 17:33 |
| *** salv-orlando has quit IRC | 17:35 | |
| pabelanger | Ya, we likely want to add centos-7 to devstack-gate for a test too. I suspect we'll need to update some ubuntuizms for centos | 17:36 |
| *** markvoelker has joined #openstack-infra | 17:46 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Revert "Switch puppet-integration-jobs for puppet-ceph to non-voting" https://review.openstack.org/500329 | 17:51 |
| pabelanger | clarkb: fungi: did we ever hear back from OVH about accounts? | 17:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: nodepoolv3.yaml: Remove setting of default values https://review.openstack.org/500377 | 17:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add rackspace clouds to nodepoolv3 https://review.openstack.org/500378 | 17:51 |
| mordred | jeblair: I added https://review.openstack.org/#/c/500374 on top of your d-g patches and it got past the cache yaml syntax error - however, now it gets to downloading etcd and hangs | 17:57 |
| mordred | jeblair: best I can tell it's actually getting past the etcd download step (the file seems to be fully there on disk and checked out) | 17:58 |
| *** thorst_afk has joined #openstack-infra | 17:58 | |
| mordred | but we stop getting logging, both on the executor AND on the node itself | 17:58 |
| *** thorst_afk has quit IRC | 18:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add rackspace clouds to nodepoolv3 https://review.openstack.org/500378 | 18:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: nodepoolv3.yaml: Switch inap to flavor-name https://review.openstack.org/500380 | 18:02 |
| *** squid has quit IRC | 18:04 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add rackspace clouds to nodepoolv3 https://review.openstack.org/500378 | 18:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add citycloud regions to nodepoolv3 https://review.openstack.org/500381 | 18:08 |
| *** olaph has quit IRC | 18:09 | |
| *** olaph has joined #openstack-infra | 18:11 | |
| *** squid has joined #openstack-infra | 18:14 | |
| *** pcaruana has quit IRC | 18:14 | |
| *** olaph1 has joined #openstack-infra | 18:15 | |
| *** olaph has quit IRC | 18:17 | |
| *** markvoelker has quit IRC | 18:19 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add vexxhost region to nodepoolv3 https://review.openstack.org/500382 | 18:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Bring centos-7 images online for nodepoolv3 https://review.openstack.org/500383 | 18:19 |
| mordred | pabelanger: I'm gonna go ahead and +A those since we're both around | 18:22 |
| pabelanger | sure | 18:23 |
| mordred | pabelanger: name-filter isn't valid anymore? | 18:23 |
| pabelanger | looking at comments it is flavor-name now | 18:23 |
| mordred | nod. fwiw you can remove that min-ram setting in a followup - if we provide flavor name we don't need it | 18:24 |
| pabelanger | mordred: Oh, I thought we still did a match on min-ram first, then filter on flavor-name | 18:25 |
| mordred | pabelanger: oh - wait - hang on | 18:25 |
| mordred | ah. you're right, I'm wrong | 18:26 |
| pabelanger | https://docs.openstack.org/infra/nodepool/feature/zuulv3/configuration.html#pool-labels | 18:26 |
| pabelanger | okay, cool | 18:26 |
| pabelanger | just wanted to make sure it didn't change | 18:26 |
| *** olaph1 is now known as olaph | 18:26 | |
| mordred | pabelanger: typo in https://review.openstack.org/#/c/500378/3 | 18:27 |
| pabelanger | k, fixing | 18:27 |
| mordred | pabelanger: same one is in https://review.openstack.org/#/c/500383 | 18:27 |
| *** salv-orlando has joined #openstack-infra | 18:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: nodepoolv3.yaml: Remove setting of default values https://review.openstack.org/500377 | 18:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add rackspace clouds to nodepoolv3 https://review.openstack.org/500378 | 18:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add citycloud regions to nodepoolv3 https://review.openstack.org/500381 | 18:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Bring centos-7 images online for nodepoolv3 https://review.openstack.org/500383 | 18:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add vexxhost region to nodepoolv3 https://review.openstack.org/500382 | 18:32 |
| openstackgerrit | Merged openstack-infra/project-config master: nodepoolv3.yaml: Switch inap to flavor-name https://review.openstack.org/500380 | 18:33 |
| *** dmellado_ has joined #openstack-infra | 18:34 | |
| *** salv-orlando has quit IRC | 18:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Switch to normal tox-py35 job with constraints https://review.openstack.org/500330 | 18:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: DNM Test with keystoneauth as required-project https://review.openstack.org/500386 | 18:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: DNM Test sibling application https://review.openstack.org/500387 | 18:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: DNM turn siblings off https://review.openstack.org/500388 | 18:38 |
| mordred | pabelanger: +3 | 18:39 |
| mordred | pabelanger: btw - that stack of 4 changes right there exercises my fancy new patch to make the tox role magically apply changes from zuul-provided git repos for unit tests if a project adds them to required-projects | 18:40 |
| mordred | pabelanger: and by exercises, I mean clearly shows that it does not work :) | 18:44 |
| pabelanger | neat | 18:46 |
| pabelanger | I'll look shortly | 18:46 |
| mordred | pabelanger: it's really close though - the upper-constraints patch totally works, fwiw | 18:46 |
| pabelanger | yay | 18:47 |
| mordred | pabelanger: that's https://review.openstack.org/#/c/500320 along with https://review.openstack.org/#/c/500328 | 18:47 |
| *** andreww has quit IRC | 18:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add rackspace clouds to nodepoolv3 https://review.openstack.org/500378 | 18:51 |
| pabelanger | Ah, I see. you added upper-constraints vars as site-variable | 18:51 |
| pabelanger | mordred: couldn't we just add http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.yaml#n29 into base also? And our tox jobs would work as expected too? | 18:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Add citycloud regions to nodepoolv3 https://review.openstack.org/500381 | 18:53 |
| openstackgerrit | Merged openstack-infra/project-config master: Add vexxhost region to nodepoolv3 https://review.openstack.org/500382 | 18:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Bring centos-7 images online for nodepoolv3 https://review.openstack.org/500383 | 18:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix typo with citycloud-sto2 https://review.openstack.org/500389 | 19:03 |
| pabelanger | mordred: ^typo with citycloud-sto2 | 19:03 |
| mordred | pabelanger: +3 | 19:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Add networking-cisco UTs against neutron stable/pike https://review.openstack.org/498801 | 19:04 |
| mordred | pabelanger: well, we could put the u_c variable in to the base job instead of into site-variables - but it needs to be separate from just putting it in env because it should only get passed as an env var if the file exists | 19:04 |
| mordred | otherwise the defaults that are in the tox files themselves will get overridden by a bogus value, and that would be sad making | 19:05 |
| *** yland has quit IRC | 19:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:07 |
| pabelanger | mordred: Ya, I guess that's the whole issue you're solving | 19:09 |
| mordred | pabelanger: although yeah - I think I like the idea of putting the variable default itself into the base job rather than site-variables | 19:10 |
| mordred | pabelanger: I'll rework to do that instead - I think it'll be more obvious to follow -thanks | 19:10 |
| pabelanger | mordred: ya, my only real concern with site-vars, is we need to wait for puppet to update file on disk. | 19:11 |
| pabelanger | everything else should be good | 19:11 |
| pabelanger | another way we could do, is inspect tox_environment for UPPER_CONSTRAINTS_FILE too, and if files doesn't exist is pop the key. But, that's basically the same approach you are doing | 19:12 |
| pabelanger | Yay, for new and exciting problems | 19:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add UPPER_CONSTRAINTS_FILE file if it exists https://review.openstack.org/500320 | 19:14 |
| *** markvoelker has joined #openstack-infra | 19:16 | |
| *** EricGonczer_ has quit IRC | 19:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo with citycloud-sto2 https://review.openstack.org/500389 | 19:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add upper-constraints plumbing to base job https://review.openstack.org/500328 | 19:19 |
| *** yolanda has joined #openstack-infra | 19:21 | |
| pabelanger | Looks like on nl01.o.o we no longer show IP address with nodepool list command | 19:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:22 |
| pabelanger | | 0000021286 | rax-ord | None | ubuntu-xenial | 8092a64e-01e7-44ac-8e5b-69afcd8d6bc8 | building | 00:00:06:10 | locked | | 19:22 |
| *** yland has joined #openstack-infra | 19:22 | |
| mordred | pabelanger: it's buiolding - it may not have an ip yet? | 19:22 |
| *** yland is now known as Guest52772 | 19:22 | |
| pabelanger | mordred: ya, however used nodes don't list them either | 19:23 |
| mordred | hrm | 19:23 |
| pabelanger | likely just missing the field now | 19:23 |
| *** rama_y has joined #openstack-infra | 19:23 | |
| mordred | pabelanger: I thought there was a way to get them ... Shrews if you look at your computer today ?? | 19:23 |
| *** pcaruana has joined #openstack-infra | 19:23 | |
| pabelanger | list --detail it seems | 19:24 |
| pabelanger | | 0000021286 | rax-ord | None | ubuntu-xenial | 8092a64e-01e7-44ac-8e5b-69afcd8d6bc8 | building | 00:00:08:48 | locked | ubuntu-xenial-rax-ord-0000021286 | None | 10.211.0.45 | None | 22 | nl01-6551-PoolWorker.rax-ord-main | None | None | | 19:24 |
| pabelanger | let me see why rax-ord isn't online yet | 19:25 |
| pabelanger | SSH timeout | 19:28 |
| pabelanger | hmm | 19:28 |
| pabelanger | trying vexxhost now | 19:29 |
| pabelanger | via IPv6 | 19:29 |
| pabelanger | nice | 19:29 |
| pabelanger | | 0000021292 | vexxhost-ca-ymq-1 | ca-ymq-2 | centos-7 | 77b643e2-eb04-4a2f-a9b6-cb1611989b00 | ready | 00:00:00:23 | unlocked | | 19:29 |
| pabelanger | HA, AZ is not what I expected :) | 19:29 |
| mordred | :) | 19:30 |
| pabelanger | launching on rax-dfw this time | 19:31 |
| *** pcaruana has quit IRC | 19:31 | |
| pabelanger | http://zuulv3.openstack.org/static/stream.html?uuid=35e2616ed2d8402983fdc46ac8cac74f&logfile=console.log | 19:32 |
| pabelanger | that doesn't look good | 19:32 |
| pabelanger | 2017-09-03 19:33:10.249344 | [ubuntu-xenial] Waiting on logger is spamming console | 19:33 |
| *** salv-orlando has joined #openstack-infra | 19:33 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:37 |
| *** salv-orlando has quit IRC | 19:38 | |
| pabelanger | okay, think I see a bug | 19:39 |
| pabelanger | config-drive isn't getting set on rackspace | 19:39 |
| pabelanger | even though we stay it is default tru | 19:39 |
| mordred | pabelanger: oh - wasn't there some issue with that? | 19:39 |
| pabelanger | unsure | 19:40 |
| pabelanger | | config_drive | | | 19:40 |
| pabelanger | is what nodepoolv3 is doing | 19:40 |
| pabelanger | maybe we are passing None? | 19:40 |
| mordred | yah- it's not default to true | 19:41 |
| mordred | we need to add the explicit calls back | 19:41 |
| mordred | shade doens't default it | 19:41 |
| mordred | and nodepool does i.config_drive = image.get('config-drive', None) | 19:42 |
| pabelanger | Ah, I thought we flipped that to True | 19:42 |
| mordred | well, our docs say we did | 19:42 |
| mordred | :) | 19:42 |
| pabelanger | ha | 19:42 |
| pabelanger | k, I'll patch project-config and we can talk about nodepool at PTG | 19:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Really set config-drive: true https://review.openstack.org/500390 | 19:44 |
| pabelanger | mordred: kind sir^ | 19:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Fix nodepool's cdocs about config-drive https://review.openstack.org/500391 | 19:44 |
| mordred | pabelanger: trade ya | 19:44 |
| pabelanger | +2 | 19:45 |
| mordred | pabelanger: so - the waiting for log thing ... what was that on? | 19:45 |
| pabelanger | looks like vexxhost | 19:45 |
| pabelanger | http://logs.openstack.org/36/500236/2/check/tox-docs/968029d/job-output.txt.gz | 19:45 |
| mordred | pabelanger: and do we perhaps have a security-group that's blocking log streaming? | 19:45 |
| pabelanger | the job passed | 19:45 |
| pabelanger | mordred: I can check, but should be managed by cloud-launcher | 19:46 |
| mordred | yah | 19:46 |
| mordred | pabelanger: oh - I wonder if we're maybe not listening on ipv6 | 19:46 |
| pabelanger | Oh, HA | 19:46 |
| mordred | although - don't we use ipv6 for infracloud? | 19:46 |
| pabelanger | I thought it was just ipv4 | 19:47 |
| mordred | you're probably right | 19:47 |
| pabelanger | but ya, this will be our first ipv6 | 19:47 |
| pabelanger | so, thats likely it | 19:47 |
| pabelanger | http://logs.openstack.org/36/500236/2/check/tox-docs/968029d/zuul-info/inventory.yaml | 19:47 |
| pabelanger | full of ipv6 hotness | 19:47 |
| mordred | s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) | 19:48 |
| mordred | s.bind(('::', port)) | 19:48 |
| mordred | it's doing the rightthing for listening on v4 and v6 | 19:48 |
| pabelanger | k | 19:48 |
| pabelanger | Um | 19:48 |
| mordred | pabelanger: maybe we don't have v6 security group rules in cloudlauncher? | 19:48 |
| pabelanger | did we open port on ipv6 firewall? | 19:48 |
| *** markvoelker has quit IRC | 19:48 | |
| mordred | :) | 19:48 |
| pabelanger | in system-config | 19:48 |
| pabelanger | checking | 19:48 |
| mordred | pabelanger: this was definitely a good thing to check :) | 19:49 |
| *** slaweq has quit IRC | 19:49 | |
| pabelanger | oh, guess it is project-config now. 1 sec | 19:49 |
| pabelanger | k, should be set | 19:50 |
| pabelanger | checking security groups | 19:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:57 |
| openstackgerrit | Merged openstack-infra/project-config master: Really set config-drive: true https://review.openstack.org/500390 | 20:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove internap clouds https://review.openstack.org/494629 | 20:02 |
| pabelanger | mordred: I think I'm getting auth errors from ansible inventory because of ^ | 20:03 |
| mordred | pabelanger: +3 | 20:04 |
| *** EricGonczer_ has joined #openstack-infra | 20:06 | |
| pabelanger | mordred: are we still using infra-files-ro for anything? looks to be swift related | 20:10 |
| mordred | pabelanger: I do not know | 20:10 |
| pabelanger | k | 20:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove entercloud credentials https://review.openstack.org/500393 | 20:12 |
| *** rama_y has quit IRC | 20:13 | |
| *** lbragstad has quit IRC | 20:16 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove internap clouds https://review.openstack.org/494629 | 20:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove datacentred credentials https://review.openstack.org/500394 | 20:19 |
| pabelanger | mordred: security group looks okay, lets see what happens once we start using config-drive | 20:19 |
| pabelanger | possible it is related | 20:19 |
| *** EricGonczer_ has quit IRC | 20:20 | |
| mordred | nod | 20:21 |
| mordred | maybe without config-drive we're not setting up something or something | 20:21 |
| mordred | pabelanger: I run out to store - but back in just a bit | 20:21 |
| pabelanger | mordred: ya, same. Going out with familiy for 90mins | 20:24 |
| *** gouthamr has quit IRC | 20:27 | |
| *** jascott1 has quit IRC | 20:27 | |
| *** rcernin has joined #openstack-infra | 20:29 | |
| *** jascott1 has joined #openstack-infra | 20:30 | |
| *** salv-orlando has joined #openstack-infra | 20:34 | |
| *** salv-orlando has quit IRC | 20:38 | |
| *** jesusaur has quit IRC | 20:40 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only run zuul-stream testing on callback changes https://review.openstack.org/500168 | 20:40 |
| *** markvoelker has joined #openstack-infra | 20:45 | |
| *** jesusaur has joined #openstack-infra | 20:46 | |
| *** slaweq has joined #openstack-infra | 20:49 | |
| *** slaweq has quit IRC | 20:55 | |
| *** jascott1 has quit IRC | 21:00 | |
| *** Goneri has joined #openstack-infra | 21:02 | |
| *** nicolasbock has quit IRC | 21:05 | |
| mordred | pabelanger: the hieradata on nl01.o.o hasn't been updated since Feb 15 | 21:11 |
| mordred | pabelanger: and it is out of data | 21:11 |
| mordred | pabelanger: however, puppet is running | 21:11 |
| mordred | wow | 21:12 |
| *** EricGonczer_ has joined #openstack-infra | 21:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add nl*.openstack.org to the nodepool group in hiera https://review.openstack.org/500395 | 21:13 |
| mordred | pabelanger: ^^ | 21:13 |
| mordred | that is the root cause of any connection error information you see from ansible/shade/nodepool from nl01 | 21:14 |
| *** EricGonczer_ has quit IRC | 21:17 | |
| *** nicolasbock has joined #openstack-infra | 21:18 | |
| *** markvoelker has quit IRC | 21:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Turn off vexxhost in v3 for now https://review.openstack.org/500397 | 21:27 |
| mordred | pabelanger: I'd like to do that ^^ while we figure out the log streaming thing | 21:28 |
| *** rossella_s has quit IRC | 21:28 | |
| *** rossella_s has joined #openstack-infra | 21:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 21:29 |
| *** salv-orlando has joined #openstack-infra | 21:35 | |
| *** mriedem has quit IRC | 21:36 | |
| *** Guest52772 has quit IRC | 21:39 | |
| *** salv-orlando has quit IRC | 21:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 21:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Turn off vexxhost in v3 for now https://review.openstack.org/500397 | 21:55 |
| openstackgerrit | Merged openstack-infra/system-config master: Add nl*.openstack.org to the nodepool group in hiera https://review.openstack.org/500395 | 21:57 |
| pabelanger | mordred: just catching up | 21:58 |
| pabelanger | mordred: okay, I'll poke at vexxhost manually, see what is going on | 21:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: nodepoolv3: Remove unused setting https://review.openstack.org/500399 | 21:59 |
| pabelanger | mordred: zuulv3.o.o/status doesn't look right. Wonder if merger is overloaded or something | 22:05 |
| pabelanger | Ya, merger is hard at work | 22:06 |
| pabelanger | Yay | 22:07 |
| pabelanger | | 0000021353 | rax-iad | None | ubuntu-xenial | d030d425-902a-4ff3-b5de-210a1cec69f0 | ready | 00:00:02:31 | unlocked | | 22:07 |
| pabelanger | mordred: Ah, now I see. somebody pushed up 20 deep patchset for nova | 22:08 |
| pabelanger | so, 1 merger is chugging to keep up | 22:08 |
| jeblair | pabelanger, mordred: mordred made a change to the status page that hides changes with no jobs running, but the changes still appear in the pipeline count, and their dynamics queues still appear as well. that may end up creating more questions than answers sometimes (in that i can see that there are 4 changes in check right now, none with jobs, but i don't know which changes or how long they've been there. i suspect there may be a ... | 22:08 |
| jeblair | ... problem, but it's hard to say what) | 22:08 |
| pabelanger | jeblair: ah, thanks. That might explain it, no jobs but merged still processing patches | 22:10 |
| pabelanger | merger* | 22:10 |
| jeblair | there's been a project-config change in there every time i've looked for the past 3 days, and now it looks an awful lot like there are 4 changes stuck | 22:10 |
| pabelanger | jeblair: okay, done. But yes, I do see a project-config still. | 22:13 |
| jeblair | ah yep | 22:13 |
| *** yland has joined #openstack-infra | 22:13 | |
| *** yland is now known as Guest46592 | 22:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 22:17 |
| pabelanger | Cool, http://logs.openstack.org/36/500236/2/check/tox-pep8/98268ee/job-output.txt.gz ran on citycloud-sto2 without an issue | 22:19 |
| pabelanger | Hmm, waiting on logger again on rax-ord I believe | 22:20 |
| pabelanger | http://zuulv3.openstack.org/static/stream.html?uuid=95b6e4a3f79c4848aac019b9d68c3122&logfile=console.log | 22:20 |
| pabelanger | mordred: jeblair: I see the issue: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/callback/zuul_stream.py?h=feature/zuulv3#n137 | 22:28 |
| pabelanger | if ip is IPV6, we'll never connect | 22:28 |
| pabelanger | let me see if I can write a patch | 22:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Support IPv6 with zuul_stream https://review.openstack.org/500401 | 22:31 |
| pabelanger | jeblair: mordred:^ I think that is it, but is that a dangourous assumption? I mean, what is zuul-executor is passed an ipv6 address, and it doesn't support ipv6 connection? | 22:32 |
| *** salv-orlando has joined #openstack-infra | 22:36 | |
| jeblair | pabelanger: you'll want to use getaddrinfo on the ip that's passed in, then use that to connect: https://docs.python.org/2/library/socket.html#socket.getaddrinfo | 22:36 |
| jeblair | pabelanger: or you might try https://docs.python.org/2/library/socket.html#socket.create_connection i haven't used that in this situation before though | 22:37 |
| jeblair | pabelanger: but that might do it in one call | 22:37 |
| pabelanger | jeblair: okay, I'll look into now. | 22:38 |
| *** claudiub|2 has joined #openstack-infra | 22:38 | |
| *** salv-orlando has quit IRC | 22:40 | |
| *** claudiub has quit IRC | 22:41 | |
| pabelanger | jeblair: question, why do we call socket.connect multiple times in the loop? I understand if we get an exception back, but if already connected, is multiple connects a noop? http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/callback/zuul_stream.py?h=feature/zuulv3#n138 | 22:46 |
| jeblair | pabelanger: we only connect once -- we keep trying to connect until it succeeds | 22:47 |
| jeblair | pabelanger: that's because that can run before the remote server has started up | 22:47 |
| jeblair | so the connection will fail until it is started | 22:47 |
| pabelanger | jeblair: okay, the part that is confusing me is line 156, if we fall into that, we loop again, wouldn't that s.connect() again? | 22:48 |
| pabelanger | I guess we'd expect line to eventually hit return | 22:49 |
| jeblair | pabelanger: line 156 is the 'else' inside the linesplit loop? | 22:49 |
| jeblair | pabelanger: i just want to make sure we're talking about the same line | 22:50 |
| pabelanger | jeblair: ya, I guess I'm confusing the loops. It doesn't look like we'd ever make it back to line 138 once inside 148 | 22:50 |
| pabelanger | jeblair: ya, 156 is else for me | 22:51 |
| jeblair | pabelanger: yes, the only wait out of the second loop is with an exception that will exit the entire function, so yes, once in there there's no way back | 22:51 |
| jeblair | s/wait out/way out/ | 22:51 |
| pabelanger | okay cool | 22:51 |
| jeblair | pabelanger: hrm, it may be that linesplit can end | 22:52 |
| *** slaweq has joined #openstack-infra | 22:52 | |
| mordred | pabelanger: oh good re: the ipv6 patch | 22:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Support IPv6 with zuul_stream https://review.openstack.org/500401 | 22:55 |
| pabelanger | okay, v2 | 22:55 |
| mordred | I'd like to get this landed: https://review.openstack.org/#/c/500168 too so that we can not be subject to that stream functional test which has become decidedly unhappy | 22:55 |
| jeblair | pabelanger: perhaps not in practice, but in theory, maybe the remote end can close the connection *without* sending one of those special strings, in which case we may actually go through the loop again | 22:55 |
| mordred | alhtough I guess that would't help with this patch :) | 22:55 |
| pabelanger | jeblair: okay, ya. That's what I thought at first glance. | 22:56 |
| jeblair | mordred: maybe you know (whether/why it's possible to exit the lines 148-160 for loop and restart the while loop at line 138 in zuul_stream | 22:56 |
| *** tosky has quit IRC | 22:56 | |
| mordred | jeblair: I don't THINK it is, no | 22:57 |
| jeblair | pabelanger: i don't *think* it *should* happen in normal circumstances, but i agree it seems like there's a theoretical possibility we could go into a loop of fetching the file over and over if the remote end is sending something weird. | 22:57 |
| jeblair | like, if the remote just sent "i'm misbehaving" and closed the socket. :) | 22:57 |
| pabelanger | Ya | 22:57 |
| *** slaweq has quit IRC | 22:58 | |
| pabelanger | okay, I wasn't sure if s.connect() was able to be called multiple times and just noop | 22:58 |
| mordred | I _think_ we could move 145 to the end out of the loop | 22:58 |
| mordred | I don't think we want to re-connect on failure at all - that would just be double-content | 22:58 |
| mordred | (which is what I think you were saying yeah jeblair ?) | 22:58 |
| mordred | jeblair: also - yah - I'm not sure the "hide things that don't have jobs" thing is working out as well as I'd want with those queue labels showing up | 22:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 23:02 |
| jeblair | mordred: why is stream func unhappy? | 23:03 |
| pabelanger | Oh, haha, because it is running on ipv6 nodes | 23:03 |
| pabelanger | and not streaming properly | 23:03 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove entercloud credentials https://review.openstack.org/500393 | 23:03 |
| jeblair | pabelanger, mordred: http://logs.openstack.org/01/500401/1/check/zuul-stream-functional/681187f/ this one is on ipv4 | 23:04 |
| jeblair | it looks like the correctly-failing test Shrews wrote | 23:04 |
| jeblair | did we land that? | 23:04 |
| pabelanger | http://logs.openstack.org/01/500401/1/check/zuul-stream-functional/681187f/stream-files/stream-job-output.txt is showing the waiting on logger spam | 23:04 |
| pabelanger | let me see why | 23:04 |
| pabelanger | ya, that is on infracloud | 23:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move ara output generation to post playbook https://review.openstack.org/500403 | 23:05 |
| pabelanger | maybe ps1 was not correct | 23:05 |
| mordred | pabelanger: ya - was just looking at the "waiting on logger" bit | 23:05 |
| pabelanger | 500401,2 passed | 23:06 |
| mordred | http://logs.openstack.org/01/500401/1/check/zuul-stream-functional/681187f/stream-files/stream-job-output.txt fwiw | 23:06 |
| mordred | oh - hah | 23:06 |
| pabelanger | let me recheck a few times to land on a ipv6 node | 23:06 |
| mordred | you had that pasted already | 23:06 |
| jeblair | mordred: 403 will mean that if ara fails, we won't get any of the zuul-stream logs | 23:06 |
| jeblair | mordred: that's okay, but only if we're supremely confident in ara, and the purpose of the zuul-stream test is not to test ara | 23:07 |
| mordred | jeblair: how about we make 2 post playbooks so the concerns are self contained and we'll get as much as is possible? | 23:07 |
| jeblair | mordred: otherwise, i'd say we should put ara generation in a block, or.... that works too | 23:07 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove datacentred credentials https://review.openstack.org/500394 | 23:08 |
| jeblair | mordred: (or make the ara *fetch* more robust) | 23:08 |
| mordred | yah | 23:08 |
| pabelanger | mordred: so, because we don't support ipv6 on centos and glean, do we need to update our clouds.yaml files and disable ipv6? I know we removed some ip related settings in nodepool in favor of clouds.yaml | 23:08 |
| mordred | pabelanger: wow, we still don't support ipv6 in glean? that's sadmaking. | 23:11 |
| pabelanger | mordred: we do on debuntu, not centos yet | 23:11 |
| mordred | pabelanger: were we forcing v4 somewhere other than clouds.yaml in v2? | 23:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move ara output generation to post playbook https://review.openstack.org/500403 | 23:12 |
| *** askb has joined #openstack-infra | 23:12 | |
| pabelanger | mordred: we had ipv6-preferred: false in nodepool.yaml http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n265 | 23:12 |
| pabelanger | I don't think that is supported in nodepoolv3 | 23:12 |
| mordred | pabelanger: kk. so - we need to add "force_ipv4" into clouds.yaml for rax then | 23:12 |
| pabelanger | k | 23:12 |
| mordred | I can do if you like | 23:12 |
| pabelanger | sure, 1 sec | 23:12 |
| pabelanger | let me make sure 500401 works on rax nodes first | 23:13 |
| mordred | kk | 23:13 |
| jeblair | wait -- we have images that we can't run on ipv6-only clouds? how did this work with osic? | 23:14 |
| mordred | jeblair: only rackspace | 23:14 |
| pabelanger | ipv6 dhcp is fine | 23:14 |
| jeblair | aaaah ok | 23:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Force IPv4 on Rackspace for nodepool https://review.openstack.org/500404 | 23:14 |
| mordred | pabelanger: ^^ there's the change, whenever it is useful | 23:15 |
| pabelanger | mordred: Oh, we might need that for vexxhost too | 23:15 |
| pabelanger | we're on glean there too | 23:15 |
| mordred | we're on glean everywhere - it's only rackspace where we don't get networking if we don't do it staticly | 23:16 |
| pabelanger | Oh, right. | 23:16 |
| mordred | btw - I verified being able to connect from ze01 to a vexxhost node over 19885 over ipv6 earlire | 23:16 |
| mordred | I may not have mentioned that | 23:16 |
| pabelanger | Ah | 23:16 |
| *** markvoelker has joined #openstack-infra | 23:16 | |
| mordred | so thanks for finding the client side issue in zuul_stream, because I was at my wit's end :) | 23:16 |
| jeblair | pabelanger: 401 lgtm assuming it checks out; since so many jobs are hitting that now, i think we should merge it and restart asap, or back out the nodepool config. | 23:17 |
| mordred | ++ | 23:17 |
| pabelanger | jeblair: okay, whatever you'd like to do wfm | 23:17 |
| jeblair | i left my +2; i'll leave this in your hands while i go back to weekending :) | 23:18 |
| mordred | also +2 from me | 23:18 |
| mordred | pabelanger: I'm down for whicher you'd like to do | 23:18 |
| pabelanger | k, lets +3 and restart | 23:18 |
| mordred | cool | 23:18 |
| mordred | as annoying as this v6 issue has been, I'm glad it was on a sunday and really only affected the two of us | 23:19 |
| *** hongbin has joined #openstack-infra | 23:20 | |
| mordred | jeblair: oh - don't know if you saw my followup patch to your d-g patch or not | 23:20 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Only run zuul-stream testing on callback changes https://review.openstack.org/500168 | 23:22 |
| pabelanger | mordred: +3 on 500404 | 23:22 |
| *** claudiub|2 has quit IRC | 23:23 | |
| mordred | \o/ | 23:23 |
| mordred | pabelanger: oh - hah - I thought I +2'd your patch earlier, but I +2'd mine :) | 23:24 |
| pabelanger | 500401 landed on rax too | 23:24 |
| mordred | pabelanger: if you hav ea sec, https://review.openstack.org/#/c/500403 | 23:24 |
| mordred | yay! | 23:24 |
| mordred | pabelanger: we got a post failure on publish docs: http://logs.openstack.org/56/d6d0f58988f426924917ec3fd8149eb9d5c4ee56/post/publish-openstack-python-docs/4002514/ | 23:26 |
| mordred | pabelanger: http://logs.openstack.org/56/d6d0f58988f426924917ec3fd8149eb9d5c4ee56/post/publish-openstack-python-docs/4002514/job-output.txt.gz#_2017-09-03_23_25_22_402117 | 23:26 |
| mordred | pabelanger: did we miss landing something? | 23:26 |
| pabelanger | hmm | 23:26 |
| *** ijw has joined #openstack-infra | 23:27 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use localhost DNS and rh1 dns as secondary https://review.openstack.org/499564 | 23:27 |
| pabelanger | mordred: Ya, I don't think we updated zuul | 23:27 |
| pabelanger | 1 sec | 23:27 |
| clarkb | pabelanger: jeblair ya we can probably scale back the workers. Maybe shutdown the logstash and gearman worker processes for a few hours and make sure the job count doesn't spike before deleting the instances | 23:27 |
| *** ijw has quit IRC | 23:27 | |
| *** ijw has joined #openstack-infra | 23:27 | |
| clarkb | mordred: iirc etcd3 was supposed to be downloaded from tarballs.o.o (which we also proxy cache) on those devstack jobs. I'm guessing if that doesn't happen that the reason is they bumped the version and its not on tarballs.o.o? | 23:28 |
| clarkb | pabelanger: no word from OVH. I expect some of them will be in Denver and we can track them down there | 23:28 |
| clarkb | also looks like xvde is happyness now | 23:28 |
| mordred | clarkb: oh - do we set a mirror location for that somewhere? lemme look | 23:29 |
| mordred | clarkb: AH - I see it | 23:29 |
| pabelanger | mordred: Oh, hmm. that is going to be a little more tricky. Let me make sure zuul_stream is happy, then move onto that | 23:29 |
| mordred | clarkb: thank you | 23:29 |
| mordred | pabelanger: cool | 23:29 |
| openstackgerrit | Merged openstack-infra/system-config master: Force IPv4 on Rackspace for nodepool https://review.openstack.org/500404 | 23:30 |
| *** ijw_ has quit IRC | 23:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Stab at fixing yaml syntax error https://review.openstack.org/500374 | 23:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Use afs_publisher_target for afs-docs https://review.openstack.org/500406 | 23:32 |
| pabelanger | mordred: ^ | 23:32 |
| pabelanger | mordred: I thought we had changed that to a job-template, but I was looking at wrong jobs | 23:32 |
| pabelanger | clarkb: cool, good to know | 23:33 |
| pabelanger | mordred: https://review.openstack.org/500401/ we might as well start landing, might take time to run jobs on ipv6 nodes | 23:36 |
| *** salv-orlando has joined #openstack-infra | 23:36 | |
| mordred | pabelanger: kk. done | 23:40 |
| mordred | pabelanger: also - I rechecked the consume-the-upper-constraints patch from shade after making the changes you suggested and it works again | 23:40 |
| *** gouthamr has joined #openstack-infra | 23:40 | |
| *** salv-orlando has quit IRC | 23:41 | |
| pabelanger | nice! | 23:41 |
| mordred | pabelanger: https://review.openstack.org/#/c/500330/ is the job using it - and you can see here http://logs.openstack.org/30/500330/4/check/tox-py35/09542e4/job-output.txt.gz#_2017-09-03_23_24_48_294328 | 23:41 |
| mordred | that it passes a local file to tox_install.sh - which is what we want | 23:41 |
| *** salv-orlando has joined #openstack-infra | 23:41 | |
| pabelanger | great, I'll add that to list to review | 23:42 |
| *** salv-orlando has quit IRC | 23:42 | |
| *** salv-orlando has joined #openstack-infra | 23:42 | |
| *** Goneri has quit IRC | 23:42 | |
| mordred | pabelanger: I think I'm just about there on the sibling pip patch too | 23:42 |
| *** salv-orlando has quit IRC | 23:42 | |
| mordred | but I'll let you konw what I've got a proven consumer :) | 23:42 |
| pabelanger | Ya, that one is going to be nice also | 23:43 |
| mordred | I think it'll be super helpful | 23:43 |
| mordred | especially since you can push up a DNM patch that adds a job with a required-project just so you can show a particular patch works | 23:43 |
| mordred | we might need to write a blog post abot that trick :) | 23:44 |
| *** mwarad has quit IRC | 23:44 | |
| *** Guest46592 has quit IRC | 23:44 | |
| *** andreww has joined #openstack-infra | 23:46 | |
| *** ijw has quit IRC | 23:46 | |
| pabelanger | Nice | 23:46 |
| pabelanger | mordred: http://logs.openstack.org/01/500401/2/check/zuul-stream-functional/ab51ccf/stream-files/stream-job-output.txt | 23:46 |
| pabelanger | job actually passed with ipv6 fixes | 23:47 |
| mordred | pabelanger: woot! it's like it fixed itself | 23:48 |
| pabelanger | indeed! | 23:48 |
| *** markvoelker has quit IRC | 23:49 | |
| *** ijw has joined #openstack-infra | 23:50 | |
| *** ijw has joined #openstack-infra | 23:50 | |
| *** slaweq has joined #openstack-infra | 23:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Revert "Turn off vexxhost in v3 for now" https://review.openstack.org/500407 | 23:54 |
| mordred | pabelanger: with v6 fixed, we should be able to revert that | 23:54 |
| pabelanger | mordred: agree | 23:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Fix nodepool's docs about config-drive https://review.openstack.org/500391 | 23:55 |
| mordred | pabelanger: and ... I fixed the commit message on that one (it used to say "Fix nodepool's cdocs" ... not sure what a cdoc is :) ) | 23:56 |
| pabelanger | +3 too | 23:56 |
| *** slaweq has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!