| *** bobh has joined #openstack-infra | 00:05 | |
| *** bobh has quit IRC | 00:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] Use testinfra from master in devel job https://review.opendev.org/656877 | 00:17 |
|---|---|---|
| *** whoami-rajat has quit IRC | 00:20 | |
| *** yamamoto has joined #openstack-infra | 00:31 | |
| ianw | it looks like testinfra 3 has broken us ... looking | 00:41 |
| *** bgmccollum has quit IRC | 00:43 | |
| *** imacdonn has quit IRC | 00:58 | |
| *** armax has quit IRC | 01:03 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add OpenAPI documentation https://review.opendev.org/535541 | 01:05 |
| *** zhangfei has joined #openstack-infra | 01:18 | |
| *** dklyle has joined #openstack-infra | 01:20 | |
| *** yamamoto has quit IRC | 01:33 | |
| ianw | it's because it's switched to using paramiko to talk to the hosts directly, and doesn't handle ipv6 | 01:39 |
| ianw | sigh, having ansible figure stuff like this out is the reason why you use ansible ... but ansible-runner makes testinfra about 7 times slower, so you can't blame people for not wanting to switch to that | 01:40 |
| ianw | just a usual example of why we can't have nice things :/ | 01:40 |
| ianw | doesn't handle ipv6 address/hostnames/ports correctly i should say ... fix in the works | 01:40 |
| *** gmann has quit IRC | 01:43 | |
| clarkb | so it is like docker :) | 01:43 |
| *** zhangfei has quit IRC | 01:59 | |
| *** bobh has joined #openstack-infra | 02:02 | |
| *** dklyle has quit IRC | 02:02 | |
| *** bobh has quit IRC | 02:07 | |
| *** armax has joined #openstack-infra | 02:08 | |
| *** hongbin has joined #openstack-infra | 02:23 | |
| *** armax has quit IRC | 02:31 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] Use testinfra from master in devel job https://review.opendev.org/656877 | 02:35 |
| *** zhurong has quit IRC | 02:35 | |
| ianw | ^ testing the pull request for the aforementioned testinfra issues | 02:36 |
| ianw | i've dropped a note in there to ask if we can start running zuul against testinfra master; i think it would help everyone | 02:37 |
| *** altlogbot_1 has quit IRC | 02:41 | |
| *** altlogbot_3 has joined #openstack-infra | 02:42 | |
| pabelanger | ianw: if you have time also, https://review.opendev.org/657126/ should fix fedora-29 issues with dbus. Left some commnets in there too. | 02:53 |
| ianw | pabelanger: sure will check in on that this afternoon | 02:56 |
| pabelanger | ianw: danke | 02:56 |
| pabelanger | now I turn in for the night | 02:56 |
| *** yamamoto has joined #openstack-infra | 03:02 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Pin testinfra < 3 https://review.opendev.org/657308 | 03:12 |
| *** hongbin has quit IRC | 03:13 | |
| *** ykarel has joined #openstack-infra | 03:15 | |
| *** psachin has joined #openstack-infra | 03:18 | |
| *** whoami-rajat has joined #openstack-infra | 03:24 | |
| *** yamamoto has quit IRC | 03:29 | |
| *** yamamoto has joined #openstack-infra | 03:29 | |
| *** altlogbot_3 has quit IRC | 03:31 | |
| *** altlogbot_0 has joined #openstack-infra | 03:32 | |
| *** yamamoto has quit IRC | 03:32 | |
| *** larainema has quit IRC | 03:42 | |
| *** yamamoto has joined #openstack-infra | 03:50 | |
| *** udesale has joined #openstack-infra | 03:53 | |
| *** raukadah has quit IRC | 04:00 | |
| *** chandankumar has joined #openstack-infra | 04:01 | |
| *** ykarel is now known as ykarel|afk | 04:21 | |
| *** ykarel|afk has quit IRC | 04:26 | |
| *** yamamoto has quit IRC | 04:32 | |
| AJaeger | ianw: did you see my message that the periodic wheel build jobs are all failing? | 04:37 |
| *** igordc has quit IRC | 04:37 | |
| *** ykarel|afk has joined #openstack-infra | 04:48 | |
| *** ykarel_ has joined #openstack-infra | 04:52 | |
| *** ykarel|afk has quit IRC | 04:55 | |
| *** yamamoto has joined #openstack-infra | 04:55 | |
| *** yamamoto has quit IRC | 04:56 | |
| *** yamamoto has joined #openstack-infra | 04:58 | |
| ianw | AJaeger: hrm, looking | 05:17 |
| ianw | oh yeah, that's *still* using my branch of pip because the upstream change never got merged | 05:18 |
| *** zhurong has joined #openstack-infra | 05:23 | |
| *** janki has joined #openstack-infra | 05:23 | |
| *** ykarel_ is now known as ykarel | 05:25 | |
| *** yamamoto has quit IRC | 05:26 | |
| ianw | AJaeger: argh, it looks like an upstream pip problem -> https://github.com/pypa/pip/pull/6449 | 05:28 |
| AJaeger | so, what can we do? Use an older pip? | 05:29 |
| *** markmcd has joined #openstack-infra | 05:31 | |
| ianw | AJaeger: i think skip editable mode ... working on it :) | 05:39 |
| AJaeger | thanks, ianw ! | 05:41 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Work around pip errors https://review.opendev.org/657316 | 05:45 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: wheel-build: Work around pip 19.0 errors https://review.opendev.org/657316 | 05:46 |
| AJaeger | thanks, ianw ! Can we test the change - or merge and monitor? | 05:49 |
| *** lpetrut has joined #openstack-infra | 05:54 | |
| *** yamamoto has joined #openstack-infra | 05:58 | |
| ianw | AJaeger: just dropped a comment, i did test locally | 06:00 |
| ianw | if you want to quick merge, i can kick off a run of it and monitor | 06:01 |
| *** yamamoto has quit IRC | 06:11 | |
| AJaeger | ianw: your call - won't be around rest of day myself... | 06:13 |
| *** florianf has joined #openstack-infra | 06:13 | |
| *** udesale has quit IRC | 06:23 | |
| *** udesale has joined #openstack-infra | 06:24 | |
| *** quiquell has joined #openstack-infra | 06:28 | |
| openstackgerrit | Merged openstack/project-config master: wheel-build: Work around pip 19.0 errors https://review.opendev.org/657316 | 06:36 |
| openstackgerrit | Merged openstack/project-config master: Remove unnecessary pyeclib configuration from zuul.d https://review.opendev.org/657147 | 06:37 |
| openstackgerrit | Merged openstack/project-config master: Add openstack/tenks Ansible dependencies https://review.opendev.org/618511 | 06:39 |
| *** yamamoto has joined #openstack-infra | 06:43 | |
| *** kopecmartin|off is now known as kopecmartin | 06:52 | |
| *** gthiemonge has quit IRC | 06:56 | |
| *** ginopc has joined #openstack-infra | 06:57 | |
| *** ccamacho has joined #openstack-infra | 06:57 | |
| *** yamamoto has quit IRC | 06:59 | |
| *** tesseract-RH has joined #openstack-infra | 07:05 | |
| *** xek has joined #openstack-infra | 07:07 | |
| *** amoralej|off is now known as amoralej | 07:09 | |
| *** pgaxatte has joined #openstack-infra | 07:12 | |
| *** altlogbot_0 has quit IRC | 07:13 | |
| *** yamamoto has joined #openstack-infra | 07:15 | |
| *** altlogbot_2 has joined #openstack-infra | 07:18 | |
| *** yamamoto has quit IRC | 07:21 | |
| *** apetrich has joined #openstack-infra | 07:22 | |
| openstackgerrit | Merged openstack/project-config master: add version map for python3.6 to 3.7 https://review.opendev.org/657044 | 07:26 |
| openstackgerrit | Merged openstack/project-config master: Retire cloud-init https://review.opendev.org/649082 | 07:26 |
| *** tosky has joined #openstack-infra | 07:28 | |
| *** arxcruz|rover is now known as arxcruz | 07:37 | |
| ianw | AJaeger: that got it building, at least | 07:40 |
| ianw | http://zuul.openstack.org/stream/8781cd38097b437d90ba5a9a5505633c?logfile=console.log | 07:40 |
| *** iurygregory has joined #openstack-infra | 07:43 | |
| *** dmellado has quit IRC | 07:44 | |
| *** dmellado has joined #openstack-infra | 07:44 | |
| *** _Cyclone_ has quit IRC | 07:45 | |
| *** kaiokmo has quit IRC | 07:45 | |
| *** jpena|off is now known as jpena | 07:49 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Only enable dbus-daemon for fedora-29 and below https://review.opendev.org/657126 | 07:49 |
| *** _Cyclone_ has joined #openstack-infra | 07:50 | |
| *** udesale has quit IRC | 07:55 | |
| *** udesale has joined #openstack-infra | 07:55 | |
| *** yamamoto has joined #openstack-infra | 08:00 | |
| *** adriancz has joined #openstack-infra | 08:01 | |
| *** rossella_s has joined #openstack-infra | 08:03 | |
| *** gthiemonge has joined #openstack-infra | 08:06 | |
| *** pkopec has joined #openstack-infra | 08:07 | |
| *** yamamoto has quit IRC | 08:07 | |
| *** ykarel is now known as ykarel|lunch | 08:08 | |
| *** pcaruana has joined #openstack-infra | 08:24 | |
| *** rpittau|afk is now known as rpittau | 08:24 | |
| *** florianf has quit IRC | 08:25 | |
| *** kjackal has joined #openstack-infra | 08:42 | |
| *** yamamoto has joined #openstack-infra | 08:45 | |
| *** ykarel|lunch is now known as ykarel | 08:50 | |
| *** yamamoto has quit IRC | 09:10 | |
| *** yamamoto has joined #openstack-infra | 09:11 | |
| *** altlogbot_2 has quit IRC | 09:11 | |
| *** altlogbot_0 has joined #openstack-infra | 09:15 | |
| *** altlogbot_0 has quit IRC | 09:15 | |
| *** altlogbot_0 has joined #openstack-infra | 09:16 | |
| *** yamamoto has quit IRC | 09:16 | |
| *** zbr is now known as zbr|rover | 09:18 | |
| *** yamamoto has joined #openstack-infra | 09:19 | |
| *** yamamoto has quit IRC | 09:19 | |
| *** roman_g has joined #openstack-infra | 09:23 | |
| *** gmann has joined #openstack-infra | 09:41 | |
| *** slaweq has joined #openstack-infra | 09:45 | |
| *** hwoarang has quit IRC | 09:49 | |
| *** hwoarang has joined #openstack-infra | 09:49 | |
| *** yamamoto has joined #openstack-infra | 09:56 | |
| *** gmann is now known as gmann_pto | 09:57 | |
| *** yamamoto has quit IRC | 10:08 | |
| *** Lucas_Gray has joined #openstack-infra | 10:21 | |
| *** slaweq has quit IRC | 10:23 | |
| *** amansi26 has joined #openstack-infra | 10:32 | |
| *** udesale has quit IRC | 10:34 | |
| *** udesale has joined #openstack-infra | 10:35 | |
| *** takamatsu has joined #openstack-infra | 10:36 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:43 | |
| *** larainema has joined #openstack-infra | 10:46 | |
| *** nicolasbock has joined #openstack-infra | 10:47 | |
| *** dpawlik has joined #openstack-infra | 11:03 | |
| *** ccamacho has quit IRC | 11:04 | |
| *** yboaron has joined #openstack-infra | 11:08 | |
| *** udesale has quit IRC | 11:17 | |
| *** yboaron_ has joined #openstack-infra | 11:18 | |
| *** altlogbot_0 has quit IRC | 11:19 | |
| *** altlogbot_0 has joined #openstack-infra | 11:20 | |
| *** yboaron has quit IRC | 11:20 | |
| *** jaosorior has joined #openstack-infra | 11:21 | |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** kjackal has quit IRC | 11:37 | |
| *** kjackal has joined #openstack-infra | 11:38 | |
| *** amansi26 has quit IRC | 11:43 | |
| *** rascasoft has quit IRC | 11:47 | |
| *** rascasoft has joined #openstack-infra | 11:49 | |
| *** rfolco|ruck has joined #openstack-infra | 11:50 | |
| *** bhavikdbavishi has quit IRC | 11:57 | |
| *** amoralej is now known as amoralej|lunch | 12:00 | |
| *** bobh has joined #openstack-infra | 12:06 | |
| *** rfarr_ has joined #openstack-infra | 12:06 | |
| *** rh-jelabarre has joined #openstack-infra | 12:07 | |
| *** yboaron_ has quit IRC | 12:07 | |
| *** bobh has quit IRC | 12:09 | |
| *** rlandy has joined #openstack-infra | 12:11 | |
| *** altlogbot_0 has quit IRC | 12:15 | |
| *** udesale has joined #openstack-infra | 12:19 | |
| *** altlogbot_1 has joined #openstack-infra | 12:20 | |
| *** udesale has quit IRC | 12:25 | |
| *** udesale has joined #openstack-infra | 12:26 | |
| *** jpena|lunch is now known as jpena | 12:29 | |
| *** pcaruana has quit IRC | 12:29 | |
| *** happyhemant has joined #openstack-infra | 12:41 | |
| *** zul has joined #openstack-infra | 12:47 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/diskimage-builder master: Bypass pip-and-virtualenv for later redhat distros https://review.opendev.org/656636 | 12:48 |
| *** yamamoto has joined #openstack-infra | 12:49 | |
| Shrews | #status log Restarted nodepool-builder on nb01 after being killed due to OOM | 12:57 |
| Shrews | infra-root: Not sure what it is with the nodepool memory issues lately (OS update?), but let's get https://review.opendev.org/656986 merged to attempt to help with some of that | 12:58 |
| *** gthiemon1e has joined #openstack-infra | 12:59 | |
| Shrews | infra-root: i'm expecting nb03 builder to die in a similar fashion very soon, too | 12:59 |
| *** aaronsheffield has joined #openstack-infra | 13:00 | |
| *** gthiemonge has quit IRC | 13:00 | |
| *** amoralej|lunch is now known as amoralej | 13:02 | |
| *** rfarr_ has quit IRC | 13:10 | |
| *** rfarr has joined #openstack-infra | 13:10 | |
| *** altlogbot_1 has quit IRC | 13:15 | |
| *** altlogbot_1 has joined #openstack-infra | 13:16 | |
| *** pcaruana has joined #openstack-infra | 13:16 | |
| *** mriedem has joined #openstack-infra | 13:19 | |
| *** altlogbot_1 has quit IRC | 13:21 | |
| *** altlogbot_3 has joined #openstack-infra | 13:24 | |
| *** altlogbot_3 has quit IRC | 13:25 | |
| *** altlogbot_3 has joined #openstack-infra | 13:26 | |
| *** psachin has quit IRC | 13:28 | |
| *** yamamoto has quit IRC | 13:31 | |
| *** yboaron_ has joined #openstack-infra | 13:32 | |
| mordred | Shrews: done | 13:34 |
| *** kjackal has quit IRC | 13:34 | |
| *** ykarel is now known as ykarel|afk | 13:35 | |
| Shrews | mordred: thx. now just need to find where we puppet the nodepool /etc/defaults file... | 13:35 |
| *** gthiemonge has joined #openstack-infra | 13:35 | |
| Shrews | ah, found it | 13:36 |
| *** gthiemon1e has quit IRC | 13:36 | |
| *** efried has joined #openstack-infra | 13:41 | |
| *** zzehring has quit IRC | 13:44 | |
| *** udesale has quit IRC | 13:46 | |
| *** ykarel|afk is now known as ykarel | 13:46 | |
| *** udesale has joined #openstack-infra | 13:47 | |
| *** zzehring has joined #openstack-infra | 13:47 | |
| *** yamamoto has joined #openstack-infra | 13:47 | |
| *** ianychoi has quit IRC | 13:48 | |
| *** kjackal has joined #openstack-infra | 13:48 | |
| openstackgerrit | David Shrewsbury proposed opendev/puppet-nodepool master: Enable jemalloc on nodepoolv3 builders + launchers https://review.opendev.org/657385 | 13:49 |
| Shrews | infra-root: i think ^ is what we need to actually use jemalloc after it's installed | 13:50 |
| *** bnemec has joined #openstack-infra | 13:50 | |
| *** Lucas_Gray has quit IRC | 13:51 | |
| *** ianychoi has joined #openstack-infra | 13:52 | |
| openstackgerrit | Merged opendev/puppet-nodepool master: Install jemalloc on nodepool servers https://review.opendev.org/656986 | 13:55 |
| *** tdasilva has joined #openstack-infra | 14:00 | |
| *** ianychoi has quit IRC | 14:04 | |
| *** janki has quit IRC | 14:07 | |
| *** sthussey has joined #openstack-infra | 14:08 | |
| *** pkopec_ has joined #openstack-infra | 14:13 | |
| *** pkopec_ has quit IRC | 14:13 | |
| *** pkopec_ has joined #openstack-infra | 14:14 | |
| *** pkopec has quit IRC | 14:14 | |
| *** quiquell is now known as quiquell|off | 14:15 | |
| *** udesale has quit IRC | 14:16 | |
| *** pkopec has joined #openstack-infra | 14:17 | |
| *** ianychoi has joined #openstack-infra | 14:18 | |
| *** udesale has joined #openstack-infra | 14:21 | |
| *** egonzalez has quit IRC | 14:22 | |
| *** egonzalez has joined #openstack-infra | 14:24 | |
| *** samueldmq has joined #openstack-infra | 14:27 | |
| *** ykarel is now known as ykarel|away | 14:27 | |
| mordred | pabelanger: feel like +A on https://review.opendev.org/657385 ? | 14:30 |
| pabelanger | done | 14:31 |
| pabelanger | I'll have to update zuul.a.c to start using them too | 14:31 |
| *** anteaya has joined #openstack-infra | 14:36 | |
| *** lpetrut has quit IRC | 14:36 | |
| Shrews | pabelanger: mordred: thx | 14:37 |
| *** dklyle has joined #openstack-infra | 14:48 | |
| *** dklyle has quit IRC | 14:49 | |
| corvus | next week i leave for barcelona to present on opendev and rook; i need a bit more time to prep for that since i haven't done a presentation like that yet. so this week i'll be spending a significant amount of time on that as well as recovering from the last 3 weeks. i'll check in periodically to see if anything is blocked, but otherwise will be afk quite a bit. | 14:49 |
| *** david-lyle has joined #openstack-infra | 14:49 | |
| openstackgerrit | Merged opendev/puppet-nodepool master: Enable jemalloc on nodepoolv3 builders + launchers https://review.opendev.org/657385 | 14:50 |
| mordred | corvus: ++ | 14:51 |
| *** armax has joined #openstack-infra | 14:53 | |
| *** imacdonn has joined #openstack-infra | 15:11 | |
| *** david-lyle is now known as dklyle | 15:13 | |
| *** gyee has joined #openstack-infra | 15:13 | |
| *** markvoelker has joined #openstack-infra | 15:16 | |
| *** gthiemonge has quit IRC | 15:16 | |
| *** yamamoto has quit IRC | 15:16 | |
| *** hrw has joined #openstack-infra | 15:19 | |
| hrw | hi | 15:19 |
| openstackgerrit | Merged zuul/zuul master: Bump tox jobs timeout to 3600 https://review.opendev.org/656771 | 15:20 |
| *** kjackal has quit IRC | 15:22 | |
| *** kjackal has joined #openstack-infra | 15:22 | |
| *** igordc has joined #openstack-infra | 15:28 | |
| corvus | mordred, fungi: it looks like 656871 is failing because we're actually running puppet in the system-config-run-* jobs now -- http://logs.openstack.org/71/656871/11/check/system-config-run-base/5b12895/trusty/syslog.txt.gz#_May__3_22_22_11 | 15:32 |
| mordred | corvus: oh. and hostnames | 15:32 |
| mordred | because puppet | 15:32 |
| *** hrw has left #openstack-infra | 15:33 | |
| corvus | yeah, that and it can't find the openstack_project puppet module | 15:33 |
| mordred | yeah | 15:33 |
| mordred | "fun" | 15:33 |
| corvus | did we miss something about putting system-config on the remote host? | 15:33 |
| corvus | i'm not sure how that's supposed to work | 15:33 |
| mordred | lemme dig - I think it might be something about setting module path | 15:33 |
| clarkb | ya fungi found that it couldnt find the systemd config modules | 15:36 |
| clarkb | despite lack of jetlag traveling and working for the last 9 days or so is exhausting today will be a sloe day for me. Also discovered a shower drain leaks so have to sort that out | 15:37 |
| *** chandankumar is now known as raukadah | 15:41 | |
| *** rpittau is now known as rpittau|afk | 15:43 | |
| mordred | corvus: ansible-role-puppet copies /opt/system-config from bridge to /opt/system-config/production on the remote host, and writes out puppet.conf and environment.conf on the remote host | 15:49 |
| mordred | corvus: we can see the copy happening herE: http://logs.openstack.org/71/656871/11/check/system-config-run-base/5b12895/job-output.txt.gz#_2019-05-03_22_22_12_957095 | 15:49 |
| *** Goneri has joined #openstack-infra | 15:50 | |
| *** ykarel|away has quit IRC | 15:53 | |
| *** yamamoto has joined #openstack-infra | 15:57 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add release note for memory leak https://review.opendev.org/657397 | 15:57 |
| openstackgerrit | Merged opendev/gear master: add missing str to bytes conversion for Python3 https://review.opendev.org/587731 | 15:58 |
| *** ekultails has joined #openstack-infra | 15:58 | |
| *** ginopc has quit IRC | 15:59 | |
| corvus | mordred: hrm... what could be different? | 15:59 |
| mordred | corvus: maybe the hostname thing is borking puppet more deeply? I'm not seeing anything wrong yet | 16:05 |
| openstackgerrit | Merged zuul/zuul master: Update timeout_handler to check for path first https://review.opendev.org/656782 | 16:10 |
| *** yamamoto has quit IRC | 16:10 | |
| *** pgaxatte has quit IRC | 16:11 | |
| *** rossella_s has quit IRC | 16:11 | |
| *** rossella_s has joined #openstack-infra | 16:12 | |
| *** ykarel has joined #openstack-infra | 16:17 | |
| *** mattw4 has joined #openstack-infra | 16:25 | |
| *** kjackal has quit IRC | 16:27 | |
| *** kjackal has joined #openstack-infra | 16:28 | |
| *** ijw_ has quit IRC | 16:30 | |
| *** igordc has quit IRC | 16:53 | |
| *** amoralej is now known as amoralej|off | 16:56 | |
| *** gmann_pto has quit IRC | 16:56 | |
| *** eernst has joined #openstack-infra | 17:02 | |
| clarkb | dmsimard: I'm still getting ara-community emails from github fwiw | 17:04 |
| *** eernst has quit IRC | 17:07 | |
| *** jpena is now known as jpena|off | 17:10 | |
| pabelanger | ianw: looks like https://review.opendev.org/657126/ is green, removal of dbus-broker in fedora-29 | 17:11 |
| openstackgerrit | Merged zuul/zuul master: Make test_playbook more robust https://review.opendev.org/656177 | 17:12 |
| *** kjackal has quit IRC | 17:12 | |
| *** jaosorior has quit IRC | 17:14 | |
| *** kjackal has joined #openstack-infra | 17:14 | |
| Shrews | ooh, we cannot use jemalloc on nb03 | 17:15 |
| Shrews | well that's a bummer | 17:15 |
| *** kopecmartin is now known as kopecmartin|off | 17:17 | |
| *** eernst has joined #openstack-infra | 17:17 | |
| *** eernst has quit IRC | 17:18 | |
| Shrews | oh, we can. just a different location | 17:19 |
| Shrews | grrrr | 17:19 |
| *** eernst has joined #openstack-infra | 17:19 | |
| Shrews | i might need some puppeting help for this | 17:19 |
| mordred | Shrews: why is it different on nb03? | 17:24 |
| *** whoami-rajat has quit IRC | 17:24 | |
| Shrews | mordred: aarch64 | 17:24 |
| mordred | oh. sigh | 17:24 |
| Shrews | yeah | 17:24 |
| *** tesseract-RH has quit IRC | 17:24 | |
| openstackgerrit | Merged zuul/zuul master: Fix race in test_job_node_failure_resume https://review.opendev.org/656178 | 17:24 |
| mordred | so we'll need to template that part I think | 17:24 |
| mordred | Shrews: I can help once I'm done with lunch-related things | 17:25 |
| *** jamesmcarthur has joined #openstack-infra | 17:25 | |
| *** ykarel has quit IRC | 17:25 | |
| *** udesale has quit IRC | 17:27 | |
| Shrews | mordred: i *think* i might have something... probably totally wrong | 17:28 |
| mordred | sweet! | 17:28 |
| openstackgerrit | David Shrewsbury proposed opendev/puppet-nodepool master: Support aarch64 builders for jemalloc usage https://review.opendev.org/657412 | 17:29 |
| clarkb | mordred: Shrews remote: https://review.opendev.org/657411 Verify image checksums after upload | 17:29 |
| Shrews | mordred: ^ | 17:29 |
| *** Swami has joined #openstack-infra | 17:29 | |
| clarkb | I haven't tested my change yet but wanted to get something up so we don't forget about that | 17:30 |
| clarkb | Shrews: that fix looks right | 17:31 |
| Shrews | \o/ | 17:31 |
| Shrews | i wasn't sure if i needed to add a new var for that | 17:32 |
| clarkb | you could use @architecture instead but this is a bit more explicit | 17:32 |
| clarkb | (so fine by me) | 17:32 |
| Shrews | clarkb: we weren't validating checksums already? weird | 17:36 |
| clarkb | Shrews: apparently glance doesn't actually take the hash as input on its api and check them | 17:37 |
| clarkb | and ya sdk/shade was only using it to check if it needs to upload a new image, not to verify if the upload was correct | 17:37 |
| clarkb | and that totally broke the ubuntu-xenial image upload to vexxhost-sjc1 at the summit | 17:37 |
| Shrews | neat | 17:38 |
| clarkb | though that may have been related to the OOMKiller so it is possible this isn't a complete fix | 17:38 |
| clarkb | it does seem that it should be checked at least | 17:38 |
| *** eernst has quit IRC | 17:39 | |
| clarkb | (Ideally glance would check it and not mark the image as successful until the values match, then it wouldn't matter if the client OOMs) | 17:40 |
| clarkb | md5 and sha256 are both incremental iirc so you can calculate as you read the data | 17:41 |
| *** Weifan has joined #openstack-infra | 17:47 | |
| *** ykarel has joined #openstack-infra | 17:48 | |
| dmsimard | clarkb: maybe because you're a watcher on the repo ? https://github.com/ansible-community/ara/watchers | 17:51 |
| clarkb | hrm maybe we automatically become watchers when we did the org addition? | 17:52 |
| dmsimard | I believe that's when the project was initially created on GitHub | 17:54 |
| dmsimard | It adds the people of the org as watchers | 17:54 |
| *** eernst has joined #openstack-infra | 17:58 | |
| tobiash | clarkb: commented on https://review.opendev.org/657411 | 17:59 |
| *** jamesmcarthur has quit IRC | 18:00 | |
| *** eernst has quit IRC | 18:02 | |
| *** eernst has joined #openstack-infra | 18:04 | |
| *** eernst has quit IRC | 18:09 | |
| clarkb | tobiash: good catch I believe it isnt set in the first get so have to swap those | 18:09 |
| *** markvoelker has quit IRC | 18:10 | |
| tobiash | clarkb: btw, I like that, I also already observed nodepool using incomplete images | 18:11 |
| *** eernst has joined #openstack-infra | 18:11 | |
| *** rossella_s has quit IRC | 18:15 | |
| openstackgerrit | Merged zuul/zuul master: Update assertNodepoolState() to retry zk requests https://review.opendev.org/656213 | 18:15 |
| *** eernst has quit IRC | 18:16 | |
| *** prometheanfire has quit IRC | 18:17 | |
| clarkb | tobiash: updated, thanks | 18:17 |
| *** eernst has joined #openstack-infra | 18:17 | |
| *** cmoura has joined #openstack-infra | 18:21 | |
| *** tjgresha_nope has quit IRC | 18:21 | |
| *** tjgresha has joined #openstack-infra | 18:21 | |
| *** tjgresha has quit IRC | 18:22 | |
| *** eernst has quit IRC | 18:22 | |
| *** Weifan has quit IRC | 18:22 | |
| *** Weifan has joined #openstack-infra | 18:26 | |
| *** yboaron_ has quit IRC | 18:28 | |
| *** yboaron_ has joined #openstack-infra | 18:28 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Match tag items against containing branches https://review.opendev.org/578557 | 18:30 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Use implied branch matcher for implied branches https://review.opendev.org/640272 | 18:31 |
| *** Weifan has quit IRC | 18:33 | |
| *** eernst has joined #openstack-infra | 18:35 | |
| *** eernst has quit IRC | 18:38 | |
| *** eernst has joined #openstack-infra | 18:38 | |
| *** Goneri has quit IRC | 18:41 | |
| *** kjackal has quit IRC | 18:42 | |
| *** Goneri has joined #openstack-infra | 18:42 | |
| *** happyhemant has quit IRC | 18:48 | |
| *** ykarel is now known as ykarel|away | 18:49 | |
| *** prometheanfire has joined #openstack-infra | 18:56 | |
| Shrews | pabelanger: mordred: either of you have a moment to push https://review.opendev.org/657412 along? | 19:03 |
| pabelanger | looking | 19:05 |
| *** Sukhdev has joined #openstack-infra | 19:10 | |
| *** jtomasek has quit IRC | 19:13 | |
| *** ykarel|away has quit IRC | 19:22 | |
| openstackgerrit | Merged opendev/puppet-nodepool master: Support aarch64 builders for jemalloc usage https://review.opendev.org/657412 | 19:26 |
| *** xek has quit IRC | 19:29 | |
| *** pkopec_ has joined #openstack-infra | 19:32 | |
| *** adriancz_ has joined #openstack-infra | 19:34 | |
| *** jamespage_ has joined #openstack-infra | 19:34 | |
| *** bdodd_ has joined #openstack-infra | 19:34 | |
| *** larainema_ has joined #openstack-infra | 19:35 | |
| *** rfarr_ has joined #openstack-infra | 19:35 | |
| *** hwoarang_ has joined #openstack-infra | 19:35 | |
| *** rh-jlabarre has joined #openstack-infra | 19:38 | |
| *** quiquell has joined #openstack-infra | 19:39 | |
| *** anteaya has quit IRC | 19:39 | |
| *** rascasoft has quit IRC | 19:39 | |
| *** pkopec has quit IRC | 19:39 | |
| *** rh-jelabarre has quit IRC | 19:39 | |
| *** yboaron_ has quit IRC | 19:39 | |
| *** Swami has quit IRC | 19:39 | |
| *** rfarr has quit IRC | 19:39 | |
| *** larainema has quit IRC | 19:39 | |
| *** hwoarang has quit IRC | 19:39 | |
| *** adriancz has quit IRC | 19:39 | |
| *** apetrich has quit IRC | 19:39 | |
| *** quiquell|off has quit IRC | 19:39 | |
| *** bdodd has quit IRC | 19:39 | |
| *** jamespage has quit IRC | 19:39 | |
| *** larainema_ is now known as larainema | 19:39 | |
| *** adriancz_ is now known as adriancz | 19:39 | |
| *** jamespage_ is now known as jamespage | 19:39 | |
| *** yboaron_ has joined #openstack-infra | 19:40 | |
| *** rascasoft has joined #openstack-infra | 19:40 | |
| *** Swami has joined #openstack-infra | 19:40 | |
| *** apetrich has joined #openstack-infra | 19:40 | |
| *** ijw has joined #openstack-infra | 19:41 | |
| openstackgerrit | Adam Spiers proposed openstack/cookiecutter master: Fix outdated references to git.openstack.org/cgit/ https://review.opendev.org/657444 | 19:46 |
| *** ijw has quit IRC | 19:46 | |
| *** ijw has joined #openstack-infra | 19:49 | |
| *** takamatsu has quit IRC | 19:51 | |
| *** Weifan has joined #openstack-infra | 19:52 | |
| *** ijw has quit IRC | 19:55 | |
| *** ijw has joined #openstack-infra | 19:55 | |
| *** Sukhdev has quit IRC | 20:00 | |
| *** hwoarang_ has quit IRC | 20:09 | |
| *** hwoarang has joined #openstack-infra | 20:15 | |
| openstackgerrit | Merged zuul/zuul master: Add release note for memory leak https://review.opendev.org/657397 | 20:15 |
| Shrews | infra-root: I want to restart all of the nodepool builders and launchers to begin using jemalloc. Objections? | 20:21 |
| clarkb | none from me | 20:27 |
| Shrews | k. going to start that | 20:28 |
| Shrews | looks like nb03 is out of disk space on /mnt | 20:30 |
| Shrews | #status log Restarted nodepool builders and launchers on commit 2e8f286c4ff3c31925cf4b0a23b0bd8104cb3b80 to begin using jemalloc | 20:34 |
| Shrews | i don't think our status logger is happy | 20:34 |
| corvus | mordred: can you take over 656871? i'm not sure how to proceed there. | 20:35 |
| *** ijw has quit IRC | 20:37 | |
| *** ijw has joined #openstack-infra | 20:37 | |
| *** zul has quit IRC | 20:47 | |
| *** pcaruana has quit IRC | 20:51 | |
| openstackgerrit | Merged zuul/zuul master: Make paused status bar blue https://review.opendev.org/655588 | 20:52 |
| Shrews | infra-root: i manually removed many dib images from nb03 to free up space. Many have been stuck in the 'deleting' state for weeks, so the on-disk images were never removed (done after deleting at the provider). Unfortunately, the logs don't go back far enough to see any errors. | 20:53 |
| Shrews | I'm not mentally prepared to think about causes/solutions just at this moment | 20:53 |
| clarkb | Shrews: I think that is because we have ti clean up the arm64 cloud(s) | 20:53 |
| clarkb | The linaro cloud remains | 20:54 |
| Shrews | clarkb what do you mean by "clean up"? | 20:54 |
| clarkb | Those clouds are shut off or in the process of being shutoff | 20:54 |
| Shrews | ah | 20:54 |
| clarkb | so we have to remove them from nodepool and remove any of our db entries for them | 20:54 |
| Shrews | i don't think that explains why we began deleting the images from the provider though, and just didn't complete | 20:55 |
| Shrews | but i'm also mentally taxed atm | 20:55 |
| clarkb | it didnt complete becausethe api calls to delete them in the cloud failed I expect | 20:55 |
| clarkb | becausethe cloud is off or half off | 20:56 |
| * Shrews crosses fingers on the jemalloc stuff making much prettier cacti graphs, and EODs | 20:59 | |
| *** mattw4 has quit IRC | 21:05 | |
| *** Lucas_Gray has joined #openstack-infra | 21:14 | |
| *** dpawlik has quit IRC | 21:15 | |
| *** pkopec_ has quit IRC | 21:16 | |
| *** dpawlik has joined #openstack-infra | 21:20 | |
| *** nicolasbock has quit IRC | 21:21 | |
| *** mattw4 has joined #openstack-infra | 21:22 | |
| *** dpawlik has quit IRC | 21:24 | |
| *** ijw has quit IRC | 21:31 | |
| *** ijw has joined #openstack-infra | 21:32 | |
| *** ijw_ has joined #openstack-infra | 21:33 | |
| *** ijw has quit IRC | 21:34 | |
| *** mattw4 has quit IRC | 21:43 | |
| *** mattw4 has joined #openstack-infra | 21:44 | |
| *** rkukura has quit IRC | 21:52 | |
| *** yboaron_ has quit IRC | 21:54 | |
| ianw | pabelanger: i don't think ihave any further thoughts on the f29 stuff, is it helpful for you to get a release with just that? | 21:56 |
| *** rfarr_ has quit IRC | 21:56 | |
| ianw | 2019-05-06 08:01:54.929596 | TASK [create-afs-token : Obtain ticket for Kerberos] | 21:59 |
| ianw | 2019-05-06 08:01:55.706800 | wheel-mirror-centos-7-python2 | kinit: Cannot contact any KDC for realm 'OPENSTACK.ORG' while getting initial credentials | 21:59 |
| ianw | clarkb: ^ ring any bells? | 21:59 |
| ianw | that was http://logs.openstack.org/6a/6affabb05381072509f382af865a54a7916b7664/periodic/publish-wheel-mirror-centos-7/8781cd3/job-output.txt.gz which i kicked off manually yesterday | 22:00 |
| *** ijw_ has quit IRC | 22:03 | |
| *** ijw has joined #openstack-infra | 22:03 | |
| openstackgerrit | sebastian marcet proposed opendev/system-config master: Fixed email configuration for openstackid https://review.opendev.org/657460 | 22:04 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add testinfra 3rd party testing https://review.opendev.org/657461 | 22:06 |
| *** Goneri has quit IRC | 22:07 | |
| *** ijw has quit IRC | 22:08 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add testinfra 3rd party testing https://review.opendev.org/657461 | 22:11 |
| clarkb | ianw: nope sorry | 22:11 |
| *** jamesmcarthur has joined #openstack-infra | 22:22 | |
| fungi | corvus: cool, i tried to provide a brief redux in irc on friday of the failures for puppet in the ansible run jobs, but it's likely lost to the annals of irc | 22:30 |
| fungi | anyway, glad to see i was on the right track | 22:30 |
| *** anteaya has joined #openstack-infra | 22:31 | |
| corvus | fungi: thanks -- i'm not sure how to proceed with that and don't have cycles at the moment, so if you have any ideas feel free to go for it | 22:32 |
| mordred | corvus: yes - I will take over 656871 | 22:33 |
| fungi | yeah, i'm not a font of ideas. i just passed out for several hours while doing nearly nothing | 22:33 |
| fungi | hopefully tomorrow will be better | 22:34 |
| mordred | fungi: I'm hoping that inspiration will strike tomorrow | 22:34 |
| fungi | but yeah, my first suspicion was that maybe install_modules.sh isn't finding the zuul-provided copy of system config for some reason | 22:35 |
| fungi | i'm sure you can all see my hands waving as i typed that | 22:35 |
| mordred | yuo | 22:35 |
| mordred | mine were waving in sync with them | 22:36 |
| corvus | carefull with all that hand-waving -- you'll put your eye out | 22:36 |
| fungi | i think i may already have done | 22:37 |
| mordred | fungi: you still have eyes? | 22:39 |
| fungi | still saving up for the bionic implants to replace them | 22:39 |
| mordred | if only we had real health care in the US | 22:41 |
| paladox | lol | 22:44 |
| openstackgerrit | Merged opendev/system-config master: Fixed email configuration for openstackid https://review.opendev.org/657460 | 22:44 |
| *** rcernin has joined #openstack-infra | 22:48 | |
| *** Weifan has quit IRC | 22:49 | |
| *** jamesmcarthur has quit IRC | 22:51 | |
| *** eernst has quit IRC | 22:53 | |
| *** Weifan has joined #openstack-infra | 22:55 | |
| *** Weifan has quit IRC | 22:56 | |
| *** Weifan has joined #openstack-infra | 22:56 | |
| *** hwoarang has quit IRC | 22:57 | |
| *** Weifan has quit IRC | 22:57 | |
| *** eernst has joined #openstack-infra | 22:59 | |
| *** hwoarang has joined #openstack-infra | 22:59 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add testinfra 3rd party testing https://review.opendev.org/657461 | 23:00 |
| *** zbr|rover has quit IRC | 23:01 | |
| *** eernst has quit IRC | 23:01 | |
| *** ijw has joined #openstack-infra | 23:01 | |
| *** tkajinam has joined #openstack-infra | 23:03 | |
| *** tdasilva has quit IRC | 23:08 | |
| *** gyee has quit IRC | 23:25 | |
| *** aaronsheffield has quit IRC | 23:26 | |
| *** tosky has quit IRC | 23:32 | |
| *** hwoarang has quit IRC | 23:41 | |
| *** hwoarang has joined #openstack-infra | 23:42 | |
| *** david-lyle has joined #openstack-infra | 23:55 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: openafs-client : update kdc servers https://review.opendev.org/657504 | 23:55 |
| *** dklyle has quit IRC | 23:55 | |
| ianw | clarkb: ^ kdc servers for the wheel jobs | 23:56 |
| ianw | puppet was updated which is why mirror-update and other puppet bits are chugging along i think | 23:57 |
| *** sthussey has quit IRC | 23:58 | |
| *** rlandy has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!