| *** diablo_rojo is now known as diablo_rojo__ | 00:00 | |
| *** diablo_rojo__ is now known as diablo_rojo | 00:00 | |
| donnyd | clarkb: whenever you get a chance... it seems like the fedora28 image is flaky with ipv6 | 00:05 |
|---|---|---|
| ianw | that's probably more my problem :) in that it just doesn't come up properly? | 00:06 |
| donnyd | yea, all of the rest of the images work fine, it would see that the fedora28 image does not. v4 works, v6 does not | 00:09 |
| donnyd | i have to go tv for a while | 00:09 |
| ianw | hrm, so the f29 images work ok? | 00:10 |
| ianw | we probably don't want to expend too much effort on f28 ... the only reason it's hung around is as a rhel8 analogue | 00:10 |
| donnyd | yea | 00:10 |
| donnyd | ok, just thought i would put it out there | 00:10 |
| * donnyd walks to tv and watches | 00:11 | |
| ianw | cool, i'm trying to get f30 up , it's lowish priority which is why it's taken so long | 00:11 |
| ianw | ttyl | 00:11 |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** michael-beaver has quit IRC | 00:12 | |
| *** slaweq has quit IRC | 00:16 | |
| *** rosmaita has quit IRC | 00:18 | |
| *** diablo_rojo has quit IRC | 00:20 | |
| *** diablo_rojo has joined #openstack-infra | 00:21 | |
| *** rosmaita has joined #openstack-infra | 00:31 | |
| *** ianychoi has joined #openstack-infra | 00:34 | |
| *** rlandy is now known as rlandy|bbl | 00:35 | |
| *** betherly has joined #openstack-infra | 00:35 | |
| *** Lucas_Gray has quit IRC | 00:36 | |
| *** dchen has joined #openstack-infra | 00:37 | |
| *** betherly has quit IRC | 00:39 | |
| *** goldyfruit has joined #openstack-infra | 00:40 | |
| *** lseki has quit IRC | 00:53 | |
| *** betherly has joined #openstack-infra | 00:55 | |
| *** happyhemant has quit IRC | 00:59 | |
| *** betherly has quit IRC | 01:00 | |
| *** diablo_rojo has quit IRC | 01:18 | |
| *** betherly has joined #openstack-infra | 01:46 | |
| *** e0ne has joined #openstack-infra | 01:48 | |
| *** ramishra has joined #openstack-infra | 01:52 | |
| *** betherly has quit IRC | 01:52 | |
| *** e0ne has quit IRC | 01:52 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:54 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 01:57 | |
| *** jamesmcarthur has joined #openstack-infra | 01:58 | |
| *** jamesmcarthur has quit IRC | 01:58 | |
| *** jamesmcarthur has joined #openstack-infra | 01:58 | |
| *** bhavikdbavishi has quit IRC | 01:58 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 01:58 | |
| *** yamamoto has joined #openstack-infra | 02:03 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: FN can now support 100 nodes - turning up quota https://review.opendev.org/674149 | 02:07 |
| *** rlandy|bbl is now known as rlandy | 02:07 | |
| *** apetrich has quit IRC | 02:08 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** rh-jelabarre has quit IRC | 02:21 | |
| *** rlandy has quit IRC | 02:29 | |
| *** ykarel|away has joined #openstack-infra | 02:32 | |
| *** yamamoto has quit IRC | 02:42 | |
| *** yamamoto has joined #openstack-infra | 02:43 | |
| *** ykarel|away has quit IRC | 02:46 | |
| *** n-saito has joined #openstack-infra | 02:48 | |
| *** bhavikdbavishi has quit IRC | 02:48 | |
| *** betherly has joined #openstack-infra | 02:59 | |
| *** betherly has quit IRC | 03:03 | |
| *** dpawlik has quit IRC | 03:04 | |
| *** betherly has joined #openstack-infra | 03:19 | |
| *** psachin has joined #openstack-infra | 03:21 | |
| *** betherly has quit IRC | 03:24 | |
| *** gyee has quit IRC | 03:26 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:33 | |
| *** psachin has quit IRC | 03:36 | |
| *** betherly has joined #openstack-infra | 03:39 | |
| *** betherly has quit IRC | 03:45 | |
| *** armax has quit IRC | 03:45 | |
| *** armax has joined #openstack-infra | 03:46 | |
| *** jamesmcarthur has quit IRC | 03:46 | |
| *** armax has quit IRC | 03:46 | |
| *** jamesmcarthur has joined #openstack-infra | 03:46 | |
| *** betherly has joined #openstack-infra | 03:50 | |
| *** jamesmcarthur has quit IRC | 03:51 | |
| *** betherly has quit IRC | 03:55 | |
| *** yamamoto has quit IRC | 04:01 | |
| *** yamamoto has joined #openstack-infra | 04:02 | |
| *** auristor has quit IRC | 04:05 | |
| *** betherly has joined #openstack-infra | 04:11 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** auristor has joined #openstack-infra | 04:14 | |
| *** slaweq has quit IRC | 04:16 | |
| *** betherly has quit IRC | 04:16 | |
| *** jamesmcarthur has joined #openstack-infra | 04:16 | |
| *** betherly has joined #openstack-infra | 04:21 | |
| *** betherly has quit IRC | 04:26 | |
| openstackgerrit | Merged openstack/project-config master: FN can now support 100 nodes - turning up quota https://review.opendev.org/674149 | 04:27 |
| *** e0ne has joined #openstack-infra | 04:30 | |
| *** whoami-rajat has joined #openstack-infra | 04:34 | |
| *** e0ne has quit IRC | 04:35 | |
| *** betherly has joined #openstack-infra | 04:41 | |
| *** betherly has quit IRC | 04:46 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Make tenant and pipeline optional in zuul-changes https://review.opendev.org/674034 | 04:51 |
| *** jamesmcarthur has quit IRC | 04:53 | |
| *** diga has joined #openstack-infra | 04:54 | |
| *** tkajinam has quit IRC | 05:00 | |
| *** betherly has joined #openstack-infra | 05:02 | |
| *** tkajinam has joined #openstack-infra | 05:02 | |
| *** udesale has joined #openstack-infra | 05:07 | |
| *** betherly has quit IRC | 05:07 | |
| *** Lucas_Gray has joined #openstack-infra | 05:10 | |
| *** betherly has joined #openstack-infra | 05:18 | |
| *** odicha has joined #openstack-infra | 05:20 | |
| *** betherly has quit IRC | 05:22 | |
| *** markvoelker has joined #openstack-infra | 05:34 | |
| *** markvoelker has quit IRC | 05:39 | |
| *** kopecmartin|off is now known as kopecmartin | 05:45 | |
| *** tkajinam_ has joined #openstack-infra | 05:53 | |
| *** tkajinam has quit IRC | 05:55 | |
| *** iurygregory has quit IRC | 05:56 | |
| *** e0ne has joined #openstack-infra | 06:04 | |
| *** janki has joined #openstack-infra | 06:04 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** Lucas_Gray has quit IRC | 06:13 | |
| *** slaweq has quit IRC | 06:15 | |
| *** pgaxatte has joined #openstack-infra | 06:23 | |
| ianw | #status log afs servers restarted without logging as kafs server currently out of rotation | 06:33 |
| openstackstatus | ianw: finished logging | 06:33 |
| *** apetrich has joined #openstack-infra | 06:34 | |
| *** janki has quit IRC | 06:34 | |
| *** janki has joined #openstack-infra | 06:35 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: AFS server restart and audit logging : helper script https://review.opendev.org/672847 | 06:38 |
| *** jaosorior has joined #openstack-infra | 06:38 | |
| *** e0ne has quit IRC | 06:42 | |
| *** pkopec has joined #openstack-infra | 06:44 | |
| *** e0ne has joined #openstack-infra | 06:45 | |
| *** witek has joined #openstack-infra | 06:51 | |
| *** jhesketh has joined #openstack-infra | 06:54 | |
| *** rcernin has quit IRC | 06:58 | |
| *** ginopc has joined #openstack-infra | 06:58 | |
| *** slaweq has joined #openstack-infra | 07:05 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: kafs support https://review.opendev.org/623974 | 07:16 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ubuntu-kernel: role to use Ubuntu mainline kernels https://review.opendev.org/665057 | 07:16 |
| *** redrobot has quit IRC | 07:17 | |
| *** tesseract has joined #openstack-infra | 07:17 | |
| *** rascasoft has joined #openstack-infra | 07:29 | |
| *** iurygregory has joined #openstack-infra | 07:30 | |
| *** jpenag is now known as jpena | 07:31 | |
| *** e0ne has quit IRC | 07:31 | |
| openstackgerrit | Merged zuul/nodepool master: builder: Log all deletions of image upload records https://review.opendev.org/674126 | 07:32 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ubuntu-kernel: role to use Ubuntu mainline kernels https://review.opendev.org/665057 | 07:37 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: kafs: allow to skip cachefilesd https://review.opendev.org/674215 | 07:37 |
| *** janki has quit IRC | 07:43 | |
| *** gfidente has joined #openstack-infra | 07:45 | |
| *** ramishra has quit IRC | 07:46 | |
| *** jtomasek has joined #openstack-infra | 07:47 | |
| *** jtomasek has quit IRC | 07:48 | |
| *** tosky has joined #openstack-infra | 07:48 | |
| *** jtomasek has joined #openstack-infra | 07:50 | |
| *** lucasagomes has joined #openstack-infra | 07:52 | |
| *** ralonsoh has joined #openstack-infra | 08:10 | |
| *** dchen has quit IRC | 08:10 | |
| openstackgerrit | ye proposed openstack/infra-manual master: Update bug status link. https://review.opendev.org/674218 | 08:12 |
| *** iurygregory has quit IRC | 08:35 | |
| *** jtomasek has quit IRC | 08:40 | |
| *** apetrich has quit IRC | 08:49 | |
| *** Lucas_Gray has joined #openstack-infra | 08:51 | |
| *** smrcascao has joined #openstack-infra | 08:52 | |
| *** Goneri has joined #openstack-infra | 08:54 | |
| *** ramishra has joined #openstack-infra | 08:57 | |
| *** tkajinam_ has quit IRC | 09:01 | |
| *** e0ne has joined #openstack-infra | 09:03 | |
| *** jtomasek has joined #openstack-infra | 09:03 | |
| *** derekh has joined #openstack-infra | 09:06 | |
| *** markvoelker has joined #openstack-infra | 09:13 | |
| *** bhavikdbavishi has quit IRC | 09:13 | |
| openstackgerrit | Mark Goddard proposed openstack/project-config master: Rename x/kayobe* to openstack/ https://review.opendev.org/669298 | 09:29 |
| *** electrofelix has joined #openstack-infra | 09:37 | |
| *** markvoelker has quit IRC | 09:46 | |
| *** ramishra has quit IRC | 09:48 | |
| *** ramishra has joined #openstack-infra | 09:48 | |
| *** Lucas_Gray has quit IRC | 09:49 | |
| *** Lucas_Gray has joined #openstack-infra | 09:51 | |
| *** ociuhandu has joined #openstack-infra | 09:54 | |
| *** Goneri has quit IRC | 09:56 | |
| *** yamamoto has quit IRC | 10:07 | |
| *** gfidente has quit IRC | 10:09 | |
| *** pkopec has quit IRC | 10:11 | |
| *** yamamoto has joined #openstack-infra | 10:13 | |
| *** priteau has joined #openstack-infra | 10:13 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** yamamoto has joined #openstack-infra | 10:15 | |
| *** yamamoto has quit IRC | 10:21 | |
| *** yamamoto has joined #openstack-infra | 10:23 | |
| *** yamamoto has quit IRC | 10:23 | |
| *** yamamoto has joined #openstack-infra | 10:24 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [WIP] Add OpenAPI description for enqueue, dequeue, autohold https://review.opendev.org/674257 | 10:25 |
| *** yamamoto has quit IRC | 10:29 | |
| *** tdasilva has quit IRC | 10:33 | |
| *** hrw has joined #openstack-infra | 10:34 | |
| hrw | morning | 10:34 |
| *** ociuhandu has quit IRC | 10:34 | |
| *** ociuhandu has joined #openstack-infra | 10:34 | |
| hrw | https://review.opendev.org/#/c/668157/ got merged month ago and debian/buster is still mirrored without backports. also without updates. | 10:35 |
| hrw | http://mirror.london.linaro-london.openstack.org/debian/dists/ http://mirror.ord.rax.opendev.org/debian/dists/ should have buster-backports from that patch iirc... | 10:36 |
| hrw | have to find also where to enable buster-updates | 10:36 |
| *** ociuhandu has quit IRC | 10:36 | |
| *** ociuhandu has joined #openstack-infra | 10:37 | |
| *** pkopec has joined #openstack-infra | 10:44 | |
| *** apetrich has joined #openstack-infra | 10:45 | |
| *** iurygregory has joined #openstack-infra | 10:47 | |
| *** Goneri has joined #openstack-infra | 10:51 | |
| *** gfidente has joined #openstack-infra | 10:54 | |
| frickler | hrw: mirror.debian and mirror.fedora haven't updated in a month according to http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1 , I didn't get to check what is happening there yet, maybe some other infra-root can step in later | 10:59 |
| hrw | frickler: thanks | 10:59 |
| *** ricolin has quit IRC | 11:00 | |
| *** ginopc has quit IRC | 11:04 | |
| *** yamamoto has joined #openstack-infra | 11:08 | |
| *** jaosorior has quit IRC | 11:08 | |
| *** yamamoto has quit IRC | 11:13 | |
| *** pkopec_ has joined #openstack-infra | 11:14 | |
| *** pkopec has quit IRC | 11:17 | |
| *** ccamacho has joined #openstack-infra | 11:18 | |
| *** tdasilva has joined #openstack-infra | 11:19 | |
| *** Wryhder has joined #openstack-infra | 11:23 | |
| *** Lucas_Gray has quit IRC | 11:23 | |
| openstackgerrit | Adam Spiers proposed openstack/project-config master: Make gerritbot notify #openstack-neutron IRC channel of stable reviews https://review.opendev.org/674273 | 11:24 |
| *** Wryhder is now known as Lucas_Gray | 11:24 | |
| hrw | fungi: ^^ | 11:24 |
| hrw | fungi: or better way: can Debian mirrors get updates more often than once per month? | 11:25 |
| *** zbr has quit IRC | 11:25 | |
| *** iurygregory has quit IRC | 11:31 | |
| *** iurygregory has joined #openstack-infra | 11:31 | |
| *** jpena is now known as jpena|lunch | 11:34 | |
| *** yamamoto has joined #openstack-infra | 11:35 | |
| *** ykarel|away has joined #openstack-infra | 11:36 | |
| *** kopecmartin is now known as kopecmartin|pto | 11:36 | |
| *** yamamoto has quit IRC | 11:36 | |
| *** guoqiao has joined #openstack-infra | 11:36 | |
| *** joeguo has quit IRC | 11:37 | |
| *** yamamoto has joined #openstack-infra | 11:43 | |
| *** takamatsu is now known as mauro|call | 11:44 | |
| *** yamamoto has quit IRC | 11:45 | |
| *** iurygregory has quit IRC | 11:47 | |
| *** iurygregory has joined #openstack-infra | 11:48 | |
| *** SotK__ has quit IRC | 11:51 | |
| *** Lucas_Gray has quit IRC | 11:51 | |
| *** guoqiao has quit IRC | 11:52 | |
| *** aedc has quit IRC | 11:54 | |
| *** rpittau|afk is now known as rpittau | 11:56 | |
| *** yamamoto has joined #openstack-infra | 11:57 | |
| *** yamamoto has quit IRC | 11:58 | |
| *** yamamoto has joined #openstack-infra | 11:59 | |
| *** markvoelker has joined #openstack-infra | 12:00 | |
| *** Lucas_Gray has joined #openstack-infra | 12:00 | |
| *** witek has quit IRC | 12:01 | |
| *** udesale has quit IRC | 12:02 | |
| *** udesale has joined #openstack-infra | 12:03 | |
| *** markvoelker has quit IRC | 12:03 | |
| *** markvoelker has joined #openstack-infra | 12:03 | |
| *** yamamoto has quit IRC | 12:03 | |
| *** markvoelker has quit IRC | 12:04 | |
| *** panda is now known as panda|lunch | 12:05 | |
| *** redrobot has joined #openstack-infra | 12:07 | |
| *** Lucas_Gray has quit IRC | 12:08 | |
| *** yamamoto has joined #openstack-infra | 12:09 | |
| *** pgaxatte has quit IRC | 12:12 | |
| *** yamamoto has quit IRC | 12:13 | |
| *** yamamoto has joined #openstack-infra | 12:13 | |
| *** markvoelker has joined #openstack-infra | 12:16 | |
| *** yamamoto has quit IRC | 12:18 | |
| *** rh-jelabarre has joined #openstack-infra | 12:22 | |
| *** ricolin has joined #openstack-infra | 12:23 | |
| *** ociuhandu has quit IRC | 12:24 | |
| *** rlandy has joined #openstack-infra | 12:25 | |
| *** dpawlik has joined #openstack-infra | 12:25 | |
| *** pgaxatte has joined #openstack-infra | 12:27 | |
| *** priteau has quit IRC | 12:28 | |
| openstackgerrit | Merged openstack/ptgbot master: Reset to OrderedDict on new day cleanup https://review.opendev.org/670577 | 12:35 |
| *** zbr has joined #openstack-infra | 12:35 | |
| openstackgerrit | Merged openstack/ptgbot master: Clean up stale data presence on a #newday command https://review.opendev.org/670578 | 12:35 |
| openstackgerrit | Merged openstack/ptgbot master: Add Python 3 Train unit tests https://review.opendev.org/670753 | 12:35 |
| *** witek has joined #openstack-infra | 12:35 | |
| *** jpena|lunch is now known as jpena | 12:39 | |
| *** jamesmcarthur has joined #openstack-infra | 12:39 | |
| *** ykarel|away has quit IRC | 12:40 | |
| hrw | jpena: can you help with debian mirroring? | 12:43 |
| jpena | hrw: I'm not an infra-root, so there's not a lot I can do to help, I'm afraid | 12:44 |
| *** n-saito has quit IRC | 12:44 | |
| jpena | dmsimard maybe? ^^ | 12:44 |
| hrw | jpena: ok. | 12:44 |
| hrw | jpena: I lost track who is who ;( | 12:44 |
| jpena | no worries :D | 12:44 |
| hrw | using Debian as a base for anything around openstack feels like asking for problems ;D | 12:45 |
| *** yamamoto has joined #openstack-infra | 12:47 | |
| *** yamamoto has quit IRC | 12:49 | |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused API SIG meeting slot https://review.opendev.org/674299 | 12:51 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused openstack-chef meeting slot https://review.opendev.org/674300 | 12:52 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused Designate meeting slot https://review.opendev.org/674301 | 12:53 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused Dragonflow meeting slot https://review.opendev.org/674303 | 12:54 |
| *** witek has quit IRC | 12:54 | |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused Gluon meeting slot https://review.opendev.org/674304 | 12:54 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused LCOO WG meeting slot https://review.opendev.org/674305 | 12:55 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused Mogan meeting slot https://review.opendev.org/674306 | 12:56 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused Murano meeting slot https://review.opendev.org/674307 | 12:57 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused BGPVPN meeting slot https://review.opendev.org/674308 | 12:58 |
| *** markvoelker has quit IRC | 12:58 | |
| *** markvoelker has joined #openstack-infra | 12:58 | |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Free up unused Telco NFV ops meeting slot https://review.opendev.org/674309 | 12:59 |
| *** rh-jelabarre has quit IRC | 13:00 | |
| *** yamamoto has joined #openstack-infra | 13:02 | |
| *** yamamoto has quit IRC | 13:02 | |
| *** yamamoto has joined #openstack-infra | 13:03 | |
| * smcginnis remembers debates about adding another meeting channel in order to fit them all | 13:03 | |
| *** sthussey has joined #openstack-infra | 13:04 | |
| *** ekultails has joined #openstack-infra | 13:06 | |
| *** panda|lunch is now known as panda | 13:06 | |
| AJaeger | smcginnis: meetings can now happen in the project channel as well... | 13:08 |
| *** yamamoto has quit IRC | 13:08 | |
| hrw | we that way in Kolla. | 13:08 |
| hrw | s/we/we do/ | 13:08 |
| smcginnis | AJaeger: Yeah, I'd say things have gotten a lot better since back then. | 13:09 |
| *** priteau has joined #openstack-infra | 13:10 | |
| *** ociuhandu has joined #openstack-infra | 13:11 | |
| *** jcoufal has joined #openstack-infra | 13:11 | |
| *** ccamacho has quit IRC | 13:18 | |
| *** pkopec_ has quit IRC | 13:18 | |
| *** eharney has joined #openstack-infra | 13:19 | |
| *** mriedem has joined #openstack-infra | 13:21 | |
| *** happyhemant has joined #openstack-infra | 13:21 | |
| *** yamamoto has joined #openstack-infra | 13:22 | |
| *** markvoelker has quit IRC | 13:25 | |
| *** goldyfruit has quit IRC | 13:25 | |
| *** markvoelker has joined #openstack-infra | 13:25 | |
| *** goldyfruit has joined #openstack-infra | 13:25 | |
| *** dciabrin__ is now known as dciabrin | 13:31 | |
| *** aaronsheffield has joined #openstack-infra | 13:32 | |
| *** ramishra has quit IRC | 13:33 | |
| *** lseki has joined #openstack-infra | 13:36 | |
| *** pkopec has joined #openstack-infra | 13:39 | |
| *** jcoufal_ has joined #openstack-infra | 13:48 | |
| *** jcoufal has quit IRC | 13:48 | |
| *** pgaxatte has quit IRC | 13:54 | |
| *** rascasoft has quit IRC | 13:56 | |
| zbr | ianw: do you happen to be core on testinfra? | 13:57 |
| *** rascasoft has joined #openstack-infra | 13:58 | |
| *** pgaxatte has joined #openstack-infra | 14:00 | |
| *** odicha has quit IRC | 14:02 | |
| *** ramishra has joined #openstack-infra | 14:03 | |
| *** jamesmcarthur has quit IRC | 14:10 | |
| fungi | frickler: hrw: i know the debian mirror problem was that the buster mirroring was added without adding the buster signing key (and stretch had been added without the stretch signing key for that matter) and so at least buster-backports is now no longer being signed by the jessie signing key. we merged a change to replace the jessie signing key with those for stretch and buster, and i meant to check | 14:10 |
| fungi | back in on the mirrors to make sure that was working but then got sidetracked. likely i still missed something. will check shortly | 14:10 |
| hrw | fungi: thanks | 14:11 |
| hrw | fungi: we want to add Debian CI job in Kolla. so far it only goes into RETRY_LIMIT (which probably could get renamed to PRE_FAILURE or something) | 14:12 |
| *** liuyulong has joined #openstack-infra | 14:15 | |
| fungi | well, pre-run playbook failures aren't the only way that a build might get retried | 14:16 |
| fungi | loss of network connection during other phases can also cause it | 14:16 |
| hrw | sure | 14:17 |
| fungi | and just because it was retried the maximum allowed number of times doesn't mean that it failed at the same place each time | 14:17 |
| *** rh-jelabarre has joined #openstack-infra | 14:19 | |
| hrw | for this one it was same place but not related to CI itself but to Debian mirrors | 14:19 |
| fungi | is it complaining about outdated indices i guess? | 14:21 |
| fungi | but yeah, it looks like reprepro during its validation phase is not finding the keys we're trying to install | 14:23 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add OpenAPI description for enqueue, dequeue, autohold https://review.opendev.org/674257 | 14:23 |
| hrw | fungi: about missing buster-updates and buster-backports | 14:23 |
| hrw | https://logs.opendev.org/40/674240/4/check/kolla-build-debian-source/0f774cf/ara-report/result/80fae6d4-f9bf-4cdc-82ab-9bdd80e45d3f/ | 14:24 |
| *** jamesmcarthur has joined #openstack-infra | 14:24 | |
| *** electrofelix has quit IRC | 14:25 | |
| fungi | there are no puppet errors about failing to install the stretch and buster archive signing keys though | 14:25 |
| fungi | so they must not be ending up where reprepro tries to find them (also not appearing in the output of `apt-key list`) | 14:25 |
| fungi | `gpg2 --list-keys` as root does show them though, so i think there's an assumption in the puppet module that reprepro is checking root's keychain rather than the system's secure apt trusted keys | 14:29 |
| fungi | which is not the case | 14:29 |
| *** jamesmcarthur has quit IRC | 14:30 | |
| fungi | or the problem may be that we're specifying the ids of the master keys but the archives are signed by subkeys | 14:32 |
| *** jamesmcarthur has joined #openstack-infra | 14:33 | |
| fungi | testing now to see whether that's it, and will push up a patch momentarily if so | 14:34 |
| fungi | hrm, nope | 14:34 |
| fungi | or it's that we're not providing the subkeys to it at all | 14:35 |
| *** dpawlik has quit IRC | 14:38 | |
| *** Lucas_Gray has joined #openstack-infra | 14:39 | |
| hrw | "curl https://somewhere/key | apt-key add -" or sth like that | 14:41 |
| *** pgaxatte has quit IRC | 14:43 | |
| *** michael-beaver has joined #openstack-infra | 14:47 | |
| *** pkopec has quit IRC | 14:48 | |
| *** goldyfruit has quit IRC | 14:50 | |
| *** ekultails has quit IRC | 14:53 | |
| *** iurygregory has quit IRC | 14:57 | |
| fungi | well, no. we directly puppet in the key material | 15:00 |
| fungi | we take keys from https://ftp-master.debian.org/keys.html | 15:00 |
| fungi | and it does indeed appear that reprepro doesn't use the trust ring `apt-key add` would manage, for obvious reasons (that would be keys trusted to sign repositories for packages installed on the local system, not keys signing repositories you want to mirror on your system) | 15:01 |
| fungi | anyway, i've narrowed it down to a mix of needed the release keys for some archives, the stable keys for others, and allowing subkeys on some of them | 15:01 |
| *** diga has quit IRC | 15:02 | |
| fungi | not entirely straightforward but it seems to be working correctly with a manual run now and i'll push up a review with the correct pieces | 15:02 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: add-build-sshkey: add centos/rhel-8 support https://review.opendev.org/674092 | 15:03 |
| fungi | hrw: i'll let you know once the mirror update run completes for the debian mirrors | 15:05 |
| fungi | it may take an hour or so with the outstanding delta | 15:05 |
| hrw | fungi: thank you for that work | 15:05 |
| *** kjackal has joined #openstack-infra | 15:05 | |
| clarkb | there is more than one key required? | 15:06 |
| *** armax has joined #openstack-infra | 15:07 | |
| clarkb | infra-root have you had a chance to look at https://etherpad.openstack.org/p/debugging-gitea08-OOM yet? I'm curious to hear what other think about that | 15:07 |
| *** Goneri has quit IRC | 15:07 | |
| *** ociuhandu has quit IRC | 15:09 | |
| *** ociuhandu has joined #openstack-infra | 15:09 | |
| *** liuyulong has quit IRC | 15:13 | |
| clarkb | double checking gitea08 for OOM in the last 12 hours (there isn't any so not a daily occurence) there is a message from conntrack complaining that something is trying to speak gre to it | 15:15 |
| clarkb | I wonder if there are machines in vexxhost scanning for gre tunnels they can join? | 15:15 |
| *** gfidente has quit IRC | 15:17 | |
| *** markvoelker has quit IRC | 15:18 | |
| *** markvoelker has joined #openstack-infra | 15:20 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Don't compare to literal True/False https://review.opendev.org/667697 | 15:22 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add test-bindep job https://review.opendev.org/674078 | 15:22 |
| *** mattw4 has joined #openstack-infra | 15:22 | |
| *** kjackal has quit IRC | 15:23 | |
| fungi | clarkb: yeah, i haven't completely nailed down which is being used where (would need to systematically pick apart the detached signatures for all the package reopsitories/suites to map it out), and the description at https://ftp-master.debian.org/keys.html of how they're applied is unfortunately vague, but i have at least hit on a working set of keys | 15:24 |
| clarkb | aspiers: happy to talk about my findings furhter either here or on the mailing list (though #zuul likely isn't the correct venue) | 15:25 |
| clarkb | aspiers: unfortunately I don't expect eatmydata to help much in those two particular cases I've identified beacuse the problems are recurrent redundant tasks and lack of memory | 15:25 |
| clarkb | fungi: seems odd that debian would go from one jessie key working across multiple releases to need multiple keys for a single release? | 15:26 |
| fungi | well, they didn't sign buster-backports with the jessie key | 15:26 |
| fungi | (also we're not mirroring jessie, it's the oldoldstable release now, so we ought to switch to newer keys anyway) | 15:27 |
| clarkb | I agree I'm just finding it weird that one key would work across jessie and stretch and then buster needs more than one key for one release? | 15:27 |
| fungi | each one is signed with multiple keys | 15:27 |
| clarkb | eg shouldn't there be one key that also works for buster | 15:28 |
| fungi | and i think the idea is that stretch needed to be signed by jessie's keys so that users could upgrade to stretch using the keys they already had under jessie | 15:28 |
| fungi | why the jessie key also worked for buster main suites i don't know | 15:28 |
| clarkb | right but why wouldn't the buster key work then? | 15:29 |
| clarkb | did something change or did we just get that bit wrong? | 15:29 |
| fungi | oh, it does work, it's just there are several of them | 15:29 |
| fungi | and they're not used ubiquitously | 15:29 |
| openstackgerrit | Merged zuul/nodepool master: builder: Remove recency table logging https://review.opendev.org/674124 | 15:30 |
| fungi | so at the moment i've got it working with a mix of keys from the "archive keys" and the "stable keys" at https://ftp-master.debian.org/keys.html | 15:30 |
| *** goldyfruit has joined #openstack-infra | 15:31 | |
| clarkb | the three buster keys listed there? | 15:32 |
| *** eharney has quit IRC | 15:32 | |
| fungi | though also they're multiple signatures in one file and modern gnupg can't handle that so it only ever verifies the first signature it finds (which is usually the one for the oldest key). i'm not sure if reprepro is using gnupg for its verification steps, but if so we may also still need to include the jessie keys as a result (because they're the first signatures in the files for stretch) | 15:32 |
| fungi | once the mirror is caught up i'll test taking it back out again, but i ended up adding it back initially to get things working | 15:33 |
| *** factor has joined #openstack-infra | 15:33 | |
| fungi | oh, and also added complication is that for some suites they're using a signing subkey, so it took some work to map those back to determine which keys still needed to be added... and also reprepro needs you to explicitly say that verifying against subkeys is okay by adding a + at the end of the key id | 15:35 |
| *** kjackal has joined #openstack-infra | 15:35 | |
| fungi | (the subkey ids are not themselves listed at https://ftp-master.debian.org/keys.html ) | 15:36 |
| clarkb | ah so we'll have to modify the reprepro config a bit too | 15:37 |
| mriedem | clarkb: this will get n-api-meta running with memcache in gate jobs again https://review.opendev.org/#/c/674025/ | 15:38 |
| mriedem | the tempest change was reverted b/c it broke ironic standalone | 15:38 |
| *** mattw4 has quit IRC | 15:38 | |
| clarkb | mriedem: the previous cache change broke things too right? | 15:38 |
| clarkb | do we know how/why? | 15:39 |
| mriedem | i only heard about the tempest change breaking ironic standalone since it didn't have nova | 15:39 |
| mriedem | this i mean https://review.opendev.org/#/c/672715/ | 15:39 |
| mriedem | reverted here https://review.opendev.org/#/c/673784/ | 15:39 |
| AJaeger | and it failed since it tried to acces /etc/nova which did not exist... | 15:40 |
| clarkb | I see by referencing NOVA_CONF it assumed the file/dir existed | 15:40 |
| clarkb | but now we push that into devstack when services are configured. Got it | 15:41 |
| clarkb | approved | 15:41 |
| rpioso | Good morning, ironicers! | 15:43 |
| mriedem | clarkb: thanks | 15:43 |
| AJaeger | rpioso: are you sure you're in the right channel? Or did I miss a joke? | 15:43 |
| * rpioso realized he's in the wrong channel. | 15:43 | |
| AJaeger | rpioso: good morning to you! | 15:43 |
| rpioso | AJaeger: o/ | 15:44 |
| * rpioso also needs more caffeine. No joke! | 15:44 | |
| * AJaeger sends an espresso to rpioso | 15:44 | |
| *** piotrowskim has quit IRC | 15:44 | |
| *** jpena is now known as jpena|off | 15:45 | |
| rpioso | AJaeger: Thank you so much :-) | 15:45 |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try) https://review.opendev.org/674334 | 15:46 |
| *** goldyfruit has quit IRC | 15:48 | |
| *** tdasilva has quit IRC | 15:49 | |
| *** tdasilva has joined #openstack-infra | 15:49 | |
| corvus | clarkb, fungi, mordred, AJaeger: i'd love to merge at least some of topic:zuul-swift today and start exercising that again | 15:51 |
| AJaeger | corvus: might take some time for me to review them today ... First glance: I liked the idea ;) | 15:52 |
| *** tdasilva has quit IRC | 15:54 | |
| clarkb | corvus: question on https://review.opendev.org/#/c/674143/1 | 15:55 |
| corvus | mnaser: ^ if you have a second | 15:56 |
| *** ociuhandu has quit IRC | 15:56 | |
| * mnaser looks at topic | 15:57 | |
| *** mattw4 has joined #openstack-infra | 15:57 | |
| openstackgerrit | Michael McCune proposed opendev/irc-meetings master: remove the early hours for api-sig https://review.opendev.org/674337 | 15:59 |
| *** ociuhandu has joined #openstack-infra | 16:00 | |
| clarkb | corvus: and a note about CORS at https://review.opendev.org/#/c/674136/3 | 16:02 |
| *** gyee has joined #openstack-infra | 16:06 | |
| *** lucasagomes has quit IRC | 16:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove in-tree jobs for neutron-classifier https://review.opendev.org/674340 | 16:06 |
| AJaeger | could I get some review for the change above, please? That allows to further update the in-tree config... | 16:07 |
| *** tdasilva has joined #openstack-infra | 16:07 | |
| *** udesale has quit IRC | 16:07 | |
| *** jamesmcarthur has quit IRC | 16:08 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add CORS support to upload-logs-swift https://review.opendev.org/674341 | 16:08 |
| corvus | clarkb: thanks, that was in the spec, but i forgot to add it :) | 16:08 |
| *** yamamoto has quit IRC | 16:11 | |
| openstackgerrit | Merged opendev/base-jobs master: Add swift base test job https://review.opendev.org/674143 | 16:13 |
| *** pcaruana has quit IRC | 16:14 | |
| *** yamamoto has joined #openstack-infra | 16:16 | |
| openstackgerrit | Merged opendev/irc-meetings master: remove the early hours for api-sig https://review.opendev.org/674337 | 16:16 |
| fungi | hrw: see if things are working better for debian jobs now. the mirror update has completed successfully | 16:17 |
| fungi | now to see if i can whittle down this list of keys some | 16:18 |
| openstackgerrit | Merged zuul/zuul-jobs master: Support Rackspace in upload-logs-swift https://review.opendev.org/674136 | 16:18 |
| *** yamamoto has quit IRC | 16:21 | |
| *** ociuhandu has quit IRC | 16:21 | |
| *** ociuhandu has joined #openstack-infra | 16:22 | |
| *** kjackal has quit IRC | 16:23 | |
| *** jbadiapa has quit IRC | 16:24 | |
| *** ociuhandu has quit IRC | 16:26 | |
| *** ociuhandu has joined #openstack-infra | 16:27 | |
| *** altlogbot_3 has quit IRC | 16:29 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Be consistent about spaces before and after vars https://review.opendev.org/667698 | 16:31 |
| *** irclogbot_1 has quit IRC | 16:33 | |
| *** altlogbot_3 has joined #openstack-infra | 16:36 | |
| *** adriancz has quit IRC | 16:38 | |
| *** dtantsur is now known as dtantsur|afk | 16:39 | |
| *** smrcascao has quit IRC | 16:41 | |
| *** irclogbot_2 has joined #openstack-infra | 16:41 | |
| *** mattw4 has quit IRC | 16:42 | |
| *** mattw4 has joined #openstack-infra | 16:43 | |
| *** rpittau is now known as rpittau|afk | 16:46 | |
| openstackgerrit | Merged opendev/base-jobs master: Upload to a swift at random https://review.opendev.org/674144 | 16:47 |
| *** mattw4 has quit IRC | 16:50 | |
| Shrews | clarkb: it seems openstacksdk on nb03 is at 0.27.0 but the other builders have upgraded to 0.32.0. I'm not sure what's different about that server | 16:52 |
| Shrews | i wonder if someone manually downgraded it at some point so that it remains at the lowest version satisfied by requirements.txt | 16:52 |
| clarkb | Shrews: I think pip will only update sdk if the requirement for it requires a change | 16:52 |
| clarkb | so if 0.27.0 is ok by the requirement it will stay put | 16:53 |
| Shrews | clarkb: yeah, but it should be the same as the other builders | 16:53 |
| Shrews | i don't understand how it would diverge | 16:53 |
| clarkb | I'm guessing that was a server we tested an sdk downgrade on to fix a problem | 16:53 |
| clarkb | then sdk updated to fix the problem and we excluded the broken version the other nodes had installed | 16:54 |
| Shrews | hrm. i suppose i should manually upgrade it then | 16:54 |
| clarkb | the exclusion meant the other servers upgraded but 0.27.0 was not excluded so remained fixed | 16:54 |
| clarkb | I seem to recall that happening with a bug in image uploads maybe? | 16:55 |
| clarkb | (the downgrade I mean) | 16:55 |
| clarkb | but nl03 doesn't upload images | 16:55 |
| *** ociuhandu has quit IRC | 16:57 | |
| Shrews | oh, hrm. a manual upgrade failed | 16:57 |
| *** ociuhandu has joined #openstack-infra | 16:57 | |
| *** tesseract has quit IRC | 16:58 | |
| Shrews | kubernetes 7.0.0 has requirement setuptools>=21.0.0, but you'll have setuptools 20.7.0 which is incompatible. | 16:58 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add CORS support to upload-logs-swift https://review.opendev.org/674341 | 16:59 |
| *** Goneri has joined #openstack-infra | 16:59 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Make all lines less than 160 characters long https://review.opendev.org/667696 | 17:00 |
| mordred | infra-root: I'm afk for the nxt couple of hours - will ping when I'm back on | 17:00 |
| clarkb | Shrews: we should be ablt to pip install -U setuptools to fix that | 17:01 |
| *** derekh has quit IRC | 17:01 | |
| clarkb | I do not know why pip wouldn't update setuptools given that requirement | 17:01 |
| clarkb | Shrews: also remember that it is pip3 that matters on those hosts | 17:02 |
| Shrews | yep, i'm using pip3 | 17:02 |
| *** ociuhandu has quit IRC | 17:02 | |
| clarkb | (that is something I frequently forget then remember when pip freeze doesn't show what I expect) | 17:03 |
| Shrews | but i'm suddenly having major connectivity issues | 17:03 |
| *** igordc has joined #openstack-infra | 17:03 | |
| openstackgerrit | Merged openstack/project-config master: Make gerritbot notify #openstack-neutron IRC channel of stable reviews https://review.opendev.org/674273 | 17:03 |
| *** rosmaita has left #openstack-infra | 17:04 | |
| *** ricolin has quit IRC | 17:07 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 17:09 |
| *** kjackal has joined #openstack-infra | 17:10 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 17:12 |
| Shrews | wow, nb03 is so different. now have issues uninstalling pyyaml, different versions of pip3, ... | 17:13 |
| clarkb | oh this is nb03 | 17:14 |
| clarkb | sorry I read it as nl03 initially | 17:14 |
| clarkb | that is our arm64 builder so ya different | 17:15 |
| AJaeger | config-core, could you review https://review.opendev.org/674340 to remove some jobs that are in-tree now, please? | 17:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: DNM: test swift logs https://review.opendev.org/674358 | 17:15 |
| clarkb | Shrews: I want to say it is still using puppet3 too beacuse puppet4 doesn't have arm64 packages | 17:15 |
| fungi | Shrews: clarkb: `pip install -U ...` with recent versions of pip uses an upgrade strategy of "only-if-needed" by default meaning it won't upgrade dependencies unless the currently-installed version is less than the required version range allows. you need to add --upgrade-strategy=eager to get it to always upgrade to the latest versions of all dependencies if that's what you want (but be forewarned | 17:15 |
| fungi | that gets complicated if you're mixing distro packages and packages from pypi) | 17:16 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: test swift logs https://review.opendev.org/674359 | 17:16 |
| fungi | so in a virtualenv/venv, eager is fine. if you're installing packages system-wide then pip will try to replace any deps which come from distro packages with newer versions from pypi | 17:17 |
| Shrews | fungi: i think i'm just going to leave it on openstacksdk 0.27.0 for now | 17:17 |
| fungi | which is annoying if you're trying to maintain some deps via distro package management | 17:17 |
| clarkb | Shrews: what version of sdk do we want? I'd like to take a look just to see what is going on | 17:18 |
| fungi | Shrews: one option if we know we always want latest openstacksdk is to just install it explicitly rather than letting it get installed as a dependency | 17:18 |
| Shrews | clarkb: latest. 0.32.0 | 17:18 |
| Shrews | clarkb: fungi: any objections if i restart nb01 and nb02 now? | 17:18 |
| fungi | no objections from me | 17:19 |
| *** panda has quit IRC | 17:19 | |
| Shrews | i'm not so concerned about nb03 for the moment | 17:19 |
| clarkb | Shrews: the yaml problem is the distutils vs pip fight | 17:19 |
| clarkb | the solution there is to uninstall pyyaml via the system deps if possible | 17:19 |
| *** dpawlik has joined #openstack-infra | 17:19 | |
| *** panda has joined #openstack-infra | 17:20 | |
| Shrews | well, i'll restart later. they both just started a dib build | 17:20 |
| fungi | the reason we install pyyaml via distro packages (or did anyway) was lack of a suitable wheel for it, which resulted in needing a full c build toolchain and headers for libyaml and such | 17:21 |
| *** openstackgerrit has quit IRC | 17:22 | |
| fungi | yeah, will still be a problem: https://pypi.org/project/PyYAML/#files | 17:22 |
| clarkb | fungi: I can't find evidence that we installed pyyaml from distro packages ourselves. Looks like cloud init depends on it | 17:22 |
| clarkb | fungi: and I think those are optional for installing pyyaml from source | 17:23 |
| fungi | also entirely probable if those images include cloud-init, yes | 17:23 |
| clarkb | if you don't have them you don't get bindings against libyaml | 17:23 |
| *** gyee has quit IRC | 17:24 | |
| clarkb | I know we attempt to uninstall cloud-init | 17:24 |
| clarkb | trying to figure out if that is happening | 17:24 |
| fungi | well, at least in the past it insisted on linking libyaml, but it detected at runtime if libyaml was available and wouldn't use the built c extensions if it wasn't | 17:24 |
| *** mattw4 has joined #openstack-infra | 17:24 | |
| fungi | or so i thought | 17:25 |
| clarkb | we remove cloud-init in launch node but don't have that as part of our 15 minute cron enforcement | 17:25 |
| fungi | did we add the cloud-init removal more recently than those systems were launched? | 17:26 |
| fungi | i guess that would explain its presence | 17:26 |
| clarkb | ya | 17:26 |
| *** ociuhandu has joined #openstack-infra | 17:26 | |
| clarkb | I think what we can do is uninstall cloud-init and python3-yaml, reboot to make sure host comes up with its networking properly, then reinstall sdk at the version you want | 17:27 |
| fungi | i concur | 17:27 |
| clarkb | Shrews: ^ that seem reasonable to you? | 17:27 |
| clarkb | if so I can do that | 17:28 |
| Shrews | sure | 17:28 |
| *** priteau has quit IRC | 17:31 | |
| clarkb | fungi: fwiw pyyaml intsalled fine with libyaml on the host but no -dev package | 17:31 |
| fungi | good! | 17:32 |
| fungi | and no glibc-dev or gcc? | 17:32 |
| clarkb | server is rebooting now | 17:32 |
| clarkb | fungi: we install gcc on all our machines iirc | 17:33 |
| clarkb | gcc and libc-dev-bin are both installed | 17:33 |
| fungi | okay, so not a huge deal then that it still has to build the sdist | 17:34 |
| *** dpawlik has quit IRC | 17:34 | |
| clarkb | `python setup.py --with-libyaml install` is how you build with libyaml bindings | 17:35 |
| clarkb | so ya I think it may just be doing pure python as is | 17:36 |
| clarkb | which for our purposes should be fine | 17:36 |
| *** gyee has joined #openstack-infra | 17:36 | |
| clarkb | you also have to explicitly import the C bound loader | 17:36 |
| fungi | oh, perfect. that used to not be an option | 17:36 |
| clarkb | Shrews: I think nb03 is all happy now | 17:38 |
| Shrews | clarkb: awesome | 17:38 |
| Shrews | i'll restart the other builders when they're less busy | 17:40 |
| Shrews | (the process, not the vm) | 17:40 |
| clarkb | sounds good | 17:40 |
| *** eharney has joined #openstack-infra | 17:41 | |
| *** dpawlik has joined #openstack-infra | 17:41 | |
| *** rfolco|ruck has quit IRC | 17:42 | |
| *** jcoufal_ has quit IRC | 17:43 | |
| *** dpawlik has quit IRC | 17:45 | |
| clarkb | cloudnull: out of curiousity were you able to further track down the docker hub 401 behavior (and why it might be cloud/region specific)? | 17:46 |
| *** Lucas_Gray has quit IRC | 17:47 | |
| *** openstackgerrit has joined #openstack-infra | 17:47 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: render console in js https://review.opendev.org/674368 | 17:47 |
| *** goldyfruit has joined #openstack-infra | 17:47 | |
| *** ociuhandu has quit IRC | 17:49 | |
| *** ociuhandu has joined #openstack-infra | 17:50 | |
| *** ociuhandu has quit IRC | 17:50 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: render console in js https://review.opendev.org/674368 | 17:50 |
| *** ralonsoh has quit IRC | 17:51 | |
| *** ramishra has quit IRC | 17:52 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Fix base-test-swift post playbook reference https://review.opendev.org/674372 | 17:52 |
| clarkb | mriedem: thinking about the low e-r classification rate, any ideas on how we might get more involvement with that? | 17:52 |
| corvus | clarkb, fungi: can we speedy merge this simple fix: https://review.opendev.org/674372 | 17:52 |
| clarkb | Maybe we can convince the various bug deputies to make that a part of their bug management process? | 17:53 |
| clarkb | corvus: done | 17:53 |
| fungi | looks fine to me too | 17:54 |
| clarkb | maybe only neutron is doing bug deputy as a formal process | 17:54 |
| *** georgk has quit IRC | 17:55 | |
| *** fdegir has quit IRC | 17:55 | |
| *** georgk has joined #openstack-infra | 17:56 | |
| *** fdegir has joined #openstack-infra | 17:56 | |
| clarkb | https://logs.opendev.org/68/672568/6/check/networking-midonet-tempest-aio-ml2-centos-7/8ab0eb2/logs/devstacklog.txt#_2019-08-02_15_30_47_679 | 17:56 |
| clarkb | slaweq: ^ do you know who might be interested in that job? | 17:57 |
| clarkb | johnsom: rm_work any idea why octavia builds happily resolve the fortnebula mirror at https://logs.opendev.org/37/673337/5/check/octavia-v2-dsvm-py2-scenario-centos-7/4e3a8b2/job-output.txt#_2019-08-02_00_41_51_583215 but then fail to do so at https://logs.opendev.org/37/673337/5/check/octavia-v2-dsvm-py2-scenario-centos-7/4e3a8b2/job-output.txt#_2019-08-02_01_03_22_701418 | 17:59 |
| johnsom | looking | 18:00 |
| *** jcoufal has joined #openstack-infra | 18:00 | |
| clarkb | slaweq: remote: https://review.opendev.org/674397 Stop running the centos-7 integration jobs. I pushed that to networking-midonet | 18:02 |
| openstackgerrit | Merged opendev/base-jobs master: Fix base-test-swift post playbook reference https://review.opendev.org/674372 | 18:03 |
| johnsom | clarkb Are the instances not using unbound anymore? https://logs.opendev.org/37/673337/5/check/octavia-v2-dsvm-py2-scenario-centos-7/4e3a8b2/controller/logs/unbound_log.txt.gz | 18:04 |
| clarkb | johnsom: they should be but we don't enable the logging by default for unbound iirc | 18:05 |
| clarkb | does devstack capture ps output still? | 18:05 |
| clarkb | should be able to check that | 18:05 |
| johnsom | I don't think so | 18:06 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 18:06 |
| johnsom | The "listener53" file shows an unbound process | 18:06 |
| clarkb | unbound by default doesn't log very verbosely | 18:06 |
| *** diablo_rojo has joined #openstack-infra | 18:06 | |
| *** diablo_rojo has quit IRC | 18:07 | |
| *** diablo_rojo has joined #openstack-infra | 18:07 | |
| clarkb | donnyd: looks like you set | maxTotalInstances | 0 | on fn cloud? | 18:08 |
| johnsom | True. I run it at home and bump the log level | 18:08 |
| clarkb | johnsom: ^ we may have to wait for that to change to debug it further as no isntances can be booted there currently | 18:08 |
| mriedem | clarkb: last time i looked at uncategorized it seemed like a lot of unit test jobs | 18:08 |
| fungi | clarkb: does 0 not mean "unlimited" i guess? | 18:09 |
| mriedem | yeah neutron functional | 18:09 |
| clarkb | fungi: -1 is unlimited | 18:09 |
| fungi | ahh | 18:09 |
| clarkb | fungi: 0 is zero | 18:09 |
| clarkb | johnsom: fwiw https://logs.opendev.org/51/662351/22/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/a76bbd1/logs/undercloud/var/log/extra/logstash.txt#_2019-08-02_15_55_57 is doing similar on a different cloud that doesn't ipv6 | 18:10 |
| clarkb | johnsom: so my initial suspicions that many it was related to ipv6 only cloud configs might not hold up | 18:10 |
| donnyd | I'm fixing cinder | 18:10 |
| *** goldyfruit has quit IRC | 18:10 | |
| clarkb | donnyd: ah | 18:10 |
| logan- | hello folks, if it is alright I'd like to re-enable the nodepool host aggregate on limestone so we can start scheduling there again. I have time over the next few days to monitor for job failures and disable/investigate further if things act up. sound alright? | 18:10 |
| clarkb | logan-: I'm ok with it | 18:11 |
| donnyd | It got very angry last night | 18:11 |
| *** e0ne has quit IRC | 18:11 | |
| johnsom | clarkb At that point we are not doing anything special, so I'm not sure it is unique to us, other than maybe installing some bindep later in the devstack than others. | 18:11 |
| johnsom | clarkb This seems a bit odd: https://logs.opendev.org/37/673337/5/check/octavia-v2-dsvm-py2-scenario-centos-7/4e3a8b2/controller/logs/syslog.txt.gz#_Aug_02_01_03_16 | 18:11 |
| clarkb | johnsom: I think iptables knows you didn't send out a request and therefore drops responses? | 18:13 |
| clarkb | that could be another instance on the same network asking for a lease | 18:13 |
| clarkb | I don't think that is a problem aprticularly with iptables doing the right thing there | 18:13 |
| johnsom | Ah, IN is eth0, so yeah, that is the right answer. | 18:14 |
| *** jtomasek has quit IRC | 18:15 | |
| clarkb | logan-: looks like max-servers is set to 50 currently so you will be controlling that via quotas? | 18:17 |
| clarkb | logan-: let me know if I can help | 18:17 |
| clarkb | mriedem: ya it is looking like a lot of functional and unittest failures | 18:17 |
| logan- | clarkb: yep I just don't have any hosts enabled for nodepools flavors, so im about to add the hosts back to the aggregate and it will start scheduling again | 18:17 |
| clarkb | in theory those unittests should never fail in the gate right? | 18:17 |
| johnsom | clarkb Yeah, I don't see why that would be happening. I think we would need to crank up the unbound logging to see why it is not able to resolve that hostname for curl. | 18:18 |
| logan- | clarkb: done now, we should see jobs go active there | 18:18 |
| mriedem | clarkb: unit tests can have races, | 18:19 |
| clarkb | mriedem: looks like some of the failures for unitests at least were caught by the bug in fetching subunit files role | 18:19 |
| mriedem | i opened two bugs, one for the sdk and one to nova for something i'm seeing in one of the nova functional failures - but likely only affects logstash indexing and subunit parsing | 18:19 |
| johnsom | clarkb I see they are using DNSSEC, I wonder if the unbound key file is out of date or the clocks are skewed | 18:19 |
| clarkb | johnsom: does unbound enforce dnssec in our config? /me checks | 18:20 |
| johnsom | they == us I guess | 18:20 |
| *** goldyfruit has joined #openstack-infra | 18:20 | |
| mriedem | clarkb: is that what this means? | 18:22 |
| mriedem | IOError: [Errno 13] Permission denied: '/opt/stack/logs/devstacklog.txt | 18:22 |
| clarkb | mriedem: the subunit thing is https://logs.opendev.org/87/673987/1/gate/cross-glance-py27/da79068/job-output.txt.gz#_2019-08-01_17_26_49_762617 | 18:23 |
| corvus | https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_58/674358/1/check/zuul-jobs-tox-linters/75b39e6/ | 18:23 |
| mriedem | idk what is causing that though | 18:23 |
| corvus | https://287243e9f3400bfe63ce-8b938dd2076b97d235f21ad4df33ebf0.ssl.cf1.rackcdn.com/674358/1/check/tox-py27/5346bab/ | 18:23 |
| mriedem | i only get 2 hits on this | 18:24 |
| mriedem | message:"cannot create" AND message:".subunit: Directory nonexistent" AND tags:"console" | 18:24 |
| mriedem | in 17 days | 18:24 |
| mriedem | *7 | 18:24 |
| clarkb | johnsom: looks like yes our unbound config sets the trust anchor file | 18:24 |
| clarkb | corvus: both seem to work for me and both are served via https | 18:25 |
| *** diablo_rojo has quit IRC | 18:25 | |
| fungi | clarkb: do we get the trust anchor file from somewhere else or are we embedding it directly and have neglected to refresh our copy? | 18:25 |
| *** diablo_rojo has joined #openstack-infra | 18:26 | |
| mriedem | this seems legit in neutron functional within the last 24 hours | 18:26 |
| mriedem | message:"IOError: [Errno 13] Permission denied: '/opt/stack/logs/devstacklog.txt" AND tags:"console" | 18:26 |
| mriedem | slaweq: ^ | 18:26 |
| *** goldyfruit has quit IRC | 18:26 | |
| clarkb | fungi: I think it may come from the distro package install. It lives at /var/lib/unbound/root.key on centos | 18:26 |
| clarkb | fungi: pretty sure we don't touch that | 18:26 |
| corvus | clarkb: there seem to be cors problems with both of those | 18:27 |
| fungi | okay, so yeah it's up to centos to keep that updated, as long as our images are fresh | 18:27 |
| clarkb | corvus: I was just checking that, ya I don't see the cors headers in the responses | 18:27 |
| clarkb | corvus: maybe we have to set it for every file upload and not just at the container level? | 18:28 |
| corvus | hrm, i thought the docs said container; i'll dig into it after lunch | 18:28 |
| *** ykarel|away has joined #openstack-infra | 18:29 | |
| mriedem | oooo i bet this is what breaks neutron functional https://github.com/openstack/devstack/commit/352d58a7afd9e2261e639af78e4fb4c99d8f9f81 | 18:29 |
| fungi | mriedem: wow, that's over 4 years old | 18:31 |
| mriedem | "Fix setup-devstack-log-dir to create the logs directory with correct permissions in the first place." | 18:31 |
| mriedem | https://bugs.launchpad.net/devstack/+bug/1838811 | 18:33 |
| openstack | Launchpad bug 1838811 in devstack "/opt/stack/devstack/tools/outfilter.py failing in neutron functional jobs since 8/2" [Undecided,New] | 18:33 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 18:35 |
| donnyd | clarkb: it would seem that i need to turn down the max_concurrent_builds from the defaults... just too much for my one little cinder-volume server | 18:40 |
| donnyd | and it seems to work, but now i am just trying to clean up the damage | 18:40 |
| clarkb | donnyd: is this related to boot from volume? | 18:40 |
| donnyd | before i bring it back online | 18:40 |
| donnyd | yea | 18:40 |
| donnyd | i have about 50 volumes stuck in detaching | 18:41 |
| donnyd | the other 300 removed just fine | 18:41 |
| *** guimaluf has joined #openstack-infra | 18:41 | |
| fungi | clarkb: i think i've confirmed that at least the reprepro version on xenial is unable to check against any other than the first signature in each Release.gpg file, so for the stretch repository we still need to use the jessie archive signing key (the stretch Release.gpg is signed first by that and secondly by the stretch stable release key). hopefully this improves on bionic when we get the mirroring | 18:49 |
| fungi | moved | 18:49 |
| clarkb | johnsom: a bit more data from logstash, ubuntu xenial, bionic and centos 7 all seem to have that problem and it affects all clouds | 18:50 |
| clarkb | johnsom: that makes me doubt the root key problem since we get that from the distros and it would be odd for them to all fail to update. However it could still be dnssec just broken a different way | 18:50 |
| clarkb | fungi: ok so we keep the old release keys and add the new ones basically | 18:51 |
| clarkb | johnsom: oh wait there are a lot of hits for could not resolve host fake | 18:51 |
| * clarkb cleans up and rechecks those assumption | 18:51 | |
| fungi | i was able to get it to work with a minimum of "VerifyRelease: 7638D0442B90D010|E0B11894F66AEC98+" (jessie archive signing key, and stretch archive signing key with subkey signatures allowed for the latter) | 18:51 |
| fungi | but yeah, we should likely include some other keys in there in preparation for when the jessie keys are retired | 18:52 |
| clarkb | johnsom: ya if I exclude the could not resolve host fake messages then I'm left with three helm tests that failed toresolve a name served by k8s and the rest are all centos7 | 18:53 |
| clarkb | johnsom: which puts centos 7 root key back into potential suspicion | 18:53 |
| *** ykarel|away has quit IRC | 18:55 | |
| *** rfolco has joined #openstack-infra | 19:00 | |
| *** rfolco is now known as rfolco|ruck | 19:01 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Re-add the Debian 8/jessie key to reprepro https://review.opendev.org/674406 | 19:05 |
| fungi | hrw: clarkb: ^ | 19:05 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 19:09 |
| clarkb | fungi: I may have tracked down part of the midonet issues. Both http://builds.midonet.org/misc/dists/stable/main/binary-all/Packages and ubuntu bionic universe provide libreswan of a version that works for midolman from http://builds.midonet.org/devstack/dists/unstable/main/binary-all/Packages is that sufficiently enough confusing to apt that it won't decide whcih version to install for you? | 19:11 |
| clarkb | https://logs.opendev.org/97/674397/1/check/networking-midonet-tempest-multinode-ml2-full/2758988/logs/devstacklog.txt.gz#_2019-08-02_18_23_59_061 is an example error | 19:12 |
| clarkb | now I guess I should look at why the fedora mirror hasn't updated in a month | 19:12 |
| *** portdirect has quit IRC | 19:13 | |
| fungi | that's a surprising behavior, if correct | 19:13 |
| *** portdirect has joined #openstack-infra | 19:13 | |
| *** e0ne has joined #openstack-infra | 19:13 | |
| clarkb | corvus: fungi ianw is http://paste.openstack.org/show/755346/ the sort of thing that happens if a vos release times out while we hold the lock? and if so how do we go about resolving that? | 19:14 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for devstack log file permissions bug 1838811 https://review.opendev.org/674408 | 19:14 |
| openstack | bug 1838811 in devstack "/opt/stack/devstack/tools/outfilter.py failing in neutron functional jobs since 8/2" [Undecided,New] https://launchpad.net/bugs/1838811 | 19:14 |
| *** kjackal has quit IRC | 19:14 | |
| fungi | clarkb: i think the problem is deeper than what you can see in "midolman : Depends: libreswan (>= 3.14-1) but it is not going to be installed" (there is likely some other conflict which is causing "libreswan (>= 3.14-1)" to be unresolveable). i'll see if i can suss it out | 19:15 |
| corvus | clarkb: yes. short version: make sure no transactions are still running, then manually remove the lock. | 19:15 |
| clarkb | fungi: I'm not sure it is super important their centos jobs don't work because the https cert for that repo expired | 19:16 |
| corvus | clarkb: i have to grab lunch, can support in more detail when i get back. | 19:16 |
| clarkb | corvus: ok I'll start reading through manpages to do that then | 19:16 |
| clarkb | corvus: k | 19:16 |
| clarkb | I'm thinking we want to disable the cron for that mirror. I'll hold the lock on mirror-update.opendev.org as a start | 19:17 |
| fungi | clarkb: in the past i have done something like `k5start -t -f /etc/afsadmin.keytab service/afsadmin -- vos unlock mirror.fedora` as root on mirror-update.o.o | 19:18 |
| clarkb | the lockfile is held on mirror-update server so we shouldn't try and write to it. I'm assuming that we'll want to do a manual reelase as updates may have been large | 19:18 |
| clarkb | (so won't release that lockfile that crontab uses until everything is super happy | 19:18 |
| clarkb | fungi: looks like vos status should tell us if there are transactions or not | 19:19 |
| fungi | yeah, i would err on the side of assuming the cronjob will not complete the refresh in the allowed timeout | 19:19 |
| clarkb | from vos listvldb I see that the volume for the fedora mirror is served by afs01.dfw and afs02.dfw | 19:22 |
| clarkb | vos stats against those two volume servers shows no transactions currently running | 19:23 |
| *** tdasilva has quit IRC | 19:23 | |
| clarkb | I assume the next step is to run vos unlock mirror.fedora | 19:23 |
| clarkb | then manually run the mirror update without the vos release, then run a vos release on the volume server with a -localauth | 19:23 |
| clarkb | I'll wait for corvus to finish lunchingand confirm | 19:23 |
| clarkb | from vos unlock manpage: "Do not user this command under normal circumstances." | 19:24 |
| fungi | yeah, i take that to mean "without confirming it's a stale lock" | 19:27 |
| clarkb | ya I just like the warning and the typo | 19:27 |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for devstack log file permissions bug 1838811 https://review.opendev.org/674408 | 19:29 |
| openstack | bug 1838811 in neutron "/opt/stack/devstack/tools/outfilter.py failing in neutron functional jobs since 8/2" [Undecided,Confirmed] https://launchpad.net/bugs/1838811 | 19:29 |
| corvus | clarkb: that sounds right; i think you're clear to proceed | 19:31 |
| clarkb | corvus: great thanks | 19:31 |
| fungi | so i suspect that something else is being installed in the midonet job which depends on an older version of libreswan | 19:31 |
| *** tosky has quit IRC | 19:32 | |
| clarkb | mirror.fedora has been unlocked. proceeding to run stuff on mirror-update to sync the mirror without doing a vos release. Then will run vos release -localauth in screen on afs01.dfw.openstack.org | 19:32 |
| *** dciabrin_ has joined #openstack-infra | 19:32 | |
| clarkb | wow that is a fairly complicated script. I guess I'll make a copy of it and remove the vos release | 19:33 |
| fungi | clarkb: i see that midonet job is also configured to get packages from http://archive.ubuntu.com/ | 19:34 |
| fungi | 2019-08-02 18:23:47.542 | Hit:13 http://archive.ubuntu.com/ubuntu bionic InRelease | 19:34 |
| *** dciabrin has quit IRC | 19:34 | |
| fungi | clarkb: oh! also the libreswan package on builds.midonet.org has an epoch in its version | 19:36 |
| *** mgoddard has quit IRC | 19:37 | |
| fungi | Version: 1:3.14-1 | 19:37 |
| *** dpawlik has joined #openstack-infra | 19:37 | |
| *** rtjure has quit IRC | 19:38 | |
| *** mgoddard has joined #openstack-infra | 19:38 | |
| fungi | so i think what's happening is that version is being preferred (because it's higher than bionic's 3.23-4) but one or more of its dependencies can't be satisfied | 19:38 |
| fungi | probably this worked on xenial and broke when we switched the default nodeset to bionic | 19:38 |
| clarkb | it is higher because the epoch says I win? | 19:38 |
| fungi | yes | 19:38 |
| clarkb | fungi: and ya I expect this broke when we switched to bionic | 19:39 |
| fungi | 1:3.14-1 > 3.23-4 | 19:39 |
| *** irclogbot_2 has quit IRC | 19:39 | |
| fungi | also the log complains the vpp package isn't going to be installed | 19:40 |
| clarkb | ya that one doesn't seem to be in ubuntu package mirrors but maybe it too has xenial deps? | 19:40 |
| *** irclogbot_1 has joined #openstack-infra | 19:43 | |
| clarkb | fungi: I'm not sure how important it is to debug I think it has been broken for a long time and I've pushed up changes to stop running those tests | 19:44 |
| clarkb | fungi: I had just noticed them popping up in e-r data and was making sure we weren't at fault and I'm afirly certain we are not | 19:44 |
| fungi | yeah, i think that's fine. looks like abandoned bitrot to me | 19:44 |
| *** goldyfruit has joined #openstack-infra | 19:50 | |
| clarkb | my fedora rsync script hasn't output much after Everything/x86_64/os/Packages/n/ many Packages/o/ is huge | 19:52 |
| *** kjackal has joined #openstack-infra | 19:53 | |
| clarkb | hrm strace says it is selecting on a fd, I assume that means it is waiting for responses from the upstream | 19:53 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 19:54 |
| clarkb | well there is a timeout. I will let it run that long and restart it if it gets killed | 19:54 |
| *** slaweq has quit IRC | 19:55 | |
| donnyd | clarkb: FN is back in op | 20:00 |
| *** whoami-rajat has quit IRC | 20:04 | |
| *** e0ne has quit IRC | 20:11 | |
| *** rh-jelabarre has quit IRC | 20:11 | |
| corvus | clarkb: i have cors working in rax cdn now (each object needs the allow header set on it). moving on to ovh next. | 20:11 |
| *** diablo_rojo has quit IRC | 20:12 | |
| *** Lucas_Gray has joined #openstack-infra | 20:12 | |
| fungi | gonna go grab early dinner, back shortly | 20:14 |
| clarkb | the fedora sync did end up timing out and then getting killed. I have restarted it | 20:17 |
| clarkb | I think the uh.edu mirror may not be online? | 20:19 |
| *** dpawlik has quit IRC | 20:19 | |
| clarkb | I can't hit it over http at least /me double checks they serve via http | 20:19 |
| corvus | timburke, mordred: i'm having trouble with cors in ovh; i've set "X-Container-Meta-Access-Control-Allow-Origin: *" on the container and verified that is returned on a HEAD. because this script also is designed to work with rax, i'm also sending "Access-Control-Allow-Origin: *" when uploading each object. but the objects don't have the allow-origin header when i fetch them. do you happen to know | 20:25 |
| corvus | of any reason that might be? | 20:25 |
| *** dciabrin_ has quit IRC | 20:26 | |
| clarkb | corvus: https://stackoverflow.com/questions/40182410/enable-permanently-cors-in-ovh-object-storage-openstack-swift | 20:27 |
| clarkb | corvus: I doubt you are using the web dashboard but maybe that is what is happening? | 20:27 |
| corvus | clarkb: no web dashboard use; newly created container via api only | 20:27 |
| *** kjackal has quit IRC | 20:28 | |
| corvus | mnaser: ^ same question re vexxhost | 20:28 |
| clarkb | re math.uh.edu fedora mirror their mirror 1 does seem to be afk. their mirror 2 responds to http however we don't use the path that they advertise at https://admin.fedoraproject.org/mirrormanager/mirrors/Fedora/30/x86_64 for mirror 1. That means I'm not quite sure how to substitute in mirror 2? | 20:29 |
| timburke | corvus, i wouldn't expect to be able to set CORS per-object; object-server will only store what's in allowed_headers: https://github.com/openstack/swift/blob/2.22.0/etc/object-server.conf-sample#L135 | 20:29 |
| clarkb | can I browse things via rsync:// | 20:29 |
| timburke | i *was* expecting the per-container setting to work, though | 20:30 |
| corvus | timburke: yeah, apparently per-object is required for rax via cdn, and does work. i'm assuming other providers would just ignore that | 20:30 |
| mordred | corvus: not off the top of my head, no | 20:31 |
| *** kjackal has joined #openstack-infra | 20:32 | |
| *** rtjure has joined #openstack-infra | 20:33 | |
| *** mriedem has quit IRC | 20:33 | |
| *** dciabrin_ has joined #openstack-infra | 20:34 | |
| donnyd | So I figured out the root issue with FN the last couple days. Apparently there are limits on how fast cinder can move with an lvm/iscsi backend, and I had to cut down max_concurrent_builds to 2. Nova was requesting around 40 volumes to be built at the same time... Well for a normal shop this wouldn't be an issue, but my poor little cinder-volume nvme server was very unhappy about this workload | 20:35 |
| clarkb | donnyd: we can adjust how quickly we make requests too, though not for oustanding build orders (its a global api rate limit in nodepool) | 20:36 |
| corvus | i tried two other things: adding x-object-meta-access-control-allow-origin to the object upload, and then removing all of the allow-origin headers from the object upload. neither helps in ovh. | 20:36 |
| clarkb | ok fedora-buffet is more stuff that fedora-enchilada and I believe the contents of fedora-buffet should be the same across all mirrors so I should be able to just change that hostname in theory | 20:36 |
| donnyd | eh, I think this will fix the issue. It can still built 14 instances every 30 seconds | 20:37 |
| donnyd | and that will go to 16 next week | 20:37 |
| donnyd | so pretty close to the chunks nodepool already seems to ask for | 20:37 |
| donnyd | just need someone to pop open a can on the ci so I can be sure | 20:38 |
| donnyd | been trying to simulate over here, and i could simulate the failure and the fix does in fact see to work just fine | 20:39 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix CORS in rackspace in upload-logs-swift https://review.opendev.org/674423 | 20:39 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix CORS in rackspace in upload-logs-swift https://review.opendev.org/674423 | 20:40 |
| clarkb | corvus: does sdk automatically rewrite index_headers to have the X-Container-Meta- prefix? | 20:41 |
| clarkb | then the other thing I noticed was case, but http headers are supposed to be case insensitive. Possibly bad software not honoring that requirement though | 20:41 |
| *** markvoelker has quit IRC | 20:42 | |
| corvus | clarkb: no rewriting should be happening | 20:42 |
| corvus | clarkb: (except possibly requests capitalizing the headers) | 20:43 |
| corvus | but the things with the meta- prefix should have them, and the things without should not | 20:43 |
| clarkb | corvus: it looks like sdk does actually add that prefix | 20:44 |
| corvus | wait what? | 20:45 |
| clarkb | but it checks first to see if the header already has it before adding it | 20:45 |
| clarkb | openstack/object_store/v1/_base.py is the code that does it then the resource types set their own _custom_metadata_prefix values | 20:46 |
| corvus | clarkb: which part of the upload script are you looking at? | 20:46 |
| clarkb | corvus: I'm looking at line 501 of https://review.opendev.org/#/c/674423/2/roles/upload-logs-swift/library/zuul_swift_upload.py | 20:47 |
| corvus | clarkb: in theory that should have no effect on ovh | 20:47 |
| corvus | only the container-level settings should matter | 20:48 |
| clarkb | ya and update_container seems to bypass all of the header prefix logic (so they are passed through as stated there) | 20:49 |
| corvus | clarkb: regarding the other thing (even though it shouldn't matter for ovh) -- that's using create_object, which is the shade method; i don't think it should be going through the method you mentioned re headers? | 20:51 |
| corvus | but i don't know how to follow the call stack there through all the proxy objects, etc | 20:51 |
| *** markvoelker has joined #openstack-infra | 20:52 | |
| clarkb | ya me either | 20:53 |
| clarkb | corvus: the shade method does it too but only on the metadata dict | 20:54 |
| clarkb | not the kwargs | 20:54 |
| corvus | ok, so we expect all the headers we're sending to be the literal values in the script | 20:54 |
| clarkb | yes, unless the shade method somehow calls the object_store set_metadata method. Which it looks like it doesn't (I think it ends up calling the put method directly instead) | 20:55 |
| mordred | corvus: yes, I think so - but let me trace through it real quick | 20:56 |
| *** markvoelker has quit IRC | 20:56 | |
| clarkb | fwiw vexxhost is ceph and ovh is swift so it is curious we get the same behavior out of both of them around this | 20:57 |
| mordred | and rax is a very old version of swift | 20:57 |
| corvus | i've sent an email to romain at ovh | 20:59 |
| clarkb | I'm going to step out for a few but then when I get back I think I want to chagne the mirror we sync fedora from MIRROR=rsync://pubmirror1.math.uh.edu/fedora-buffet/fedora/linux to rsync://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux | 21:00 |
| clarkb | if anyone knows of a reason to not do that please let me know | 21:00 |
| *** yamamoto has joined #openstack-infra | 21:00 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Reduce the swift upload targets https://review.opendev.org/674424 | 21:01 |
| corvus | that change makes me sad ^ | 21:01 |
| corvus | clarkb, fungi, mordred: can you review https://review.opendev.org/674423 and https://review.opendev.org/674424 | 21:02 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Rework some bugs https://review.opendev.org/674425 | 21:02 |
| mordred | corvus: {'headers': {'X-Container-Meta-Web-Index': 'index.html', 'X-Container-Meta-Access-Control-Allow-Origin': '*'}} - those headers are passed through unaltered | 21:02 |
| mordred | :q | 21:02 |
| mordred | gah | 21:02 |
| *** Lucas_Gray has quit IRC | 21:03 | |
| mordred | corvus: OH! | 21:03 |
| mordred | corvus: wait - no | 21:03 |
| *** yamamoto has quit IRC | 21:05 | |
| mordred | corvus: I'm seeing what I think is a bug - let me verify but I think we may not need to do 674424 | 21:06 |
| corvus | i'm deleting a bunch of old containers in rax from previous attempts at storing logs | 21:09 |
| *** Lucas_Gray has joined #openstack-infra | 21:09 | |
| *** slaweq has joined #openstack-infra | 21:11 | |
| *** kjackal has quit IRC | 21:11 | |
| *** phalmos has joined #openstack-infra | 21:12 | |
| corvus | there are 17402 objects in the "images" container in dfw. similar in ord and iad. total space used is 43.36 TB | 21:12 |
| corvus | Shrews: ^ there may be some more optimization we can do with nodepool/sdk in rackspace | 21:12 |
| Shrews | corvus: what are those? images made by nodepool builder? | 21:14 |
| corvus | Shrews: yep | 21:14 |
| mordred | corvus, Shrews: they are the swift objects used in creating the images | 21:15 |
| *** dciabrin_ has quit IRC | 21:15 | |
| mordred | once the image is imported successfully, they are no longer needed and can be deleted | 21:15 |
| *** slaweq has quit IRC | 21:15 | |
| *** _erlon_ has joined #openstack-infra | 21:15 | |
| Shrews | oh, so something sdk leaves behind then | 21:16 |
| clarkb | corvus: both changes lgtm but will let mordred confirm presence of bug before approving anything | 21:18 |
| donnyd | friday afternoons are no good for load testing | 21:18 |
| mordred | corvus: the headers thing is definitely an SDK bug. I have a half-fix - half because if fixes the immediate bug but is, I'm pretty sure, incomplete | 21:19 |
| donnyd | with that I am pretty sure i have this thing dialed in, so I am going to punch out for the weekend | 21:19 |
| corvus | mordred: can you elaborate? | 21:19 |
| corvus | (i'm unsure what "headers thing" is) | 21:19 |
| mordred | oh - sorry - the lack fo the access-control headers being set on the objects even though you are setting them in your script | 21:20 |
| corvus | i was not aware that wasn't happening | 21:20 |
| corvus | i'm pretty sure rax cdn requires those to function, and it did not function before i added the header on the object upload, and then i added that header, and then it functioned. | 21:21 |
| clarkb | ok I've switched to pubmirror2 and am rerunning fedora mirro script | 21:21 |
| mordred | ok. that's even weirder | 21:21 |
| corvus | mordred: note that setting the access control header on objects is expected and required in rax, whereas other swifts should ignore that. | 21:22 |
| corvus | so depending on what you're testing against, you'll see different behavior there | 21:23 |
| clarkb | ok we successfully rsynced against pubmirror2 going to run the vos release -localauth in screen on afs01.dfw.o.o | 21:24 |
| mordred | corvus: yeah - I have no idea how anything is actually setting that header on rackspace for yhou | 21:26 |
| corvus | mordred: can you walk me through why you think there's a problem? | 21:27 |
| mordred | yes - but give me just a sec so I can fully collect my thoughts here and not walk you down a dark alley | 21:27 |
| *** EmilienM is now known as EmilienM|afk | 21:28 | |
| *** EmilienM|afk is now known as EmilienM | 21:28 | |
| corvus | ok. just to start things off, we're talking about this header https://review.opendev.org/#/c/674423/2/roles/upload-logs-swift/library/zuul_swift_upload.py@581 going through this method https://review.opendev.org/#/c/674423/2/roles/upload-logs-swift/library/zuul_swift_upload.py@599 right? | 21:31 |
| *** jcoufal_ has joined #openstack-infra | 21:31 | |
| *** jcoufal has quit IRC | 21:31 | |
| corvus | mordred: also, any chance you could +3 https://review.opendev.org/674423 and https://review.opendev.org/674424 so they can work through while we dig in? | 21:31 |
| *** rlandy has quit IRC | 21:31 | |
| mordred | corvus: done | 21:33 |
| *** aedc has joined #openstack-infra | 21:34 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Switch fedora mirroring to pubmirror2.math.uh.edu https://review.opendev.org/674428 | 21:37 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Trim fedora mirror https://review.opendev.org/674429 | 21:37 |
| clarkb | infra-root ^ changes related to getting fedora mirror working again | 21:38 |
| clarkb | the second change is a cleanup not necessary for functionality | 21:38 |
| mordred | corvus: yes. in local testing that header is not being set, and tracing through the code it makes sense that it's not being set - but you're saying you are seeing it set on things properly? | 21:38 |
| corvus | mordred: yep | 21:39 |
| clarkb | mordred: note that corvus us using teh shade methods not the proxy methods | 21:39 |
| mordred | yeah | 21:39 |
| corvus | mordred: and this is only expected to work on rax | 21:39 |
| fungi | back and catching up | 21:39 |
| mordred | OH FOR THE LOVE OF | 21:39 |
| mordred | clarkb: yes - I knew he was doing that and was debugging that - except that my test script wasn't because I'm dumb | 21:40 |
| mordred | corvus: let's shelve that issue for now :) | 21:40 |
| mordred | there IS an issue with the proxy layer, but we don't care about that for these purposes | 21:40 |
| clarkb | unless the shade layer goes through the proxy layer (which is difficult to trace) | 21:41 |
| mordred | corvus: the other issue is that you are setting the headers on the container, and that isn't working to cause the cors headers to actually get sent | 21:41 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix CORS in rackspace in upload-logs-swift https://review.opendev.org/674423 | 21:41 |
| corvus | mordred: okay, summary: script should work fine because it uses shade methods, the sdk/proxy equivalent might not work? | 21:41 |
| openstackgerrit | Merged opendev/base-jobs master: Reduce the swift upload targets https://review.opendev.org/674424 | 21:42 |
| mordred | yes. and I want to fix the sdk/proxy layer, but that's irrelevant to the current question | 21:42 |
| corvus | mordred: and no, the only current issue is that neither ovh nor vexxhost appears to honor the x-container-meta-access-control-allow-origin setting. afaict, i am setting it on the container correctly, and when i HEAD the container, i get the setting back, indicating that it was received and stored. | 21:43 |
| corvus | subsequent fetches of objects in the container, however, do not have an access-control-allow-origin header. | 21:44 |
| mordred | corvus: ok. and it seems that sdk is also doing what we're asking there | 21:45 |
| mordred | *phew* - I thought there was more broken sdk-side than there is | 21:45 |
| corvus | i have an email out to romain, and an irc ping to mnaser | 21:45 |
| mordred | I mean, don't get me wrong, there's some broken things - and also I'm not happy that setting the headers isn't working ... | 21:45 |
| corvus | romain says he'll look monday | 21:45 |
| mordred | corvus: reading swift docs, I agree that the things you are setting are the right thigns to set | 21:48 |
| *** markvoelker has joined #openstack-infra | 21:48 | |
| corvus | whew | 21:48 |
| clarkb | that stackoverflow article implies they have their own management around this stuff | 21:49 |
| clarkb | its possible the bug is entirely on their side I suppose | 21:49 |
| corvus | i wish we had more swifts | 21:49 |
| mordred | there's also: | 21:50 |
| mordred | "In addition the the values set in container metadata, some cluster-wide values may also be configured using the strict_cors_mode, cors_allow_origin and cors_expose_headers in proxy-server.conf. See proxy-server.conf-sample for more information." | 21:50 |
| mordred | so there is apparently some server-side config that *might* be in play - I have no idea whether that's true or not | 21:50 |
| corvus | we have 3, each of which is completely different, one isn't swift at all, another is a fork. so it's impossible to triangulate any behavior differences. | 21:50 |
| corvus | a swift without cors is pretty useless to us | 21:51 |
| corvus | s/us// | 21:51 |
| mordred | corvus: https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/object_gateway_guide_for_red_hat_enterprise_linux/object_gateway_swift_api | 21:51 |
| mordred | corvus: indicates that radosgw doesn't support cors | 21:51 |
| corvus | i guess mnaser is off the hook | 21:52 |
| clarkb | I can actually check that really quickly. one sec | 21:52 |
| mordred | more importantly - https://github.com/ceph/ceph/blob/master/doc/radosgw/swift.rst | 21:52 |
| mordred | master of ceph says the same thing | 21:52 |
| clarkb | openstack.org uses a bunch of resources hosted in vexxhost ceph/swift | 21:53 |
| clarkb | and sure enough I don't see headers for cors | 21:53 |
| clarkb | so how does that work at all? | 21:54 |
| clarkb | is it because none of them are scripts? | 21:54 |
| clarkb | they are all image objects | 21:54 |
| mordred | yeah - none of them are being requested by the javascript | 21:54 |
| mordred | they're just, you know, image links | 21:55 |
| clarkb | but ya sure enough no headers there | 21:55 |
| corvus | https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#What_requests_use_CORS | 21:55 |
| corvus | fonts is an interesting thing i wouldn't have guessed | 21:55 |
| *** betherly has joined #openstack-infra | 21:55 | |
| corvus | because, of course, licensing | 21:55 |
| mordred | corvus: and setting arbitrary metadata on objects to be returned in headers only works for headers tarting with x-meta-object right? | 21:55 |
| corvus | mordred: correct -- i did see that work on ... i think ovh? because i set an x-meta-object-... header. but of course, there is no such header that's useful to us | 21:56 |
| corvus | or, x-object-meta i think | 21:57 |
| clarkb | the expiration value is sent as a header too | 21:57 |
| mordred | yeah | 21:57 |
| mordred | x-object-meta | 21:57 |
| corvus | yeah, timburke linked a list of allowed object headers earlier https://github.com/openstack/swift/blob/2.22.0/etc/object-server.conf-sample#L135 | 21:57 |
| mordred | corvus, clarkb: wanna be even more annoyed? | 21:59 |
| corvus | mordred: it's not likely to make a difference at this point | 22:00 |
| *** jcoufal_ has quit IRC | 22:01 | |
| *** betherly has quit IRC | 22:01 | |
| mordred | corvus, clarkb: cors IS supported in the ceph s3 gateway | 22:01 |
| *** Lucas_Gray has quit IRC | 22:02 | |
| *** jistr has quit IRC | 22:02 | |
| * fungi groans | 22:03 | |
| corvus | mordred: wow | 22:03 |
| clarkb | are the same containers/objects able to be manipulated by both apis or are they separate (we could maybe hate ourselves and set cors via that api if so) | 22:06 |
| mordred | clarkb: I do not know | 22:07 |
| fungi | i don't know that there's any guarantee that the 23 compat api is exposed/enabled? | 22:07 |
| clarkb | " The S3 and Swift APIs share a common namespace, so you may write data with one API and retrieve it with the other." | 22:07 |
| clarkb | it is possible that would work | 22:07 |
| fungi | er, s/23/s3/ | 22:07 |
| mordred | vexxhost runs swift_s3 in their catalog | 22:07 |
| fungi | so seems possible at least | 22:07 |
| mordred | I don't know if that's the ceph s3 proxy or anything about it | 22:07 |
| *** Lucas_Gray has joined #openstack-infra | 22:08 | |
| corvus | are the public urls different? | 22:08 |
| mordred | no | 22:09 |
| *** jistr has joined #openstack-infra | 22:09 | |
| mordred | so conceivably you could try making s3 api requests to the swift endpoint, however that works | 22:09 |
| mordred | [{'endpoints': [{'region_id': 'ca-ymq-1', 'url': 'https://object-storage-ca-ymq-1.vexxhost.net', 'region': 'ca-ymq-1', 'interface': 'admin', 'id': '4f1fafb25e58421e91a2139dc0382c53'}, {'region_id': 'ca-ymq-1', 'url': 'https://object-storage-ca-ymq-1.vexxhost.net', 'region': 'ca-ymq-1', 'interface': 'internal', 'id': '5808ab12037f4ab49c37cba0ac9cd58e'}, {'region_id': 'ca-ymq-1', 'url': | 22:10 |
| mordred | 'https://object-storage-ca-ymq-1.vexxhost.net', 'region': 'ca-ymq-1', 'interface': 'public', 'id': 'a680b7674b3b430c9f5e008f72f3dfe4'}], 'type': 's3', 'id': '13f4cf29c7394a7a801d8aeacca24378', 'name': 'swift_s3'}] | 22:10 |
| mordred | ftr | 22:10 |
| corvus | zuul-jobs can/should have an upload-logs-s3 role. is opendev willing to use it? | 22:11 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| clarkb | corvus: I worry that the swift proper clouds won't support it | 22:12 |
| clarkb | (I haven't checked if they run swifts3 or similar) | 22:12 |
| clarkb | so that might get ugly from a management perspective | 22:12 |
| corvus | clarkb: right, i wouldn't suggest using it on a cloud that supported swift | 22:12 |
| corvus | but we could use upload-logs-swift on some clouds, upload-logs-s3 on others | 22:13 |
| fungi | i don't have any objection to using a publicly-spec'ed api to talk to free and open source software | 22:13 |
| clarkb | corvus: ya I think in that case it would probably be fine. It would suck to have to debug two sets of apis but its not like we don't already juggle more | 22:14 |
| fungi | even if that api happens to be primarily associated with a proprietary service, it's apparently implemented more widely | 22:14 |
| *** slaweq has quit IRC | 22:15 | |
| *** armax has quit IRC | 22:16 | |
| clarkb | fungi: mordred https://review.opendev.org/#/c/674428/ and its child would be good to review | 22:17 |
| clarkb | I probably won't release the fedora mirror lock until after that first one merges and applies | 22:17 |
| clarkb | (just to avoid wasted effort talking to down host | 22:17 |
| *** jistr has quit IRC | 22:17 | |
| *** armax has joined #openstack-infra | 22:18 | |
| *** jistr has joined #openstack-infra | 22:19 | |
| clarkb | also not sure if we want to manaully run vos release after it deletes a bunch of data | 22:19 |
| clarkb | if so then the first run after merging the child should probably be manual too | 22:19 |
| corvus | i would recommend that | 22:19 |
| mordred | corvus: I agree wtih the words clarkb and fungi said - I do not think it would be bad at this point to talk to a ceph using upload-logs-s3 | 22:20 |
| clarkb | corvus: ok I'll definitely hold the lock then | 22:20 |
| mordred | and it would provide a somewhat convenient way for us to know that zuul's upload-logs-s3 works without having to actually upload logs to the aws s3 service | 22:20 |
| *** eharney has quit IRC | 22:20 | |
| *** markvoelker has quit IRC | 22:21 | |
| corvus | mordred: yeah, assuming it really is compatible :) | 22:21 |
| corvus | swift != swift, so i'm not going to bet that rados-swift_s3 == s3 :) | 22:21 |
| mordred | corvus: indeed | 22:22 |
| corvus | SpamapS pushed this up long time ago: https://review.opendev.org/599931 | 22:22 |
| corvus | once you take out the index stuff, it's mostly just a single call to s3_sync | 22:22 |
| corvus | aside from index generation, the swift upload role does: threaded parallel upload, content-type setting, content-encoding and streaming gzip | 22:23 |
| corvus | it's conceivable that s3_sync does all of that and we can just use it | 22:23 |
| corvus | but if it doesn't, then we may want to adopt the approach in the swift upload role | 22:23 |
| mordred | yeah. although it's also conceivable that s3_sync has no capabilities to be used against a non-amazon region - so we should check that | 22:24 |
| corvus | true. that would make a pretty compelling reason not to use it | 22:24 |
| mordred | yeah. | 22:25 |
| *** goldyfruit has quit IRC | 22:25 | |
| *** pfallenop has quit IRC | 22:25 | |
| * mordred needs to go eat the dinner ... there is some sushi with his name on it | 22:25 | |
| fungi | this is also probably seeds of useful feedback to the swift maintainers | 22:27 |
| corvus | fungi: well, i don't know that we have any swift feedback yet | 22:27 |
| corvus | ovh runs swift, and isn't working. next week we'll figure out if that's an ovh problem, a swift problem, or a corvus problem. | 22:28 |
| fungi | fair, if rax is not really swift and vexxhost is definitely ceph reimplementing swift apis and ovh may be somehow intercepting calls... | 22:28 |
| clarkb | we do however have ceph feedback | 22:28 |
| corvus | yep | 22:28 |
| clarkb | "please add cors support to swift api" | 22:28 |
| fungi | heh | 22:28 |
| fungi | that is true | 22:28 |
| corvus | i'm puzzled at how it could be less work to have written the table entry that says it's not supported than to actually support it. | 22:29 |
| corvus | the implementation in swift itself is 4 lines. | 22:29 |
| clarkb | corvus: also the s3 version of it is a bunch of xml so probably far more difficult to add support for | 22:29 |
| cloudnull | clarkb I have not been able to track down why we were seeing the 401 in the ovh region, however, I did put up a review to force our tools to reauth whenever they encounter a 401 - https://review.opendev.org/#/c/674097 | 22:30 |
| *** pfallenop has joined #openstack-infra | 22:32 | |
| timburke | corvus, it's a few more than that ;-) https://review.opendev.org/#/c/528106/ gives a decent idea of the scope of the feature | 22:32 |
| timburke | i should really revive https://review.opendev.org/#/c/533028/ at some point... | 22:32 |
| clarkb | https://github.com/ceph/ceph/blob/master/src/rgw/rgw_cors_s3.cc is the ceph s3 implementation | 22:35 |
| corvus | timburke: i get that i'm being naive here, but why is all that necessary? browsers themselves implement cors evaluation, so why is simply returning the configured value in the header not sufficient? | 22:36 |
| clarkb | corvus: I was just going to ask that, I didn't realized servers were enforcing cors | 22:36 |
| timburke | a large part of it is serving OPTIONS responses iirc. been a while since i thought much about it, honestly | 22:38 |
| corvus | (the 4 lines i was thinking of were 148--154 of cors.py, which, granted, are actually 5 lines after removing wrapping) | 22:38 |
| openstackgerrit | Merged opendev/system-config master: Switch fedora mirroring to pubmirror2.math.uh.edu https://review.opendev.org/674428 | 22:43 |
| clarkb | w3c spec says that server side may elect to do the enforcement too | 22:43 |
| corvus | yeah was just reading https://www.w3.org/TR/cors | 22:43 |
| clarkb | "This extension enables server-side applications to enforce limitations (e.g. returning nothing) on the cross-origin requests that they are willing to service." | 22:43 |
| timburke | allow-origin's actually a pretty small part of what swift needs to worry about. there's allow-method (in case you wanted the browser to be issuing PUTs, DELETEs, etc.), expose-headers (so you can see metadata), there's concerns about how browsers are going to cache whatever responses you're sending... | 22:44 |
| clarkb | timburke: ya I was just surprised that you are enforcing too since typically I'ev seen that as the browser's job | 22:44 |
| clarkb | but people may make requests from outside a browser too | 22:45 |
| clarkb | and if the intent is to protect the content and not just to protect the dynamic execution environment within the browser than blocking at the server makes sense | 22:45 |
| corvus | clarkb: well... that might be able to be bypassed by simply sending an "Origin:" header. | 22:46 |
| clarkb | corvus: if you can guess which origin is allowed (I agree it isn't the bset method of protecting the content) | 22:46 |
| corvus | something that a browser can enforce is done correctly, but probably becomes ineffective outside of that environment | 22:47 |
| clarkb | it might also reduce total network bw | 22:47 |
| clarkb | which may be a major plus for swift users? | 22:47 |
| clarkb | (and maybe you can timing attack that string comparison to figure out an allowed origin) | 22:48 |
| *** roman_g has quit IRC | 22:52 | |
| openstackgerrit | Merged opendev/system-config master: Trim fedora mirror https://review.opendev.org/674429 | 22:52 |
| *** rascasoft has quit IRC | 23:00 | |
| *** rascasoft has joined #openstack-infra | 23:02 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** slaweq has quit IRC | 23:16 | |
| *** markvoelker has joined #openstack-infra | 23:20 | |
| *** _erlon_ has quit IRC | 23:25 | |
| *** betherly has joined #openstack-infra | 23:26 | |
| *** betherly has quit IRC | 23:31 | |
| *** sthussey has quit IRC | 23:43 | |
| *** roman_g has joined #openstack-infra | 23:46 | |
| *** betherly has joined #openstack-infra | 23:47 | |
| *** betherly has quit IRC | 23:51 | |
| *** markvoelker has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!