| rpioso | fungi: It's a new symptom from an system which was able to successfully clone just a couple of days ago. | 00:00 |
|---|---|---|
| rpioso | fungi: There are others who are pointing the finger you know where ;-) | 00:00 |
| fungi | yeah, network devices you're connecting through can suddenly become overwhelmed by other misbehaving hosts you share it with | 00:01 |
| *** rfolco has joined #openstack-infra | 00:01 | |
| rpioso | or the network device can be changed and introduce problems. | 00:01 |
| fungi | in my days doing internet engineering we saw no end of issues when backbone peers would dissolve or downgrade their connections with each other over contract disputes about who was going to cover transit costs, and network connectivity between different parts of the internet would just fall apart | 00:02 |
| rpioso | clarkb: Checking at home could rule out opendev.org server issues, no? If so, the light might then shine brighter on something closer to the client. | 00:03 |
| fungi | so it may not even be that local to you, just a suddenly overloaded peering point along your preferred route | 00:03 |
| clarkb | rpioso: yes, but I've basically already done that | 00:03 |
| clarkb | did clones in a loop witu no issue | 00:03 |
| rpioso | fungi: I'm referring to something much simpler, a new switch. | 00:03 |
| fungi | rpioso: that is equally likely, yep | 00:04 |
| fungi | especially if some network admin didn't properly configure the memory allocation for the bridge table or something | 00:04 |
| rpioso | clarkb: Understood, and agree. | 00:04 |
| fungi | and so teh switch is periodically reverting to flooding all frames to all ports when it gets slightly overwhelmed | 00:05 |
| fungi | or loop protection has been disabled and someone accidentally connected a couple switchports to each other | 00:05 |
| rpioso | It's supposedly a bigger, faster, and badder (oops) switch. | 00:05 |
| fungi | or forgot to configure a lag and so multiple connections are being treated as separate, causing loops | 00:06 |
| fungi | i've seen more ways a switch replacement can go wrong than i care to count | 00:06 |
| rpioso | fungi: I'll pass those possibilities along. | 00:07 |
| fungi | that's really just the tip of the iceberg | 00:07 |
| rpioso | fungi: Understood | 00:07 |
| fungi | but anyway, one thing you could do is some extended ping tests and see if you're getting any significant amount of packet loss | 00:08 |
| fungi | just a little packet loss can cause tcp retransmits to spiral out of control | 00:08 |
| fungi | and drag your throughput waaaay down | 00:08 |
| rpioso | fungi: How is the gitea server instance selected? Is it fixed from a given host? | 00:09 |
| * rpioso guesses a hash of the IP address | 00:09 | |
| fungi | yes, hash by source address | 00:09 |
| fungi | mainly to address issues where related requests for a git operation could otherwise wind up farmed out to different backens which are slightly out of sync | 00:10 |
| fungi | s/backens/backends/ | 00:10 |
| fungi | we *could* take gitea02 down in the haproxy so that your connections wind up getting mapped to a different backend server instead, as a test, though it could be mildly disruptive (not that much if we take it out of the pool gracefully) | 00:12 |
| rpioso | fungi: If it's not too disruptive and could be done quickly, I'm game. | 00:13 |
| fungi | conveniently i have the command queued up in shell history | 00:13 |
| * rpioso would like to soon call it a day and catch some Boston Bruins hockey. | 00:13 | |
| rpioso | fungi: Ready when you are. | 00:14 |
| fungi | infra-root: any objection to me temporarily disabling balance_git_https/gitea02.opendev.org in the haproxy? | 00:14 |
| clarkb | not from me | 00:14 |
| fungi | okay, so as not to eat into the bruins game, i'll go ahead ;) | 00:15 |
| rpioso | fungi: Thank you! Let's go Bruins! | 00:15 |
| fungi | rpioso: when you make a request now from that client do you see a cert from a different backend? | 00:15 |
| fungi | let it not be said i am unkind to beantown, or lack a respect for hockey | 00:16 |
| rpioso | fungi: Yes, I do see a new cert, gitea05.opendev.org | 00:17 |
| fungi | okay, let's see if you can reproduce your issue | 00:17 |
| fungi | if you can, then it's unlikely to be backend-specific | 00:17 |
| rpioso | fungi: It's already looking like it's been reproduced. On the first clone! Let's give it a few minutes to confirm. | 00:18 |
| fungi | k | 00:19 |
| rpioso | fungi: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated. | 00:22 |
| fungi | ouch | 00:22 |
| rpioso | fatal: The remote end hung up unexpectedly. | 00:22 |
| rpioso | I saw that earlier. | 00:22 |
| fungi | yeah, that's typical for a prematurely-terminated socket | 00:22 |
| rpioso | Just prior to our convo. | 00:23 |
| fungi | but does mean some pretty nasty network problems usually | 00:23 |
| rpioso | The second clone isn't proceeding normally. | 00:23 |
| rpioso | Or switch misconfig ? | 00:23 |
| fungi | well, i'm not speculating on what could cause the "network problems" but... switches are network devices ;) | 00:24 |
| rpioso | Within 24 hours, the symptom appeared. And I wasn't cloning all 24 of 'em. | 00:24 |
| rpioso | Perhaps it's coincidental. | 00:24 |
| rpioso | fungi: Please feel free to revert the cluster. | 00:24 |
| fungi | i'm going to check resource utilization graphs for gitea05 just to make sure it's not having trouble we don't know about | 00:25 |
| fungi | nominal traffic levels and system load, no appreciable memory pressure | 00:26 |
| clarkb | fungi: ya 02 was in a similar boat when I tested it yesterday | 00:26 |
| clarkb | it had work it was doing but nothing crazy | 00:26 |
| fungi | some failed tcp attempts on the open statistics | 00:26 |
| *** jamesmcarthur has joined #openstack-infra | 00:27 | |
| rpioso | fungi: The first two clones failed in the same way. Then clones 3-6 rocked normally. Now clone 7 is stuck. | 00:27 |
| fungi | rpioso: this sounds consistent with your earlier observations? | 00:28 |
| rpioso | fungi: Indeed! | 00:28 |
| fungi | i'll go ahead and reenable gitea02 in the pool | 00:28 |
| fungi | thanks for testing! | 00:28 |
| rpioso | clarkb, fungi: Terminated bandwidth hungry loop. Thanks, again, for all of your assistance. | 00:29 |
| fungi | the failed tcp opens on the graph look consistent with the other servers too, so i suspect that's just our equivalent of cosmic background radiation | 00:30 |
| fungi | more likely it's showing the liveliness probes from haproxy | 00:31 |
| fungi | i bet it's doing syn,syn/ack,poof | 00:31 |
| fungi | would explain why it's so steady and consistent across all backends | 00:31 |
| *** ijw has joined #openstack-infra | 00:32 | |
| fungi | rpioso: sorry we don't have a better answer for you, but... go bruins! | 00:32 |
| rpioso | clarkb, fungi: As always, you guys were a great help. Thank you, thank you. Let's go Bruins! | 00:37 |
| clarkb | fungi: is poof an actual tcp verb? :) | 00:43 |
| *** rfolco has quit IRC | 00:44 | |
| *** jamesmcarthur has quit IRC | 00:45 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Don't search github api for projects not in Zuul https://review.opendev.org/694390 | 00:45 |
| *** efried has quit IRC | 00:48 | |
| *** ociuhandu has joined #openstack-infra | 00:48 | |
| *** efried has joined #openstack-infra | 00:49 | |
| *** ociuhandu has quit IRC | 00:52 | |
| *** jamesmcarthur has joined #openstack-infra | 00:53 | |
| *** mriedem has quit IRC | 00:56 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Look for depends-on lines in github dependency search https://review.opendev.org/697595 | 01:00 |
| *** rlandy|afk has quit IRC | 01:03 | |
| *** jamesmcarthur has quit IRC | 01:04 | |
| *** jamesmcarthur has joined #openstack-infra | 01:05 | |
| *** ricolin has joined #openstack-infra | 01:10 | |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** rfolco has joined #openstack-infra | 01:13 | |
| *** jamesmcarthur has joined #openstack-infra | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:15 | |
| *** jamesmcarthur has joined #openstack-infra | 01:15 | |
| *** ociuhandu has joined #openstack-infra | 01:17 | |
| *** michael-beaver has quit IRC | 01:19 | |
| *** ociuhandu has quit IRC | 01:23 | |
| *** rfolco has quit IRC | 01:33 | |
| *** rfolco has joined #openstack-infra | 01:33 | |
| *** jamesmcarthur has quit IRC | 01:35 | |
| *** rfolco has quit IRC | 01:38 | |
| *** jamesmcarthur has joined #openstack-infra | 01:39 | |
| *** jamesmcarthur has quit IRC | 01:44 | |
| *** ociuhandu has joined #openstack-infra | 01:57 | |
| *** jamesmcarthur has joined #openstack-infra | 02:02 | |
| *** ociuhandu has quit IRC | 02:03 | |
| *** jamesmcarthur has quit IRC | 02:13 | |
| *** jamesmcarthur has joined #openstack-infra | 02:13 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Also build sibling container images https://review.opendev.org/697393 | 02:25 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 02:25 |
| *** ociuhandu has joined #openstack-infra | 02:26 | |
| *** gyee has quit IRC | 02:30 | |
| *** ociuhandu has quit IRC | 02:31 | |
| *** ijw has quit IRC | 02:54 | |
| *** ijw has joined #openstack-infra | 02:55 | |
| *** ijw has quit IRC | 03:15 | |
| *** surpatil has joined #openstack-infra | 03:15 | |
| *** jamesmcarthur has quit IRC | 03:18 | |
| *** surpatil is now known as surpatil||127_0_ | 03:28 | |
| *** surpatil||127_0_ is now known as surpatil | 03:29 | |
| *** surpatil is now known as surpatil|127_0_0 | 03:29 | |
| *** surpatil|127_0_0 is now known as surpatil | 03:29 | |
| *** surpatil is now known as surpatil|wrf | 03:32 | |
| *** surpatil|wrf is now known as surpatil|wfr | 03:32 | |
| *** surpatil|wfr is now known as surpatil|wfh | 03:33 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix up siblings copy https://review.opendev.org/697614 | 03:34 |
| ianw | there are more problems in that than i'd like to admit :) | 03:34 |
| ianw | statusbot is gone again | 03:40 |
| *** psachin has joined #openstack-infra | 03:40 | |
| *** openstackstatus has joined #openstack-infra | 03:41 | |
| *** ChanServ sets mode: +v openstackstatus | 03:41 | |
| *** ociuhandu has joined #openstack-infra | 03:52 | |
| *** udesale has joined #openstack-infra | 04:00 | |
| *** ykarel|away has joined #openstack-infra | 04:01 | |
| *** ociuhandu has quit IRC | 04:02 | |
| *** dave-mccowan has quit IRC | 04:04 | |
| *** ricolin has quit IRC | 04:16 | |
| *** ricolin has joined #openstack-infra | 04:18 | |
| *** ykarel|away is now known as ykarel | 04:21 | |
| *** rishabhhpe has joined #openstack-infra | 04:36 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix up siblings copy https://review.opendev.org/697614 | 04:49 |
| *** tkajinam has quit IRC | 05:02 | |
| *** tkajinam has joined #openstack-infra | 05:04 | |
| *** jamesmcarthur has joined #openstack-infra | 05:20 | |
| *** raukadah is now known as chkumar|ruck | 05:20 | |
| *** jamesmcarthur has quit IRC | 05:25 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** tkajinam_ has joined #openstack-infra | 05:35 | |
| *** ociuhandu has quit IRC | 05:36 | |
| *** tkajinam has quit IRC | 05:38 | |
| *** surpatil|wfh has quit IRC | 05:45 | |
| *** rh-jelabarre has quit IRC | 05:48 | |
| openstackgerrit | Merged zuul/zuul master: Log item warning messages at info level https://review.opendev.org/697571 | 05:54 |
| openstackgerrit | Merged zuul/zuul master: Fix potential wedge with provides/requires/dependencies https://review.opendev.org/697579 | 06:01 |
| *** rcernin has quit IRC | 06:03 | |
| *** jtomasek has joined #openstack-infra | 06:12 | |
| openstackgerrit | Merged zuul/zuul master: Pagure - add the enqueue_ref unit test https://review.opendev.org/687351 | 06:17 |
| rishabhhpe | Hi ALL , We are not able to see results published on CI-Watch screen from yesterday . Also Devstack installation is failing at very early stage of setup.. Is there anyone else facing the same issue or someone who can help us understand the root cause of issue? | 06:22 |
| rishabhhpe | In success log we are able to see /opt/stack/data/CA while in failure log we are not able to see the same directory. For reference we have pasted the logs here http://paste.openstack.org/show/787215/ | 06:22 |
| *** ociuhandu has joined #openstack-infra | 06:25 | |
| *** soniya29 has joined #openstack-infra | 06:39 | |
| *** ociuhandu has quit IRC | 06:42 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/697622 | 06:47 |
| *** ociuhandu has joined #openstack-infra | 07:05 | |
| *** diablo_rojo has quit IRC | 07:23 | |
| *** rcernin has joined #openstack-infra | 07:25 | |
| *** pgaxatte has joined #openstack-infra | 07:30 | |
| *** ociuhandu has quit IRC | 07:42 | |
| *** ykarel is now known as ykarel|lunch | 07:46 | |
| *** yamamoto has joined #openstack-infra | 07:50 | |
| *** hashar has joined #openstack-infra | 07:54 | |
| *** rishabhhpe has quit IRC | 07:59 | |
| *** sikkar_ has joined #openstack-infra | 07:59 | |
| *** dpawlik has joined #openstack-infra | 08:02 | |
| *** pcaruana has joined #openstack-infra | 08:11 | |
| *** tobias-urdin has joined #openstack-infra | 08:15 | |
| *** tkajinam_ has quit IRC | 08:16 | |
| *** slaweq_ has joined #openstack-infra | 08:16 | |
| *** slaweq_ is now known as slaweq | 08:21 | |
| *** rpittau|afk is now known as rpittau | 08:21 | |
| *** ociuhandu has joined #openstack-infra | 08:27 | |
| *** witek_ has joined #openstack-infra | 08:28 | |
| *** tesseract has joined #openstack-infra | 08:31 | |
| *** ociuhandu has quit IRC | 08:33 | |
| *** iurygregory has joined #openstack-infra | 08:38 | |
| *** ralonsoh has joined #openstack-infra | 08:55 | |
| *** ykarel|lunch is now known as ykarel | 08:57 | |
| *** hashar has quit IRC | 09:05 | |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: Documentation fixes https://review.opendev.org/697635 | 09:06 |
| *** gfidente|afk is now known as gfidente | 09:11 | |
| *** lucasagomes has joined #openstack-infra | 09:17 | |
| *** ociuhandu has joined #openstack-infra | 09:21 | |
| *** ociuhandu has quit IRC | 09:26 | |
| *** ociuhandu has joined #openstack-infra | 09:27 | |
| *** aedc has joined #openstack-infra | 09:28 | |
| *** dchen has quit IRC | 09:31 | |
| *** derekh has joined #openstack-infra | 09:32 | |
| *** hashar has joined #openstack-infra | 09:33 | |
| *** ociuhandu has quit IRC | 09:56 | |
| *** yamamoto has quit IRC | 09:56 | |
| *** ociuhandu has joined #openstack-infra | 09:56 | |
| *** aarents has quit IRC | 10:01 | |
| *** ociuhandu has quit IRC | 10:01 | |
| *** sikkar_ has quit IRC | 10:05 | |
| *** rishabhhpe has joined #openstack-infra | 10:06 | |
| *** derekh has quit IRC | 10:12 | |
| *** rcernin has quit IRC | 10:12 | |
| *** rcernin has joined #openstack-infra | 10:15 | |
| *** derekh has joined #openstack-infra | 10:18 | |
| *** pkopec has joined #openstack-infra | 10:26 | |
| *** dtantsur|afk is now known as dtantsur | 10:28 | |
| *** rfolco has joined #openstack-infra | 10:28 | |
| dtantsur | morning folks! is there a way to disable IPv6 in devstack? | 10:32 |
| *** witek_ has quit IRC | 10:33 | |
| *** rcernin has quit IRC | 10:33 | |
| *** ociuhandu has joined #openstack-infra | 10:36 | |
| *** ociuhandu has quit IRC | 10:40 | |
| rishabhhpe | Hi ALL , We are not able to see results published on CI-Watch screen from yesterday . Also Devstack installation is failing at very early stage of setup.. Is there anyone else facing the same issue or someone who can help us understand the root cause of issue? | 10:40 |
| rishabhhpe | In success log we are able to see /opt/stack/data/CA while in failure log we are not able to see the same directory. For reference we have pasted the logs here http://paste.openstack.org/show/787215/ | 10:40 |
| frickler | dtantsur: are you talking about infra jobs or running devstack locally? for the latter please come along to #openstack-qa | 10:46 |
| dtantsur | I'm not even sure.. | 10:47 |
| frickler | for infra jobs, some of our node providers only have external connectivity via IPv6, so disabling that would be difficult | 10:47 |
| dtantsur | frickler: we see increasing number of things like https://d38d0367c1ca70558cf0-3059cf1890ea1358c70d952067d56657.ssl.cf5.rackcdn.com/693017/20/check/ironic-standalone/1e582b4/controller/logs/ironic-bm-logs/node-2_console_log.txt.gz | 10:47 |
| dtantsur | which is, in a V4 job a "baremetal" node only gets V6 addresses | 10:47 |
| *** rishabhhpe has quit IRC | 10:49 | |
| *** derekh has quit IRC | 10:53 | |
| frickler | dtantsur: hmm, how is that node implemented? it should be only internal to the testing environment, not connected to the external interfaces of the test nodes, because you couldn't provide dhcp there anyway | 10:55 |
| dtantsur | yep, it's an internal VM (used for ironic testing) | 10:55 |
| *** derekh has joined #openstack-infra | 10:55 | |
| dtantsur | so DHCP is provided by neutron (and it may be a neutron problem in the end) | 10:56 |
| *** pkopec has quit IRC | 10:58 | |
| *** derekh has quit IRC | 10:59 | |
| *** derekh has joined #openstack-infra | 10:59 | |
| *** ramishra has quit IRC | 11:15 | |
| *** yamamoto has joined #openstack-infra | 11:27 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 11:28 | |
| *** ramishra has joined #openstack-infra | 11:31 | |
| *** ociuhandu has joined #openstack-infra | 11:39 | |
| *** pkopec has joined #openstack-infra | 11:41 | |
| *** ociuhandu has quit IRC | 11:44 | |
| *** zbr has quit IRC | 11:47 | |
| *** zbr has joined #openstack-infra | 11:47 | |
| *** tosky has joined #openstack-infra | 11:50 | |
| *** yamamoto has quit IRC | 11:50 | |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: Documentation fixes https://review.opendev.org/697635 | 11:56 |
| *** yamamoto has joined #openstack-infra | 11:58 | |
| *** yamamoto has quit IRC | 11:58 | |
| *** udesale has quit IRC | 12:00 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix up siblings copy https://review.opendev.org/697614 | 12:02 |
| *** rfolco is now known as rfolco|bbl | 12:03 | |
| *** yamamoto has joined #openstack-infra | 12:25 | |
| *** witek has joined #openstack-infra | 12:27 | |
| *** Lucas_Gray has joined #openstack-infra | 12:31 | |
| *** Lucas_Gray has quit IRC | 12:40 | |
| *** Lucas_Gray has joined #openstack-infra | 12:40 | |
| *** Lucas_Gray has quit IRC | 12:45 | |
| *** Lucas_Gray has joined #openstack-infra | 12:49 | |
| *** rlandy has joined #openstack-infra | 13:03 | |
| *** jamesmcarthur has joined #openstack-infra | 13:06 | |
| *** hashar has quit IRC | 13:07 | |
| *** rh-jelabarre has joined #openstack-infra | 13:09 | |
| *** Lucas_Gray has quit IRC | 13:14 | |
| *** Lucas_Gray has joined #openstack-infra | 13:15 | |
| *** jamesmcarthur has quit IRC | 13:28 | |
| *** jamesmcarthur has joined #openstack-infra | 13:29 | |
| *** jamesmcarthur has quit IRC | 13:34 | |
| *** mriedem has joined #openstack-infra | 13:41 | |
| *** mgoddard has quit IRC | 13:44 | |
| *** mgoddard has joined #openstack-infra | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| *** ociuhandu has joined #openstack-infra | 13:47 | |
| *** pkopec has quit IRC | 13:50 | |
| *** ociuhandu has quit IRC | 13:52 | |
| *** soniya29 has quit IRC | 13:55 | |
| *** mgoddard has quit IRC | 13:58 | |
| *** ociuhandu has joined #openstack-infra | 14:07 | |
| *** mgoddard has joined #openstack-infra | 14:13 | |
| *** eharney has joined #openstack-infra | 14:15 | |
| *** hashar has joined #openstack-infra | 14:22 | |
| *** ociuhandu has quit IRC | 14:27 | |
| *** ociuhandu has joined #openstack-infra | 14:28 | |
| *** aedc has quit IRC | 14:28 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/697622 | 14:28 |
| *** aedc has joined #openstack-infra | 14:28 | |
| *** jamesmcarthur has quit IRC | 14:29 | |
| *** jamesmcarthur has joined #openstack-infra | 14:30 | |
| *** jamesmcarthur has quit IRC | 14:35 | |
| *** Goneri has joined #openstack-infra | 14:38 | |
| openstackgerrit | Merged openstack/ptgbot master: Add common navbar and display MOTD on all pages https://review.opendev.org/694326 | 14:39 |
| *** goldyfruit___ has joined #openstack-infra | 14:41 | |
| *** pkopec has joined #openstack-infra | 14:41 | |
| *** liuyulong has quit IRC | 14:47 | |
| *** ykarel is now known as ykarel|afk | 14:50 | |
| *** hashar has quit IRC | 14:53 | |
| *** jklare has quit IRC | 14:56 | |
| *** jklare has joined #openstack-infra | 14:59 | |
| *** jamesmcarthur has joined #openstack-infra | 15:00 | |
| *** chkumar|ruck is now known as raukadah | 15:01 | |
| *** aedc has quit IRC | 15:02 | |
| *** dave-mccowan has joined #openstack-infra | 15:05 | |
| *** goldyfruit___ is now known as goldyfruit | 15:07 | |
| *** jamesmcarthur has quit IRC | 15:08 | |
| *** dave-mccowan has quit IRC | 15:10 | |
| *** jamesmcarthur has joined #openstack-infra | 15:22 | |
| *** KeithMnemonic has joined #openstack-infra | 15:29 | |
| *** jamesmcarthur has quit IRC | 15:31 | |
| *** jamesmcarthur has joined #openstack-infra | 15:32 | |
| *** ykarel|afk is now known as ykarel|away | 15:38 | |
| *** jamesmcarthur has quit IRC | 15:46 | |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.3 https://review.opendev.org/694622 | 15:47 |
| openstackgerrit | David Moreau Simard proposed opendev/system-config master: DNM: Test ara 1.3 for bridge https://review.opendev.org/695108 | 15:48 |
| *** ykarel|away has quit IRC | 15:49 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul-jobs master: Make ara-report role to zuul_return an artifact https://review.opendev.org/697681 | 15:50 |
| openstackgerrit | David Moreau Simard proposed opendev/system-config master: DNM: Test ara-report artifact https://review.opendev.org/697683 | 16:00 |
| *** aedc has joined #openstack-infra | 16:03 | |
| *** ociuhandu has quit IRC | 16:06 | |
| *** rfolco|bbl is now known as rfolco | 16:12 | |
| *** dpawlik has quit IRC | 16:14 | |
| fungi | infra-root: ovh has sent an announcement that effective february 18 they'll be removing subregion-specific swift endpoints from their keystone catalog, and that customers should switch to generic region-named swift endpoints | 16:14 |
| fungi | i guess this means we need to update our zuul logs config | 16:14 |
| *** apetrich has quit IRC | 16:15 | |
| fungi | i'm not quite sure, but i think that means adding new "gra" and "bhs" regions in clouds.yaml and then switching base-jobs to reach swift there | 16:16 |
| fungi | jenkins@o.o got a copy of the announcement, but in french because i've never been able to figure out how to convince ovh to send english messages to that account's contact | 16:17 |
| fungi | (or at least the last time i tried, they limited the available languages based on your billing address, which in the case of that account is set up in a funky way due to it being gratis) | 16:19 |
| clarkb | fungi: I think shade and sdk are supposed to figure out region specific catalogs | 16:21 |
| *** edausq has joined #openstack-infra | 16:22 | |
| fungi | oh? | 16:22 |
| clarkb | I wonderif that mightthis a but more seamless for us? | 16:22 |
| *** efried is now known as efried_afk | 16:22 | |
| fungi | so maybe it'll "just work" then | 16:22 |
| clarkb | ya there is some templating mechansim they get from somewhere | 16:22 |
| clarkb | mordred: ^ would knoe | 16:22 |
| clarkb | iirc citycloud uses this feature | 16:22 |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| fungi | i suppose we could just wait for february and see if it breaks | 16:23 |
| rledisez | fungi: hi, i'm from ovh, managing object storage. if you need any help just ping me. (don't wait february, it will break for sure ;)) | 16:23 |
| fungi | or is there some way to see what endpoint the executors are accessing? | 16:23 |
| fungi | rledisez: thanks, we're trying to work out whether openstacksdk (which we're inferring our swift endpoints from) will find the correct swift endopoint in the catalog automatically | 16:24 |
| rledisez | fungi: current situation is you have both endpoint in your catalog (eg: BHS and BHS1). in february, we will remove BHS1 | 16:25 |
| fungi | or whether we need to configure it to talk to a completely different keystone region to find the swift endpoint | 16:25 |
| rledisez | so, you should be able to change your config to use BHS right now it should work. if you do so, you're done | 16:26 |
| *** pkopec_ has joined #openstack-infra | 16:26 | |
| fungi | okay, so we don't need to add a clouds.yaml entry to tell our systems to talk to a different "bhs" keystone endpoint to find the new bhs swift | 16:26 |
| rledisez | no. just specify the new region name and it should be transparent | 16:26 |
| fungi | if the bhs1 keystone endpoint already has a bhs swift in its catalog then we may not need to do anything at all | 16:26 |
| rledisez | do you have a pointer to your current config? | 16:27 |
| rledisez | *link | 16:27 |
| fungi | it sounds like we don't specify a swift region, we just ask keystone to tell us where to reach swift | 16:27 |
| clarkb | fungi: we do specify a region | 16:27 |
| clarkb | link in a sec | 16:27 |
| rledisez | ouch, bad idea | 16:27 |
| rledisez | thx clarkb | 16:27 |
| fungi | ahh, then i misunderstood what you meant about openstacksdk guessing the correct swift endpoint, clarkb | 16:28 |
| clarkb | https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/secrets.yaml#L173 | 16:28 |
| clarkb | fungi: there is functionality in keystone (I think) where if you talk to one keystone it can tell you there are other keystones allowing you to run a catalog per region rather than one cloud global catalog. I confused what ovh is doing with how citycloud uses this feature | 16:28 |
| rledisez | ok, so it seems you just need to replace: | 16:29 |
| rledisez | region_name: BHS1 | 16:29 |
| rledisez | by | 16:29 |
| rledisez | region_name: BHS | 16:29 |
| clarkb | fungi: in this case I think ovh is simply syaing BHS is our swift region not bhsX | 16:29 |
| rledisez | and same for GRA | 16:29 |
| clarkb | rledisez: thanks. | 16:29 |
| *** pkopec has quit IRC | 16:29 | |
| rledisez | BUT, it only applies for object storage. instances stays on BHS1/GRA1 | 16:29 |
| fungi | okay, but we at least don't need different credentials and a different keystone endpoint for the new region. thanks rledisez | 16:29 |
| rledisez | I don't know if you can make the difference in your config (if you need to) | 16:29 |
| rledisez | fungi: right creds stay the same | 16:30 |
| fungi | yeah, should just be changing a couple lines like what clarkb linked there | 16:30 |
| clarkb | rledisez: yup those secrets are curerntly only used for swift. I'll push up a test change to confirm all that in a sec | 16:30 |
| rledisez | clarkb: perfect then, you should be good | 16:30 |
| mordred | clarkb: no - sdk will not fix the ovh shift for us | 16:33 |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Test ovh swift region update https://review.opendev.org/697692 | 16:34 |
| clarkb | fungi: mordred ^ if we land that we can test with a base job udpate in say zuul-jobs then if that works we roll out an update globally | 16:34 |
| * mordred read more scrollback - yes - I agree with teh scrollback | 16:34 | |
| *** Lucas_Gray has quit IRC | 16:35 | |
| *** lpetrut has quit IRC | 16:35 | |
| mordred | and that patch looks good | 16:35 |
| fungi | test playbook change lgtm | 16:35 |
| *** sgw has quit IRC | 16:36 | |
| *** pkopec_ has quit IRC | 16:37 | |
| *** pkopec has joined #openstack-infra | 16:39 | |
| *** yamamoto has quit IRC | 16:40 | |
| *** yamamoto has joined #openstack-infra | 16:41 | |
| *** yamamoto has quit IRC | 16:41 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Set correct python version for non-chroot scripts https://review.opendev.org/697211 | 16:41 |
| openstackgerrit | Merged openstack/pbr master: Switch to Ussuri jobs https://review.opendev.org/691456 | 16:44 |
| *** pkopec has quit IRC | 16:44 | |
| *** rlandy is now known as rlandy|ruck | 16:44 | |
| *** ociuhandu has joined #openstack-infra | 16:45 | |
| *** ociuhandu has quit IRC | 16:45 | |
| *** ociuhandu has joined #openstack-infra | 16:46 | |
| openstackgerrit | Merged opendev/base-jobs master: Test ovh swift region update https://review.opendev.org/697692 | 16:47 |
| *** gyee has joined #openstack-infra | 16:48 | |
| *** rpittau is now known as rpittau|afk | 16:51 | |
| *** ociuhandu has quit IRC | 16:51 | |
| *** sgw has joined #openstack-infra | 16:51 | |
| *** lucasagomes has quit IRC | 16:54 | |
| *** psachin has quit IRC | 16:55 | |
| *** pgaxatte has quit IRC | 16:57 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Set correct python version for non-chroot scripts https://review.opendev.org/697211 | 16:58 |
| *** eernst has joined #openstack-infra | 16:58 | |
| *** jamesmcarthur has joined #openstack-infra | 16:59 | |
| *** efried_afk is now known as efried | 17:00 | |
| *** derekh has quit IRC | 17:03 | |
| openstackgerrit | Jeremy Stanley proposed opendev/storyboard master: Change py3 testing to py36 https://review.opendev.org/697549 | 17:09 |
| openstackgerrit | Jeremy Stanley proposed opendev/storyboard master: Build container images https://review.opendev.org/611191 | 17:09 |
| fungi | mordred: i had to move a couple lines from the second change in that series to the first to get the job to work ^ | 17:09 |
| *** hashar has joined #openstack-infra | 17:09 | |
| clarkb | infra-root https://review.opendev.org/#/c/680178/ has been restored to test the ovh swift update | 17:13 |
| *** iurygregory has quit IRC | 17:14 | |
| *** jamesmcarthur has quit IRC | 17:14 | |
| fungi | Job base-test not defined | 17:14 |
| fungi | need to add opendev/base-jobs as a required project? | 17:15 |
| fungi | er, i guess as a project in the zuul tenant config? | 17:15 |
| openstackgerrit | Merged zuul/zuul-jobs master: use-buildset-registry: Vendor pytoml and remarshal https://review.opendev.org/695050 | 17:15 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add build-container-image role https://review.opendev.org/695251 | 17:15 |
| corvus | fungi: that's from a different zuul | 17:15 |
| fungi | i would have expected it to be implied already though | 17:15 |
| clarkb | ya we can ignore software factory ci for this subset of testing | 17:15 |
| fungi | d'oh! | 17:16 |
| *** ykarel|away has joined #openstack-infra | 17:16 | |
| clarkb | if we were testing roles we shouldn't ignore them :) | 17:16 |
| fungi | i totally didn't spot the username, just saw a zuul config error | 17:16 |
| * fungi sighs | 17:16 | |
| fungi | thanks :/ | 17:16 |
| fungi | clearly my coffee was defective this morning | 17:16 |
| *** jamesmcarthur has joined #openstack-infra | 17:18 | |
| *** apetrich has joined #openstack-infra | 17:18 | |
| corvus | fungi: try coffee again? | 17:19 |
| mordred | fungi: I +2 things in that stack - but maybe you should too - and each of our +2 is a half of a +2 | 17:19 |
| *** witek has quit IRC | 17:21 | |
| *** aedc has quit IRC | 17:21 | |
| *** KeithMnemonic has quit IRC | 17:22 | |
| openstackgerrit | Merged openstack/project-config master: starlingx: add oidc-auth-armada-app https://review.opendev.org/697576 | 17:22 |
| *** hashar has quit IRC | 17:22 | |
| *** hashar_ has joined #openstack-infra | 17:23 | |
| fungi | mordred: wfm | 17:23 |
| *** pcaruana has quit IRC | 17:23 | |
| fungi | clarkb: looks like post_failure in such a way as we don't get any logs. did you happen to watch the console stream for one? | 17:25 |
| *** jamesmcarthur has quit IRC | 17:25 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Record job build attempts in inventory https://review.opendev.org/697548 | 17:26 |
| clarkb | fungi: I did not | 17:26 |
| *** jamesmcarthur has joined #openstack-infra | 17:26 | |
| fungi | i'll recheck | 17:26 |
| clarkb | fungi: k | 17:26 |
| openstackgerrit | Merged zuul/zuul-jobs master: use-buildset-registry: Add podman support https://review.opendev.org/695051 | 17:28 |
| *** jamesmcarthur has quit IRC | 17:28 | |
| *** rlandy|ruck is now known as rlandy|ruck|brb | 17:29 | |
| *** jamesmcarthur has joined #openstack-infra | 17:29 | |
| fungi | clarkb: https://zuul.opendev.org/t/zuul/stream/672362c11c10432ca7dd6dde2cef85b8?logfile=console.log | 17:31 |
| fungi | though i suppose it's possible the errors won't show up at the end of the stream | 17:32 |
| clarkb | fungi: ya in which case we can find the logs on the executor | 17:33 |
| fungi | yup | 17:33 |
| *** hashar_ has quit IRC | 17:33 | |
| fungi | it's on to the post tasks now | 17:34 |
| *** hashar has joined #openstack-infra | 17:34 | |
| fungi | openstack.exceptions.ConfigException: Region BHS is not a valid region name for cloud defaults. Valid choices are BHS1,GRA1,SBG1. Please note that region names are case sensitive. | 17:35 |
| *** ociuhandu has joined #openstack-infra | 17:35 | |
| *** tesseract has quit IRC | 17:35 | |
| clarkb | from ze01 | 17:35 |
| fungi | so openstacksdk wants to find that region in clouds.yaml anyway | 17:35 |
| fungi | is my guess for that exception | 17:36 |
| fungi | or it needs to be told a proxy region name there? | 17:36 |
| clarkb | fungi: I'm guessing the secret data is actually clouds.yaml content and it lists explicit regions | 17:36 |
| *** ociuhandu has quit IRC | 17:37 | |
| fungi | (get defaults from the bhs1 entry, but use bhs as the region name in the reqyest) | 17:37 |
| clarkb | fungi: we probably need new secret data sets regions to BHS and GRA instead of BHS1 and GRA1 | 17:37 |
| clarkb | (maybe) | 17:37 |
| clarkb | mordred: ^ | 17:37 |
| fungi | oh | 17:37 |
| mordred | oh | 17:37 |
| mordred | wait | 17:37 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: openshift speculative containers https://review.opendev.org/696939 | 17:37 |
| mordred | we need to update sdk | 17:37 |
| mordred | one sec | 17:37 |
| *** ociuhandu has joined #openstack-infra | 17:38 | |
| clarkb | oh is this part of the vendored ovh data? | 17:38 |
| fungi | bug? | 17:38 |
| fungi | ahh | 17:38 |
| clarkb | the "I know how to talk to ovh let me help you" json file | 17:38 |
| fungi | which we also serve from a vhost on files.o.o right? | 17:38 |
| *** bnemec is now known as beekneemech | 17:38 | |
| mordred | clarkb yes fungi no | 17:38 |
| fungi | oh, right, service-types is different | 17:39 |
| mordred | yeah | 17:39 |
| mordred | it's so awesome that this is all so awesome | 17:39 |
| mordred | this is the thing tha we added the ability for cloud vendors to publish themselves ... but so far mnaser is the only one to do so | 17:39 |
| *** ociuhandu has quit IRC | 17:39 | |
| *** ociuhandu has joined #openstack-infra | 17:39 | |
| fungi | right-o | 17:40 |
| mordred | there was already a patch up to sdk for this - but it had a json error | 17:40 |
| mordred | https://review.opendev.org/692961 update OVH vendor entry | 17:40 |
| fungi | maybe rledisez would be interested in that as additional related feedback | 17:40 |
| clarkb | ok so we wait for sdk to update and release then we recheck that zuul-jobs change | 17:40 |
| mordred | yeah. although I think we should be able to override that? | 17:41 |
| * mnaser is happy we did that | 17:41 | |
| clarkb | mordred: ya I think we can if we provide our own complete clouds.yaml entry for it (I and I don't know what is in that secret right now because it is encrypted) | 17:41 |
| mnaser | i still think its pretty cool - https://vexxhost.com/.well-known/openstack/api | 17:41 |
| *** yamamoto has joined #openstack-infra | 17:41 | |
| clarkb | mordred: but we have until february so waiting a cuple days is probably fine? | 17:41 |
| mnaser | we did goof once and lose that file | 17:41 |
| clarkb | (mostly I don't want to decrypt the secret and instead want to eat breakfast :) if someone else wants to see what we've configured there go for it) | 17:42 |
| openstackgerrit | Merged opendev/storyboard master: Change py3 testing to py36 https://review.opendev.org/697549 | 17:43 |
| mordred | clarkb: yah - no rush | 17:43 |
| fungi | i have decrypted the one for bhs1 | 17:48 |
| fungi | the username, password and project ids are all just raw strings containing their expected values | 17:48 |
| clarkb | ok so it isnt yaml | 17:49 |
| clarkb | in that case the sdk update us probably easiest | 17:49 |
| fungi | nope, the actual username, actual password and numerical project id | 17:49 |
| fungi | no yaml | 17:49 |
| fungi | no additional contextual data | 17:49 |
| *** yamamoto has quit IRC | 17:49 | |
| mordred | clarkb: you wanna +A https://review.opendev.org/#/c/692961 ? | 17:50 |
| fungi | so yeah, seems like we do need updated sdk or at least updated json blob or overrides in clouds.yaml | 17:50 |
| mordred | we landed an sdk patch the other day that mriedem needs in nova too - so it would be good to have a new release anyway | 17:50 |
| *** armstrong has joined #openstack-infra | 17:50 | |
| clarkb | mordred: that change removes bhs1 and gra1? | 17:51 |
| clarkb | I think we need to keep those too? | 17:51 |
| mordred | clarkb: oh - good catch | 17:52 |
| fungi | yeah, otherwise that's going to break us worse | 17:52 |
| mordred | clarkb, fungi: updated | 17:53 |
| mordred | of course - now this makes me want to ponder modelling service-specific regions | 17:54 |
| *** rlandy|ruck|brb is now known as rlandy|ruck | 17:54 | |
| mordred | but - I think that's a can of worms best left alone | 17:54 |
| *** ricolin has quit IRC | 17:55 | |
| clarkb | you should get a proper error message for when you mismatch region service combos | 17:55 |
| clarkb | probably good enough for now | 17:55 |
| mordred | although it feels like this is stepping ovh towards perhaps using openstack azs within a region? like having BHS as the region then BHS1 and BHS5 as azs | 17:56 |
| mordred | clarkb: yeah - I agree | 17:56 |
| fungi | their announcement hinted that they may transition more services to this model in the future | 17:57 |
| *** jamesmcarthur has quit IRC | 18:02 | |
| *** jamesmcarthur has joined #openstack-infra | 18:02 | |
| mordred | yeah. back at my last employer before they gave up on deploying a public cloud - we were trying to convince people to just use the region/az model in openstack as it existed to provide a similar model | 18:07 |
| mordred | than ran up against "but I want to complain and do things differently and write my own thing and fail" as the overall strategy | 18:07 |
| * mordred totally does not harbor any bitterness | 18:08 | |
| *** jamesmcarthur has quit IRC | 18:17 | |
| openstackgerrit | Merged zuul/zuul master: Improve functionality and docs around ansible installation https://review.opendev.org/675403 | 18:19 |
| *** ralonsoh has quit IRC | 18:29 | |
| sgw | clarkb: I see that the new oidc-auth-armada-app repo and associated core group has been created. Can you please add robert.church@windriver.com to the group as it's initial member? Maybe that in progress by someone ? | 18:33 |
| clarkb | sgw: done | 18:36 |
| sgw | clarkb: thanks! | 18:37 |
| *** ociuhandu has quit IRC | 18:51 | |
| *** ociuhandu has joined #openstack-infra | 18:52 | |
| *** ociuhandu has quit IRC | 18:57 | |
| *** hashar has quit IRC | 19:03 | |
| *** tosky has quit IRC | 19:18 | |
| *** jtomasek has quit IRC | 19:28 | |
| *** ociuhandu has joined #openstack-infra | 19:29 | |
| *** ociuhandu has quit IRC | 19:35 | |
| *** beekneemech has quit IRC | 19:39 | |
| *** armax has joined #openstack-infra | 19:40 | |
| *** bnemec has joined #openstack-infra | 19:40 | |
| *** bnemec has quit IRC | 19:47 | |
| *** bnemec has joined #openstack-infra | 19:52 | |
| *** bnemec is now known as beekneemech | 19:53 | |
| openstackgerrit | Antony Messerli proposed openstack/diskimage-builder master: Adds support for Fedora 31 https://review.opendev.org/697713 | 19:59 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Look for depends-on lines in dependency searches https://review.opendev.org/697595 | 20:07 |
| *** lpetrut has joined #openstack-infra | 20:11 | |
| *** jtomasek has joined #openstack-infra | 20:13 | |
| fungi | gonna pop out to run some errands and grab an early dinner, but should be back in a couple of hours | 20:23 |
| *** dirk has quit IRC | 20:26 | |
| *** dirk has joined #openstack-infra | 20:32 | |
| *** lourot has quit IRC | 20:40 | |
| *** armstrong has quit IRC | 20:40 | |
| *** lourot has joined #openstack-infra | 20:41 | |
| *** lourot has joined #openstack-infra | 20:42 | |
| *** lpetrut has quit IRC | 20:56 | |
| *** diablo_rojo has joined #openstack-infra | 20:57 | |
| *** jamesmcarthur has joined #openstack-infra | 20:59 | |
| *** jamesmcarthur has quit IRC | 21:04 | |
| *** rlandy|ruck has quit IRC | 21:11 | |
| *** Zara has quit IRC | 21:15 | |
| clarkb | mordred: looks like sdk change has merged. Next step is a release? | 21:20 |
| mordred | clarkb: yup | 21:28 |
| mordred | lemme submit patch | 21:28 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Increase udp buffer sizes when installing afs client https://review.opendev.org/697589 | 21:28 |
| mordred | clarkb: remote: https://review.opendev.org/697721 Release 0.39.0 of openstacksdk | 21:29 |
| *** jamesmcarthur has joined #openstack-infra | 21:30 | |
| *** rfolco has quit IRC | 21:33 | |
| openstackgerrit | Clark Boylan proposed opendev/puppet-openafs master: Increase udp buffer sizes when running afs https://review.opendev.org/697588 | 21:35 |
| *** jamesmcarthur has quit IRC | 21:36 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: DO NOT MERGE test cleanup phase playbook https://review.opendev.org/680178 | 21:37 |
| jrosser | can i get a bit of advice with this http://paste.openstack.org/show/787262/ | 21:38 |
| jrosser | then i go here and see they just released a new version https://pypi.org/project/colorama/0.4.3/#history | 21:38 |
| clarkb | jrosser: that looks like a broken package (it is missing a file that the setup.py expects to be there) | 21:40 |
| jrosser | yeah, whats the right thing to do, is there a way to pin back to 0.4.1 | 21:41 |
| * jrosser not quite sure how this works | 21:41 | |
| clarkb | you can use constraints for force an older version. Or pip lockfile. Or try !=0.4.1 | 21:42 |
| clarkb | in any case you should file a bug with them as they should be including that file in the sdist if setup.py requires it | 21:42 |
| jrosser | it's blown up whilst building rally, not sure how many others that will affect | 21:45 |
| *** irclogbot_3 has quit IRC | 21:45 | |
| clarkb | jrosser: looks like they made a 0.4.3 | 21:45 |
| clarkb | and that sdist has readme.rst in it | 21:46 |
| jrosser | oh hahah https://github.com/tartley/colorama/issues/240 | 21:46 |
| clarkb | I think what you want to do is set !=0.4.2 in your requirements for that requirement | 21:46 |
| openstackgerrit | Merged zuul/zuul master: Fix job result metric name for builds to retry https://review.opendev.org/697248 | 21:46 |
| jrosser | they broke a lot of people | 21:46 |
| *** yamamoto has joined #openstack-infra | 21:47 | |
| openstackgerrit | Antony Messerli proposed openstack/diskimage-builder master: Adds support for Fedora 31 https://review.opendev.org/697713 | 21:47 |
| *** slaweq has quit IRC | 21:50 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: openshift speculative containers https://review.opendev.org/696939 | 21:50 |
| *** ociuhandu has joined #openstack-infra | 21:50 | |
| *** yamamoto has quit IRC | 21:52 | |
| *** slaweq has joined #openstack-infra | 21:54 | |
| *** hashar has joined #openstack-infra | 21:55 | |
| *** hashar has quit IRC | 21:55 | |
| openstackgerrit | Merged zuul/zuul master: Pagure: Fix handling of threshold_reached: None https://review.opendev.org/691254 | 22:00 |
| *** ykarel|away has quit IRC | 22:11 | |
| *** jtomasek has quit IRC | 22:12 | |
| *** goldyfruit has quit IRC | 22:12 | |
| *** slaweq has quit IRC | 22:18 | |
| *** slaweq has joined #openstack-infra | 22:20 | |
| *** diablo_rojo has quit IRC | 22:31 | |
| *** jamesmcarthur has joined #openstack-infra | 22:33 | |
| *** tosky has joined #openstack-infra | 22:35 | |
| *** jamesmcarthur has quit IRC | 22:38 | |
| *** ijw has joined #openstack-infra | 22:47 | |
| openstackgerrit | Merged zuul/zuul master: Don't search github api for projects not in Zuul https://review.opendev.org/694390 | 22:48 |
| *** slaweq has quit IRC | 22:50 | |
| *** kaisers has quit IRC | 22:50 | |
| *** ociuhandu has quit IRC | 22:50 | |
| openstackgerrit | Merged zuul/zuul master: Look for depends-on lines in dependency searches https://review.opendev.org/697595 | 23:03 |
| *** mriedem has quit IRC | 23:21 | |
| *** ijw has quit IRC | 23:22 | |
| *** ijw has joined #openstack-infra | 23:25 | |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| *** jamesmcarthur has joined #openstack-infra | 23:34 | |
| *** ociuhandu has quit IRC | 23:35 | |
| *** jamesmcarthur has quit IRC | 23:38 | |
| *** ramishra has quit IRC | 23:48 | |
| openstackgerrit | Merged zuul/nodepool master: Documentation fixes https://review.opendev.org/697635 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!