| *** matbu has quit IRC | 00:19 | |
| *** matbu has joined #openstack-infra | 00:19 | |
| *** quasar has quit IRC | 00:29 | |
| *** gyee has quit IRC | 00:53 | |
| *** iurygregory has quit IRC | 01:26 | |
| *** Limech has joined #openstack-infra | 01:27 | |
| *** Goneri has quit IRC | 02:31 | |
| *** dansmith has quit IRC | 02:34 | |
| *** dansmith has joined #openstack-infra | 02:34 | |
| *** kaisers has quit IRC | 02:44 | |
| *** rcernin has quit IRC | 02:54 | |
| *** rcernin has joined #openstack-infra | 03:13 | |
| *** ramishra has quit IRC | 04:00 | |
| *** ramishra has joined #openstack-infra | 04:04 | |
| *** thedaveking has quit IRC | 04:58 | |
| *** ykarel|away has joined #openstack-infra | 05:01 | |
| *** ykarel|away is now known as ykarel | 05:02 | |
| *** vishalmanchanda has joined #openstack-infra | 05:32 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-infra | 05:33 | |
| *** rcernin has quit IRC | 05:36 | |
| *** rcernin has joined #openstack-infra | 05:43 | |
| *** rcernin has quit IRC | 05:57 | |
| *** rcernin has joined #openstack-infra | 06:03 | |
| *** user_19173783170 has joined #openstack-infra | 06:06 | |
| user_19173783170 | I'm setting up my 3rd party ci system, but when i testing communication Between upstream and my ci system | 06:10 |
|---|---|---|
| user_19173783170 | there's a trouble in the zuul log which was "zuul.Scheduler: Project opendev/ci-sandbox not found" | 06:12 |
| *** matt_kosut has joined #openstack-infra | 06:13 | |
| user_19173783170 | can someone give me some idea? | 06:20 |
| *** rcernin has quit IRC | 06:21 | |
| *** rpittau|afk is now known as rpittau | 06:47 | |
| *** lmiccini has joined #openstack-infra | 06:52 | |
| *** ysandeep|away is now known as ysandeep|ruck | 06:53 | |
| *** ralonsoh has joined #openstack-infra | 06:56 | |
| *** rcernin has joined #openstack-infra | 07:03 | |
| *** Tengu has joined #openstack-infra | 07:09 | |
| *** Tengu has quit IRC | 07:12 | |
| *** ramishra has quit IRC | 07:19 | |
| *** sboyron has joined #openstack-infra | 07:19 | |
| *** ramishra has joined #openstack-infra | 07:20 | |
| *** ramishra has quit IRC | 07:21 | |
| *** ramishra has joined #openstack-infra | 07:21 | |
| *** Tengu has joined #openstack-infra | 07:22 | |
| *** ociuhandu has joined #openstack-infra | 07:37 | |
| *** dklyle has quit IRC | 07:37 | |
| *** ociuhandu has quit IRC | 07:41 | |
| *** jcapitao has joined #openstack-infra | 07:42 | |
| *** sboyron has quit IRC | 07:43 | |
| *** jacksonier has joined #openstack-infra | 08:04 | |
| *** eolivare has joined #openstack-infra | 08:05 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 08:07 | |
| *** jacksonier has quit IRC | 08:07 | |
| *** jacksonier has joined #openstack-infra | 08:07 | |
| *** iurygregory_ has joined #openstack-infra | 08:09 | |
| *** rcernin has quit IRC | 08:09 | |
| *** sboyron has joined #openstack-infra | 08:10 | |
| *** andrewbonney has joined #openstack-infra | 08:11 | |
| *** jacksonier has quit IRC | 08:14 | |
| *** jacksonier has joined #openstack-infra | 08:14 | |
| *** iurygregory_ is now known as iurygregory | 08:14 | |
| *** dtantsur|afk is now known as dtantsur | 08:42 | |
| *** dchen has quit IRC | 08:45 | |
| *** rcernin has joined #openstack-infra | 08:52 | |
| *** jcapitao has quit IRC | 08:53 | |
| *** jcapitao has joined #openstack-infra | 08:55 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** jcapitao is now known as jcapitao_afk | 09:02 | |
| *** rcernin has quit IRC | 09:06 | |
| *** jacksonier has quit IRC | 09:06 | |
| *** jacksonier has joined #openstack-infra | 09:06 | |
| *** jacksonier has quit IRC | 09:07 | |
| *** jacksonier has joined #openstack-infra | 09:07 | |
| *** ociuhandu has joined #openstack-infra | 09:17 | |
| *** ociuhandu has quit IRC | 09:29 | |
| *** ociuhandu has joined #openstack-infra | 09:30 | |
| *** zxiiro has quit IRC | 09:30 | |
| *** ociuhandu has quit IRC | 09:34 | |
| *** ociuhandu has joined #openstack-infra | 09:34 | |
| *** jcapitao_afk is now known as jcapitao | 09:39 | |
| *** derekh has joined #openstack-infra | 09:40 | |
| *** jcapitao has quit IRC | 09:44 | |
| *** ysandeep|lunch is now known as ysandeep|ruck | 09:45 | |
| *** jcapitao has joined #openstack-infra | 09:46 | |
| *** jacksonier has quit IRC | 09:51 | |
| *** jacksonier has joined #openstack-infra | 09:51 | |
| *** sboyron has quit IRC | 09:54 | |
| *** sboyron has joined #openstack-infra | 09:54 | |
| *** slaweq has joined #openstack-infra | 09:56 | |
| *** ociuhandu has quit IRC | 10:09 | |
| *** ociuhandu has joined #openstack-infra | 10:10 | |
| *** user_19173783170 has quit IRC | 10:27 | |
| *** Ajohn has joined #openstack-infra | 10:39 | |
| *** gibi has joined #openstack-infra | 10:44 | |
| gibi | clarkb: hi! is there a way to see why e-r indexing is slow? | 10:44 |
| gibi | clarkb: we have a bug https://bugs.launchpad.net/nova/+bug/1902276 that as a side effect generates 100MBs of libvirtd.log | 10:45 |
| openstack | Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New] | 10:45 |
| gibi | clarkb: could this be the cause of the slow logstash indexing? | 10:45 |
| *** ociuhandu has quit IRC | 10:47 | |
| *** hashar has joined #openstack-infra | 10:54 | |
| *** jacksonier has quit IRC | 10:57 | |
| *** jacksonier has joined #openstack-infra | 10:58 | |
| *** dtantsur is now known as dtantsur|bbl | 11:13 | |
| *** ociuhandu has joined #openstack-infra | 11:14 | |
| *** ociuhandu has quit IRC | 11:18 | |
| *** ociuhandu has joined #openstack-infra | 11:20 | |
| *** ociuhandu has quit IRC | 11:30 | |
| *** ysandeep|ruck is now known as ysandeep|afk | 11:30 | |
| *** ociuhandu has joined #openstack-infra | 11:33 | |
| *** jacksonier has quit IRC | 11:38 | |
| *** rcernin has joined #openstack-infra | 11:40 | |
| *** ociuhandu has quit IRC | 11:40 | |
| dulek | Folks, had anything changed regarding Google DNS accessibility on gate VMs? | 11:44 |
| dulek | We're seeing quite lot of "2020-11-05T09:51:02.430Z [ERROR] plugin/errors: 2 svc-server.network-policy-4259. A: read udp 10.1.0.210:37572->8.8.8.8:53: i/o timeout" in our coredns instances. | 11:45 |
| *** jacksonier has joined #openstack-infra | 11:45 | |
| *** jcapitao is now known as jcapitao_lunch | 11:49 | |
| *** ociuhandu has joined #openstack-infra | 11:55 | |
| *** rfolco has joined #openstack-infra | 12:06 | |
| *** jacksonier has quit IRC | 12:08 | |
| *** rcernin has quit IRC | 12:08 | |
| *** ociuhandu has quit IRC | 12:10 | |
| *** janders has quit IRC | 12:19 | |
| *** janders has joined #openstack-infra | 12:20 | |
| *** janders has quit IRC | 12:20 | |
| *** ysandeep|afk is now known as ysandeep | 12:25 | |
| *** janders has joined #openstack-infra | 12:29 | |
| *** ociuhandu has joined #openstack-infra | 12:30 | |
| *** janders has quit IRC | 12:32 | |
| *** janders has joined #openstack-infra | 12:33 | |
| *** janders has quit IRC | 12:33 | |
| *** jpena is now known as jpena|lunch | 12:36 | |
| *** janders has joined #openstack-infra | 12:36 | |
| *** ykarel has quit IRC | 12:40 | |
| *** ykarel has joined #openstack-infra | 12:41 | |
| fungi | dulek: are you communicating with that directly, or going through the unbound dns cache running on the node? we use a caching recursive resolver on nodes to reduce query volume precisely because the public dns servers and connectivity to them are unreliable | 12:43 |
| *** ysandeep is now known as ysandeep|ruck | 12:48 | |
| *** jamesdenton has quit IRC | 12:48 | |
| *** jcapitao_lunch is now known as jcapitao | 12:59 | |
| *** tosky has joined #openstack-infra | 13:07 | |
| *** jpena|lunch is now known as jpena | 13:32 | |
| *** Ajohn has quit IRC | 13:36 | |
| *** ociuhandu has quit IRC | 13:37 | |
| *** Ajohn has joined #openstack-infra | 13:38 | |
| *** d34dh0r53 has quit IRC | 13:43 | |
| *** ysandeep|ruck is now known as ysandeep|brb | 13:53 | |
| *** ociuhandu has joined #openstack-infra | 13:56 | |
| dulek | fungi: Ah, well this is multilevel issue then. | 13:57 |
| dulek | fungi: So we're doing that directly. | 13:57 |
| dulek | Because we need coredns running on our K8s cluster and it needs to forward non-local queries somewhere. | 13:58 |
| dulek | We could just make it forward to the DNS taken from the VMs resolv.conf. | 13:58 |
| *** ociuhandu has quit IRC | 13:59 | |
| dulek | But we were told it'll sometimes be IPv6 and we don't implement dual stack for pods (coredns runs in a pod). | 13:59 |
| *** ociuhandu has joined #openstack-infra | 13:59 | |
| *** d34dh0r53 has joined #openstack-infra | 14:01 | |
| fungi | dulek: yes, that's part of the reason too... in some of our providers there's no global ipv4 addressing for nodes so they go through a layer-4 overload nat/pat to a common global ipv4 address for reaching v4-only resources on the internet. that nat can get strained and is increasingly likely to cause v4 protocols like udp, which have no retransmission signalling, to fail in ugly ways | 14:02 |
| fungi | in those providers, dns over ipv6 is likely to perform far better because the nodes can use their individual global v6 addresses to communicate and not have to deal with the dangers of nat | 14:03 |
| dulek | fungi: Can you tell me which providers have that so I can try to check if this is the culprit? | 14:04 |
| fungi | limestone and openedge at the very least, let me see if i can spot some others | 14:05 |
| donnyd | fungi: No jobs are running on OE at this point because my edge rtr is about to die | 14:07 |
| dulek | fungi: Recently we seen the issue on rax-iad and rax-dfw | 14:07 |
| *** eharney_ has joined #openstack-infra | 14:07 | |
| fungi | also linaro i think | 14:07 |
| fungi | rackspace is dual-stack with global ipv4 and global ipv6 so no nat there | 14:08 |
| *** kaisers has joined #openstack-infra | 14:08 | |
| *** ociuhandu has quit IRC | 14:10 | |
| *** eharney has quit IRC | 14:10 | |
| *** dtantsur|bbl is now known as dtantsur | 14:20 | |
| *** hashar has quit IRC | 14:23 | |
| dulek | So it's something else then. :/ | 14:24 |
| *** jamesdenton has joined #openstack-infra | 14:32 | |
| fungi | notably, rackspace is the provider which drove us to not use provider-supplied recursive resolvers, because they blacklist misbehaving instance ip addresses from using their resolvers, but their blacklist lags ip allocation to new instances, so we would regularly boot a new node and find it unable to perform any dns queries | 14:36 |
| *** Guest5484 has quit IRC | 14:40 | |
| *** Goneri has joined #openstack-infra | 14:46 | |
| *** artom has quit IRC | 14:48 | |
| *** ociuhandu has joined #openstack-infra | 14:49 | |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Create inspur/venus repository https://review.opendev.org/761751 | 14:50 |
| *** ociuhandu has quit IRC | 14:55 | |
| *** ysandeep|brb is now known as ysandeep|ruck | 15:00 | |
| *** vishalmanchanda has quit IRC | 15:12 | |
| *** ysandeep|ruck is now known as ysandeep|away | 15:15 | |
| *** ociuhandu has joined #openstack-infra | 15:16 | |
| *** rpittau is now known as rpittau|afk | 15:18 | |
| *** zxiiro has joined #openstack-infra | 15:31 | |
| *** artom has joined #openstack-infra | 15:41 | |
| *** dklyle has joined #openstack-infra | 15:55 | |
| *** gfidente has joined #openstack-infra | 16:05 | |
| *** sshnaidm_ has joined #openstack-infra | 16:12 | |
| *** sshnaidm|afk has quit IRC | 16:13 | |
| *** lmiccini has quit IRC | 16:19 | |
| *** artom has quit IRC | 16:20 | |
| *** sshnaidm_ is now known as sshnaidm|afk | 16:21 | |
| *** eolivare has quit IRC | 16:23 | |
| *** gyee has joined #openstack-infra | 16:33 | |
| *** ociuhandu_ has joined #openstack-infra | 16:52 | |
| *** ociuhandu has quit IRC | 16:55 | |
| *** ociuhandu_ has quit IRC | 16:56 | |
| *** Lewis has joined #openstack-infra | 16:57 | |
| *** Lewis is now known as Guest93652 | 16:58 | |
| *** artom has joined #openstack-infra | 17:02 | |
| *** ykarel has quit IRC | 17:05 | |
| *** ykarel_ has joined #openstack-infra | 17:05 | |
| *** dtantsur is now known as dtantsur|afk | 17:13 | |
| *** ykarel_ has quit IRC | 17:19 | |
| *** jcapitao has quit IRC | 17:33 | |
| *** pcrews has joined #openstack-infra | 17:35 | |
| *** derekh has quit IRC | 17:38 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Run publish-openstack-artifacts on ubuntu-focal https://review.opendev.org/761776 | 17:49 |
| *** jpena is now known as jpena|off | 18:02 | |
| *** andrewbonney has quit IRC | 18:07 | |
| *** ralonsoh has quit IRC | 18:13 | |
| *** Tengu has quit IRC | 18:22 | |
| *** Tengu has joined #openstack-infra | 18:22 | |
| *** ade_lee has quit IRC | 18:41 | |
| *** ade_lee has joined #openstack-infra | 18:41 | |
| *** ade_lee has quit IRC | 18:42 | |
| *** gfidente has quit IRC | 18:44 | |
| *** dave-mccowan has quit IRC | 18:49 | |
| *** kaisers has quit IRC | 18:59 | |
| *** lxkong has quit IRC | 19:39 | |
| *** raissa has joined #openstack-infra | 20:48 | |
| *** ociuhandu has joined #openstack-infra | 20:53 | |
| *** ociuhandu has quit IRC | 21:02 | |
| *** melwitt is now known as jgwentworth | 21:26 | |
| *** lxkong has joined #openstack-infra | 21:58 | |
| *** rfolco has quit IRC | 22:14 | |
| *** raissa has quit IRC | 22:14 | |
| *** raissa has joined #openstack-infra | 22:22 | |
| *** matt_kosut has quit IRC | 22:31 | |
| *** hamalq has joined #openstack-infra | 23:05 | |
| *** dave-mccowan has joined #openstack-infra | 23:07 | |
| *** matt_kosut has joined #openstack-infra | 23:09 | |
| *** matt_kosut has quit IRC | 23:16 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!