| *** yamamoto has joined #openstack-infra | 00:00 | |
| *** hongbin has quit IRC | 00:04 | |
| *** yamamoto has quit IRC | 00:05 | |
| *** hongbin has joined #openstack-infra | 00:05 | |
| *** gyee has quit IRC | 00:21 | |
| *** hamalq has quit IRC | 00:36 | |
| *** hongbin has quit IRC | 00:41 | |
| *** hongbin has joined #openstack-infra | 01:06 | |
| *** rcernin has quit IRC | 01:11 | |
| *** rcernin has joined #openstack-infra | 01:11 | |
| *** pkopec has quit IRC | 01:15 | |
| *** eandersson has quit IRC | 01:29 | |
| *** eandersson has joined #openstack-infra | 01:29 | |
| *** ldenny has quit IRC | 01:48 | |
| *** ldenny has joined #openstack-infra | 01:49 | |
| *** yamamoto has joined #openstack-infra | 01:52 | |
| *** openstackgerrit has joined #openstack-infra | 02:10 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: A pyca/cryptography to Zuul tenant https://review.opendev.org/745990 | 02:10 |
|---|---|---|
| *** lbragstad has quit IRC | 02:32 | |
| *** rcernin has quit IRC | 03:06 | |
| *** armax has quit IRC | 03:08 | |
| *** psachin has joined #openstack-infra | 03:31 | |
| *** ramishra has joined #openstack-infra | 03:45 | |
| *** rcernin has joined #openstack-infra | 03:52 | |
| *** ricolin has joined #openstack-infra | 04:04 | |
| *** eharney has quit IRC | 04:05 | |
| *** eharney has joined #openstack-infra | 04:24 | |
| *** hongbin has quit IRC | 04:26 | |
| *** vishalmanchanda has joined #openstack-infra | 04:32 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| *** yamamoto has quit IRC | 04:36 | |
| *** yamamoto has joined #openstack-infra | 04:39 | |
| *** ykarel has joined #openstack-infra | 04:49 | |
| *** TViernion has quit IRC | 05:14 | |
| *** TViernion has joined #openstack-infra | 05:17 | |
| *** lmiccini has joined #openstack-infra | 05:32 | |
| *** apetrich has joined #openstack-infra | 05:42 | |
| *** eolivare has joined #openstack-infra | 06:05 | |
| *** ysandeep|away is now known as ysandeep | 06:06 | |
| *** jaicaa has quit IRC | 06:07 | |
| *** jaicaa has joined #openstack-infra | 06:10 | |
| ykarel | Is there some issue in opendev.org? git clone or browsing not working | 06:25 |
| ykarel | working now but too slow | 06:28 |
| Tengu | I wasn't able to open the website. | 06:29 |
| Tengu | guess they have an issue at some point. | 06:29 |
| *** Limech has quit IRC | 06:31 | |
| ianw | hrm, yes i saw it | 06:32 |
| ianw | it seems the gita container on gitea04 where i was being sent had restarted | 06:32 |
| Tengu | hm... the entry LB should have probes in order to prevent this kind of things... ? | 06:33 |
| ianw | it's responding now, i'm just watching it | 06:33 |
| ianw | Tengu: it does, but it seems like it was opening a ssh connection but the git communication was hanging | 06:34 |
| ykarel | hmm its' improving | 06:34 |
| Tengu | ianw: ok. well, it happens ;). | 06:34 |
| ianw | gitea04 seems ok. let me know if still seeing issues and tell me your ip and i can see what backend it went to | 06:36 |
| prometheanfire | git review is stalling, because above reasons | 06:37 |
| abhishekk | I am also having trouble in accessing https://opendev.org/openstack/requirements.git | 06:38 |
| prometheanfire | heh, that's what I'm submitting my review for :D | 06:38 |
| prometheanfire | I'll check back in the morning, my review isn't important enough to keep me from sleep | 06:39 |
| ianw | Swap: 8.0G 5.7G 2.3G | 06:40 |
| ianw | hrm | 06:41 |
| ianw | Swap: 8.0G 8.0G 24K | 06:41 |
| ianw | that does not seem god | 06:41 |
| ianw | good | 06:41 |
| ianw | 2020/08/13 06:42:39 ...c/net/http/server.go:3090:logf() [I] http: TLS handshake error from 38.108.68.124:33468: read tcp 38.108.68.131:3000->38.108.68.124:33468: read: connection reset by peer | 06:42 |
| *** markvoelker has joined #openstack-infra | 06:45 | |
| *** slaweq has joined #openstack-infra | 06:46 | |
| *** markvoelker has quit IRC | 06:49 | |
| *** hiwkby has joined #openstack-infra | 06:59 | |
| *** ryohayakawa has quit IRC | 07:04 | |
| *** zxiiro has quit IRC | 07:06 | |
| *** markvoelker has joined #openstack-infra | 07:35 | |
| *** markvoelker has quit IRC | 07:39 | |
| *** tosky has joined #openstack-infra | 07:42 | |
| *** andrewbonney has joined #openstack-infra | 07:48 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Create pyca/infra https://review.opendev.org/746014 | 07:49 |
| *** markvoelker has joined #openstack-infra | 07:50 | |
| *** markvoelker has quit IRC | 07:51 | |
| *** jpena|off is now known as jpena | 07:57 | |
| *** Lucas_Gray has joined #openstack-infra | 08:01 | |
| *** hashar has joined #openstack-infra | 08:03 | |
| *** dchen_ has quit IRC | 08:04 | |
| *** lucasagomes has joined #openstack-infra | 08:04 | |
| *** nightmare_unreal has joined #openstack-infra | 08:05 | |
| *** ryohayakawa has joined #openstack-infra | 08:09 | |
| *** Wryhder has joined #openstack-infra | 08:14 | |
| *** Lucas_Gray has quit IRC | 08:15 | |
| *** Wryhder is now known as Lucas_Gray | 08:15 | |
| *** matt_kosut has joined #openstack-infra | 08:35 | |
| *** gfidente has joined #openstack-infra | 08:41 | |
| *** rcernin has quit IRC | 09:05 | |
| *** ysandeep is now known as ysandeep|lunch | 09:20 | |
| *** Lucas_Gray has quit IRC | 09:22 | |
| *** pkopec has joined #openstack-infra | 09:28 | |
| *** Lucas_Gray has joined #openstack-infra | 09:30 | |
| *** ysandeep|lunch is now known as ysandeep | 09:35 | |
| *** hashar has quit IRC | 09:46 | |
| *** Lucas_Gray has quit IRC | 09:48 | |
| *** Wryhder has joined #openstack-infra | 09:48 | |
| *** Wryhder is now known as Lucas_Gray | 09:49 | |
| jpena | are we having a sync issue between the gitea servers? When cloning the ironic-python-agent repo from a machine in the US, I get redirected to gitea06.opendev.org and it's missing the latest commit. However, from Spain I get redirected to gitea01 and get all commits | 09:49 |
| *** rcernin has joined #openstack-infra | 09:57 | |
| *** nightmare_unreal has quit IRC | 10:40 | |
| *** rcernin has quit IRC | 10:40 | |
| *** yolanda has quit IRC | 10:42 | |
| *** ykarel_ has joined #openstack-infra | 10:43 | |
| *** ykarel has quit IRC | 10:45 | |
| *** yolanda has joined #openstack-infra | 10:46 | |
| *** rcernin has joined #openstack-infra | 10:47 | |
| *** rcernin has quit IRC | 11:00 | |
| *** yamamoto has quit IRC | 11:01 | |
| *** lucasagomes is now known as lucas-hungry | 11:03 | |
| *** lucas-hungry is now known as lucasagomes | 11:03 | |
| *** apetrich has quit IRC | 11:11 | |
| *** yolanda has quit IRC | 11:13 | |
| *** yolanda has joined #openstack-infra | 11:20 | |
| *** apetrich has joined #openstack-infra | 11:23 | |
| *** yamamoto has joined #openstack-infra | 11:23 | |
| *** yamamoto has quit IRC | 11:29 | |
| *** eolivare_ has joined #openstack-infra | 11:31 | |
| *** eolivare has quit IRC | 11:32 | |
| *** jpena is now known as jpena|lunch | 11:37 | |
| *** ykarel_ is now known as ykarel | 11:55 | |
| *** yamamoto has joined #openstack-infra | 12:03 | |
| *** rfolco has joined #openstack-infra | 12:05 | |
| *** lmiccini has quit IRC | 12:07 | |
| *** yamamoto has quit IRC | 12:11 | |
| *** hashar has joined #openstack-infra | 12:12 | |
| *** ryohayakawa has quit IRC | 12:18 | |
| *** rlandy has joined #openstack-infra | 12:21 | |
| *** lmiccini has joined #openstack-infra | 12:27 | |
| *** marios|ruck has joined #openstack-infra | 12:35 | |
| *** hiwkby has quit IRC | 12:36 | |
| *** jpena|lunch is now known as jpena | 12:50 | |
| *** dklyle has quit IRC | 13:07 | |
| *** ociuhandu has quit IRC | 13:09 | |
| *** lbragstad has joined #openstack-infra | 13:13 | |
| *** ociuhandu has joined #openstack-infra | 13:24 | |
| *** ociuhandu has quit IRC | 13:29 | |
| *** d34dh0r53 has joined #openstack-infra | 13:29 | |
| *** tkajinam has quit IRC | 13:32 | |
| *** ykarel has quit IRC | 13:59 | |
| *** dave-mccowan has quit IRC | 13:59 | |
| *** ykarel has joined #openstack-infra | 13:59 | |
| *** dave-mccowan has joined #openstack-infra | 14:00 | |
| *** ociuhandu has joined #openstack-infra | 14:02 | |
| *** ykarel_ has joined #openstack-infra | 14:03 | |
| *** ykarel has quit IRC | 14:05 | |
| clarkb | jpena: the earliermentioned outageslikely caused some replication events to get missed. We will need to ensure gitea is healthy first then force a resync from gerrit | 14:13 |
| *** zxiiro has joined #openstack-infra | 14:18 | |
| *** ociuhandu_ has joined #openstack-infra | 14:19 | |
| *** dave-mccowan has quit IRC | 14:21 | |
| *** ociuhandu has quit IRC | 14:22 | |
| *** ykarel__ has joined #openstack-infra | 14:30 | |
| *** ykarel_ has quit IRC | 14:32 | |
| *** dklyle has joined #openstack-infra | 14:33 | |
| *** ykarel__ is now known as ykarel | 14:35 | |
| *** armax has joined #openstack-infra | 14:40 | |
| *** rlandy is now known as rlandy|mtg | 14:44 | |
| *** hashar has quit IRC | 14:45 | |
| *** rlandy|mtg is now known as rlandy | 15:15 | |
| *** lmiccini has quit IRC | 15:18 | |
| *** vishalmanchanda has quit IRC | 15:32 | |
| *** ysandeep is now known as ysandeep|away | 15:35 | |
| clarkb | jpena: we think whatever caused it has subsided and have rebooted all the backends to ensure the OOMing isn't going to cause persistent issues. WIth that done we've also forced a complete rereplication | 15:37 |
| clarkb | I'll let you know when replication is done | 15:37 |
| clarkb | and in the meantime we continue to dig into likely causes | 15:37 |
| jpena | thanks clarkb | 15:42 |
| *** priteau has joined #openstack-infra | 15:46 | |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Trigger service-eavesdrop when gerritbot channels change https://review.opendev.org/746168 | 15:52 |
| *** mgoddard has quit IRC | 15:54 | |
| *** mgoddard has joined #openstack-infra | 16:01 | |
| *** eolivare_ has quit IRC | 16:10 | |
| *** jpena is now known as jpena|off | 16:12 | |
| *** rlandy is now known as rlandy|ruck | 16:22 | |
| *** psachin has quit IRC | 16:25 | |
| *** marios|ruck is now known as marios|out | 16:26 | |
| *** lucasagomes has quit IRC | 16:33 | |
| *** Lucas_Gray has quit IRC | 16:36 | |
| *** gyee has joined #openstack-infra | 16:40 | |
| *** ykarel is now known as ykarel|away | 16:42 | |
| *** ociuhandu_ has quit IRC | 16:43 | |
| *** tosky has quit IRC | 16:43 | |
| *** ociuhandu has joined #openstack-infra | 16:44 | |
| *** marios|out has quit IRC | 16:44 | |
| *** ociuhandu has quit IRC | 16:50 | |
| *** aluria has quit IRC | 16:54 | |
| *** aluria has joined #openstack-infra | 16:56 | |
| *** pkopec has quit IRC | 16:56 | |
| *** ociuhandu has joined #openstack-infra | 16:57 | |
| *** ociuhandu has quit IRC | 17:04 | |
| *** ykarel|away has quit IRC | 17:10 | |
| *** gfidente has quit IRC | 17:14 | |
| *** JayF has quit IRC | 17:15 | |
| *** andrewbonney has quit IRC | 17:31 | |
| *** priteau has quit IRC | 17:36 | |
| *** priteau has joined #openstack-infra | 17:45 | |
| *** priteau has quit IRC | 17:53 | |
| clarkb | jpena|off: should be all synced up now | 18:20 |
| openstackgerrit | Merged openstack/project-config master: Trigger service-eavesdrop when gerritbot channels change https://review.opendev.org/746168 | 18:36 |
| *** JayF has joined #openstack-infra | 18:55 | |
| openstackgerrit | Merged openstack/project-config master: Gerritbot: only comment on stable:follows-policy repos https://review.opendev.org/744947 | 18:59 |
| *** openstackgerrit has quit IRC | 19:02 | |
| fungi | looks like it restarted! | 19:06 |
| fungi | container init running since 19:02 according to ps | 19:07 |
| *** hashar has joined #openstack-infra | 19:10 | |
| *** tosky has joined #openstack-infra | 20:10 | |
| *** yamamoto has joined #openstack-infra | 20:13 | |
| *** yamamoto has quit IRC | 20:18 | |
| *** dave-mccowan has joined #openstack-infra | 20:23 | |
| *** slaweq has quit IRC | 20:37 | |
| *** slaweq has joined #openstack-infra | 20:43 | |
| *** slaweq has quit IRC | 20:48 | |
| *** rfolco has quit IRC | 20:51 | |
| *** rchanter has joined #openstack-infra | 20:55 | |
| *** rlandy|ruck has quit IRC | 21:50 | |
| *** yamamoto has joined #openstack-infra | 21:55 | |
| *** mgoddard has quit IRC | 22:00 | |
| *** rcernin has joined #openstack-infra | 22:00 | |
| *** mgoddard has joined #openstack-infra | 22:05 | |
| *** armax has quit IRC | 22:07 | |
| *** rfolco has joined #openstack-infra | 22:12 | |
| melwitt | I notice e-r indexing is quite far behind http://status.openstack.org/elastic-recheck is/was there an incident? do we expect it to catch up? | 22:13 |
| fungi | clarkb mentioned a recent outage where we lost too many stripes to be able to reconstruct the data which was there, but i would have expected that had mostly aged out by now | 22:17 |
| *** rchanter has quit IRC | 22:20 | |
| *** rchanter has joined #openstack-infra | 22:23 | |
| melwitt | ack | 22:28 |
| *** openstackgerrit has joined #openstack-infra | 22:29 | |
| openstackgerrit | Merged openstack/project-config master: Create pyca/infra https://review.opendev.org/746014 | 22:29 |
| clarkb | its been struggling lately and it semes like we fix one thing and another happens | 22:29 |
| clarkb | I'm guessing there is another giant log file that is killing things | 22:30 |
| clarkb | neutron had been profiling a function call in logs and that function call was called a lot but I think it got fixed | 22:30 |
| *** yamamoto has quit IRC | 22:31 | |
| melwitt | ack, thanks | 22:39 |
| *** armax has joined #openstack-infra | 22:49 | |
| *** d34dh0r53 has quit IRC | 22:58 | |
| *** tkajinam has joined #openstack-infra | 22:58 | |
| *** ociuhandu has joined #openstack-infra | 23:00 | |
| *** ociuhandu has quit IRC | 23:05 | |
| *** tosky has quit IRC | 23:09 | |
| *** hashar has quit IRC | 23:53 | |
| *** hashar has joined #openstack-infra | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!