| *** dwilde has quit IRC | 00:04 | |
| *** tosky has quit IRC | 00:12 | |
| *** mlavalle has quit IRC | 00:15 | |
| ianw | fungi: hrm, i'd have to check if we have that turned on ATM | 00:19 |
|---|---|---|
| ianw | fungi: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/apache-ua-filter/files/ua-filter.conf | 00:25 |
| ianw | semrush is definitely something that's come up before. but i don't think we're explicitly blocking it | 00:26 |
| fungi | yeah, i also think it's just statistical odds that crawlers would be the things which will show up in small cross-sections of our traffic | 01:02 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gitea token test https://review.opendev.org/c/opendev/system-config/+/782887 | 01:08 |
| fungi | ianw: experimenting with auth offload? | 01:14 |
| ianw | fungi: yeah, after reading those bugs on the password hash, it seems like maybe if we use an api token we bypass that? | 01:14 |
| fungi | token auth seemed to be the primary recommendation, yes | 01:15 |
| ianw | shouldn't be too hard. famous last words :) | 01:15 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Remove review-dev https://review.opendev.org/c/opendev/zone-opendev.org/+/782889 | 01:21 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gitea token test https://review.opendev.org/c/opendev/system-config/+/782887 | 01:38 |
| openstackgerrit | Merged opendev/system-config master: Create review-staging group https://review.opendev.org/c/opendev/system-config/+/780698 | 01:42 |
| ianw | infra-root: can we remove opendev-k8s-master opendev-k8s-1,2,3 in vexxhost ca-ymq-1 region? i'm trying to start a new review server but we're out of quota | 02:12 |
| ianw | that woudl be 32 gb ram | 02:13 |
| ianw | Requested 131072, but already used 71680 of 102400 ram | 02:13 |
| ianw | i guess we need more at any rate | 02:14 |
| *** hamalq has quit IRC | 02:24 | |
| ianw | hrm, so it seems you get the token sha only once, from then on the api reports it as blank. i guess this makes sense but it isn't documented | 03:02 |
| *** hemanth_n has joined #opendev | 03:16 | |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add review02.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/782893 | 03:17 |
| *** ykarel|away has joined #opendev | 03:42 | |
| *** gothicserpent has joined #opendev | 03:44 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gitea token test https://review.opendev.org/c/opendev/system-config/+/782887 | 04:20 |
| *** ykarel|away is now known as ykarel | 05:04 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gitea token test https://review.opendev.org/c/opendev/system-config/+/782887 | 05:17 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: launch-node : cap to 8gb swap https://review.opendev.org/c/opendev/system-config/+/782898 | 05:35 |
| ianw | gitea never went above ~6% ram usage with ^^ on local testing here | 05:38 |
| *** ysandeep|away is now known as ysandeep | 05:56 | |
| *** marios has joined #opendev | 06:12 | |
| *** marios has quit IRC | 06:38 | |
| *** fressi has joined #opendev | 06:40 | |
| *** marios has joined #opendev | 06:46 | |
| *** ralonsoh has joined #opendev | 06:49 | |
| *** DSpider has joined #opendev | 06:52 | |
| *** DSpider has quit IRC | 06:54 | |
| *** jaicaa has quit IRC | 07:04 | |
| *** jaicaa has joined #opendev | 07:08 | |
| *** lpetrut has joined #opendev | 07:12 | |
| *** slaweq has joined #opendev | 07:18 | |
| *** tkajinam has quit IRC | 07:33 | |
| *** eolivare has joined #opendev | 07:33 | |
| *** tkajinam has joined #opendev | 07:33 | |
| *** ysandeep is now known as ysandeep|lunch | 07:51 | |
| *** jpena|off is now known as jpena | 07:59 | |
| *** fressi has joined #opendev | 08:09 | |
| *** andrewbonney has joined #opendev | 08:10 | |
| *** amoralej|off is now known as amoralej | 08:11 | |
| *** rpittau|afk is now known as rpittau | 08:13 | |
| *** sboyron has joined #opendev | 08:19 | |
| *** hashar has joined #opendev | 08:25 | |
| *** ykarel_ has joined #opendev | 08:36 | |
| *** ykarel has quit IRC | 08:38 | |
| *** ykarel_ is now known as ykarel | 08:43 | |
| *** tosky has joined #opendev | 08:43 | |
| *** ysandeep|lunch is now known as ysandeep | 08:45 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Auto find greatest Fedora cloud image sub-release https://review.opendev.org/c/openstack/diskimage-builder/+/755992 | 08:50 |
| openstackgerrit | Merged opendev/irc-meetings master: Move the Masakari meeting one hour back https://review.opendev.org/c/opendev/irc-meetings/+/782838 | 08:50 |
| *** ykarel_ has joined #opendev | 08:53 | |
| *** ykarel has quit IRC | 08:55 | |
| openstackgerrit | Merged opendev/irc-meetings master: Remove usused Heat meeting slot https://review.opendev.org/c/opendev/irc-meetings/+/777184 | 09:01 |
| openstackgerrit | Merged opendev/irc-meetings master: Remove usused Charms meeting slot https://review.opendev.org/c/opendev/irc-meetings/+/777188 | 09:01 |
| *** ykarel__ has joined #opendev | 09:02 | |
| *** ykarel_ has quit IRC | 09:05 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Add Fedora 33 support https://review.opendev.org/c/openstack/diskimage-builder/+/755750 | 09:07 |
| *** sshnaidm|afk is now known as sshnaidm | 09:24 | |
| *** ykarel__ has quit IRC | 09:30 | |
| *** ykarel__ has joined #opendev | 09:34 | |
| *** ykarel__ has quit IRC | 09:34 | |
| *** hashar has quit IRC | 10:20 | |
| *** hashar has joined #opendev | 10:39 | |
| *** ykarel has joined #opendev | 10:39 | |
| *** ralonsoh_ has joined #opendev | 10:48 | |
| *** ralonsoh has quit IRC | 10:48 | |
| *** lpetrut has quit IRC | 10:56 | |
| *** dtantsur|afk is now known as dtantsur | 11:35 | |
| fungi | #status log Restarted the haproxy container on gitea-lb01.opendev.org because it had been stuck in 100% cpu consumption since 04:00 UTC | 11:41 |
| openstackstatus | fungi: finished logging | 11:41 |
| *** ysandeep is now known as ysandeep|afk | 11:48 | |
| *** jpena is now known as jpena|lunch | 11:48 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder master: simple-init: support installing Glean from packages https://review.opendev.org/c/openstack/diskimage-builder/+/782299 | 11:49 |
| *** lpetrut has joined #opendev | 11:52 | |
| fungi | and now that haproxy is behaving again, something is destroying one of the backends | 11:54 |
| fungi | will take gitea08 out of the pool | 11:54 |
| *** jonher has joined #opendev | 11:55 | |
| fungi | a similar event hit gitea02 starting at roughly the same time haproxy went crazy | 11:55 |
| *** ysandeep|afk is now known as ysandeep | 11:57 | |
| fungi | and 05 and 06 | 12:00 |
| *** stephenfin has quit IRC | 12:00 | |
| *** ralonsoh_ is now known as ralonsoh | 12:02 | |
| fungi | no overlap in addresses between 02, 05 and 06 between 04:00 and 06:00 so i think we're looking at multiple address sources triggering this | 12:03 |
| *** hashar is now known as hasharLunch | 12:10 | |
| *** hemanth_n has quit IRC | 12:16 | |
| *** rfayan has joined #opendev | 12:23 | |
| *** jpena|lunch is now known as jpena | 12:29 | |
| *** lbragstad has quit IRC | 13:01 | |
| *** amoralej is now known as amoralej|lunch | 13:04 | |
| *** lbragstad has joined #opendev | 13:05 | |
| *** klonn has joined #opendev | 13:09 | |
| *** klonn has quit IRC | 13:14 | |
| *** dwilde has joined #opendev | 13:19 | |
| corvus | ianw: i don't think there's any k8s stuff in use; is probably fine to delete | 13:32 |
| *** amoralej|lunch is now known as amoralej | 14:07 | |
| *** hasharLunch is now known as hashar | 14:09 | |
| mordred | ianwcorvus : ^^ I agree - that _was_ where we were doing k8s for the gitea/ceph cluster | 14:12 |
| openstackgerrit | Slawek Kaplonski proposed openstack/project-config master: Add neutron stable cores +2 powers in stable neutron-lib branches https://review.opendev.org/c/openstack/project-config/+/783011 | 14:14 |
| fungi | gitea08 seems to have recovered, i'm putting it back in the haproxy pool | 14:25 |
| *** mlavalle has joined #opendev | 14:48 | |
| openstackgerrit | Merged openstack/project-config master: Add neutron stable cores +2 powers in stable neutron-lib branches https://review.opendev.org/c/openstack/project-config/+/783011 | 14:49 |
| *** dwilde has quit IRC | 15:04 | |
| *** dwilde has joined #opendev | 15:06 | |
| fungi | memory consumption on 08 is climbing again | 15:17 |
| fungi | i'm going to take it back out of the pool until it calms back down | 15:17 |
| mtreinish | random gerrit question, I just noticed that the Reply-to field in the notification emails looks like it contains every email address that's subscribed to changes on a repo. | 15:23 |
| mtreinish | Was it always like this and I just didn't notice until now, or did something change recently? | 15:23 |
| fungi | that's neat | 15:23 |
| fungi | no idea if that's new in 3.2 | 15:23 |
| mtreinish | ah, yeah I think it's probably a 3.2 change, looking at the user agent string at least in my mailbox the giant reply-to fields started with 'Gerrit/3.2.5.1-60-gbb9ea229fb-dirty' | 15:26 |
| *** fressi has quit IRC | 15:26 | |
| *** ysandeep is now known as ysandeep|dinner | 15:42 | |
| *** ysandeep|dinner is now known as ysandeep | 15:46 | |
| *** mlavalle has quit IRC | 15:48 | |
| *** _mlavalle_1 has joined #opendev | 15:48 | |
| *** ysandeep is now known as ysandeep|dinner | 15:48 | |
| *** lpetrut has quit IRC | 15:57 | |
| *** roman_g has joined #opendev | 15:59 | |
| *** ykarel has quit IRC | 16:24 | |
| *** ykarel has joined #opendev | 16:26 | |
| weshay|ruck | eeek.. can I get infra to peek at a patch in the gate.. 774500,1 | 16:32 |
| weshay|ruck | seems stuck to me, but not sure | 16:32 |
| weshay|ruck | oh.. nvrmind. | 16:32 |
| weshay|ruck | goes to infra channel | 16:32 |
| *** ysandeep|dinner is now known as ysandeep | 16:37 | |
| *** stephenfin has joined #opendev | 16:39 | |
| fungi | yeah, saw your earlier question there, so that's where i've been following up with analysis | 16:40 |
| *** ykarel has quit IRC | 16:40 | |
| *** hamalq has joined #opendev | 16:47 | |
| *** jpena is now known as jpena|off | 16:49 | |
| *** hamalq_ has joined #opendev | 16:49 | |
| *** hamalq has quit IRC | 16:52 | |
| *** rpittau is now known as rpittau|afk | 17:06 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:11 | |
| *** marios is now known as marios|out | 17:15 | |
| *** ysandeep is now known as ysandeep|away | 17:15 | |
| *** otherwiseguy has quit IRC | 17:18 | |
| *** jpena|off has quit IRC | 17:19 | |
| *** janders has quit IRC | 17:19 | |
| *** dmellado has quit IRC | 17:20 | |
| *** mugsie has quit IRC | 17:20 | |
| *** jhesketh has quit IRC | 17:20 | |
| *** Dmitrii-Sh has quit IRC | 17:20 | |
| *** arxcruz has quit IRC | 17:20 | |
| *** amorin has quit IRC | 17:20 | |
| *** gibi has quit IRC | 17:20 | |
| *** osmanlicilegi has quit IRC | 17:20 | |
| *** valleedelisle has quit IRC | 17:20 | |
| *** smcginnis has quit IRC | 17:20 | |
| *** chrome0 has quit IRC | 17:20 | |
| *** JohnnyRa1 has quit IRC | 17:20 | |
| *** ttx has quit IRC | 17:20 | |
| *** logan- has quit IRC | 17:20 | |
| *** mkowalski has quit IRC | 17:20 | |
| *** radez has quit IRC | 17:20 | |
| *** slittle1 has quit IRC | 17:20 | |
| *** mugsie has joined #opendev | 17:20 | |
| *** amorin has joined #opendev | 17:20 | |
| *** ttx has joined #opendev | 17:20 | |
| *** janders has joined #opendev | 17:20 | |
| *** smcginnis has joined #opendev | 17:20 | |
| *** gibi has joined #opendev | 17:20 | |
| *** JohnnyRa1 has joined #opendev | 17:20 | |
| *** jpena|off has joined #opendev | 17:20 | |
| *** chrome0 has joined #opendev | 17:20 | |
| *** Dmitrii-Sh has joined #opendev | 17:20 | |
| *** mkowalski has joined #opendev | 17:20 | |
| *** arxcruz has joined #opendev | 17:20 | |
| *** valleedelisle has joined #opendev | 17:20 | |
| *** osmanlicilegi has joined #opendev | 17:20 | |
| *** otherwiseguy has joined #opendev | 17:20 | |
| *** jhesketh has joined #opendev | 17:20 | |
| *** ttx has quit IRC | 17:20 | |
| *** Eighth_Doctor has quit IRC | 17:20 | |
| *** stevebaker has quit IRC | 17:20 | |
| *** JayF has quit IRC | 17:20 | |
| *** akahat has quit IRC | 17:20 | |
| *** dpawlik has quit IRC | 17:20 | |
| *** calcmandan_ has quit IRC | 17:20 | |
| *** calcmandan has joined #opendev | 17:20 | |
| *** dpawlik has joined #opendev | 17:21 | |
| *** dmellado has joined #opendev | 17:21 | |
| *** JayF has joined #opendev | 17:21 | |
| *** mordred has quit IRC | 17:21 | |
| *** logan- has joined #opendev | 17:21 | |
| *** ttx has joined #opendev | 17:22 | |
| *** slittle1 has joined #opendev | 17:23 | |
| *** akahat has joined #opendev | 17:30 | |
| fungi | i've put gitea08 back into the haproxy pool again | 17:32 |
| fungi | still keeping an eye on all the servers | 17:32 |
| *** JayF has quit IRC | 17:42 | |
| *** hamalq_ has quit IRC | 17:42 | |
| *** tkajinam has quit IRC | 17:42 | |
| *** eolivare has quit IRC | 17:42 | |
| *** cloudnull has quit IRC | 17:42 | |
| *** dmsimard has quit IRC | 17:42 | |
| *** paladox has quit IRC | 17:42 | |
| *** mgagne has quit IRC | 17:42 | |
| *** jmorgan has quit IRC | 17:42 | |
| *** hillpd has quit IRC | 17:42 | |
| *** aprice has quit IRC | 17:42 | |
| *** zaro has quit IRC | 17:42 | |
| *** dmellado has quit IRC | 17:42 | |
| *** dpawlik has quit IRC | 17:42 | |
| *** Dmitrii-Sh has quit IRC | 17:42 | |
| *** chrome0 has quit IRC | 17:42 | |
| *** sboyron has quit IRC | 17:42 | |
| *** sshnaidm|afk has quit IRC | 17:42 | |
| *** icey has quit IRC | 17:42 | |
| *** redrobot has quit IRC | 17:42 | |
| *** SWAT has quit IRC | 17:42 | |
| *** dviroel has quit IRC | 17:42 | |
| *** jroll has quit IRC | 17:42 | |
| *** ildikov has quit IRC | 17:42 | |
| *** bhagyashris has quit IRC | 17:42 | |
| *** eharney has quit IRC | 17:42 | |
| *** stephenfin has quit IRC | 17:42 | |
| *** lbragstad has quit IRC | 17:42 | |
| *** ricolin has quit IRC | 17:42 | |
| *** frickler has quit IRC | 17:42 | |
| *** artom has quit IRC | 17:42 | |
| *** yoctozepto has quit IRC | 17:42 | |
| *** openstackgerrit has quit IRC | 17:42 | |
| *** dhellmann has quit IRC | 17:42 | |
| *** chkumar|ruck has quit IRC | 17:42 | |
| *** jonher has quit IRC | 17:42 | |
| *** prometheanfire has quit IRC | 17:42 | |
| *** noonedeadpunk has quit IRC | 17:42 | |
| *** swest has quit IRC | 17:42 | |
| *** weshay|ruck has quit IRC | 17:42 | |
| *** spotz has quit IRC | 17:42 | |
| *** dardelean has quit IRC | 17:42 | |
| *** akahat has quit IRC | 17:42 | |
| *** slittle1 has quit IRC | 17:42 | |
| *** JohnnyRa1 has quit IRC | 17:42 | |
| *** gothicserpent has quit IRC | 17:42 | |
| *** owalsh has quit IRC | 17:42 | |
| *** zbr|rover has quit IRC | 17:42 | |
| *** mrunge has quit IRC | 17:42 | |
| *** ianw has quit IRC | 17:42 | |
| *** corvus has quit IRC | 17:42 | |
| *** kevinz has quit IRC | 17:42 | |
| *** gouthamr has quit IRC | 17:42 | |
| *** snbuback has quit IRC | 17:42 | |
| *** gmann has quit IRC | 17:42 | |
| *** diablo_rojo_phon has quit IRC | 17:42 | |
| *** portdirect has quit IRC | 17:42 | |
| *** rm_work has quit IRC | 17:42 | |
| *** Open10K8S has quit IRC | 17:42 | |
| *** zbr|rover has joined #opendev | 17:42 | |
| *** mrunge has joined #opendev | 17:42 | |
| *** _mlavalle_1 has quit IRC | 17:42 | |
| *** corvus has joined #opendev | 17:42 | |
| *** paladox has joined #opendev | 17:42 | |
| *** owalsh has joined #opendev | 17:42 | |
| *** stephenfin has joined #opendev | 17:42 | |
| *** eolivare has joined #opendev | 17:43 | |
| *** sshnaidm|afk has joined #opendev | 17:43 | |
| *** sboyron has joined #opendev | 17:43 | |
| *** hamalq has joined #opendev | 17:43 | |
| *** lbragstad has joined #opendev | 17:43 | |
| *** jonher has joined #opendev | 17:43 | |
| *** sboyron has quit IRC | 17:43 | |
| *** prometheanfire has joined #opendev | 17:43 | |
| *** SWAT has joined #opendev | 17:43 | |
| *** chrome0 has joined #opendev | 17:43 | |
| *** lbragstad has quit IRC | 17:43 | |
| *** rm_work has joined #opendev | 17:44 | |
| *** sboyron has joined #opendev | 17:44 | |
| *** swest has joined #opendev | 17:44 | |
| *** tkajinam has joined #opendev | 17:44 | |
| *** ianw has joined #opendev | 17:44 | |
| *** lbragstad has joined #opendev | 17:44 | |
| *** dhellmann has joined #opendev | 17:44 | |
| *** amoralej is now known as amoralej|off | 17:44 | |
| *** dmellado has joined #opendev | 17:44 | |
| *** icey has joined #opendev | 17:44 | |
| *** parallax has quit IRC | 17:45 | |
| *** gmann has joined #opendev | 17:45 | |
| *** bhagyashris has joined #opendev | 17:45 | |
| *** frickler has joined #opendev | 17:47 | |
| *** frickler_ has joined #opendev | 17:47 | |
| frickler | freenode seems to take https://www.openssl.org/news/secadv/20210325.txt rather seriously, makes me wonder whether we should, too | 17:47 |
| *** parallax has joined #opendev | 17:48 | |
| *** gothicserpent has joined #opendev | 17:48 | |
| *** 7YUAAAAAD has joined #opendev | 17:49 | |
| *** frickler_ has quit IRC | 17:49 | |
| *** eolivare has quit IRC | 17:49 | |
| *** akahat has joined #opendev | 17:50 | |
| *** cloudnull has joined #opendev | 17:51 | |
| *** jhesketh has quit IRC | 17:52 | |
| *** mordred has joined #opendev | 17:52 | |
| *** jhesketh has joined #opendev | 17:52 | |
| *** dwilde has quit IRC | 17:53 | |
| *** jmorgan has joined #opendev | 17:54 | |
| *** hashar has quit IRC | 17:57 | |
| *** auristor has quit IRC | 17:57 | |
| *** SotK has quit IRC | 17:57 | |
| *** mwhahaha has quit IRC | 17:57 | |
| *** johnsom has quit IRC | 17:57 | |
| *** TheJulia has quit IRC | 17:57 | |
| *** melwitt has quit IRC | 17:57 | |
| *** CeeMac has quit IRC | 17:57 | |
| *** mnaser has quit IRC | 17:57 | |
| *** clayg has quit IRC | 17:57 | |
| *** TheJulia has joined #opendev | 17:57 | |
| *** mwhahaha has joined #opendev | 17:57 | |
| *** CeeMac has joined #opendev | 17:57 | |
| *** hashar has joined #opendev | 17:57 | |
| *** clayg has joined #opendev | 17:57 | |
| *** auristor has joined #opendev | 17:57 | |
| *** melwitt has joined #opendev | 17:58 | |
| *** jpena|off has quit IRC | 17:58 | |
| *** dtantsur has quit IRC | 17:58 | |
| *** iurygregory has quit IRC | 17:58 | |
| *** donnyd has quit IRC | 17:58 | |
| *** rpittau|afk has quit IRC | 17:58 | |
| *** mattmceuen has quit IRC | 17:58 | |
| *** jrosser has quit IRC | 17:58 | |
| *** erbarr has quit IRC | 17:58 | |
| *** seongsoocho has quit IRC | 17:58 | |
| *** bbezak has quit IRC | 17:58 | |
| *** knikolla has quit IRC | 17:58 | |
| *** clarkb has quit IRC | 17:58 | |
| *** walshh_ has quit IRC | 17:58 | |
| *** fungi has quit IRC | 17:58 | |
| *** SotK has joined #opendev | 17:58 | |
| *** mnaser has joined #opendev | 17:58 | |
| *** johnsom has joined #opendev | 17:58 | |
| *** knikolla has joined #opendev | 17:58 | |
| *** seongsoocho has joined #opendev | 17:58 | |
| *** bbezak has joined #opendev | 17:58 | |
| *** clarkb has joined #opendev | 17:58 | |
| *** dtantsur has joined #opendev | 17:58 | |
| *** erbarr has joined #opendev | 17:58 | |
| *** donnyd has joined #opendev | 17:58 | |
| *** rpittau|afk has joined #opendev | 17:58 | |
| *** jrosser has joined #opendev | 17:58 | |
| *** walshh_ has joined #opendev | 17:58 | |
| *** mattmceuen has joined #opendev | 17:58 | |
| *** jpena|off has joined #opendev | 17:58 | |
| *** fungi has joined #opendev | 17:58 | |
| *** JayF has joined #opendev | 17:58 | |
| *** eharney has joined #opendev | 17:59 | |
| *** openstack has joined #opendev | 18:03 | |
| *** ChanServ sets mode: +o openstack | 18:03 | |
| *** clarkb has quit IRC | 18:03 | |
| *** hashar has quit IRC | 18:03 | |
| *** jmorgan has quit IRC | 18:03 | |
| *** akahat has quit IRC | 18:03 | |
| *** gothicserpent has quit IRC | 18:03 | |
| *** dmellado has quit IRC | 18:03 | |
| *** mnasiadka has quit IRC | 18:03 | |
| *** guilhermesp has quit IRC | 18:03 | |
| *** erbarr has quit IRC | 18:03 | |
| *** donnyd has quit IRC | 18:03 | |
| *** dtantsur has quit IRC | 18:03 | |
| *** jrosser has quit IRC | 18:03 | |
| *** mwhahaha has quit IRC | 18:03 | |
| *** TheJulia has quit IRC | 18:03 | |
| *** parallax has quit IRC | 18:03 | |
| *** mgoddard has quit IRC | 18:03 | |
| *** persia has quit IRC | 18:03 | |
| *** jmorgan has joined #opendev | 18:04 | |
| *** openstackstatus has joined #opendev | 18:04 | |
| *** ChanServ sets mode: +v openstackstatus | 18:04 | |
| *** guilhermesp has joined #opendev | 18:04 | |
| *** hashar has joined #opendev | 18:04 | |
| *** clarkb has joined #opendev | 18:04 | |
| *** jrosser has joined #opendev | 18:04 | |
| *** donnyd has joined #opendev | 18:04 | |
| *** erbarr has joined #opendev | 18:04 | |
| *** dtantsur has joined #opendev | 18:04 | |
| *** mwhahaha has joined #opendev | 18:04 | |
| *** TheJulia has joined #opendev | 18:04 | |
| *** parallax has joined #opendev | 18:04 | |
| *** mgoddard has joined #opendev | 18:04 | |
| *** persia has joined #opendev | 18:04 | |
| *** mnasiadka has joined #opendev | 18:04 | |
| *** dmellado has joined #opendev | 18:04 | |
| *** donnyd has quit IRC | 18:05 | |
| *** erbarr has quit IRC | 18:05 | |
| *** dtantsur has quit IRC | 18:05 | |
| *** jrosser has quit IRC | 18:05 | |
| *** mwhahaha has quit IRC | 18:05 | |
| *** TheJulia has quit IRC | 18:05 | |
| *** parallax has quit IRC | 18:05 | |
| *** mgoddard has quit IRC | 18:05 | |
| *** persia has quit IRC | 18:05 | |
| *** persia has joined #opendev | 18:05 | |
| *** TheJulia has joined #opendev | 18:05 | |
| *** mordred has quit IRC | 18:06 | |
| *** frickler has quit IRC | 18:06 | |
| *** tkajinam has quit IRC | 18:06 | |
| *** gmann has quit IRC | 18:06 | |
| *** Jeffrey4l has quit IRC | 18:06 | |
| *** Jeffrey4l has joined #opendev | 18:06 | |
| *** frickler_ has joined #opendev | 18:06 | |
| *** donnyd has joined #opendev | 18:06 | |
| *** tkajinam has joined #opendev | 18:06 | |
| *** gmann has joined #opendev | 18:07 | |
| *** erbarr has joined #opendev | 18:07 | |
| *** mwhahaha has joined #opendev | 18:07 | |
| *** Guest76033 has joined #opendev | 18:07 | |
| *** jrosser has joined #opendev | 18:08 | |
| *** auristor has joined #opendev | 18:08 | |
| *** andrewbonney has quit IRC | 18:08 | |
| *** dtantsur has joined #opendev | 18:09 | |
| *** gothicserpent has joined #opendev | 18:09 | |
| *** akahat has joined #opendev | 18:10 | |
| fungi | frickler_: yeah, i need to read the advisory more closely, today has been a frenzy | 18:10 |
| *** 7YUAAAAAD is now known as weshay|ruck | 18:11 | |
| *** mordred has joined #opendev | 18:12 | |
| *** ralonsoh has joined #opendev | 18:13 | |
| fungi | #status log Restarted nodepool-launcher container on nl04.openstack.org in hopes of clearing stuck node requests from what looks like brief disruption in ovh-bhs1 around 03:30 UTC | 18:13 |
| openstackstatus | fungi: finished logging | 18:13 |
| *** parallax has joined #opendev | 18:13 | |
| *** dtantsur is now known as dtantsur|afk | 18:13 | |
| *** Eighth_Doctor has joined #opendev | 18:14 | |
| *** ildikov has joined #opendev | 18:15 | |
| *** ralonsoh has quit IRC | 18:15 | |
| *** ttx has quit IRC | 18:16 | |
| *** calcmandan has quit IRC | 18:16 | |
| *** otherwiseguy has quit IRC | 18:16 | |
| *** mkowalski has quit IRC | 18:16 | |
| *** valleedelisle has quit IRC | 18:16 | |
| *** gibi has quit IRC | 18:16 | |
| *** amorin has quit IRC | 18:16 | |
| *** mugsie has quit IRC | 18:16 | |
| *** marios|out has quit IRC | 18:16 | |
| *** gnuoy has quit IRC | 18:16 | |
| *** elod has quit IRC | 18:16 | |
| *** priteau has quit IRC | 18:16 | |
| *** mtreinish has quit IRC | 18:16 | |
| *** amoralej|off has quit IRC | 18:16 | |
| *** tristanC has quit IRC | 18:16 | |
| *** amotoki has quit IRC | 18:16 | |
| *** ozzzo has quit IRC | 18:16 | |
| *** odyssey4me has quit IRC | 18:16 | |
| *** Eighth_Doctor has quit IRC | 18:16 | |
| *** hamalq has quit IRC | 18:16 | |
| *** logan- has quit IRC | 18:16 | |
| *** osmanlicilegi has quit IRC | 18:16 | |
| *** smcginnis has quit IRC | 18:16 | |
| *** janders has quit IRC | 18:16 | |
| *** roman_g has quit IRC | 18:16 | |
| *** rfayan has quit IRC | 18:16 | |
| *** tosky has quit IRC | 18:16 | |
| *** slaweq has quit IRC | 18:16 | |
| *** jaicaa has quit IRC | 18:16 | |
| *** irclogbot_3 has quit IRC | 18:16 | |
| *** jbryce has quit IRC | 18:16 | |
| *** ShadowJonathan has quit IRC | 18:16 | |
| *** gibi has joined #opendev | 18:16 | |
| *** gnuoy has joined #opendev | 18:16 | |
| *** calcmandan has joined #opendev | 18:16 | |
| *** ttx has joined #opendev | 18:16 | |
| *** mugsie has joined #opendev | 18:16 | |
| *** odyssey4me has joined #opendev | 18:16 | |
| *** valleedelisle has joined #opendev | 18:17 | |
| *** amotoki has joined #opendev | 18:17 | |
| *** jaicaa has joined #opendev | 18:17 | |
| *** hamalq has joined #opendev | 18:17 | |
| *** rfayan has joined #opendev | 18:17 | |
| *** mkowalski has joined #opendev | 18:17 | |
| *** tosky has joined #opendev | 18:17 | |
| *** otherwiseguy has joined #opendev | 18:18 | |
| *** osmanlicilegi has joined #opendev | 18:18 | |
| *** mtreinish has joined #opendev | 18:18 | |
| *** logan- has joined #opendev | 18:18 | |
| *** Eighth_Doctor has joined #opendev | 18:19 | |
| *** iurygregory has joined #opendev | 18:20 | |
| *** frickler_ is now known as frickler | 18:20 | |
| *** irclogbot_3 has joined #opendev | 18:20 | |
| *** cgoncalves has joined #opendev | 18:21 | |
| *** smcginnis has joined #opendev | 18:21 | |
| *** tristanC has joined #opendev | 18:21 | |
| *** Eighth_Doctor has quit IRC | 18:29 | |
| *** mordred has quit IRC | 18:29 | |
| *** dwilde has joined #opendev | 18:31 | |
| *** roman_g has joined #opendev | 18:35 | |
| *** elod has joined #opendev | 18:37 | |
| fungi | looks like the irc disruption has concluded, i'm going to restart gerritbot because it seems to think it's joined to at least one channel that it isn't | 18:43 |
| fungi | #status log Restarted gerritbot, since in the wake of IRC netsplits it seems to have forgotten it's not joined to at least some channels | 18:45 |
| openstackstatus | fungi: finished logging | 18:45 |
| *** Eighth_Doctor has joined #opendev | 18:52 | |
| *** mordred has joined #opendev | 18:55 | |
| *** roman_g has quit IRC | 18:55 | |
| *** jralbert has joined #opendev | 19:09 | |
| jralbert | Trying to keep up with the chat logs from eavesdrop, so I'm not sure if this has been raised or not, but we're once again unable to git clone from opendev. We've got ~600 compute nodes to upgrade, and the process is stalled on trying to get source | 19:11 |
| *** mlavalle has joined #opendev | 19:12 | |
| fungi | jralbert: starting around 16:00 utc yesterday we've been getting periodic bursts of traffic overloading our git server backends and load balancer, i've been trying to stay on top of it, but haven't been able to nail down specific source ip addresses triggering the issue yet | 19:16 |
| jralbert | Is it possible that it's us? | 19:16 |
| fungi | jralbert: it's possible. did you start trying yesterday? are you cloning repositories in unusual ways? you're deploying via git checkouts i guess? | 19:17 |
| jralbert | We started the compute node component of our upgrade yesterday | 19:17 |
| jralbert | We're using openstack-ansible, which checks out from git on each node to build python wheels for nova and neutron. We're just working on nova right now | 19:18 |
| fungi | looks like something started hitting us again around 18:55 utc, winding up at the gitea06 backend and overloading it. i'm disabling that in our load balancer pool now | 19:18 |
| jrosser | jralbert: if you can clone the repos locally you can override the URL in the OSA config to pick up your local copies | 19:19 |
| jrosser | or you can search/replace the opendev.org for github in the repo specs | 19:19 |
| jralbert | jrosser: so this would be an effect of osa's move to per-node builds, yes? It used to do these checkouts once on the repo containers and push out built wheels from there? | 19:20 |
| fungi | ideally we should be able to handle git clone operations, but nova takes a lot of ram to serve a complete clone, and our load balancing has to be persisted on source address for now unfortunately, so if tons of requests all come from the same ip address (for example behind a many-to-one nat) they all get directed to the same backend | 19:21 |
| jrosser | jralbert: if you have a repo server in your deployment the wheels are only built once | 19:21 |
| jrosser | there should not be a per-node build at all | 19:21 |
| jralbert | curious: we do have repo containers, but it certainly does look like every node is trying to check out. Would ansible limit affect that? | 19:22 |
| jrosser | i want to say no, but..... :) | 19:22 |
| jralbert | at the scale of our environment, the compute plays take so long to run that we split them out and run them in their own limit after the control plane is done | 19:23 |
| jrosser | jralbert: if you have some wierd behaviour there stick it in a paste and drop it in #openstack-ansible | 19:23 |
| jralbert | I'm not sure what to show you? | 19:24 |
| jrosser | anything that appears to need to clone the upstream repo once per compute node, thats certainly not as intended | 19:24 |
| *** artom has joined #opendev | 19:25 | |
| fungi | looks like we ended up with a fairly large memory spike on gitea04 after i took 06 out of the pool, but it seems to be backing off now | 19:29 |
| fungi | or maybe not, it's erratic. if it's still high for another 5-minute sample i'll take it out temporarily too | 19:29 |
| *** mgoddard has joined #opendev | 19:33 | |
| fungi | yeah, okay, it seems to have stopped in the last 5 minutes so not taking it out of the pool after all | 19:36 |
| fungi | 06 looks like it's going to need a while to get its memory usage back under control though, so leaving it out for now | 19:36 |
| *** rfayan has quit IRC | 19:52 | |
| *** stevebaker has joined #opendev | 19:58 | |
| *** roman_g has joined #opendev | 20:00 | |
| *** zigo has joined #opendev | 20:03 | |
| *** priteau has joined #opendev | 20:36 | |
| *** slaweq has joined #opendev | 20:49 | |
| *** hashar has quit IRC | 21:11 | |
| ianw | corvus / mordred : thanks, i will clean up those resources | 21:19 |
| ianw | fungi: a tale of two gitea tests, told in images https://imgur.com/a/YrlDNcd :) | 21:30 |
| corvus | ianw: for api access? | 21:48 |
| corvus | (for the repo creation/update stuff?) | 21:48 |
| ianw | corvus: yep, the second was switched to use an api token | 22:03 |
| *** ralonsoh has joined #opendev | 22:10 | |
| fungi | ianw: if i know anything about comparing shapes, one of those is definitely bigger than the other | 22:25 |
| *** sboyron has quit IRC | 22:26 | |
| johnsom | Cloning into 'python-designateclient'... | 22:31 |
| johnsom | fatal: unable to access 'https://opendev.org/openstack/python-designateclient.git/': GnuTLS recv error (-110): The TLS connection was non-properly terminated. | 22:31 |
| johnsom | Are we rolling endpoints for the openssl thing? | 22:32 |
| fungi | i don't think so, ot | 22:33 |
| fungi | it's probably an overloaded backend again... checking | 22:33 |
| johnsom | This worked fine about 10 minutes ago, but I can't clone now. | 22:33 |
| fungi | well, i don't see any runaway memory usage on any of the backends, nor cpu slammed on the lb this time | 22:35 |
| fungi | i'm able to clone that | 22:35 |
| fungi | johnsom: can you `echo|openssl s_client -connect opendev.org:https 2>/dev/null|openssl x509 -text|grep Subject:` | 22:36 |
| fungi | and tell me what backend that says | 22:36 |
| fungi | from the same machine where you're getting the error | 22:37 |
| johnsom | Subject: CN = gitea05.opendev.org | 22:37 |
| *** Eighth_Doctor is now known as Conan_Kudo | 22:37 | |
| johnsom | 01 seems to work | 22:37 |
| johnsom | On the host with 05, it won't clone | 22:38 |
| fungi | checking out 05 in closer detail then, thanks | 22:38 |
| fungi | it's certainly slow to log in via ssh | 22:38 |
| fungi | very slow | 22:38 |
| *** Conan_Kudo is now known as Eighth_Doctor | 22:39 | |
| fungi | yikes, load average is 111 | 22:40 |
| fungi | swap thrashing like mad | 22:40 |
| fungi | i wonder why the snmp memory graph in cacti doesn't reflect it | 22:40 |
| fungi | taking it out of the pool now | 22:40 |
| fungi | johnsom: try again, you'll likely hit a different backend now | 22:40 |
| *** Eighth_Doctor is now known as Conan_Kudo | 22:40 | |
| johnsom | Yeah, got 04, works like a charm | 22:41 |
| fungi | hah, now i see, the graph went from all normal and unassuming to unresponsive between 5-minute polls: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66725&rra_id=all | 22:41 |
| fungi | hard to spot that | 22:41 |
| *** Conan_Kudo is now known as Eighth_Doctor | 22:41 | |
| fungi | whatever's been going on, it's picked back up again | 22:42 |
| johnsom | o/ thanks. Back to churning out patches. Let me know if you need anything else from me. | 22:42 |
| fungi | in good news, gitea06 seems to have completely recovered so i've added it back to the pool | 22:42 |
| fungi | johnsom: nope, all set, thanks for letting us know! | 22:43 |
| mordred | fungi: you know - back in my mysql days I would tell clients to disable swap because it was better to have the db server hard-fail than degrade. ... I wonder - now that we have these in containers, instead of disabling swap on the server, maybe we should set a cgroup ram limit on the gitea container | 22:46 |
| mordred | that's just me pondering out loud | 22:46 |
| fungi | also wish haproxy noticed when these fell over, but i think in layer 4 mode it will only notice if connections start being rejected or timing out | 22:47 |
| fungi | so connection tarpits continue to get an equal number of connections sent their way | 22:48 |
| *** janders has joined #opendev | 22:54 | |
| *** openstackgerrit has joined #opendev | 22:55 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gitea token test https://review.opendev.org/c/opendev/system-config/+/782887 | 22:55 |
| fungi | haproxy cpu has gone nuts again. restarting its container now | 22:58 |
| fungi | #status log Restarted haproxy container on gitea-lb01 due to runaway CPU consumption | 22:59 |
| openstackstatus | fungi: finished logging | 22:59 |
| ianw | fungi: sorry, trying to keep up ... this haproxy behaviour is new? | 23:00 |
| fungi | since wednesday, yes | 23:00 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66609&rra_id=all | 23:00 |
| fungi | short story is we see "something" happen which begins to skyrocket memory consumption on one or more backends. minutes later haproxy consumes all available cpu (~6 system ~2 user), probably an interrupt storm | 23:01 |
| fungi | it will continue like that indefinitely until we restart the container apparently, at least it has each time so far | 23:02 |
| fungi | after restarting the haproxy container it returns to normal, but whatever problem connections are happening continue to pound the backends, and taking an exhausted backend out of the pool frequently means one of the others will suddenly see memory skyrocket once that happens | 23:03 |
| fungi | i've been trying to correlate which source addresses are being directed to which backends and track which ones move across backend disablement to new backends exhibiting problems, but so far haven't found a clear culprit | 23:04 |
| *** amorin has joined #opendev | 23:05 | |
| fungi | i'm starting to think there are multiple or changing source addresses involved, or the problem isn't tied to specific requests (though the fact that it moves from a disabled backend to another available backend suggests it is related to connections) | 23:05 |
| ianw | we should really move the haproxy logging into separate files | 23:05 |
| fungi | i've basically just opened cacti graphs for lb cpu and memory on each of the 8 backends and am watching them constantly | 23:06 |
| fungi | right now gitea05 is disabled in haproxy while it calms down from this latest incident, bit the other 7 are taking connections | 23:07 |
| fungi | memory is climbing steeply on gitea08, i may disable it momentarily if it continues | 23:12 |
| johnsom | Are the haproxy connection logs scrolling and are they timestamped behind? | 23:13 |
| fungi | it's being logged to syslog | 23:14 |
| fungi | rsyslog | 23:14 |
| fungi | which is timestamping them | 23:14 |
| johnsom | Yeah, but is it a very high rate per second? Sometime it can get io bound trying to write logs and you will see a system spike like that. | 23:15 |
| fungi | oh, that might be, yeah | 23:15 |
| fungi | though when i log in, it's the haproxy process with high system cpu, not rsyslog | 23:15 |
| fungi | maybe writing to the rsyslog unix socket is the chokepoint there | 23:16 |
| fungi | snmp is definitely reporting it as system not iowait though | 23:17 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: haproxy: write to container log files https://review.opendev.org/c/opendev/system-config/+/783120 | 23:17 |
| fungi | which is why i suspected interrupt handling | 23:17 |
| fungi | i used to see this same sort of cpu utilization pattern on firewalls when they'd get overrun with high volumes of tiny packets (udp datagrams, tcp synfloods, tcp reset reflection attacks, et cetera) | 23:19 |
| fungi | though in this case i have a feeling it's some client malfunctioning in strange ways after the backend its being persisted to turns into a tarpit | 23:19 |
| johnsom | That is why I was curious what your requests per second is. | 23:20 |
| johnsom | This is a updated version of focal? | 23:20 |
| fungi | i'll try to get a count | 23:21 |
| fungi | johnsom: it's haproxy running in a container, so not sure how to answer | 23:21 |
| johnsom | I was just trying to get an idea of the version of haproxy | 23:21 |
| ianw | the host is bionic, HA-Proxy version 2.3.7-2d39ce3 2021/03/16 | 23:22 |
| johnsom | Oh, home grown haproxy then | 23:22 |
| ianw | well, from their container | 23:22 |
| johnsom | ah, ok | 23:23 |
| *** gibi has quit IRC | 23:23 | |
| ianw | https://grafana.opendev.org/d/ZQmopePMz/opendev-load-balancer?orgId=1 is another thing to look at | 23:23 |
| ianw | it definitely shows when some of the servers get unhappy | 23:24 |
| fungi | johnsom: so... cacti is able to poll the load balancer when this is happening... new theory, the tarpitting is causing established connection count to skyrocket past some sane threshold and haporxy goes crazy polling/context switching: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66611&rra_id=all | 23:24 |
| fungi | traffic volume and packet rate are reasonably normal the whole time | 23:24 |
| johnsom | Yeah, looking at those graphs, this is not a connection rate issue or the logging issue. | 23:24 |
| fungi | it would explain why the haproxy cpu utilization trails the memory exhaustion on the backend... takes that long for tarpitted connections to pile up to the point where haproxy can't easily track them | 23:26 |
| johnsom | So, the "HTTPS Server Average Session Time" graph seems to align to the gitea05 issue I saw. | 23:27 |
| fungi | i'd be happier if i could figure out what requests are eating so much memory on the gitea backends | 23:28 |
| johnsom | Yeah, I know a heck of a lot more about haproxy than gitea. lol | 23:30 |
| *** gibi has joined #opendev | 23:30 | |
| johnsom | There is a 2.3.8 build of haproxy from today though I can't point to an obvious fix in the changelog. | 23:31 |
| fungi | new openssl? ;) | 23:33 |
| johnsom | Doesn't look like the container is on dockerhub yet though. | 23:33 |
| johnsom | http://git.haproxy.org/?p=haproxy-2.3.git;a=commit;h=e572195c76ee79c5780adecb2560b6e6e36a7d1e | 23:33 |
| *** roman_g has quit IRC | 23:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!