| *** tosky has quit IRC | 00:03 | |
| *** matt_kosut has joined #openstack-infra | 00:03 | |
| *** jamesmcarthur has joined #openstack-infra | 00:06 | |
| *** matt_kosut has quit IRC | 00:08 | |
| *** jamesmcarthur has quit IRC | 00:20 | |
| *** jamesmcarthur has joined #openstack-infra | 00:24 | |
| *** jamesmcarthur has quit IRC | 00:34 | |
| *** dwilde has quit IRC | 00:40 | |
| *** zzzeek has quit IRC | 00:44 | |
| *** zzzeek has joined #openstack-infra | 00:46 | |
| *** jamesmcarthur has joined #openstack-infra | 01:00 | |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** jamesmcarthur has joined #openstack-infra | 01:10 | |
| *** armax has joined #openstack-infra | 01:44 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** jamesmcarthur has joined #openstack-infra | 01:50 | |
| *** jamesmcarthur has quit IRC | 01:55 | |
| *** jamesmcarthur has joined #openstack-infra | 02:00 | |
| *** matt_kosut has joined #openstack-infra | 02:04 | |
| *** matt_kosut has quit IRC | 02:09 | |
| *** soniya29 has quit IRC | 02:17 | |
| *** benj_ has quit IRC | 02:18 | |
| *** ramishra has quit IRC | 02:18 | |
| *** benj_ has joined #openstack-infra | 02:19 | |
| *** yamamoto_ has joined #openstack-infra | 02:19 | |
| *** yamamoto has quit IRC | 02:19 | |
| *** sshnaidm has quit IRC | 02:19 | |
| *** jamesmcarthur has quit IRC | 02:21 | |
| *** jamesmcarthur has joined #openstack-infra | 02:22 | |
| *** soniya29 has joined #openstack-infra | 02:22 | |
| *** amoralej|off has quit IRC | 02:23 | |
| *** tobberydberg has quit IRC | 02:23 | |
| *** jamesmcarthur has quit IRC | 02:26 | |
| *** sshnaidm has joined #openstack-infra | 02:29 | |
| *** tobberydberg has joined #openstack-infra | 02:35 | |
| *** rcernin has quit IRC | 02:41 | |
| *** jamesmcarthur has joined #openstack-infra | 02:52 | |
| *** rcernin has joined #openstack-infra | 02:58 | |
| *** rcernin has quit IRC | 03:03 | |
| *** rcernin has joined #openstack-infra | 03:03 | |
| *** jamesmcarthur has quit IRC | 03:08 | |
| *** jamesmcarthur has joined #openstack-infra | 03:08 | |
| *** svyas has joined #openstack-infra | 03:22 | |
| *** soniya29 has quit IRC | 03:22 | |
| *** tomaw_ has quit IRC | 03:22 | |
| *** tomaw has joined #openstack-infra | 03:22 | |
| *** zzzeek has quit IRC | 03:22 | |
| *** stevebaker has quit IRC | 03:22 | |
| *** zzzeek has joined #openstack-infra | 03:23 | |
| *** ricolin_ has joined #openstack-infra | 03:25 | |
| *** psachin has joined #openstack-infra | 03:33 | |
| *** stevebaker has joined #openstack-infra | 03:48 | |
| *** ramishra has joined #openstack-infra | 04:00 | |
| *** matt_kosut has joined #openstack-infra | 04:05 | |
| *** matt_kosut has quit IRC | 04:09 | |
| *** ykarel has joined #openstack-infra | 04:24 | |
| *** vishalmanchanda has joined #openstack-infra | 05:10 | |
| *** zzzeek has quit IRC | 05:10 | |
| *** zzzeek has joined #openstack-infra | 05:14 | |
| *** psachin has quit IRC | 05:18 | |
| *** psachin has joined #openstack-infra | 05:22 | |
| *** jamesmcarthur has quit IRC | 05:24 | |
| *** jamesmcarthur has joined #openstack-infra | 05:29 | |
| *** jamesmcarthur has quit IRC | 05:34 | |
| *** jamesmcarthur has joined #openstack-infra | 05:52 | |
| *** ykarel has quit IRC | 05:58 | |
| *** ykarel has joined #openstack-infra | 06:05 | |
| *** matt_kosut has joined #openstack-infra | 06:06 | |
| *** matt_kosut has quit IRC | 06:09 | |
| *** matt_kosut has joined #openstack-infra | 06:09 | |
| *** jamesmcarthur has quit IRC | 06:12 | |
| *** jamesmcarthur has joined #openstack-infra | 06:18 | |
| *** jamesmcarthur has quit IRC | 06:23 | |
| *** dychen has joined #openstack-infra | 06:31 | |
| *** dchen has quit IRC | 06:33 | |
| *** jamesmcarthur has joined #openstack-infra | 06:35 | |
| *** jamesmcarthur has quit IRC | 06:49 | |
| *** jamesmcarthur has joined #openstack-infra | 06:52 | |
| *** jamesmcarthur has quit IRC | 06:56 | |
| ykarel | hberaud, frickler now i see Forbidden https://tarballs.opendev.org/openstack/os-refresh-config/, is this a new issue now? | 07:04 |
|---|---|---|
| *** lpetrut has joined #openstack-infra | 07:14 | |
| *** jamesmcarthur has joined #openstack-infra | 07:20 | |
| *** AJaeger has joined #openstack-infra | 07:27 | |
| *** ralonsoh has joined #openstack-infra | 07:28 | |
| Jeffrey4l | infra, https://docs.openstack.org/ is down? | 07:29 |
| *** jamesmcarthur has quit IRC | 07:29 | |
| *** jamesmcarthur has joined #openstack-infra | 07:29 | |
| *** jamesmcarthur has quit IRC | 07:29 | |
| *** jamesmcarthur has joined #openstack-infra | 07:30 | |
| AJaeger | Yes, it is. | 07:34 |
| AJaeger | (or better: our backing store is down) | 07:34 |
| Jeffrey4l | got, thanks | 07:35 |
| *** jamesmcarthur has quit IRC | 07:35 | |
| *** jcapitao has joined #openstack-infra | 07:35 | |
| *** ykarel_ has joined #openstack-infra | 07:45 | |
| *** lpetrut_ has joined #openstack-infra | 07:45 | |
| *** ykarel has quit IRC | 07:45 | |
| *** lpetrut_ has quit IRC | 07:45 | |
| *** ykarel_ is now known as ykarel | 07:45 | |
| *** lpetrut_ has joined #openstack-infra | 07:46 | |
| *** lpetrut has quit IRC | 07:48 | |
| *** danpawlik5 has quit IRC | 07:51 | |
| *** Ajohn has joined #openstack-infra | 07:51 | |
| ykarel | Ok content visible now, but taraballs still missing | 07:52 |
| ykarel | for python-cinderlib-3.0.0, os-refresh-config-12.0.0, ansible-tripleo-ipsec-10.0.0, os-collect-config-12.0.0 | 07:53 |
| *** dpawlik7 has joined #openstack-infra | 07:53 | |
| *** sboyron has joined #openstack-infra | 07:53 | |
| *** ociuhandu has joined #openstack-infra | 07:53 | |
| *** rcernin has quit IRC | 07:53 | |
| *** JanZerebecki[m] has joined #openstack-infra | 07:56 | |
| *** ricolin_ has quit IRC | 07:57 | |
| *** rcernin has joined #openstack-infra | 07:57 | |
| *** ykarel_ has joined #openstack-infra | 07:58 | |
| *** ociuhandu has quit IRC | 07:58 | |
| *** rpittau|afk is now known as rpittau | 07:59 | |
| *** AJaeger has quit IRC | 07:59 | |
| *** jamesmcarthur has joined #openstack-infra | 08:00 | |
| *** ykarel has quit IRC | 08:01 | |
| *** dciabrin has joined #openstack-infra | 08:01 | |
| *** ykarel_ is now known as ykarel | 08:03 | |
| *** slaweq has joined #openstack-infra | 08:04 | |
| *** jamesmcarthur has quit IRC | 08:05 | |
| *** andrewbonney has joined #openstack-infra | 08:16 | |
| *** jamesmcarthur has joined #openstack-infra | 08:20 | |
| *** eolivare has joined #openstack-infra | 08:24 | |
| *** dtantsur|afk is now known as dtantsur | 08:32 | |
| ykarel | frickler, ^ isn't due to a node failure yesterday we are a bit behind with syncing afs volumes. hopefully should get resolved later today finshed? | 08:32 |
| *** tosky has joined #openstack-infra | 08:44 | |
| *** mgoddard has joined #openstack-infra | 08:46 | |
| frickler | ykarel: seems like sadly this still isn't finished, no. but let's keep that topic in #opendev in order to avoid repeating things | 08:47 |
| *** gfidente has joined #openstack-infra | 08:47 | |
| *** matt_kosut has quit IRC | 08:50 | |
| *** matt_kosut has joined #openstack-infra | 08:50 | |
| *** hashar has joined #openstack-infra | 08:52 | |
| *** jpena|off is now known as jpena | 08:56 | |
| *** lucasagomes has joined #openstack-infra | 08:56 | |
| *** matt_kosut has quit IRC | 08:58 | |
| *** ociuhandu has joined #openstack-infra | 09:01 | |
| *** yamamoto_ has quit IRC | 09:01 | |
| *** amoralej has joined #openstack-infra | 09:08 | |
| amoralej | it seems that nodepool images are failing to build because of lack of space somewhere? https://nb01.opendev.org/ | 09:09 |
| amoralej | https://nb01.opendev.org/centos-8-stream-0000086819.log | 09:10 |
| amoralej | and it seems it's trying to rebuild all in a loop | 09:10 |
| *** rcernin has quit IRC | 09:13 | |
| ysandeep | #openstack-infra is their an ongoing issue with mirrors? | 09:13 |
| ysandeep | I see multiple jobs failing to reach mirror - Failed to fetch https://mirror-int.dfw.rax.opendev.org/ubuntu/dists/bionic/main/binary-amd64/Packages 403 Forbidden | 09:13 |
| ysandeep | https://e3407683a5f2e0a0f3ab-465cb76ef00671c120c8a8ba2200c18d.ssl.cf2.rackcdn.com/771157/3/check/openstack-tox-py36/882aa09/job-output.txt | 09:13 |
| *** matt_kosut has joined #openstack-infra | 09:14 | |
| *** matt_kosut has quit IRC | 09:24 | |
| *** rcernin has joined #openstack-infra | 09:24 | |
| *** derekh has joined #openstack-infra | 09:38 | |
| *** yamamoto has joined #openstack-infra | 09:39 | |
| *** yamamoto has quit IRC | 09:39 | |
| *** yamamoto has joined #openstack-infra | 09:40 | |
| *** zxiiro has quit IRC | 09:47 | |
| *** yamamoto has quit IRC | 09:47 | |
| *** yamamoto has joined #openstack-infra | 09:47 | |
| *** yamamoto has quit IRC | 09:48 | |
| *** matt_kosut has joined #openstack-infra | 09:52 | |
| *** ysandeep is now known as ysandeep|afk | 09:53 | |
| *** matt_kosut has quit IRC | 09:54 | |
| *** matt_kosut has joined #openstack-infra | 09:54 | |
| *** matt_kosut has quit IRC | 09:59 | |
| *** ociuhandu has quit IRC | 10:00 | |
| *** ociuhandu has joined #openstack-infra | 10:01 | |
| *** ociuhandu has quit IRC | 10:01 | |
| *** ociuhandu has joined #openstack-infra | 10:01 | |
| *** yamamoto has joined #openstack-infra | 10:15 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** yamamoto has joined #openstack-infra | 10:15 | |
| *** yamamoto has quit IRC | 10:20 | |
| rpittau | I gues the "forbidden" issue is also known for the docs pages | 10:49 |
| ykarel | frickler, sure, Thanks | 10:49 |
| *** ociuhandu has quit IRC | 10:53 | |
| *** ociuhandu has joined #openstack-infra | 10:53 | |
| *** matt_kosut has joined #openstack-infra | 10:54 | |
| *** icey has joined #openstack-infra | 10:59 | |
| *** icey has quit IRC | 10:59 | |
| *** ociuhandu has quit IRC | 11:00 | |
| *** icey has joined #openstack-infra | 11:00 | |
| *** ociuhandu has joined #openstack-infra | 11:01 | |
| *** dpawlik7 has quit IRC | 11:06 | |
| *** ociuhandu has quit IRC | 11:06 | |
| *** ociuhandu has joined #openstack-infra | 11:07 | |
| *** dpawlik7 has joined #openstack-infra | 11:09 | |
| *** ociuhandu has quit IRC | 11:13 | |
| *** ociuhandu has joined #openstack-infra | 11:17 | |
| *** ociuhandu has quit IRC | 11:34 | |
| *** ociuhandu has joined #openstack-infra | 11:35 | |
| *** xek has joined #openstack-infra | 11:45 | |
| sean-k-mooney | clarkb: corvus just had someone hit the fact fedora 33 blocked sha1 based keys for ssh | 11:45 |
| sean-k-mooney | any plans to update the gerrit ones so that f33 works out of the box | 11:46 |
| sean-k-mooney | https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 | 11:46 |
| sean-k-mooney | i belive this is caused by the host key not the clinet key | 11:47 |
| sean-k-mooney | but not sure if gerrit allows you to update that | 11:47 |
| *** ociuhandu has quit IRC | 11:49 | |
| *** ociuhandu has joined #openstack-infra | 11:49 | |
| sean-k-mooney | its pretty easy to work around just adding PubkeyAcceptedKeyTypes ssh-rsa to your ssh config for review.opendev.org | 11:50 |
| sean-k-mooney | but it would be nice if we could rotate them to not need that | 11:50 |
| *** jcapitao is now known as jcapitao_lunch | 11:54 | |
| *** ociuhandu has quit IRC | 11:54 | |
| *** ysandeep|afk is now known as ysandeep | 11:55 | |
| *** yamamoto has joined #openstack-infra | 11:55 | |
| *** ociuhandu has joined #openstack-infra | 12:20 | |
| *** lpetrut__ has joined #openstack-infra | 12:22 | |
| *** yamamoto has quit IRC | 12:24 | |
| *** lpetrut_ has quit IRC | 12:25 | |
| *** yamamoto has joined #openstack-infra | 12:28 | |
| *** jpena is now known as jpena|lunch | 12:28 | |
| *** svyas is now known as soniya29 | 12:29 | |
| *** rcernin has quit IRC | 12:36 | |
| *** ociuhandu has quit IRC | 12:38 | |
| *** ociuhandu has joined #openstack-infra | 12:38 | |
| *** eandersson has quit IRC | 12:44 | |
| *** eandersson has joined #openstack-infra | 12:44 | |
| *** Wasaac has quit IRC | 12:44 | |
| *** ociuhandu has quit IRC | 12:46 | |
| *** bhagyashris has quit IRC | 12:47 | |
| *** bhagyashris has joined #openstack-infra | 12:47 | |
| *** sshnaidm is now known as sshnaidm|ruck | 12:52 | |
| *** Wasaac has joined #openstack-infra | 12:53 | |
| *** rlandy has joined #openstack-infra | 12:54 | |
| *** rcernin has joined #openstack-infra | 12:55 | |
| *** hashar has quit IRC | 12:56 | |
| *** vishalmanchanda has quit IRC | 12:56 | |
| *** tobberydberg has quit IRC | 12:56 | |
| *** jrosser has quit IRC | 12:56 | |
| *** snapiri has quit IRC | 12:56 | |
| *** cgoncalves has quit IRC | 12:56 | |
| *** ysandeep has quit IRC | 12:56 | |
| *** ddurst has quit IRC | 12:56 | |
| *** logan- has quit IRC | 12:56 | |
| *** dychen has quit IRC | 12:56 | |
| *** ociuhandu has joined #openstack-infra | 12:57 | |
| *** hashar has joined #openstack-infra | 12:59 | |
| *** vishalmanchanda has joined #openstack-infra | 12:59 | |
| *** tobberydberg has joined #openstack-infra | 12:59 | |
| *** jrosser has joined #openstack-infra | 12:59 | |
| *** snapiri has joined #openstack-infra | 12:59 | |
| *** cgoncalves has joined #openstack-infra | 12:59 | |
| *** ysandeep has joined #openstack-infra | 12:59 | |
| *** ddurst has joined #openstack-infra | 12:59 | |
| *** logan- has joined #openstack-infra | 12:59 | |
| *** arne_wiebalck has quit IRC | 12:59 | |
| *** logan- has quit IRC | 12:59 | |
| *** logan- has joined #openstack-infra | 13:00 | |
| *** rcernin has quit IRC | 13:00 | |
| *** arne_wiebalck has joined #openstack-infra | 13:01 | |
| *** amoralej is now known as amoralej|lunch | 13:02 | |
| *** wolsen has quit IRC | 13:03 | |
| *** JanZerebecki[m] has quit IRC | 13:03 | |
| *** ociuhandu has quit IRC | 13:05 | |
| *** mordred has quit IRC | 13:05 | |
| fungi | sean-k-mooney: clarkb has filed a bug with upstream gerrit about it. apparently they don't implement the negotiation protocol necessary for clients to discover that rsa-sha2 is supported by the server, and fedora still configures openssh to assume sha1 if ssh-rsa is attempted and no negotiation can be done | 13:08 |
| sean-k-mooney | fungi: ya the fedora behavior i think is a bug | 13:09 |
| fungi | also it's only the sha1 host key hashing that openssh is deprecating, while fedora decided to break sha1 for auth as well | 13:09 |
| sean-k-mooney | but ok | 13:09 |
| sean-k-mooney | well its more they seam to have brokend assl ssh-rsa | 13:10 |
| sean-k-mooney | im not sure it will work if you use sha2 or not | 13:10 |
| sean-k-mooney | it will for now i think | 13:10 |
| *** jcapitao_lunch is now known as jcapitao | 13:10 | |
| sean-k-mooney | but i think they are planning to go to ECDHE, DHE only | 13:10 |
| sean-k-mooney | based on the FUTURE policy they declared | 13:10 |
| fungi | interesting | 13:11 |
| sean-k-mooney | in https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2#Detailed_Description | 13:11 |
| sean-k-mooney | anyway no worries its a 30 second irc converation when it somes up to get people working again | 13:12 |
| fungi | elliptic curve crypto is still a bit green compared to more established and vetted schemes, and the only reason to avoid rsa at this point is the fear that science fiction, er i mean quantum computers, will be able to factor very large primes with ease | 13:12 |
| *** rcernin has joined #openstack-infra | 13:13 | |
| sean-k-mooney | ya although factoring elpictic curve contol points is equally simple in principal with a quantum computer | 13:13 |
| Tengu | (elliptic pub keys are smaller and more convenient to share, also) | 13:13 |
| sean-k-mooney | so mech | 13:13 |
| sean-k-mooney | *meh | 13:13 |
| fungi | Tengu: sure, there is a convenience factor, but that would be a questionable reason to stop supporting rsa | 13:13 |
| sean-k-mooney | Tengu: give its generally of the oder of KB im not sure that matters | 13:14 |
| sean-k-mooney | ECs are faster though you are right | 13:14 |
| Tengu | sure. and breaking things with new policies like it was done lately isn't nice at all ;). anyway. I'm with that ECs team for some years now :) | 13:14 |
| Tengu | sean-k-mooney: faster, in rhel, isn't really useful with the GssAPI set to ON by default X( | 13:15 |
| Tengu | always the first thing I disable on my systems here. | 13:15 |
| sean-k-mooney | hehe i would not know i never us rhel | 13:15 |
| Tengu | well, it's the same on CentOS afaik :) | 13:16 |
| sean-k-mooney | i barely even touch centos or fedora | 13:16 |
| Tengu | and probably fedora | 13:16 |
| Tengu | ah, well. in that case... those are the only ones I know with that stupid option activated by default... | 13:16 |
| sean-k-mooney | yet anouther reason to not like fedora | 13:16 |
| Tengu | my pro + private laptops are on fedora - my wife's tower is on ubuntu.... some of my servers are on centos, some others on fedora... :) | 13:17 |
| sean-k-mooney | im on fedora currently but that likely wont last for much longer | 13:17 |
| Tengu | just a matter of getting the right thing in my ansible playbooks.. | 13:17 |
| Tengu | anyway. this digresses from the initial topic - sorry for that ;) | 13:17 |
| fungi | given the relative ages of ecc and rsa, there's slightly more of a risk that someone will come up with a novel way to unwind ecc, whereas rsa has been around the block a few times already and taken many beatings, which for me is as much of a reason to prefer rsa as qc is a reason to prefer ecc | 13:18 |
| sean-k-mooney | fungi: well i think tye are still going to support RSA keys | 13:18 |
| sean-k-mooney | but require DHE for key exchange | 13:19 |
| *** wolsen has joined #openstack-infra | 13:20 | |
| fungi | it looked like they still had support for rsa-sha2-256 and rsa-sha2-512 enabled, our gerrit server just doesn't provide the negotiation mechanism openssh would need to be able to discover that | 13:21 |
| sean-k-mooney | ya i think they intended that to work but the negociation interaction was not taken into account | 13:22 |
| *** eharney has joined #openstack-infra | 13:25 | |
| sean-k-mooney | i think they would prefer we moved to DHE-RSA-AES256-SHA256 for example instead of rsa-sha2-256" | 13:26 |
| fungi | and upstream openssh plans to make something like rsa-sha2-256 the default once ssh-rsa sha1 is really disabled there | 13:26 |
| fungi | which would have solved this if fedora had done that | 13:27 |
| sean-k-mooney | ya im not sure if they updated the default or not | 13:27 |
| *** rcernin has quit IRC | 13:27 | |
| sean-k-mooney | im still on fedora 32 so dont have the new behavior | 13:27 |
| fungi | apparently they didn't | 13:28 |
| *** jpena|lunch is now known as jpena | 13:28 | |
| fungi | based on our investigation last week | 13:28 |
| *** ociuhandu has joined #openstack-infra | 13:32 | |
| *** wolsen has quit IRC | 13:35 | |
| *** ociuhandu has quit IRC | 13:36 | |
| *** ociuhandu has joined #openstack-infra | 13:36 | |
| *** mordred has joined #openstack-infra | 13:55 | |
| *** zxiiro has joined #openstack-infra | 13:59 | |
| *** zul has joined #openstack-infra | 14:01 | |
| *** lbragstad has joined #openstack-infra | 14:03 | |
| *** wolsen has joined #openstack-infra | 14:05 | |
| *** amoralej|lunch is now known as amoralej | 14:09 | |
| *** lbragstad has quit IRC | 14:16 | |
| *** d34dh0r53 has quit IRC | 14:17 | |
| *** lbragstad has joined #openstack-infra | 14:19 | |
| *** ysandeep is now known as ysandeep|afk | 14:19 | |
| *** lbragstad has quit IRC | 14:20 | |
| *** lbragstad has joined #openstack-infra | 14:20 | |
| *** sboyron has quit IRC | 14:27 | |
| *** d34dh0r53 has joined #openstack-infra | 14:29 | |
| *** ociuhandu has quit IRC | 14:29 | |
| *** ociuhandu has joined #openstack-infra | 14:30 | |
| *** ociuhandu has quit IRC | 14:34 | |
| *** ociuhandu has joined #openstack-infra | 14:34 | |
| *** zbr2 has joined #openstack-infra | 14:35 | |
| *** zbr has quit IRC | 14:37 | |
| *** zbr2 is now known as zbr | 14:37 | |
| *** ysandeep|afk is now known as ysandeep | 14:45 | |
| *** dpawlik7 has quit IRC | 14:52 | |
| *** ykarel has quit IRC | 14:57 | |
| *** nightmare_unreal has joined #openstack-infra | 15:12 | |
| *** psachin has quit IRC | 15:14 | |
| *** dpawlik7 has joined #openstack-infra | 15:17 | |
| *** hberaud has quit IRC | 15:21 | |
| *** dpawlik7 has quit IRC | 15:26 | |
| *** lpetrut__ has quit IRC | 15:27 | |
| *** dpawlik9 has joined #openstack-infra | 15:27 | |
| *** dpawlik9 is now known as dpawlik | 15:27 | |
| *** hberaud has joined #openstack-infra | 15:27 | |
| *** ricolin has quit IRC | 15:29 | |
| *** ysandeep is now known as ysandeep|away | 15:32 | |
| *** ociuhandu has quit IRC | 15:32 | |
| *** ociuhandu has joined #openstack-infra | 15:33 | |
| openstackgerrit | Cole Walker proposed openstack/project-config master: Add PTP Notification app to StarlingX https://review.opendev.org/c/openstack/project-config/+/771235 | 15:33 |
| *** ociuhandu has quit IRC | 15:37 | |
| dansmith | fungi: 31s to submit this patch this morning: 1 file changed, 33 insertions(+) | 15:43 |
| dansmith | that was a trivial update to that patch, actually, not even creating a new review | 15:43 |
| fungi | dansmith: moments ago? | 15:46 |
| fungi | looks like we're hitting a pretty large load spike again just now: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=26&rra_id=all | 15:47 |
| dansmith | fungi: moments before I said that, so 7:43ish PST | 15:47 |
| fungi | yeah, looks like system load shot up over 30 in th elast 5-minute sample | 15:47 |
| dansmith | how big is that worker? average of 9 ain't peanuts | 15:48 |
| fungi | 16 vcpus | 15:48 |
| dansmith | definitely higher average load than last year before the transition right?, like by a lot | 15:48 |
| dansmith | okay | 15:48 |
| fungi | yeah, system load overall is higher now that it's storing basically all data in git repositories rather than older gerrit which used mysql (which we offloaded to a trove instance) | 15:49 |
| dansmith | yeah, just wondering if (worried that) this is the new normal | 15:50 |
| *** sboyron has joined #openstack-infra | 15:50 | |
| fungi | but most of the performance impact seems to be lock contention, i guess file control in the git repositories | 15:50 |
| dansmith | maybe, but submitting to tempest and glance seem as slow as nova | 15:50 |
| fungi | well, we've done some performance tuning and still have a few more things to try, and are working with upstream gerrit off and on to try to get ideas for how to improve | 15:50 |
| dansmith | can you scale gerrit to multiple machines or is it monolithic? | 15:51 |
| dansmith | I guess you'd need a shared fs now at least | 15:51 |
| clarkb | you can but its largely as active standby aiui | 15:51 |
| dansmith | ack, that's not what I meant by "scale" :P | 15:51 |
| fungi | yeah, the actions are handled by queue worker threads, so pushing to tempest could be blocked by other actions in progress | 15:52 |
| clarkb | well you can do read actions against the standby | 15:52 |
| clarkb | but ya | 15:52 |
| clarkb | there is a very new ha plugin but it seems people have nothing but problems with it reading the mailing list threads about it | 15:52 |
| fungi | but again, high availability is not necessarily the same thing as load distribution either | 15:53 |
| fungi | sometimes you find them together, but they're distinct characteristics of a system | 15:54 |
| dansmith | how much load does zuul and the jobs create on gerrit? I would think not much, | 15:54 |
| dansmith | but we're also headed for deep queues again this morning it looks like | 15:55 |
| dansmith | and just wondering if addressing one with additional paring down of what we run would address both | 15:55 |
| openstackgerrit | Cole Walker proposed openstack/project-config master: Add PTP Notification app to StarlingX https://review.opendev.org/c/openstack/project-config/+/771235 | 15:56 |
| clarkb | it is some load, zuul has to fetch changes as well as query change state and relationships. We've done a fair bit to improve that (asked third party CIs to update Zuul to use a more efficienty query and enabled git v2 protocol in gerrit and on our zuul deployment) | 15:56 |
| fungi | we do think ci systems put a fair amount of strain on the gerrit server, yes. particularly if they're not intelligent about the queries they perform or the frequency with which they perform them | 15:56 |
| clarkb | it definitely seems like the v2 protocol change may have helped a lot | 15:56 |
| dansmith | okay I wasn't eventhinking about external systems, more our own jobs, but that makes sense | 15:58 |
| dansmith | can we isolate workload by account? like, can we see that spikes correlate to lots of activity by the 3rd part CI accounts? | 15:58 |
| *** rlandy is now known as rlandy|mtg | 15:58 | |
| dansmith | I know I'm just complaining and not helping, I'm just.. worried about where we're headed | 15:59 |
| clarkb | yes, that is how we were reaching out and talking to individual third party ci systems to get them to update | 15:59 |
| clarkb | dansmith: I mean I;m worried too | 15:59 |
| clarkb | but we've been head down on this since like august | 15:59 |
| clarkb | and gotten very little help outside our tiny group | 15:59 |
| dansmith | ack | 16:00 |
| clarkb | it does seem we're making improvements, but they are slow going because we have a million other fires to deal with too | 16:00 |
| clarkb | like openafs deciding that january 14 - 31 was a good time to stop working due to a bitmask error | 16:01 |
| dansmith | I guess I keep thinking back to what you said about google running bigger gerrits than us and just wonder what they're doing, other than throwing vertical resources at it | 16:01 |
| *** sboyron has quit IRC | 16:01 | |
| fungi | google's gerrit isn't really gerrit | 16:02 |
| dansmith | clarkb: sorry, I'm really not trying to complain about you .. I don't know what else to do other than to try to understand what the changes are, highlight when I see problems, and maybe help brainstorm | 16:02 |
| fungi | they made their own private changes, like completely different proprietary storage backend | 16:02 |
| clarkb | vertical resources is how most people tackle this aiui | 16:02 |
| clarkb | 256GB memory gerrit installs with 64 cpus are common | 16:02 |
| clarkb | but also people seem to have accepted that new gerrit is slower than old gerrit | 16:03 |
| fungi | hello java ;) | 16:03 |
| clarkb | there are a number of threads on this topic on their mailing list and the consensus seems to be "thats how it is" | 16:03 |
| clarkb | even pushing to gerrit's gerrit is slow | 16:03 |
| dansmith | *sadface* | 16:03 |
| dansmith | what was the killer feature that moving to git-as-db brought? | 16:03 |
| dansmith | because again, that seemed like a great idea on a napkin, but... | 16:03 |
| *** ociuhandu has joined #openstack-infra | 16:04 | |
| clarkb | I don't recall all the motivations when they started it, but I think a lot of it had to do with replication and the active standby stuff I was talking about | 16:04 |
| clarkb | it allows you to sync the system state in a straightforward way | 16:05 |
| dansmith | heh, if we slow this mofo down a lot, we can replicate it easier for... $profit? | 16:05 |
| dansmith | still seems like git as the file backend for that wouldn't have been the right move, but.. okay.. I thought maybe it had to do with some desire to capture comments in the same repo as code or something | 16:06 |
| clarkb | I think that was more of a side effect (that is how it works) | 16:06 |
| clarkb | dansmith: https://bugs.chromium.org/p/gerrit/issues/detail?id=13621 https://bugs.chromium.org/p/gerrit/issues/detail?id=13514 for a fun side effect fungi and I found too | 16:07 |
| fungi | i get the impression folks at google are also not fans of having data in a rdbms | 16:08 |
| fungi | maybe it's the oracle rivalry | 16:08 |
| dansmith | fungi: of course not.. rdbms are boring because they work and people understand them | 16:08 |
| *** ociuhandu has quit IRC | 16:09 | |
| fungi | indeed | 16:10 |
| clarkb | related there is also a thread on the gerrit mailing list on why people are migrating off of gerrit | 16:11 |
| dansmith | ugh | 16:11 |
| clarkb | from a workflow perspective I think gerrit is still the tool that gets things right, but from an operations perspective a lot of people have struggled with it. And I imagine that the maintainers of it running a fork of it don't help much | 16:11 |
| *** sboyron has joined #openstack-infra | 16:11 | |
| dansmith | yeah that seems like a bad sign | 16:12 |
| fungi | though they've basically done that forever | 16:14 |
| clarkb | ya it isn't a new situation, but I think people have grown grumpy over the ops side not improving | 16:14 |
| dansmith | also a bad sign :) | 16:14 |
| clarkb | wikimedia in particular | 16:15 |
| dansmith | "this has never been good enough for us, unforked" | 16:15 |
| *** jamesmcarthur has quit IRC | 16:16 | |
| *** jamesmcarthur has joined #openstack-infra | 16:17 | |
| *** ociuhandu has joined #openstack-infra | 16:29 | |
| *** ociuhandu has quit IRC | 16:33 | |
| *** eolivare_ has joined #openstack-infra | 16:42 | |
| *** eandersson8 has joined #openstack-infra | 16:43 | |
| *** zbr1 has joined #openstack-infra | 16:43 | |
| *** Wasaac has quit IRC | 16:44 | |
| *** slaweq has quit IRC | 16:44 | |
| *** Wasaac has joined #openstack-infra | 16:45 | |
| *** slaweq has joined #openstack-infra | 16:45 | |
| *** zbr has quit IRC | 16:45 | |
| *** eandersson has quit IRC | 16:45 | |
| *** eolivare has quit IRC | 16:45 | |
| *** zzzeek has quit IRC | 16:45 | |
| *** stevebaker has quit IRC | 16:45 | |
| *** ociuhandu has joined #openstack-infra | 16:45 | |
| *** zbr1 is now known as zbr | 16:45 | |
| *** eandersson8 is now known as eandersson | 16:45 | |
| *** zzzeek has joined #openstack-infra | 16:46 | |
| *** hashar has quit IRC | 16:48 | |
| *** rpittau is now known as rpittau|afk | 16:49 | |
| *** ociuhandu has quit IRC | 16:53 | |
| *** rlandy|mtg is now known as rlandy | 16:54 | |
| *** lucasagomes has quit IRC | 16:56 | |
| *** ociuhandu has joined #openstack-infra | 17:00 | |
| *** ociuhandu has quit IRC | 17:01 | |
| *** ociuhandu has joined #openstack-infra | 17:02 | |
| *** takamatsu has quit IRC | 17:02 | |
| *** ociuhandu has quit IRC | 17:02 | |
| *** matt_kosut has quit IRC | 17:04 | |
| *** zul has quit IRC | 17:05 | |
| *** ykarel has joined #openstack-infra | 17:07 | |
| *** matt_kosut has joined #openstack-infra | 17:14 | |
| *** ykarel has quit IRC | 17:15 | |
| *** ociuhandu has joined #openstack-infra | 17:35 | |
| *** jcapitao has quit IRC | 17:36 | |
| *** ociuhandu has quit IRC | 17:40 | |
| *** ociuhandu has joined #openstack-infra | 17:41 | |
| openstackgerrit | Clark Boylan proposed openstack/openstack-zuul-jobs master: Build openafs 1.8.7 on centos https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/771257 | 17:41 |
| *** amoralej is now known as amoralej|off | 17:44 | |
| *** ociuhandu has quit IRC | 17:51 | |
| *** ociuhandu has joined #openstack-infra | 17:53 | |
| *** ociuhandu has quit IRC | 17:58 | |
| *** ralonsoh has quit IRC | 17:58 | |
| *** vishalmanchanda has quit IRC | 17:59 | |
| *** jpena is now known as jpena|off | 18:07 | |
| *** ociuhandu has joined #openstack-infra | 18:22 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Build openafs 1.8.7 on centos https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/771257 | 18:28 |
| *** eolivare_ has quit IRC | 18:30 | |
| *** ociuhandu has quit IRC | 18:35 | |
| *** dwilde has joined #openstack-infra | 18:50 | |
| *** armax has quit IRC | 18:52 | |
| *** ociuhandu has joined #openstack-infra | 18:53 | |
| *** ociuhandu has quit IRC | 19:01 | |
| *** dwilde has quit IRC | 19:06 | |
| *** jamesmcarthur has quit IRC | 19:22 | |
| *** jamesmcarthur has joined #openstack-infra | 19:22 | |
| *** jamesmcarthur has quit IRC | 19:23 | |
| *** jamesmcarthur has joined #openstack-infra | 19:23 | |
| *** andrewbonney has quit IRC | 19:25 | |
| *** jamesmcarthur has quit IRC | 19:25 | |
| *** jamesmcarthur has joined #openstack-infra | 19:26 | |
| *** nightmare_unreal has quit IRC | 19:29 | |
| *** vishalmanchanda has joined #openstack-infra | 19:30 | |
| *** elod has quit IRC | 19:33 | |
| *** elod has joined #openstack-infra | 19:35 | |
| *** ociuhandu has joined #openstack-infra | 19:56 | |
| *** ociuhandu has quit IRC | 20:01 | |
| *** jamesmcarthur has quit IRC | 20:06 | |
| *** jamesmcarthur has joined #openstack-infra | 20:07 | |
| *** jamesmcarthur has quit IRC | 20:12 | |
| *** jamesmcarthur has joined #openstack-infra | 20:15 | |
| *** gfidente is now known as gfidente|afk | 20:21 | |
| *** sboyron has quit IRC | 20:21 | |
| *** slaweq has quit IRC | 20:24 | |
| *** stevebaker has joined #openstack-infra | 20:28 | |
| *** gfidente|afk has quit IRC | 20:56 | |
| *** slaweq has joined #openstack-infra | 20:56 | |
| *** ociuhandu has joined #openstack-infra | 20:58 | |
| *** dciabrin_ has joined #openstack-infra | 21:00 | |
| *** weshay|ruck has joined #openstack-infra | 21:01 | |
| weshay|ruck | hrm.. may have a stuck job in the queue | 21:01 |
| weshay|ruck | 769595,1 | 21:01 |
| weshay|ruck | tripleo-ci-centos-8-undercloud-upgrade-ussuri looks done, but is not closing out | 21:01 |
| *** dciabrin has quit IRC | 21:02 | |
| fungi | did you see it finish? right now zuul isn't connecting to the console stream but that could be because the streaming agent is dead on that particular executor | 21:04 |
| fungi | so the build could also still be running and there's just no console stream | 21:05 |
| fungi | i'll see if i can find any executors missing console streaming processes | 21:06 |
| weshay|ruck | oh | 21:06 |
| weshay|ruck | no.. just looked at the zuul console.. k .. makes sense | 21:07 |
| weshay|ruck | thanks | 21:07 |
| weshay|ruck | thanks fungi | 21:07 |
| fungi | yeah, i have a feeling that build is still in progress and just taking longer than average | 21:07 |
| fungi | but wi;; find out | 21:07 |
| weshay|ruck | k k | 21:07 |
| weshay|ruck | :) | 21:07 |
| * weshay|ruck hoping that queue does NOT reset | 21:07 | |
| sshnaidm|ruck | fungi, I can see it finished: https://eddc062afbf7d654edc5-b2957c5cb819a93ff1b9b4818f5b88e8.ssl.cf5.rackcdn.com/769595/1/gate/tripleo-ci-centos-8-standalone-upgrade-ussuri/5b8ddac/ | 21:09 |
| sshnaidm|ruck | fungi, it appears twice, as running and finished | 21:09 |
| *** ociuhandu has quit IRC | 21:09 | |
| sshnaidm|ruck | fungi, oops, it's a different job, ignore me | 21:10 |
| *** ociuhandu has joined #openstack-infra | 21:10 | |
| *** ociuhandu has quit IRC | 21:13 | |
| *** ociuhandu has joined #openstack-infra | 21:13 | |
| fungi | ze08 and ze12 are both missing a listener on the finger port, so if the build is running from one of those two executors then that's almost certainly the explanation | 21:15 |
| fungi | which i'll know shortly | 21:15 |
| *** jamesmcarthur has quit IRC | 21:16 | |
| fungi | yep, running from ze08, so that's it | 21:16 |
| fungi | i'll make a note to try to restart those two executors at some point when load dies down a bit, don't want to restart 17% of the running builds | 21:17 |
| *** xek has quit IRC | 21:26 | |
| *** ociuhandu has quit IRC | 21:26 | |
| *** jamesmcarthur has joined #openstack-infra | 21:29 | |
| *** jamesmcarthur has quit IRC | 21:36 | |
| *** vishalmanchanda has quit IRC | 22:09 | |
| *** slaweq has quit IRC | 22:20 | |
| *** slaweq has joined #openstack-infra | 22:24 | |
| *** rcernin has joined #openstack-infra | 22:39 | |
| *** slaweq has quit IRC | 23:14 | |
| *** lxkong has quit IRC | 23:28 | |
| *** lxkong has joined #openstack-infra | 23:29 | |
| *** armax has joined #openstack-infra | 23:34 | |
| *** jamesmcarthur has joined #openstack-infra | 23:37 | |
| *** derekh has quit IRC | 23:41 | |
| *** armax has quit IRC | 23:41 | |
| *** jamesmcarthur has quit IRC | 23:42 | |
| *** rlandy has quit IRC | 23:44 | |
| *** dchen has joined #openstack-infra | 23:46 | |
| *** lxkong has quit IRC | 23:48 | |
| *** ociuhandu has joined #openstack-infra | 23:49 | |
| *** dychen has joined #openstack-infra | 23:55 | |
| *** ociuhandu has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!