| dansmith | I know that the answer to this is "of course", | 00:00 |
|---|---|---|
| dansmith | well, | 00:00 |
| dansmith | I'm trying to decide how important I think it is to prove the speedup in the gate workers, vs. relying on my local consistent numbers and pushing to get it into the firehose to prove it out | 00:01 |
| dansmith | I could spend a bunch of time trying to track individual worker runtimes, rechecking like a mad person (which currently means basically one or two data points per day), | 00:01 |
| dansmith | or I could just say "look, it's faster." | 00:02 |
| fungi | i favor bold assertions | 00:02 |
| clarkb | I think if you can show it isn't any slower in the gate that the other outliers but can show its faster locally you could sell it as a local dev improement | 00:02 |
| fungi | but i don't decide what patches merge to devstack ;) | 00:02 |
| dansmith | I can see being super iop constrained being the big deal here, but this is much faster on rust even, without a caching layer and ssds underneath | 00:02 |
| dansmith | do we know if vexx, ovh, etc are on rust or silicon? | 00:03 |
| clarkb | vexx is all ceph on I don't know what. Not sure about any of hte other clouds | 00:03 |
| dansmith | because especially without head thrashing, more in the queue should be better for overall throughput | 00:04 |
| dansmith | with this config, I see multiple places where openstack client is burning 100% cpu across multiples, and things like nova-manage and glance-manage spinning wheels generating sql before mysql ever starts fsyncing() the disk | 00:04 |
| *** jamesmcarthur has joined #openstack-infra | 00:11 | |
| *** jamesmcarthur has quit IRC | 00:40 | |
| *** dychen has joined #openstack-infra | 01:03 | |
| *** dychen has quit IRC | 01:05 | |
| *** armax has joined #openstack-infra | 01:11 | |
| *** dave-mccowan has quit IRC | 01:24 | |
| *** jamesdenton has quit IRC | 01:33 | |
| *** jamesden_ has joined #openstack-infra | 01:33 | |
| *** hamalq has quit IRC | 01:40 | |
| *** armax has quit IRC | 01:44 | |
| *** ricolin_ has joined #openstack-infra | 02:14 | |
| *** lbragstad_ is now known as lbragstad | 02:15 | |
| *** dchen has quit IRC | 02:22 | |
| *** dchen has joined #openstack-infra | 02:23 | |
| *** dychen has joined #openstack-infra | 02:46 | |
| *** matt_kosut has joined #openstack-infra | 03:55 | |
| *** matt_kosut has quit IRC | 04:00 | |
| *** dychen has joined #openstack-infra | 04:04 | |
| *** dychen has quit IRC | 04:17 | |
| *** ricolin_ has quit IRC | 04:20 | |
| *** ajitha has joined #openstack-infra | 04:24 | |
| *** ricolin has joined #openstack-infra | 04:24 | |
| *** yamamoto_ has joined #openstack-infra | 04:51 | |
| *** yamamoto has quit IRC | 04:52 | |
| *** ykarel has joined #openstack-infra | 04:52 | |
| *** gfidente|afk has quit IRC | 04:53 | |
| *** ricolin has quit IRC | 05:11 | |
| *** Ajohn has quit IRC | 05:19 | |
| *** ricolin has joined #openstack-infra | 05:24 | |
| *** zzzeek has quit IRC | 05:41 | |
| *** zzzeek has joined #openstack-infra | 05:42 | |
| *** matt_kosut has joined #openstack-infra | 05:57 | |
| *** matt_kosut has quit IRC | 05:58 | |
| *** vishalmanchanda has joined #openstack-infra | 05:58 | |
| *** matt_kosut has joined #openstack-infra | 05:58 | |
| *** ysandeep|away is now known as ysandeep | 06:06 | |
| *** rcernin has quit IRC | 06:07 | |
| *** ykarel_ has joined #openstack-infra | 06:16 | |
| *** ykarel has quit IRC | 06:19 | |
| *** nightmare_unreal has joined #openstack-infra | 06:41 | |
| *** ramishra has quit IRC | 06:45 | |
| *** gyee has quit IRC | 06:46 | |
| openstackgerrit | Rico Lin proposed openstack/project-config master: Add ubuntu bionic and focal with xxxlarge flavor https://review.opendev.org/c/openstack/project-config/+/771565 | 07:02 |
| *** ramishra has joined #openstack-infra | 07:08 | |
| *** lpetrut has joined #openstack-infra | 07:10 | |
| *** tkajinam_ has joined #openstack-infra | 07:19 | |
| *** tkajinam has quit IRC | 07:20 | |
| *** ykarel_ is now known as ykarel | 07:27 | |
| *** dklyle has quit IRC | 07:31 | |
| *** eolivare has joined #openstack-infra | 07:39 | |
| *** ralonsoh has joined #openstack-infra | 07:41 | |
| *** jcapitao has joined #openstack-infra | 07:49 | |
| *** jpena|off is now known as jpena | 07:52 | |
| *** sboyron has joined #openstack-infra | 07:57 | |
| *** slaweq has joined #openstack-infra | 07:58 | |
| *** ociuhandu has joined #openstack-infra | 08:00 | |
| *** ociuhandu has quit IRC | 08:05 | |
| *** jamesden_ has quit IRC | 08:05 | |
| *** jamesdenton has joined #openstack-infra | 08:06 | |
| *** amoralej|off is now known as amoralej | 08:07 | |
| *** lucasagomes has joined #openstack-infra | 08:08 | |
| *** rpittau|afk is now known as rpittau | 08:11 | |
| *** sboyron has quit IRC | 08:14 | |
| *** tosky has joined #openstack-infra | 08:16 | |
| *** gfidente has joined #openstack-infra | 08:22 | |
| *** hashar has joined #openstack-infra | 08:23 | |
| *** sboyron has joined #openstack-infra | 08:24 | |
| *** andrewbonney has joined #openstack-infra | 08:24 | |
| *** xek has joined #openstack-infra | 08:31 | |
| *** sboyron has quit IRC | 08:37 | |
| *** ociuhandu has joined #openstack-infra | 08:58 | |
| *** ociuhandu has quit IRC | 09:04 | |
| *** ociuhandu has joined #openstack-infra | 09:10 | |
| *** ociuhandu has quit IRC | 09:14 | |
| *** jaicaa has quit IRC | 09:16 | |
| *** jaicaa has joined #openstack-infra | 09:18 | |
| *** matt_kos_ has joined #openstack-infra | 09:34 | |
| *** lpetrut_ has joined #openstack-infra | 09:34 | |
| *** xek_ has joined #openstack-infra | 09:34 | |
| *** ykarel_ has joined #openstack-infra | 09:34 | |
| *** tosky has quit IRC | 09:36 | |
| *** matt_kosut has quit IRC | 09:36 | |
| *** tosky_ has joined #openstack-infra | 09:36 | |
| *** xek has quit IRC | 09:37 | |
| *** lpetrut has quit IRC | 09:37 | |
| *** ykarel has quit IRC | 09:37 | |
| *** ykarel_ is now known as ykarel | 09:39 | |
| *** yamamoto_ has quit IRC | 09:39 | |
| *** tosky_ is now known as tosky | 09:43 | |
| *** derekh has joined #openstack-infra | 09:46 | |
| *** jamesdenton has quit IRC | 10:00 | |
| *** jamesdenton has joined #openstack-infra | 10:01 | |
| *** ysandeep is now known as ysandeep|afk | 10:14 | |
| *** sboyron has joined #openstack-infra | 10:14 | |
| *** ociuhandu has joined #openstack-infra | 10:18 | |
| *** yamamoto has joined #openstack-infra | 10:22 | |
| *** yamamoto has quit IRC | 10:28 | |
| *** lamt has quit IRC | 10:43 | |
| *** hashar is now known as hasharAway | 10:53 | |
| *** dchen has quit IRC | 10:58 | |
| *** dtantsur|afk is now known as dtantsur | 10:58 | |
| *** rcernin has joined #openstack-infra | 11:23 | |
| *** jcapitao is now known as jcapitao_lunch | 11:26 | |
| *** ysandeep|afk is now known as ysandeep | 11:36 | |
| *** yamamoto has joined #openstack-infra | 11:56 | |
| *** rcernin has quit IRC | 11:57 | |
| *** ricolin has quit IRC | 12:06 | |
| *** jamesdenton has quit IRC | 12:18 | |
| *** jamesdenton has joined #openstack-infra | 12:19 | |
| *** jpena is now known as jpena|lunch | 12:30 | |
| *** jcapitao_lunch is now known as jcapitao | 12:39 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** rlandy has joined #openstack-infra | 12:46 | |
| *** ociuhandu has quit IRC | 12:49 | |
| openstackgerrit | Rico Lin proposed openstack/project-config master: Add ubuntu bionic and focal with xxxlarge flavor https://review.opendev.org/c/openstack/project-config/+/771565 | 12:55 |
| *** tosky has quit IRC | 13:06 | |
| *** tosky has joined #openstack-infra | 13:06 | |
| *** ricolin has joined #openstack-infra | 13:08 | |
| *** amoralej is now known as amoralej|lunch | 13:09 | |
| *** yamamoto has joined #openstack-infra | 13:13 | |
| ricolin | mnaser, fungi can you help on review https://review.opendev.org/c/openstack/project-config/+/771565 we need it as experiment on arm64+devstack job with larger cpu set | 13:16 |
| *** yamamoto has quit IRC | 13:18 | |
| *** yamamoto has joined #openstack-infra | 13:18 | |
| *** ociuhandu has joined #openstack-infra | 13:23 | |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** owalsh has quit IRC | 13:48 | |
| *** amoralej|lunch is now known as amoralej | 14:05 | |
| *** owalsh has joined #openstack-infra | 14:08 | |
| *** ykarel is now known as ykarel|mtg | 14:10 | |
| *** systemd is now known as systemc | 14:10 | |
| *** markmcclain has quit IRC | 14:26 | |
| fungi | ricolin: i've approved it, but keep in mind we only have one provider for those nodes, and not much quota | 14:30 |
| fungi | you may end up with node_error results or builds queued indefinitely under some circumstances | 14:31 |
| *** vishalmanchanda has quit IRC | 14:38 | |
| *** dave-mccowan has joined #openstack-infra | 14:39 | |
| *** zul has joined #openstack-infra | 14:41 | |
| openstackgerrit | Merged openstack/project-config master: Add ubuntu bionic and focal with xxxlarge flavor https://review.opendev.org/c/openstack/project-config/+/771565 | 14:42 |
| *** markmcclain has joined #openstack-infra | 14:46 | |
| ricolin | fungi, thanks for the heads up | 14:48 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 14:49 | |
| fungi | yw | 14:51 |
| *** ykarel|mtg is now known as ykarel | 15:08 | |
| amoralej | fungi, who may help with https://review.opendev.org/c/zuul/zuul-jobs/+/771105/2 and https://review.opendev.org/c/zuul/zuul-jobs/+/770815 ? | 15:13 |
| fungi | amoralej: maybe me, but i'll need to look when i get a break between meetings (in two simultaneously for most of the next 4 hours) | 15:13 |
| amoralej | ack, thanks | 15:14 |
| *** hasharAway has quit IRC | 15:18 | |
| *** wolsen has quit IRC | 15:19 | |
| *** JanZerebecki[m] has quit IRC | 15:19 | |
| *** mordred has quit IRC | 15:20 | |
| *** irclogbot_3 has quit IRC | 15:21 | |
| *** irclogbot_2 has joined #openstack-infra | 15:22 | |
| *** armax has joined #openstack-infra | 15:25 | |
| *** mordred has joined #openstack-infra | 15:31 | |
| fungi | amoralej: so it's possible we're at the point where the gentoo images are finally updating again, i'll check. i know some fixes got merged into diskimage-builder, a release of that was tagged, we updated our nodepool image builds to pull in that new release, and those should in theory have been deployed to our servers and restarted | 15:36 |
| amoralej | ok | 15:36 |
| amoralej | fungi, i retried the job some hours ago, but i'm not sure if there has been any progress since then or how close we are of having those in place | 15:37 |
| fungi | yeah, i'm looking to figure out whether the image builds are succeeding, if not i'll ask in #zuul for an additional reviewer on the change to temporarily stop running on it | 15:38 |
| fungi | prometheanfire: so it's getting a lot farther now, but still breaking: https://nb02.opendev.org/gentoo-17-0-systemd-0000164758.log "eselect python set python3.8 !!! Error: Can't load module python" | 15:40 |
| *** lpetrut_ has quit IRC | 15:42 | |
| *** armax has quit IRC | 15:48 | |
| fungi | amoralej: i've asked for folks in #zuul to weigh in on temporarily dropping the gentoo role tests for zuul-jobs changes | 15:51 |
| amoralej | ack, thanks fungi | 15:51 |
| *** dklyle has joined #openstack-infra | 15:56 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 15:56 | |
| *** dklyle has quit IRC | 15:57 | |
| *** matt_kos_ has quit IRC | 15:58 | |
| *** hashar has joined #openstack-infra | 15:59 | |
| *** armax has joined #openstack-infra | 16:06 | |
| *** wolsen has joined #openstack-infra | 16:06 | |
| *** JanZerebecki[m] has joined #openstack-infra | 16:06 | |
| *** dklyle has joined #openstack-infra | 16:08 | |
| prometheanfire | fungi: ya, I saw something about that, it just happened within the last day or two (app-eselect/eselect-python was removed from my system too) | 16:22 |
| fungi | oh fun | 16:23 |
| fungi | a never-ending source of entertainment ;) | 16:23 |
| prometheanfire | yep | 16:23 |
| *** ysandeep is now known as ysandeep|away | 16:31 | |
| *** gyee has joined #openstack-infra | 16:32 | |
| prometheanfire | so far the answer I got is that I (and others) should have paid more attention to things | 16:41 |
| *** lpetrut has joined #openstack-infra | 16:42 | |
| prometheanfire | we can manually edit a file or manually install eselect python | 16:42 |
| *** matt_kosut has joined #openstack-infra | 16:43 | |
| *** lucasagomes has quit IRC | 16:47 | |
| *** ociuhandu_ has joined #openstack-infra | 16:49 | |
| *** lpetrut has quit IRC | 16:50 | |
| *** jamesmcarthur has joined #openstack-infra | 16:51 | |
| *** armax has quit IRC | 16:52 | |
| prometheanfire | python order of preference may be set by python_targets in make.conf, but I'm getting confirmation on that assumption | 16:52 |
| *** ociuhandu has quit IRC | 16:53 | |
| clarkb | fungi: do we need to pause the gentoo builds? | 16:54 |
| *** ociuhandu_ has quit IRC | 16:54 | |
| clarkb | (just noticing that it is one of the images the builders are trying to build but sounds like they wn't work?) | 16:54 |
| prometheanfire | I think it'll need at least another dib release... | 16:54 |
| prometheanfire | likely to remove 'eselect python' calls | 16:54 |
| clarkb | looks like they run for at least 25 minutes before failing though so it isn't in a very tight loop | 16:57 |
| fungi | an unlike last time, it doesn't time out and leak, filling up the drive | 16:57 |
| fungi | but if folks do want to pause it again, https://review.opendev.org/771104 | 16:57 |
| clarkb | I've +2'd that since it sounds like another dib release may be necessary | 16:58 |
| *** jpena is now known as jpena|off | 16:59 | |
| fungi | yeah, we went ahead and merged the change in zuul-jobs to stop testing roles on it for now | 17:07 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Revert "Revert "Revert "Temporarily stop booting nodes in inap-mtl01""" https://review.opendev.org/c/openstack/project-config/+/771857 | 17:09 |
| *** ykarel has quit IRC | 17:09 | |
| *** ykarel has joined #openstack-infra | 17:11 | |
| *** amoralej is now known as amoralej|off | 17:11 | |
| *** ykarel has quit IRC | 17:17 | |
| *** jcapitao has quit IRC | 17:21 | |
| openstackgerrit | Artom Lifshitz proposed openstack/openstack-zuul-jobs master: WIP: Generate SSH keys in PEM format https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/771858 | 17:22 |
| *** gfidente is now known as gfidente|afk | 17:29 | |
| *** dave-mccowan has quit IRC | 17:31 | |
| *** hamalq has joined #openstack-infra | 17:33 | |
| *** lpetrut has joined #openstack-infra | 17:40 | |
| *** diablo_rojo has quit IRC | 17:41 | |
| *** rpittau is now known as rpittau|afk | 17:42 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Revert "Revert "Temporarily stop booting nodes in inap-mtl01""" https://review.opendev.org/c/openstack/project-config/+/771857 | 17:42 |
| *** rlandy is now known as rlandy|biab | 17:43 | |
| *** ralonsoh has quit IRC | 17:52 | |
| *** rcernin has joined #openstack-infra | 17:54 | |
| *** lpetrut has quit IRC | 17:55 | |
| *** rcernin has quit IRC | 17:59 | |
| prometheanfire | next portage tarball time (sometime tonight) should have the eselect python dep readded (dev removed it from stable packages...) | 18:09 |
| fungi | longer term, should we be preinstalling it, or was that a mistake on their part? | 18:20 |
| *** dtantsur is now known as dtantsur|afk | 18:32 | |
| *** amoralej|off is now known as amoralej | 18:40 | |
| *** rlandy|biab is now known as rlandy | 18:41 | |
| *** nightmare_unreal has quit IRC | 18:42 | |
| *** jamesdenton has quit IRC | 18:43 | |
| *** jamesdenton has joined #openstack-infra | 18:43 | |
| *** andrewbonney has quit IRC | 18:55 | |
| *** amoralej is now known as amoralej|off | 19:24 | |
| *** dougwig has quit IRC | 19:27 | |
| *** dougwig has joined #openstack-infra | 19:28 | |
| *** xek_ has quit IRC | 19:30 | |
| *** xek_ has joined #openstack-infra | 19:31 | |
| *** jamesmcarthur has quit IRC | 19:52 | |
| *** dave-mccowan has joined #openstack-infra | 19:53 | |
| *** slaweq has quit IRC | 19:55 | |
| *** hashar has quit IRC | 20:00 | |
| *** jamesmcarthur has joined #openstack-infra | 20:03 | |
| *** jamesdenton has quit IRC | 20:06 | |
| *** jamesdenton has joined #openstack-infra | 20:06 | |
| *** sboyron has quit IRC | 20:09 | |
| *** irclogbot_2 has quit IRC | 20:14 | |
| *** tosky has quit IRC | 20:14 | |
| *** tosky has joined #openstack-infra | 20:15 | |
| *** irclogbot_3 has joined #openstack-infra | 20:16 | |
| *** eolivare has quit IRC | 20:17 | |
| *** ajitha has quit IRC | 20:56 | |
| prometheanfire | it was a mistake | 20:56 |
| prometheanfire | dev just said they are going to keep it as a dependency of python for the forseeable future, that said, even when that future comes the dib patch I just proposed 'fixes' it | 20:57 |
| fungi | cool | 21:13 |
| *** rlandy is now known as rlandy|biab | 21:17 | |
| *** owalsh has quit IRC | 21:26 | |
| *** derekh has quit IRC | 21:37 | |
| *** matt_kosut has quit IRC | 21:40 | |
| *** owalsh has joined #openstack-infra | 21:44 | |
| *** jamesmcarthur_ has joined #openstack-infra | 21:45 | |
| *** jamesmcarthur has quit IRC | 21:49 | |
| *** rcernin has joined #openstack-infra | 21:54 | |
| *** rcernin has quit IRC | 21:59 | |
| *** jamesmcarthur has joined #openstack-infra | 22:02 | |
| *** xek_ has quit IRC | 22:05 | |
| *** jamesmcarthur_ has quit IRC | 22:05 | |
| *** derekh has joined #openstack-infra | 22:06 | |
| *** owalsh has quit IRC | 22:16 | |
| *** rlandy|biab is now known as rlandy | 22:20 | |
| *** owalsh has joined #openstack-infra | 22:25 | |
| *** d34dh0r53 has quit IRC | 22:37 | |
| *** rcernin has joined #openstack-infra | 22:38 | |
| *** d34dh0r53 has joined #openstack-infra | 22:48 | |
| *** jamesmcarthur_ has joined #openstack-infra | 22:51 | |
| *** jamesmcarthur has quit IRC | 22:55 | |
| *** rcernin has quit IRC | 22:56 | |
| *** rcernin has joined #openstack-infra | 22:57 | |
| *** jamesmcarthur has joined #openstack-infra | 23:02 | |
| *** yamamoto has quit IRC | 23:04 | |
| *** yamamoto has joined #openstack-infra | 23:04 | |
| *** jamesmcarthur_ has quit IRC | 23:05 | |
| *** jamesmcarthur has quit IRC | 23:12 | |
| *** jamesmcarthur has joined #openstack-infra | 23:16 | |
| *** jamesmcarthur has quit IRC | 23:16 | |
| *** jamesmcarthur has joined #openstack-infra | 23:16 | |
| *** ociuhandu has joined #openstack-infra | 23:24 | |
| *** rlandy has quit IRC | 23:26 | |
| *** ociuhandu has quit IRC | 23:29 | |
| *** dchen has joined #openstack-infra | 23:33 | |
| *** jamesmcarthur has quit IRC | 23:55 | |
| *** jamesmcarthur has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!