Tuesday, 2023-03-28

fungioh cool00:11
opendevreviewSteve Kowalik proposed opendev/bindep master: Support and prefer unittest.mock  https://review.opendev.org/c/opendev/bindep/+/87873302:17
fricklerClark[m]: ildikov: codesearch found lots of references like this, not sure if they're actually being used https://opendev.org/airship/tempest-plugin/src/branch/master/.zuul.yaml#L5304:25
ildikovfrickler: thank you, I’ll check them tomorrow!04:40
opendevreviewMerged openstack/project-config master: Add metal3-io/metal3-dev-env to openstack tenant  https://review.opendev.org/c/openstack/project-config/+/87822305:40
*** amoralej|off is now known as amoralej06:17
opendevreviewIan Wienand proposed zuul/zuul-jobs master: registry-tag-remove: role to delete tags from registry  https://review.opendev.org/c/zuul/zuul-jobs/+/87861406:18
opendevreviewIan Wienand proposed zuul/zuul-jobs master: promote-container-image: use generic tag removal role  https://review.opendev.org/c/zuul/zuul-jobs/+/87874006:18
priteauHello. Did something change with regards to topics in Gerrit? I have removed the topic from https://review.opendev.org/c/openstack/kolla-ansible/+/871279 to change it, and I don't see any UI element to add a topic back.07:15
priteauNevermind, it is hidden under SHOW ALL07:15
*** jpena|off is now known as jpena07:20
fricklerpriteau: yes, I had the same issue the other day, not sure if that is a recent UI "optimization"07:25
mnasiadkaSpeaking about Gerrit - in Kolla-Ansible I see Review-Priority ticked as green even if it's not voted (i.e. 0) - where do we fix that?07:42
fricklermnasiadka: that's not a bug from gerrit pov, it only considers submit-requirements. since review-priority is non-blocking, this is satisfied with RP=007:53
fricklerI think clarkb had some more context for that07:53
mnasiadkaok, that makes sense07:53
fricklerinfra-root: looks like we may need to force-merge some horizon zuul config changes. they neglected to remove the xstatic-font-awesome repo before its retirement and now patches to drop it fail because of issues in different branches09:04
fricklerhttps://review.opendev.org/c/openstack/horizon/+/878751 vs. https://review.opendev.org/c/openstack/horizon/+/87532609:04
fungimnasiadka: frickler: yes, basically if review-priority doesn't require a positive vote to merge then it will show a green checkmark indicating that requirement is satisfied, since apparently that's what gerrit upstream thinks the change list should be showing to people (whether or not a change is ready to merge)12:17
*** amoralej is now known as amoralej|lunch12:18
lecris[m]I've noticed that the opendev documentation says that we can host projects on OpenDev, but I can't find any instructions of where to get started12:18
lecris[m]The Get Started page is about developing on already present projects, not on making a new one12:19
lecris[m]Or alternatively is there another way to get a zuul workflow for a project?12:21
*** gthiemon1e is now known as gthiemonge12:24
fricklerfungi: yes, technically this makes sense, but it destroyed the intended use of the RP field, unless one is using custom dashboards12:28
fungilecris[m]: zuul is open source, anyone can deploy and run the requisite services and connect them to the supported code review platform of their choice. there are also companies like vexxhost who offer managed zuul-as-a-service and other project hosting platforms platforms like softwarefactory which also provide zuul services to projects hosted with them. as for opendev's project creation12:30
fungiprocess, we have it documented in our infrastructure manual here: https://docs.opendev.org/opendev/infra-manual/latest/creators.html12:30
fungiit's basically all done via code review adding project information to structured data files in a configuration repository, and then project creation in gerrit and related systems is handled through post-merge continuous deployment automation (also driven by zuul)12:32
lecris[m]Thanks for the link might be helpful to link this on the front page: https://opendev.org/12:34
fungigood point, though mostly we cater to projects that want to be hosted in opendev due to their maintainers having already contributed to other projects hosted in opendev and who therefore have some familiarity with the required workflows and processes12:37
fungisince we're essentially a community collective of projects who pool our resources to help maintain a common set of open source collaboration tools for each other12:38
lecris[m]So currently we cannot make user level projects, and it needs to be reviewed. I'll give it a thought. There is no free/trial version for hosted zuul to play around?12:38
fungilecris[m]: the opendev collaboratory isn't zuul-as-a-service, if that was your impression. we're running an integrated project hosting platform which happens to include zuul as its ci/cd component, but it's a necessarily opinionated set of connected tools and workflows, not simply a means of getting access to a zuul someone has hosted for you12:40
lecris[m]Indeed, I am vaguely familiar with the openstack projects on opendev, I was wondering about the other zuul services you've mentioned 12:41
fungilecris[m]: at https://zuul-ci.org/start there's a link to vexxhost which has a commercial zuul-as-a-service offering, not sure if they have a free trial12:43
lecris[m]Thanks I'll check it ouy12:45
fungilecris[m]: also red hat runs https://softwarefactory-project.io/ which is a hosting platform that incorporates zuul (they run one projects can be hosted in, but also they make a sort of distribution from it)12:47
fungilecris[m]: as for hosting in opendev, yes we discourage creating throwaway test projects, but are happy to host an open source project you have some plan to maintain12:49
lecris[m]Ok, I've checked them both, but it seems making a self-managed zuul to be more accessible than those12:50
fungiif you're just wanting to play around with zuul, there's a "quickstart" all-in-one container based (docker-compose) deployment in the zuul/zuul repository which includes a running gerrit container connected to it and some rudimentary log hosting through apache12:50
fungithat "quick start installation" linked from https://zuul-ci.org/start12:51
lecris[m]Is there a playbook for it?12:51
fungiyeah, it actually gets run on every proposed change for the zuul project as a full integration test... i'll dig up the link12:52
fungilecris[m]: https://opendev.org/zuul/zuul/src/branch/master/playbooks/tutorial12:53
fungiif you're curious, here's how it gets run as part of a zuul job too: https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L181-L19412:54
fricklerlecris[m]: if you are done with the quickstart and want a more production like deployment, maybe have a look at https://github.com/osism/ansible-collection-services/tree/main/roles/zuul, this is what I (co-)wrote and use for our downstram zuul12:54
fricklerdownstream even12:55
lecris[m]Thanks for the links 🎉12:55
lecris[m]So even the playbook uses docker-compose I see12:56
lecris[m]Still works on podman though right?12:57
fungiyes, i think we actually use podman for that, but would need to check12:58
*** amoralej|lunch is now known as amoralej13:00
fungimmm, https://opendev.org/zuul/zuul/src/branch/master/playbooks/tutorial/pre.yaml#L4 calls the ensure-docker role: https://zuul-ci.org/docs/zuul-jobs/container-roles.html#role-ensure-docker but it may "just work" if switched to https://zuul-ci.org/docs/zuul-jobs/container-roles.html#role-ensure-podman13:03
mnasiadkaIs there a way to set a storyboard project as read only? (we'd like to move Kayobe to launchpad)13:24
fungimnasiadka: yes, i should be able to switch it to disabled, just a sec13:25
fungiconfirmed, there's an is_active column in the projects table. looks like we switched that to 0 and updated the description for the project when we did this in the past13:27
fungimnasiadka: so we should be able to do it to kayobe, are you wanting that done now or on a certain day?13:28
mnasiadkafungi: I think we'd like to create a launchpad project first - I'll reach out with a request to do it once we have it13:29
mnasiadkaspeaking of which - is there some special magic to create a launchpad project?13:29
fungimnasiadka: i haven't done it in a long time, so you're likely to figure it out as fast as i do. just make sure you set the owner to a group that's owned by the openstack administrators group, like how nova's is configured, that allows me or someone else in that group to step in and handle things like ptl replacement if something happens and you're not available to conduct a smooth transition of13:31
lecris[m]I never understood why some projects have storyboard, while most are on launchpad. Historical reasons or are the good usecases for storyboard?13:31
mnasiadkafungi: will do13:32
fungilecris[m]: launchpad was unmaintained for a very long time when we started creating an alternative. also there were lots of projects using closed-source kanban boards for task tracking and prioritization (trello especially) and we wanted to offer them an open source alternative to that13:33
fungihowever the people doing sb development came and went, and we were never able to build a critical mass of people using it who also wanted to help develop/maintain the code for it13:34
fungiprobably the biggest challenge is that most of our projects, traditionally, were python based and web frameworks had popularly started shifting to javascript which very few contributors had significant experience or comfort level with13:35
lecris[m]Hmm, but why not gitea issues? Launchpad doesn't have kanban either right?13:35
fungigitea wasn't even a twinkle in anyone's eye in 201313:35
lecris[m]Oh, that long ago. But eventually opendev adopted gitea, so why not then?13:36
fungialso gitea issues will require a pretty heavy lift to how we currently deploy and run gitea, right now we just have a farm of independent read-only gitea servers we replicate commits into from gerrit, no user account management13:37
fricklerhttps://lists.opendev.org/pipermail/service-discuss/2022-November/000382.html has some context about that13:38
fungiwe used gitea as a replacement for cgit, because people preferred it for code browsing. right now it's really just set up as a code browser interface for us13:38
lecris[m]Oh yeah. Probably will be hard until federated users is a thing 🤔13:38
fungi(and we had been using cgit as a replacement for gitweb)13:38
fungieven solving the account management for gitea, we'll need some sort of shared backend because we get too much traffic for a single gitea server13:40
fungii don't think we've ruled it out, we're just operating on a skeleton crew trying to continue maintaining things, and have limited time for significant feature work unless more people start helping out with general maintenance tasks first13:41
lecris[m]Hmm, I thought that would be automatically handled by the git ssh which is not (or at least can be configured to) managed by the gitea server13:42
fungiwe've learned that volunteers to maintain a new service or implement some new feature doesn't help much, because people get bored and move on quickly and leave the rest of us stuck maintaining whatever they built and abandoned13:42
fungilecris[m]: what would be handled? offloading the request volume?13:43
fungior the shared data backend?13:43
lecris[m]git traffic through ssh. That is the main bottleneck right? Not the actual frontend management13:44
fungiwe don't serve git repositories over ssh, they're over https13:44
lecris[m]Oh, right, that would be a strain on gitea directly13:45
fungissh doesn't work from lots of places... corporate networks where the firewall admins think ssh means hackers have taken control of your workstation, mainland china, et cetera13:46
lecris[m]What about for the backend?13:47
fungibut also we get a lot more non-git web traffic than you might expect, particularly crawlers13:47
fungisynchronizing git content between the servers is a solved problem, we do that already, the challenge is other features (which we currently have disabled) like issues, wiki, anything that needs user management13:48
fungikeeping all that synchronized between multiple gitea servers historically wasn't trivial, though maybe it is now13:49
lecris[m]Ok, I've found probably the [relevant issue](https://gitea.com/gitea/helm-chart/pulls/205) on the HA setup13:50
fungilooks like that design relies on a shared block device writeable from multiple systems13:51
lecris[m]Well, one place that has it solved would be gitlab, but probably too much of a hustle to migrate there13:53
fricklerwhich might in fact be available in vexxhost. but certainly would require a good amount of testing13:53
fungiand it's not clear that it's a solution to load distribution (active-active clustering), that may be purely for failover (active-standby)13:53
fungilecris[m]: also there's the problem that gitlab is open-core, and we try to avoid open-core projects13:54
lecris[m]Indeed. Didn't gitea start to move to open-core?13:55
fricklerat least there is a fork that seems to have gained some traction13:56
fungithey created a company, i don't think they (yet) said they were going to make a commercial distribution with features absent from the open source version and reject similar patches to add those features to the open source version13:56
fungiand also, yes, there's a separate community fork of gitea which we've been tracking as an alternative if that were to happen13:57
lecris[m]What's the fork named?13:58
lecris[m]Thnx. Funny how we have a fork of a fork14:00
fungigranted, many open source projects are forks of forks of forks dating back decades14:01
fungihttps://forgejo.org/ seems to be the current url for it14:02
clarkbfwiw mordred and corvus sort of pioneered the run gitea as an active active cluster. All the history of that remains in opendev/system-config. We didn't realize until after the initial deployment that the gitea processes were trampling over each other via the code search index14:25
clarkbit took a while but gitea eventually added support for an external index like elasticsearch in response to that, but by then we had abandoned the gitea cluster in favor of the independent giteas and switching back is a bit of work at this point14:26
clarkbin particular the original design relied on kubernetes and we don't currently have a good way to deploy kubernetes except by hand14:26
lecris[m]Main/all infrastructure is ansible based?14:28
clarkblecris[m]: I'm not sure what Main/all is. But a number of years ago https://opendev.org/opendev/system-config/src/branch/master/kubernetes deployed a (broken due to code indexes) active active gitea cluster on k8s14:28
clarkbbugs were filed with gitea with what was learned and we changed the deployment as gitea didn't support that deployment setup14:29
lecris[m]I mean for example, gitea is deployed via ansible for opendev.org instead of kubernetes or other such deployments?14:31
clarkblecris[m]: the current deployment system of our production giteas is ansible managing docker-compose/docker14:31
lecris[m]Ok, so more or less similar to openstack deployments afaiu?14:33
clarkblecris[m]: https://review.opendev.org/c/opendev/system-config/+/877541 gives a good window into what the ongoing maintenance is like. THis change if/when landed will build and publish new gitea 1.19 docker images, then followup jobs will run ansible to upgrade our giteas in a rolling fashion behind a load balancer which should result in no user visible downtime14:33
clarkblecris[m]: it has nothing to do with openstack deploymens14:33
clarkbsome people also use docker compose and docker and ansible to deploy openstack, but there isn't any shared machinery other than using the same tools14:33
lecris[m]I thought nowadays, only the ansible+docker based developments are actively maintained for openstack. Tripleo, kayobe, kolla14:35
clarkblecris[m]: I don't think that is accurate. vexxhost has their operator based system iirc. Charms still receive changes and so on14:35
clarkbBut even if that were the case that doesn't give openstack a monopoly and deploying things with docker and ansible. My point is that we may share common tools but the deployment systems and playbooks and roles used don't have any overlap. Its not openstack related14:36
fungiand rdo has rhel packages, debian has deb packages, ubuntu has uca with packages and their charms orchestration...14:37
clarkbI would say the choices we have made are very much not kolla like :)14:37
clarkbkolla spends a lot of energy on building images on different platforms from source and distro packages and we explicitly do not. We build a single image using common base tools for the language or operating environment14:38
fungioh, actually the main debian deployment solution relies on puppet modules now for a lot, i think14:38
lecris[m]Yes indeed, just that knowing a bit on the whole stack of one, helps with the other. I've basically learned how to debug docker containers, ansible workflows, etc from tripleo.14:39
corvuslecris: hi, i'm the original author and one of the maintainers of zuul.  i also provide support for self-hosted zuul via my company Acme Gating.  i also have a zuul configurator on the acme website that might help you get started quickly with several of zuul's code hosting options and commercial cloud providers (i also support openstack of course, but that's not as easy to package up :)  here's a link:14:41
lecris[m]clarkb: Speaking of which, would still love to see Fedora support someday :D. TripleO at least has a bootstrapping issue from centos to Fedora due to missing `btrfs`14:42
fricklerclarkb: fungi: corvus: any thoughts on the horizon issue I mentioned earlier?14:43
clarkbfrickler: in https://review.opendev.org/c/openstack/horizon/+/878751 can the missing job also be fixed at the same time?14:43
clarkbfrickler: same question for https://review.opendev.org/c/openstack/horizon/+/875326 looks like. Basically you should be able to remove the missing job from the config so zuul doesn't complain about it while also removing the repo?14:44
lecris[m]Thanks, I want to get my feet wet with some test projects before committing to one or another. Thanks for the information though, I'll keep it in mind14:44
corvusmakes sense.  feel free to use the configurator there -- it's very similar to the zuul quickstart (which i also wrote) but includes cloud and code review configuration out of the box14:46
corvusbasically, the zuul quickstart is designed to teach you how to bootstrap a zuul system using fundamental concepts.  the configurator is designed to get a zuul system up and running immediately (without teaching you anything).14:47
fricklerclarkb: the ussuri patch removes the job that the master patch complains about, not sure what else is to be removed. each patch is still blocked by be other one14:48
fungiso cross-branch deadlock basically, requiring one or the other to bypass testing?14:50
lecris[m]Hmm, probably will play around with the quickstart and the role since I am also writing a playbook for my test cluster. Btw, why not also a openstack cloud settings? Too much variability?14:50
fricklerfungi: that's how I read it, yes14:50
fungiand this was the result of removing the project from the tenant config i guess?14:52
frickler"Zuul encountered a syntax error while parsing its configuration in the repo openstack/horizon on branch master." on the ussuri patch and vice versa14:52
clarkbthe master job that the ussuri change complains about is present though14:52
clarkbbut I guess it doesn't see that because the master config is broken so that new config isn't loadale14:52
corvuslecris: exactly -- with the configurator, i was trying to make something that works out of the box for the most people.  i have customers using openstack and help them with it, but it's not something that's easy to pre-package.14:52
clarkbfrickler: fungi: in that case we might be able to force merge just the master change and then ussuri would see the config as valid and be mergable normally?14:53
fricklerclarkb: ah, o.k., I didn't check that deep. then maybe the job removal isn't actually needed, just the project drop14:53
fricklerbut that would also explain how this situation could happen, since I was wondering how the job could get removed without zuul complaining in the first place14:54
clarkbya I think my suggestion would be to force merge the xstatic project removal from horizon master. Then recheck ussuri14:54
corvusfrickler: it would be interesting to see the results of just proposing a change with the project removal14:55
corvusthere's a possibility it might not trip zuul's "this change made this error" check14:56
fricklerI'll update the ussuri patch. https://review.opendev.org/c/openstack/horizon/+/875326 is already just that14:56
corvusoh then clarkb 's suggestion is probably best.  alternatively, we could enable circular deps, but i think we'd probably prefer to force-merge :)14:58
lecris[m]Speaking of horizon, is there a roadmap for skyline if it will replace it at some time? What's the main advantage of developing that?14:58
fungilecris[m]: we don't really get that deep into the projects being hosted in opendev, it's probably more of a question for the #openstack-tc channel or the openstack-discuss mailing list14:59
lecris[m]Oh, ok15:00
fungiwe do use a variety of openstack deployments, but mainly via api integration and cli, so not all that up on the webui scene for it15:01
fricklercorvus: still the some failure. maybe one of you can do the merge, I'm still in a PTG session in parallel, otherwise I can do it my morning15:01
mnasiadkaGetting NODE_FAILURE for all aarch64 jobs that Kolla runs - https://review.opendev.org/c/openstack/kolla/+/87866315:26
clarkbmnasiadka: looking at https://grafana.opendev.org/d/391eb7bb3c/nodepool-linaro?orgId=1 and https://grafana.opendev.org/d/42e9cd74b4/nodepool-osuosl?orgId=1 it doesn't seem to be a complete failure. We probably need to look at the nb03.opendev.org launcher service logs to see more15:28
clarkbmnasiadka: is it specific to a particular distro or distro release maybe?15:28
mnasiadkacentos9, debian, ubuntu and rocky9 is probably enough of a mix to assume at least one would be working ;-)15:29
clarkbI'm in a meeting now but will log in to check logs as soon as this is done15:33
lecris[m]Can someone point me to where I can get the spec files for rdo? I want to port some of them to copr to test some basic Fedora migration15:33
clarkblecris[m]: you'll need to ask rdo.15:34
fungithey have a #rdo irc channel, but looks like you just found it15:40
clarkbmnasiadka: linaro failed 300-0020850007 with "no valid host found" and OSUOSL failed with "No fixed IP addresses available"15:48
clarkbthe linaro failure occured at ~14:40 which is when linaro was busy15:49
clarkbso the no valid host is possibly a valid failure and we've got our quota limits set improperly?15:49
fungior a leak15:49
clarkbfungi: yes that is possible too.15:50
clarkbon the osuosl side I suspect we share the ip pool with other users and there isn't much we can do about that15:50
clarkbanyway I think a recheck would be appropriate and may generate more useful datapoints15:50
clarkbok ya the arm64 pipeline is running jobs now and the grafana graphs reflect nodes are booting for that16:05
clarkbI think this may have been an unfortunate timing issue combining lack of resources in both clouds16:05
clarkbfrickler: fungi: when yo uget a chance can I get reviews on https://review.opendev.org/c/opendev/zone-opendev.org/+/878710 to finish up gitea01-04 cleanups16:10
*** jpena is now known as jpena|off16:22
fricklerRamereth: ^^ maybe you can check that IP failure16:27
Ramerethfrickler: looking16:39
Ramerethlooks like we're running low on public IPs on that network16:41
clarkbwe can dial back max-servers if necessary. Also nodepool will respect quota changes if you sa you change the instance quota on the cloud side16:43
Ramerethdo you require public IP addresses? I was going to work on deploying a new NAT provider network16:44
Ramereth(not sure why I didn't do that in the first place a long time ago and just rely on floating IPs, but dealing with that now...)16:45
*** amoralej is now known as amoralej|off16:49
clarkbRamereth: we do require it as it vastly simplifies our multicloud deployment16:53
clarkbRamereth: we can do ipv6 only test nodes for public IPs though and then outbound can NAT through a single ipv4 address16:53
clarkbunfortunately ipv4 still required due to github and other resources not having ipv6 addresses16:53
clarkbbasiclly if we get ipv6 then I think we end up only needing ~1.5 IPv4 addrs. 1 for our mirror/proxy caching node and the other for general test node outbound NAT which can be shared with the rest of the cloud hence the .516:54
*** sfinucan is now known as stephenfin17:20
Ramereth[m]I will see what we can do. I have limited hours this week as I have company visiting. But last I looked there were only 3 IPs available. So if you limit your jobs to that, it should get you by at least. 17:24
opendevreviewMerged opendev/zone-opendev.org master: Remove gitea01-04 server DNS records  https://review.opendev.org/c/opendev/zone-opendev.org/+/87871017:40
*** dhill is now known as Guest916720:11
clarkbfungi: is there any way to tell pip to not cache wheels it builds for certain packages? In particular zuul's use of libre2 ends up causing problems on tumbleweed for me because it updates often enough that the .so moves and the wheel is invalid21:00
fungiclarkb: there's a --no-cache-dir option according to --help21:03
fungilabeled as "Disable the cache."21:04
clarkbI think that disables the entire cache which I don't want21:04
funginot sure about on a per-project basis though21:04
clarkbits relaly only re2 linked things that I regularly have to run a find in .cache/pip/wheels to delete21:04
fungii suppose you could selectively scrub the cache with a script between invocations21:04
clarkbya maybe thats the best thing just store the find incantation to delete them somewhere21:04
fungii wipe the cache every time i rebuild my default python, but sounds like you're seeing issues more often than that21:05
clarkbyes, in this case its the libre2(-devel) package updating creating a new .so file breaking existing wheels21:05
fungii suppose you could also wipe it any time you libre2 is upgraded21:05
clarkbbut pip has no validation of wheel external links before using the wheels21:05
fungiwell, also wheels don't build binary extensions, they only ship prebuilt ones21:06
fungii guess you're building re2 from sdist on install due to lack of sufficient wheel for your platform?21:07
clarkbthere are no wheels published on pypi iirc21:07
fungipip does cache what it builds, and that could lead to what you're seeing21:07
clarkbso ya its building from scratch and then caching that21:07
clarkbthen 3 months later I'm rebuilding the zuul test env and it explodes21:07
funginot a lot of options for specifically disabling the built cache vs the download cache21:08
clarkbwith a stable distro release this is never an issue bceause the .so file never moves21:11
clarkband  should remain compatible though I suppose its possible a security update should cause you to need to rebuild too? probably very unlikely though21:11
fungiyeah, sover bumps are verboten within a debian stable release, for example21:16
fungibut i run on debian unstable, and also i build my python interpreters from upstream source anyway, so i end up manually invalidating my pip cache extremely often21:17
opendevreviewIan Wienand proposed zuul/zuul-jobs master: promote-image-container: do not delete tags  https://review.opendev.org/c/zuul/zuul-jobs/+/87861221:30
opendevreviewIan Wienand proposed zuul/zuul-jobs master: registry-tag-remove: role to delete tags from registry  https://review.opendev.org/c/zuul/zuul-jobs/+/87861421:30
opendevreviewIan Wienand proposed zuul/zuul-jobs master: promote-container-image: use generic tag removal role  https://review.opendev.org/c/zuul/zuul-jobs/+/87874021:30
opendevreviewMerged zuul/zuul-jobs master: build-container-image: directly push with buildx  https://review.opendev.org/c/zuul/zuul-jobs/+/87848721:50
Ramerethclarkb: I'm going to setup another public provider network, but it will be a few days (likely not until next week) until it's ready21:58
clarkbRamereth: no rush. This is why we've got the othe rcloud to supplment things. As I mentioned I think the timing was just really bad for that one chnage and rerunning things seeemd to go much better. THank you for looking at it and that sounds like a good plan to mitigate this22:03
ianwclarkb: yeah that gets me *every* time running zuul tests22:55

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!