| rdogerrit | rdo-trunk rdo-trunk proposed nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch2.17-release https://review.rdoproject.org/r/c/nfvinfo/+/55304 | 00:03 |
|---|---|---|
| rdogerrit | rdo-trunk rdo-trunk proposed nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch3.1-release https://review.rdoproject.org/r/c/nfvinfo/+/55305 | 00:03 |
| rdogerrit | rdo-trunk rdo-trunk proposed nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch3.2-release https://review.rdoproject.org/r/c/nfvinfo/+/55306 | 00:03 |
| rdogerrit | rdo-trunk rdo-trunk proposed nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch3.3-release https://review.rdoproject.org/r/c/nfvinfo/+/55307 | 00:03 |
| rdogerrit | rdo-trunk rdo-trunk proposed nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch3.4-release https://review.rdoproject.org/r/c/nfvinfo/+/55308 | 00:03 |
| rdogerrit | rdo-trunk rdo-trunk proposed rdoinfo master: Promote CBS tags update for caracal-9s-release https://review.rdoproject.org/r/c/rdoinfo/+/55309 | 00:05 |
| rdogerrit | Merged nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch2.17-release https://review.rdoproject.org/r/c/nfvinfo/+/55304 | 07:27 |
| rdogerrit | Merged nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch3.2-release https://review.rdoproject.org/r/c/nfvinfo/+/55306 | 07:27 |
| rdogerrit | Merged nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch3.4-release https://review.rdoproject.org/r/c/nfvinfo/+/55308 | 07:27 |
| rdogerrit | Merged nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch3.1-release https://review.rdoproject.org/r/c/nfvinfo/+/55305 | 07:27 |
| rdogerrit | Merged rdoinfo master: Promote CBS tags update for caracal-9s-release https://review.rdoproject.org/r/c/rdoinfo/+/55309 | 07:28 |
| rdogerrit | Merged nfvinfo master: Promote CBS tags update for nfvsig-9s-openvswitch3.3-release https://review.rdoproject.org/r/c/nfvinfo/+/55307 | 07:28 |
| jcapitao[m] | karolinku: hello, could you review https://review.rdoproject.org/r/c/gating_scripts/+/55302 please ? | 07:30 |
| rdogerrit | Joel Capitao proposed rdoinfo master: Unpin kolla to master branch https://review.rdoproject.org/r/c/rdoinfo/+/55310 | 07:47 |
| rdogerrit | Joel Capitao proposed rdoinfo master: DNM test adding CS10 deps https://review.rdoproject.org/r/c/rdoinfo/+/55311 | 07:57 |
| rdogerrit | Joel Capitao proposed rdoinfo master: DNM test adding CS10 deps https://review.rdoproject.org/r/c/rdoinfo/+/55311 | 08:55 |
| jcapitao[m] | karolinku: and also https://review.rdoproject.org/r/c/rdoinfo/+/55310 when you have a chance | 08:57 |
| rdogerrit | Joel Capitao proposed rdoinfo master: Promote first batch of deps in epoxy-10s-testing https://review.rdoproject.org/r/c/rdoinfo/+/55311 | 09:13 |
| rdogerrit | Merged rdoinfo master: Unpin kolla to master branch https://review.rdoproject.org/r/c/rdoinfo/+/55310 | 09:16 |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Create new component optimize for watcher service https://review.rdoproject.org/r/c/rdoinfo/+/55312 | 09:18 |
| rdogerrit | rdo-trunk rdo-trunk proposed openstack/kolla-distgit dalmatian-rdo: openstack-kolla-19.0.0-0.1.0rc2 https://review.rdoproject.org/r/c/openstack/kolla-distgit/+/55313 | 09:22 |
| jcapitao[m] | amoralej: FYI https://issues.redhat.com/browse/RDO-420 | 09:28 |
| amoralej | thanks! | 09:31 |
| rdogerrit | Joel Capitao proposed gating_scripts master: Add Support for handling CentOS Stream 10 tags https://review.rdoproject.org/r/c/gating_scripts/+/55299 | 09:37 |
| rdogerrit | Merged openstack/kolla-distgit dalmatian-rdo: openstack-kolla-19.0.0-0.1.0rc2 https://review.rdoproject.org/r/c/openstack/kolla-distgit/+/55313 | 10:05 |
| rdogerrit | Merged rdoinfo master: Create new component optimize for watcher service https://review.rdoproject.org/r/c/rdoinfo/+/55312 | 10:16 |
| rdogerrit | Alfredo Moralejo proposed openstack/watcher-distgit rpm-master: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55315 | 11:03 |
| rdogerrit | Alfredo Moralejo proposed openstack/watcher-distgit dalmatian-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55136 | 11:10 |
| rdogerrit | Alfredo Moralejo proposed openstack/watcher-distgit dalmatian-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55136 | 11:11 |
| rdogerrit | Alfredo Moralejo proposed openstack/watcher-distgit caracal-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55316 | 11:14 |
| rdogerrit | Alfredo Moralejo proposed openstack/watcher-distgit bobcat-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55317 | 11:14 |
| rdogerrit | Alfredo Moralejo proposed openstack/watcher-distgit antelope-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55318 | 11:16 |
| rdogerrit | Merged openstack/watcher-distgit antelope-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55318 | 11:38 |
| rdogerrit | Merged openstack/watcher-distgit caracal-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55316 | 11:39 |
| rdogerrit | Merged openstack/watcher-distgit dalmatian-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55136 | 11:39 |
| rdogerrit | Merged gating_scripts master: Add support of CS10 in deps validate and final build job https://review.rdoproject.org/r/c/gating_scripts/+/55302 | 11:49 |
| rdogerrit | Alfredo Moralejo proposed openstack/watcher-distgit bobcat-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55317 | 11:56 |
| rdogerrit | Merged openstack/watcher-distgit rpm-master: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55315 | 12:10 |
| rdogerrit | Merged openstack/watcher-distgit bobcat-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55317 | 12:43 |
| rdogerrit | Bogdan Dobrelya proposed rdo-jobs master: Fix zuul parser warnings https://review.rdoproject.org/r/c/rdo-jobs/+/55291 | 13:02 |
| rdogerrit | Bogdan Dobrelya proposed rdo-jobs master: Add a multi-cell non-voting job https://review.rdoproject.org/r/c/rdo-jobs/+/53192 | 13:04 |
| rdogerrit | Alfredo Moralejo proposed openstack/tobiko-distgit rpm-master: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55319 | 14:07 |
| rdogerrit | Alfredo Moralejo proposed openstack/tobiko-distgit dalmatian-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55320 | 14:26 |
| rdogerrit | Alfredo Moralejo proposed openstack/tobiko-distgit bobcat-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55321 | 14:27 |
| rdogerrit | Alfredo Moralejo proposed openstack/tobiko-distgit caracal-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55322 | 14:28 |
| rdogerrit | Alfredo Moralejo proposed openstack/tobiko-distgit antelope-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55323 | 14:29 |
| rdogerrit | Joel Capitao proposed openstack/watcher-distgit zed-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55324 | 14:40 |
| rdogerrit | Joel Capitao proposed openstack/watcher-distgit yoga-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55325 | 14:41 |
| rdogerrit | Joel Capitao proposed openstack/tobiko-distgit zed-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55326 | 14:42 |
| rdogerrit | Joel Capitao proposed openstack/tobiko-distgit yoga-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55327 | 14:42 |
| amoralej | jcapitao[m] thanks ^ for some reason i though we had stopped zed and yoga builders | 14:42 |
| jcapitao[m] | amoralej: I think we will stop building for those releases soon though | 14:44 |
| rdogerrit | Merged openstack/tobiko-distgit antelope-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55323 | 14:45 |
| rdogerrit | Merged openstack/tobiko-distgit dalmatian-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55320 | 14:46 |
| rdogerrit | Merged openstack/tobiko-distgit bobcat-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55321 | 14:46 |
| rdogerrit | Merged openstack/tobiko-distgit caracal-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55322 | 14:46 |
| rdogerrit | Merged openstack/tobiko-distgit rpm-master: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55319 | 14:47 |
| rdogerrit | Merged openstack/tobiko-distgit zed-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55326 | 14:58 |
| rdogerrit | Merged openstack/tobiko-distgit yoga-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/55327 | 14:58 |
| rdogerrit | Merged openstack/watcher-distgit yoga-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55325 | 14:59 |
| rdogerrit | Merged openstack/watcher-distgit zed-rdo: Dummy commit to rebuild https://review.rdoproject.org/r/c/openstack/watcher-distgit/+/55324 | 15:00 |
| rdogerrit | Alfredo Moralejo proposed openstack/watcher-dashboard-distgit rpm-master: DNM Built in-flight patch https://review.rdoproject.org/r/c/openstack/watcher-dashboard-distgit/+/55328 | 16:24 |
| fungi | does anybody happen to know why trunk-builder-centos9.rdoproject.org would be constantly hammering opendev.org's git servers with requests for puppet-openstack repositories? and maybe could it be slowed down a bit? it's hitting us so hard that our gitea backends are getting knocked offline and taken out of our load balancer pool | 17:03 |
| fungi | spotz[m]: ^ maybe you know who would be able to look into that? | 17:04 |
| rdogerrit | rdo-trunk rdo-trunk proposed openstack/kolla-distgit dalmatian-rdo: openstack-kolla-19.0.0-1 https://review.rdoproject.org/r/c/openstack/kolla-distgit/+/55329 | 17:07 |
| spotz[m] | <fungi> "does anybody happen to know..." <- Let me see what I can find out | 17:22 |
| fungi | thanks spotz[m]!!! | 17:23 |
| spotz[m] | I've pinged folk but it might be their off hours already | 17:24 |
| spotz[m] | fungi: Infra says it's not them trying to track further | 17:49 |
| fungi | spotz[m]: cool, thanks for checking into it. the requests are originating from 38.129.56.237 which is trunk-builder-centos9.rdoproject.org in dns, user agent string claims the client tool is git/2.43.5 and we're seeing periodic bursts from it grabbing lots of puppet-openstack git repos, like maybe it re-clones them from scratch multiple times | 17:58 |
| spotz[m] | Fabian (CentOS infra) traced it further and it's hosted at Vexxhost | 18:00 |
| jcapitao[m] | fungi: I'm afk but I'll take a look later on | 18:02 |
| fungi | jcapitao[m]: thanks!!! | 18:02 |
| fungi | spotz[m]: yes, the ip address is in a block assigned to vexxhost, traceroutes indicate it's in their ca-ymq-1 region | 18:03 |
| apevec | yes, it is our private cloud managed by Vexxhost, where we run our public infra for RDO project | 18:28 |
| apevec | but puppet-openstack shouldn't be special, we have this RDO Trunk service doing basically git pull of all openstack/* repos that we continuously build | 18:30 |
| apevec | it's a periodic cron job and should not be that often, lemme check | 18:31 |
| fungi | in this case we were seeing git requests for primarily openstack/puppet-* repos hit between 15:50 and 16:00 utc at such a rapid rate that the backend it was betting balanced to stopped responding to ssl socket setup requests long enough to get removed from the load balancer pool | 18:32 |
| jcapitao[m] | so I checked the logs of our trunk-builder-centos9.rdoproject.org, there is no anormal activity | 18:55 |
| jcapitao[m] | we are pulling every 5 minutes OpenStack repos to build the new commits | 18:56 |
| apevec | jcapitao: looking at crontabs, all DLRN builders run at default */5 * maybe we could stagger and lower frequency | 18:57 |
| jcapitao[m] | fungi so the activity you noticed between 15:50 and 16:00 utc happens every 5 minutes (maybe in that period, the potential caching mechanism were in bad state) | 18:58 |
| jcapitao[m] | apevec: yeah we could change to */10, another solution is to move to GH mirror repo | 18:59 |
| apevec | ah yes, in rdoinfo | 18:59 |
| apevec | but we could then hit GH rate limiting | 19:00 |
| jcapitao[m] | right.. | 19:00 |
| jcapitao[m] | we should at least lower frequency of older Trunk builder I'd say | 19:01 |
| fungi | yeah, i'm not saying it's new, in opendev we're digging into delays users are reporting and while digging into logs i noticed that server is hitting git hard enough that the backends become so unresponsive they get removed from haproxy, so i'm trying to figure out what it's doing to result in that behavior | 19:02 |
| fungi | is it re-cloning the repos over and over? or just doing a ref update to pull new commits into an existing git cache? | 19:02 |
| fungi | gitea is tuned for handling ref updates well, but mass git clone operations are fairly taxing on it, so definitely please avoid repeatedly running git clone for the same repositories over and over if you can | 19:08 |
| apevec | it is git pull, it should not git clone from scratch every time unless we have a bug in DLRN | 19:09 |
| apevec | in rdoinfo we have upstream: https://opendev.org/openstack/%(project)s maybe git over HTTPS is not the most efficient? | 19:12 |
| jcapitao[m] | fungi: we don't git clone, but git fetch and git reset index and working tree if the repo already cloned | 19:16 |
| jcapitao[m] | i.e https://github.com/softwarefactory-project/DLRN/blob/55ceda49e3241ad65702114f72984bf9c7dd909c/dlrn/repositories.py#L75 | 19:17 |
| jcapitao[m] | and https://github.com/softwarefactory-project/DLRN/blob/55ceda49e3241ad65702114f72984bf9c7dd909c/dlrn/repositories.py#L139 | 19:17 |
| fungi | cool, are you updating repos in parallel or serially? and if serially, do you incorporate a delay between actions for each repo? | 19:19 |
| fungi | (and if in parallel, could you look at switching to serializing it?) | 19:20 |
| jcapitao[m] | we are serializing that part without delay | 19:20 |
| fungi | but also, depending on how long it's taking your cron to complete, i wonder if running every 5 minutes is causing overlap | 19:20 |
| jcapitao[m] | we have a lock file that prevent running twice (thus overlapping) | 19:21 |
| jcapitao[m] | we can add a "sleep" between git request | 19:22 |
| jcapitao[m] | but as apevec mentioned, maybe using SSH as protocol might be better | 19:23 |
| jcapitao[m] | at least for the Web UI enduser | 19:24 |
| apevec | but we have all DLRN builders running at the same time... so there's some parallel hammering | 19:25 |
| apevec | jcapitao: maybe we could switch non-master builders to run at lower frequency, every few hours? | 19:26 |
| jcapitao[m] | good point | 19:28 |
| jcapitao[m] | apevec: let's do that, and also we can change the minute value for each builder | 19:29 |
| jcapitao[m] | https://github.com/rdo-infra/ansible-role-dlrn/blob/bdcd8c58c4c67f5a1a67a72a883cbd4f4882ec7a/tasks/worker.yml#L363 | 19:29 |
| jcapitao[m] | the ansible role gives the ability to do so ^ | 19:30 |
| fungi | thanks! i'll try to check back up on our haproxy logs next week and see if the impact has lessened | 19:31 |
| jcapitao[m] | I'd say every 10 minutes for non master trunk, with 1 minute delay between each non master builder | 19:32 |
| jcapitao[m] | I'll do the change tomorrow, and will notify you fungi once done | 19:33 |
| fungi | much obliged | 19:33 |
| fungi | i take it the various dlrn builders all appear to originate from the same client ip address | 19:41 |
| jcapitao[m] | yes, one DLRN builder is a system user on the same host | 19:44 |
| jcapitao[m] | c.f https://github.com/softwarefactory-project/sf-infra/blob/master/playbooks/group_vars/dlrn-centos9#L22-L326 | 19:45 |
| fungi | makes sense, that does jive with what i was seeing in our logs in that case | 19:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!