Thursday, 2021-04-15

*** mlavalle has quit IRC00:04
*** CeeMac has quit IRC00:24
*** hamalq has quit IRC01:00
ianw2021-04-15 00:55:22,992 ERROR nodepool.driver.NodeRequestHandler[]: [e: 73669cd29bfa4e80adf58a9eedb5e450] [node_request: 300-0013700920] Declining node request due to exception in NodeRequestHandler:01:00
ianwException: Unable to find flavor with min ram: 800001:00
ianwi may have messed up the flavors...01:00
ianwor, i may have screwed up the password?01:02
ianwthat couldn't be though, because it uploaded the imgaes01:03
openstackgerritIan Wienand proposed opendev/system-config master: OSU OSL : fix typo in project id
openstackgerritIan Wienand proposed openstack/project-config master: OSU OSL : use correct flavor name
ianw^ that should fix it, looking for the wrong name01:09
*** ysandeep|away is now known as ysandeep01:16
openstackgerritMerged openstack/project-config master: OSU OSL : use correct flavor name
*** brinzhang_ is now known as brinzhang02:06
ianwok, next problem02:20
ianw409: Client Error for url:, Multiple possible networks found, use a Network ID to be more specific.02:20
ianwthis might be interesting02:25
ianwwe have two subnets &
ianwi guess the /27 has enough room for our current 15 hosts02:28
ianwmaybe?  if we do need to alternate, i'm not sure how to specify that02:28
fungia /27 yields 32 total addresses (minus base, broadcast, and probably a gateway)02:31
fungi64 for a /2602:31
openstackgerritIan Wienand proposed opendev/system-config master: OSU OSL : add default network
openstackgerritMerged opendev/system-config master: OSU OSL : fix typo in project id
ianwyeah, i've reached out to find out what the difference is from their end.  i'll manually patch in the first (called "public4") and see02:34
*** hemanth_n has joined #opendev02:40
openstackgerritMerged opendev/system-config master: nodepool-builder: configure upload workers, reduce nb03
*** gry has left #opendev03:28
ianwNo fixed IP addresses available for network: 48bfc43c-c99e-4395-afbd-97d02ef0116a, not rescheduling.03:36
*** ykarel has joined #opendev03:45
openstackgerritIan Wienand proposed opendev/system-config master: OSU OSL : add default network
*** snapdeal has joined #opendev04:16
openstackgerritIan Wienand proposed openstack/project-config master: Fix OSU OSL typo (again!  this is so easy to turn around!)
openstackgerritMerged openstack/project-config master: Fix OSU OSL typo (again!  this is so easy to turn around!)
*** brinzhang_ has joined #opendev04:45
*** brinzhang has quit IRC04:48
*** whoami-rajat has joined #opendev04:50
*** marios has joined #opendev05:03
openstackgerritIan Wienand proposed openstack/project-config master: nodepool: make ARM64 config names consistent
openstackgerritMerged opendev/system-config master: OSU OSL : add default network
*** vishalmanchanda has joined #opendev05:41
*** sshnaidm|pto has quit IRC05:47
*** sshnaidm|pto has joined #opendev05:48
*** slaweq has quit IRC05:55
*** slaweq_ has joined #opendev05:55
*** slaweq_ is now known as slaweq05:55
fricklerfungi: since you didn't generate the new signing keys yet (or did I just look wrong?), maybe we can discuss key parameters. my initial idea was to move to rsa4096, which I seemed to remember having discussed some time ago already, but couldn't find it in my logs05:59
*** ykarel has quit IRC06:00
*** ykarel has joined #opendev06:00
fricklerthen I looked at your original spec, which mentions a possible transition to ed25519 at some point, which makes me wonder whether it would be feasible as a first step to have both key types and do two signatures for everything06:01
*** eolivare has joined #opendev06:09
*** ralonsoh has joined #opendev06:10
*** sboyron has joined #opendev06:29
*** slaweq_ has joined #opendev06:30
*** icey has quit IRC06:34
*** icey has joined #opendev06:35
*** slaweq has quit IRC06:36
*** slaweq_ is now known as slaweq06:36
*** fressi has joined #opendev06:47
*** ykarel_ has joined #opendev06:49
*** ykarel has quit IRC06:51
*** jpena|off is now known as jpena06:55
*** amoralej|off is now known as amoralej07:05
*** jaicaa has quit IRC07:10
*** jaicaa has joined #opendev07:13
*** andrewbonney has joined #opendev07:24
*** ykarel_ is now known as ykarel07:30
*** rpittau|afk is now known as rpittau07:45
*** tosky has joined #opendev07:46
*** dtantsur|afk is now known as dtantsur07:59
*** hemanth_n has quit IRC08:32
*** ykarel is now known as ykarel|lunch08:33
*** ysandeep is now known as ysandeep|lunch08:49
*** slaweq has quit IRC08:51
*** slaweq has joined #opendev08:51
*** vishalmanchanda has quit IRC09:27
*** ysandeep|lunch is now known as ysandeep09:32
*** DSpider has joined #opendev09:35
*** ykarel|lunch is now known as ykarel09:38
*** dpawlik9 has quit IRC09:46
*** vishalmanchanda has joined #opendev09:48
*** snapdeal has quit IRC09:52
*** dpawlik4 has joined #opendev09:59
*** ykarel_ has joined #opendev10:29
*** ykarel has quit IRC10:32
*** hrw has joined #opendev10:35
hrwkevinz: INFO:kolla.common.utils.placement-base:E: Failed to fetch  403  Forbidden [IP: 2604:1380:4111:3e54:f816:3eff:fe17:6b17 80]10:36
hrwok, works again10:38
hrwis there a way to recheck only check-arm64 queue?10:43
*** ykarel_ has quit IRC10:44
*** ykarel_ has joined #opendev10:45
*** ykarel_ is now known as ykarel11:15
*** jpena is now known as jpena|lunch11:31
*** ykarel_ has joined #opendev11:54
*** ykarel has quit IRC11:56
*** ykarel_ is now known as ykarel12:02
*** CeeMac has joined #opendev12:05
*** jpena|lunch is now known as jpena12:33
*** amoralej is now known as amoralej|lunch12:36
*** snapdeal has joined #opendev12:37
*** snapdeal has quit IRC12:42
*** snapdeal has joined #opendev12:44
fungifrickler: i haven't generated it yet, no, was probably going to do that today. switching the key type would be easy enough, but signing with two keys would need changes to the underlying ansible role13:06
fungialso what's wrong with 3072-bit rsa? seems to still be gnupg's default13:08
*** marios is now known as marios|call13:15
fungiaha, looks like gnupg 2.3.0 changes the default to rsa409613:18
*** amoralej|lunch is now known as amoralej13:29
fungier, actually that's the python-gnupg docs, not the gnupg docs13:34
*** klonn has joined #opendev13:39
*** klonn has quit IRC13:42
*** marios|call is now known as marios13:55
*** diablo_rojo has joined #opendev13:56
mordredLinkedIn advertised this to me today: ... I totally thought it was a joke, because it sure does look like a cooking show. but it does, indeed, seem to be a c++ course, and they sure do look like they're going to be using food and cooking metaphors as they write multi-threaded code14:25
mordred(the intro seems to be available without login or subscription)14:25
fungialso when they reach the point where it's time to put the source into the compiler, they just pull out one they compiled earlier to save time14:26
fungicooking show methods totally work for programming classes14:26
fungihrw: yes! you want "check arm64" according to
fungihrw: for those 403 errors from the mirror, we see evidence of intermittent network connectivity in that cloud, ianw was looking into it yesterday14:29
*** avass has quit IRC14:42
*** avass has joined #opendev14:43
*** snapdeal has quit IRC14:44
hrwfungi: thanks15:25
fungimy pleasure15:26
*** ysandeep is now known as ysandeep|dinner15:36
*** mlavalle has joined #opendev15:36
fungifrickler: okay, so gnupg 2.3.0 (2021-04-07) switched the default key algorithm to ed25519/cv25519, that's all the convincing i need:;a=blob;f=NEWS;h=6aec353e359b2502018b572b0bc869e3dac518cb;hb=refs/heads/master#l2715:40
fungiwilling to give that a try as long as the 2.2.4 on bridge has support (which i believe it does)15:41
fungii'll know shortly15:43
*** ykarel has quit IRC15:47
johnsomCloning into 'releases'...15:49
johnsomfatal: unable to access '': GnuTLS recv error (-110): The TLS connection was non-properly terminated.15:49
fungithat doesn't look good15:50
fungii'll check the servers, see if one is having a bad day15:50
johnsomHmmm, that was on a focal host that updated yesterday. So, not sure if it's just me or ...15:50
johnsomsubject: CN=gitea02.opendev.org15:51
fungiestablished tcp connections on the lb shot waaay up in the past few minutes, suggesting one of the backends has turned into a tarpit15:51
fungiyep, 02 looks about to oom15:52
fungii'll take it out of the lb pool15:52
fungi#status log Temporarily disabled the gitea02 backend in haproxy due to impending memory exhaustion15:53
openstackstatusfungi: finished logging15:53
fungijohnsom: working now?15:53
fungii have a feeling, based on what we saw from the openstackansible user a few weeks back, that the wallaby release is going to trigger a bunch of this15:54
johnsomHmm, nope, the TLS isn't completing for me now15:54
clarkbya we're just redirecting the hose when we do that15:54
johnsomAh, just slow. Got 6 this time.15:54
fungi01 is also being slammed15:54
fungimight be all of them15:54
johnsomYeah, not cloning15:55
fungi03 as well15:55
fungiseems we're under a distributed denial of service attack. i'll see if i can map them all to a single netblock15:55
clarkbfungi: let me know if I can help15:55
fungii've put 02 back into the pool for now while i dig deeper15:56
fungiunfortunately nothing as simple as connection count is going to tell us who's doing it15:57
fungipretty sure this is going to wind up being another case of "i told my 1000 node cluster to upgrade openstack, so all the servers are cloning nova now"15:58
fungii guess i'll try to sample some clone requests from one of the backends and then map that in the haproxy log to client addresses15:59
*** hamalq has joined #opendev16:00
fungimade much harder by the fact that they're in swap thrash now16:01
openstackgerritClark Boylan proposed opendev/system-config master: Upgrade gitea to 1.13.7
*** marios is now known as marios|out16:01
clarkbfungi: ^ I don't think that will help, but always a good reminder when this sort of thing happens16:01
clarkbfungi: ya that is roughly what I have done in the past, look at the access log to see which requests are very slow/large on a gitea server. Then map back to load balancer16:02
*** hamalq has quit IRC16:02
*** hamalq has joined #opendev16:02
fungii have a feeling this might be wal-mart16:05
*** jpena is now known as jpena|off16:05
fungineed to map more samples back, but this is somewhat tedious and takes time16:06
*** rpittau is now known as rpittau|afk16:09
fungiit seems like nobody besides us uses reverse dns these days16:11
johnsomLooks like there is a gitea patch proposed to add PROXY protocol support, so there is hope in sight that you can get the actual client addresses in the near-ish future16:12
clarkboh that is great news16:13
fungijohnsom: well, we don't do layer 7 proxying at the moment though, so likely won't help us16:13
fungithis is straight up layer 4, tls is terminated on the backends16:13
johnsomPROXY protocol works with all protocols via HAProxy16:13
fungioh, wait that right16:14
johnsomYeah, that is not a problem16:14
fungii was thinking of x-forwarded-for16:14
clarkbya this is an haproxy thing16:14
clarkband then you need webservers that also support it (not sure if apache does out of the box but one problem at a time)16:14
johnsomWell, almost everything supports it now, but HAProxy started it, yes16:14
fungiso not plain tcp, but apache can parse it16:14
fungia wrapper basically16:15
fungi(tcp-in-tcp essentially)16:15
johnsomIt's prefix bytes to the TCP connection16:15
johnsomSo, right before the TLS in your case16:15
fungiyep, i remember investigating it before for something else16:15
fungianyway, we could be using it now, we don't actually terminate ssl with gitea, we terminate it with an apache running on the gitea servers16:16
johnsomAh, yeah, you can totally switch that on now if you have a frontend web server for gitea16:16
fungiso we could just do it between haproxy and apache16:17
fungiwe're relying on apache to be able to implement user agent based filters for certain really annoying botnets16:17
clarkbmod remoteip apparently16:18
johnsomYep, mod_remoteip16:21
johnsommod_proxy_protocol is getting merged into mod_remoteip16:21
clarkbfungi: fwiw I can hit 01 now without any issue16:21
clarkbperhaps whatever it was has settled down?16:21
johnsomYeah, I just got a clone16:22
*** marios|out has quit IRC16:22
fungiunfortunately the investigation on gitea02 so far indicates that between 15:40 and 16:00 utc, the most frequent requesters of an openstack/nova git-upload-pack were a couple of addresses whois says are registered to wal-mart16:23
fungii'll check 06 for a second data point16:23
*** mlavalle has quit IRC16:23
*** _mlavalle_1 has joined #opendev16:24
fungiyep, two different addresses in the same netblock16:24
clarkbsince things seem to be settling I'm going to look at my todo list again which says it is time to clean up external ids16:24
johnsomFYI, I recommend using proxy protocol v2, the config in haproxy is send-proxy-v2. There are additional options if you are TLS offloading, but it sounds like you are not.16:25
clarkbping me if you need help with gitea (or anything else) but I'm going to focus on running the externalid cleanups for the accounts I recently retired then will also rerun consistency chceks on gerrit16:25
fungiright, it's probably too late to go blocking them now, but i've also got a method to be able to map these a little faster next time it hits16:25
fungiestablished connections in haproxy is falling sharply at this point so i expect it's over for the moment16:26
fungiin the span of 20 minutes we saw what seemed to be ~500 attempts to clone nova from addresses in that netblock16:27
fungiextrapolating across our backends16:27
smcginnisLooks like still issues. I've been waiting for to load for several minutes.16:32
clarkbsmcginnis: ya I think its falling off, not necessarily completely happy yet16:34
fungiwell, in the latest sample, looks like it could be spiking back up again16:34
fungiif this is similar to what we saw with the osa user a few weeks back, they tried to deploy an upgrade, that broke with a bunch of git errors, so once they saw the failures reported back they tried again16:34
fungiand again16:35
fungiand again16:35
clarkbah yup there is a recent spike on cacti16:35
fungii'll see if it's still the same addresses16:35
clarkbit was falling off :)16:35
*** dtantsur is now known as dtantsur|afk16:36
fungiyep, same class c network16:37
clarkbthe external id cleanups ran and log file has been put in the normal spot on review16:37
clarkbI'm going to run consistecny checking next16:38
fungii'm blocking it temporarily on the lb16:38
clarkbfungi: that seems reasonable and maybe they will reach otu and we can ask tehm to not ddos us16:38
jrosseri wonder if jmccrory is still at walmart16:38
fungi#status log Temporarily blocked in iptables on to limit impact from excessive git clone requests16:39
openstackstatusfungi: finished logging16:39
fungijrosser: are they likely to also be using osa, like our previous incident? maybe hitting the same bug?16:39
jrosserwell i just put 2 and 2 together and get 5 maybe as jimmy was with walmart and also a known OSA contributor/user16:41
fungiahh, okay16:41
*** _mlavalle_1 has quit IRC16:42
jrosserfwiw the behaviour that the folk from had where it cloned everything 100s of times is totally wrong and i've reached out to them since to try and unpick whats happening in their environment16:45
fungimemory utilization seems to be falling on the backends again, so seems like those addresses were probably the source of the expensive requests16:46
clarkbfungi: connection count is falling way off on the lb too16:46
fungithough that's a secondary indicator16:46
fungibasically backends start responding very slowly and all the normal incoming requests pile up16:47
fungiin theory we'll see the swap thrash subside on the backends before the connection count recovers16:47
*** amoralej is now known as amoralej|off16:48
clarkb281 is not unique errors now (down from 334)16:49
fungifor the gerrit account collisions?16:51
clarkband that is down overall from 64316:51
clarkb334 was the previous state before this round of cleanups16:52
fungineed to take a paperwork break to scan some paper records, but i'll check back in on the gitea farm in a bit16:52
*** eolivare has quit IRC16:55
clarkbdoesn't seem to be spiking back up again16:56
fungiyep, load averages have also all fallen <1 now16:58
fungiso whatever needed to be paged back in has been i think16:58
fungiand caches are rewarmned16:58
*** eolivare has joined #opendev16:59
clarkbI've got my audit script rerunning so that I don't see the old data for the accounst that were just cleaned up. When that is done I'm going to context switch to reviewing zuul changes and booting new zk servers though. Don't think I'll get to the next batch of account cleanups for a little bit17:04
*** mlavalle has joined #opendev17:05
*** ralonsoh has quit IRC17:12
*** eolivare has quit IRC17:14
*** ysandeep|dinner is now known as ysandeep17:16
clarkbcacti continues to make gitea look stable17:20
fungithough i feel bad for the wal-mart sysadmins, i hope they reach out soon17:23
*** zul has joined #opendev17:26
clarkbaudit has completed and it looks about how I expect it. I think I'll call that done for now as everyting looks good17:33
*** andrewbonney has quit IRC17:50
mnaserinfra-root, infra-core: docker hub registry is fully down right now -- so just a heads up in case jobs start reporting failures =)18:00
clarkbmnaser: do you know if they have a status page where they will track that? usually we pass that sort of thing along to people who ask about when it might be fixed18:03
clarkb(I can google for it too when I finish these reviews)18:03
mnaserclarkb: :)18:04
mnaserand more specifically, is the incident18:04
fungithat's certainly fun18:08
* fungi bets it's all the kolla users upgrading to wallaby18:08
fungithough i think kolla hasn't tagged wallaby yet18:08
fungithings are still looking okay with the git servers, it seems18:10
clarkbI'm starting the boot,, and I don't expect issues related to the docker hub outage as we don't do docker things at launch18:14
clarkbthen tomorrow we can start stumbling through replacement of the old servers18:15
clarkb reports they are operational now18:16
clarkbso from at least 17:55UTC to ~18:16UTC there could be problems in job logs18:17
clarkbI suspect it may have started a bit before 17:55UTC too but that is what they recorded on the incident page18:17
*** vishalmanchanda has quit IRC18:27
*** sboyron has quit IRC18:42
openstackgerritClark Boylan proposed opendev/ master: Add new zookeeper servers to DNS
*** hamalq has quit IRC19:17
openstackgerritClark Boylan proposed opendev/system-config master: Add
clarkbI'm WIP'ing ^ but that should be the first change we need to implement option A at
clarkbcorvus: ^ for that zk work one of the steps in my document is to update the client configs by hand to see all three new servers when the first two are rotated in. This way we only need to do one restart of the clients. The only clients that currently matter are zuul-scheduler, nodepool launchers and builders? Should I plan to do the mergers and executors too?19:22
clarkbI did just confirm that 02 is still the leader so the order there is correct at least for now19:24
*** hamalq has joined #opendev19:31
corvusclarkb: currently scheduler and nodepool yes.  i expect us to add executors to that very soon (next week?), but will require executor restarts of course, so as long as system-config tracks your by-hand work closely (so the new config is on disk when the executors are restarted), shouldn't be an issue.19:43
clarkbcorvus: yup I believe zuul will be writing out the updates as the group membership changes19:45
clarkbso it should track, the by hand step is merely there to ensure we don't have to restart frequently (we can lie and tell zuul/nodepool there is a future config state we haven't quite reached yet )19:45
clarkbfwiw I did spend some time today double checking the dynamic reconfiguration support in zk and it seems complicated enough that not relying on it for now seems ideal19:46
clarkb(there are new config files involved and you have to configure acls etc)19:46
clarkbthat said, their docs give clients a short recipe for tracking those changes and we might want to consider switching to it in the future and then updating zuul and nodepool to auto shift their connections based on watches of the config data19:47
*** whoami-rajat has quit IRC19:47
corvusclarkb: note if we do that we may need to look into details about how we're handling the config in the containers19:48
clarkbcorvus: yup that is why I'm punting, it seems like a neat feature to support but also complicated so I didn't want to mix that into this process19:48
corvuswe're mostly steamrolling over the dynamic update feature to make our "git is authoritative" approach work19:49
clarkbcorvus: it is also disabled by default due to security concerns19:49
clarkbwhcih is another aspect to consider19:49
clarkbI've also started thinking about what a zuul scheduelr replacement looks like. It appears that we don't actually auto start the zuul scheduler when deploying it. I think that means we can deploy a, have ansible write out configs and lay down dirs as well as pull containers. Then we stop 01's zuul, sync secrets (and maybe even git repos) to 02, then start zuul on 0219:50
clarkbupdate DNS and in theory we've done a fairly seemless upgrade there19:50
*** auristor has quit IRC19:52
clarkbI think we can also shrink the new zuul server. We may run into OOM problems more quickly but that may even be a good thing?19:53
clarkbthe new zk servers should be the same flavor as the old ones fwiw19:53
*** diablo_rojo has quit IRC19:54
*** slittle1 has joined #opendev19:57
*** auristor has joined #opendev19:59
slittle1Hmmm... having trouble creating our StarlingX release branchs19:59
slittle1My notes say this used to work  ...20:00
*** ysandeep is now known as ysandeep|away20:01
slittle1git push --tags gerrit r/stx.5.020:01
clarkbslittle1: branches need to be created through the web ui20:02
clarkbthe command you have pasted looks like one to push tags20:02
clarkb(I suppose you could do it via the rest api too, but branch creation via git requires force push access and that isn't typically available)20:03
openstackgerritClark Boylan proposed opendev/gerritlib master: Add function to set project parent
fungibranch creation via git requires direct push rights yes (but not push --force)20:09
openstackgerritClark Boylan proposed opendev/jeepyb master: Set gerrit project parents
slittle1I thought membership in group starlingx-release gave be the power20:10
clarkbfungi: ^ I don't know that those are complete and it may be worth WIPing the jeepyb change until we're happy with it20:10
slittle1if not, tell me more about the rest api20:11
clarkbslittle1: it will depend on the exact acl config you've got in place. But typically the power you have allows you to create branches via the web ui or the rest api but not through git20:11
fungislittle1: membership in group starlingx-release currently gives power to create branches via the webui or rest api, but you need a different permission to do it via git push (and it would also allow you to completely bypass code review)20:11
*** diablo_rojo has joined #opendev20:11
clarkbslittle1: is the first thing to look at for the rest api as you will need to authenticate for this.20:11
clarkbslittle1: then you can use to create the branch20:12
fungii wouldn't recommend setting direct push permissions for a group unless the members of that group use separate accounts for it so they won't accidentally bypass code review and wind up pushing code directly to a branch20:12
clarkbfungi: I have WIP'd 78650120:13
fungiclarkb: ooh, thanks! i'll take a look after i finish dinner20:13
slittle1the main idea here is to have a script that walks over all our git repos, pushes a new branch, a new tag, and a .gitreview update more without review20:14
clarkbslittle1: creating the branch and tag you can do without review (in fact there isn't a way to review those). Then you could push the .gitreview change up for review and autoapprove it20:15
fungislittle1: you could do it via curl and an api password in that case, or use a dedicated account which isn't normally used for interactive git activity to avoid accidents20:15
fungithe gerrit rest api is scriptable20:16
openstackgerritClark Boylan proposed opendev/system-config master: Add
clarkbneat if you push a new ps to a wip change it stays wip by default20:21
clarkbslittle1: that is a python script I wrote semi recently thatuses the rest api for other purposes but does both reads and writes and may be helpful20:23
slittle1hmm, Found the release branch scripts of an old colleague... he was using 'git push' the same way i was20:27
slittle1At some point, that must have been permitted for our release group20:28
fungiwhat repository are you trying to create a branch on?20:28
fungiand was that script previously used to create branches in gerrit or on the earlier starlingx github repos?20:29
clarkbalso possible there was a behavior change we didn't expect in the gerrit upgrade20:29
clarkbbut the behavior slittle1 describes now is what I would've expected pre upgrade too (and I expect it now)20:29
slittle1pretty much all the starlingx/* repos ... e.g starlingx/tools.git20:30
clarkbslittle1: is the acl config for that repo20:32
clarkbslittle1: it says you can create the branch (which is what allows you to do rest api or web ui branch creation), but there is no push permission20:32
clarkbyou would need a push permission on refs/heads/* to push a new branch to it that way using git20:33
clarkbare you sure the old script you are looking at wasn't just pushing tags?20:33
clarkbbecause the command you pasted would be the sort of thing I would expect for someone pushing a tag20:33
slittle1It did both20:34
clarkband does allow pushing tags20:34
fungieven on the older version of gerrit we were running (2.13) the only way to allow branch creation via git push was to grant push in the [access "refs/heads/*"] section of the acl. the create permission which is currently there in your acls has only permitted creation via the webui and rest api. the access control docs point out the difference (and also indicate that push rights allow bypassing code review for20:34
fungipushing commits directly to branches):
slittle1look at release/ within starlingx/tools.git20:34
slittle1perhaps we had the power a few years ago and lost it during one of the upgrades20:35
fungiit looks to me like that script expected the target branch to be precreated in gerrit, and it's pushing the release tags for it?20:36
fungiahh, it's referencing a tag for a local branch20:37
slittle1No, it creates the branch it not found, which was the normal case20:39
clarkb I think that is the key clue?20:39
clarkbit does seem toe expect SRC_BRANCH and BRANCH to be identical?20:40
clarkboh wiat nevermind I see what that is saying20:40
clarkbupdate the gitreview file if the branch has shifted20:40
slittle1No, they should be differnt20:40
clarkbso ya I'm not sure how that would've ever worked looking at the acl configs for a few starlingx projects20:40
clarkbthe git push to create a branch requires permissions that are not there20:40
slittle1The stuff you are keying in on was just there to resume a run that failed part way through20:41
clarkbhowever, the permissions that are there allow you to create the branch via a different method20:41
slittle1yep, gotta figure out the new method20:41
clarkbtoo bad dtroyer doesn't appear to be on irc anymore, we could ask :)20:43
fungislittle1: there is also an ssh cli command you can use with the current permissions:
fungiif you're authenticating other things via ssh, that may be the easiest solution20:44
clarkbfungi: oh good catch20:45
slittle1ssh -p 29418 gerrit create-branch   >20:46
fungissh -p 29418 gerrit create-branch starlingx/tools newbranch deadbeef12345678...20:48
fungirelies on the create reference permission, which is what the acl has for the starlingx-release group20:49
clarkbfungi: maybe we should think about resurrecting re the gitea sadness earlier today20:56
clarkbthe trick with that chagne remains figuring out a viable limit that allows NAT'd users through while also restricting floods20:56
slittle1is there a create-tag as well ?21:00
clarkbslittle1: no, that you git push21:01
fungitags need local key material to sign anyway, so gerrit couldn't technically "create" those anyway21:01
slittle1git push  gerrit test_tag_123:test_tag_12321:02
clarkbthe rest api does have a method for that but that must be an unsigned tag21:02
clarkb`git push gerrit tag test_tag_123` iirc21:03
clarkb"tag <tag> means the same as refs/tags/<tag>:refs/tags/<tag>" from the manpage for git push21:03
clarkbso that is shorthand for `git push gerrit refs/tags/test_tag_123:refs/tags/test_tag_123`21:03
slittle1git push  gerrit tag test_tag_12321:04
slittle1 ! [remote rejected] test_tag_123 -> test_tag_123 (prohibited by Gerrit: not permitted: create)21:04
fungigit push gerrit test_tag_12321:04
fungibut also it needs to be a signed tag21:04
fungistarlingx-release has permission to push signed tags, not unsigned tags21:04
clarkbNote you won't be able to delete that tag if you push it21:05
clarkb(note sure how much you'll care about that)21:05
clarkbas a side note, I think git has the ability to tell you where a branch diverged from its parent. If that is what this tag is for it may not be necessary (though perhaps simpler to just check the tag value than inspect history)21:10
fungispelunking through the infra-manual history, back in 2014 we added instructions for openstack stable branch creation which look like they involved openstack's release managers using git push to create them (but i don't recall that ever working unless they were also administrators). unfortunately then in march of last year a change was made to the manual copying this option into the general section on branch21:25
clarkbThe only time I can remember promoting release managers was for deletions not creations21:26
fungiso up until roughly a year ago, we at least had a sentence in the manual saying branch creation by git push isn't expected to work, but that was lost in the branch creation section refactor21:29
clarkbfungi: looks like it may be working to set the project parent21:31
clarkbwe should double check that jeepyb is being run multiple times against that project to ensure the cache udpate is working. We may also want to start inspecting the resulting state a bit better21:32
fungiit was in the feature branch section before: "To get started with a feature branch you will need to create the new branch in Gerrit with the 'feature/' prefix. Note that Gerrit ACLs do not allow for pushing of new branches via git, but specific groups of Gerrit users can create new branches."21:32
fungiadded by in 201421:32
fungistrangely the git push recommendation for creating proposed/.* branches was added to the manual that same month by
fungiclarkb: i don't think 774023 would have helped much today since it was lots of different addresses spread out over the entire cluster... i expect they were each only doing a connection or two at a time, but retrying after not getting any response21:54
clarkbI see so not one doing a bunch of concurrent requests * bignumber but one doing one request * big number21:55
fungihowever the git-upload-pack requests were probably continuing to be processed by the backend even after the client disconnected and retried21:55
fungionce it gets into a sad state, it likely starts a chain reaction which won't end until the client gives up and stops retrying to clone21:56
clarkbfungi: do you think we can land that one in prep for using those new zk servers sometime soon (kinda sounds like airship might be delaying more ...)22:00
fungilgtm, approved it22:04
clarkbthe WIP change to do the swapout of the first node passes testing now too22:06
openstackgerritMerged opendev/ master: Add new zookeeper servers to DNS
openstackgerritClark Boylan proposed opendev/gerritlib master: Add function to set project parent
clarkbfungi: ^ that adds a bit more testing confirmation of state we want. I've rechecked the jeepyb change too22:23
fungigit status22:29
fungiheh, you're not my command shell!22:29
clarkbon branch irc22:29
clarkbYour branch is up to date with freenode/irc22:29
openstackgerritJeremy Stanley proposed opendev/infra-manual master: Update branch creation for PolyGerrit
fungicapturing the earlier conversation while it's fresh in our minds ^22:31
*** tosky has quit IRC22:34
openstackgerritClark Boylan proposed opendev/gerritlib master: Add function to set project parent
*** gothicserpent has quit IRC23:02
*** cenne is now known as cenne|out23:08
clarkbcool ^ that fails the way I expect it to now. The jeepyb side should pass23:12
clarkbhrm jeepyb fails with the same error23:13
openstackgerritClark Boylan proposed opendev/gerritlib master: Add function to set project parent
*** gothicserpent has joined #opendev23:16
clarkband the latest batch of testing hit the dockerhub rate limit. Might be time to call it a day23:26
fungidockerhub sounds the shift change whistle23:31
fungii've still gotta stick around to at least see the end of the tc vacancy poll through23:32
fungibut that's just in a little over 10 minutes at this point23:32

Generated by 2.17.2 by Marius Gedminas - find it at!