Thursday, 2021-02-04

ianwnot related to current discussion, but do we know why infra-prod base is failing?
ianwi'm looking now00:05
ianwhrm, seems involved00:06
ianwECDSA host key for has changed and you have requested strict checking.00:08
*** hamalq has quit IRC00:12
*** hamalq has joined #opendev00:12
openstackgerritIan Wienand proposed opendev/system-config master: Update airship mirror address
clarkbI think the cloud launcher is also having trouble recently. I was going to look into that as part of inmotion cloud spinup.00:15
clarkbI did start that this morning and ran into a few issues. Emailed them and just recently got a response back saying things should be better, so I'll see if Ic an giveit another go tomorrow00:16
clarkbthinking about the haproxy thing a bit more we could also try rate limiting00:18
clarkbhaproxy does support it though you have to build it up out of some pretty basic primitives (so it doesn't look very intuitive)00:18
clarkbI am now learning me an haproxy rate limit setup00:19
clarkbmost of the examples people have written up seem to be http specific, but it seems that since you build up with simple primitives a version for tcp could be devices00:20
*** rchurch has quit IRC00:23
*** mlavalle has quit IRC00:26
*** rchurch has joined #opendev00:26
ianwis that rate-limiting connections, or throughput?00:26
clarkball of the examples use connections. If there is a primitive for throughput we can probably make that work too00:27
clarkbI'm trying to find where haproxy documents all of these built in primitives that you use to build up the table00:27
*** mlavalle has joined #opendev00:27
clarkbhttp_req_rate(10s) <- is the http request rate over a 10 second period for example00:27
clarkbthen you can compare that to some constant you define as the limit and decide to allow new requests or not. I assume there are similar things for tcp and probably throughput but haven't found the docs for that yet00:28
clarkbianw: yup there is conn_cur, conn_rate, and bytes_out_rate00:30
clarkbfwiw I haven't decided if I think this is a good idea, but wanted to read up on it to understand what is possible00:30
ianwsimilar to the http redirect stuff, it's probably good to have tested and there behind a switch for when we really need it00:31
ianwat least I mean ... if the limits are reasonable and it's working, i guess we should just keep it on00:32
clarkbsounds like the bytes_out_rate is more typcially used for CDN billing type purposes ( you record the data in the same tables and instaed of rejecting connections based on that you bill with it? )00:33
clarkbI think the risk with using something like that to rate limit is you actually want transfers to go as quickly as possible and complete00:33
clarkb has been the most useful doc so far fwiw00:34
fungiclarkb: i thought we already added logging of the forward source port in haproxy for this very reason?00:37
clarkbfungi: for rate limiting?00:38
fungino, source port for the forwarded socket00:40
fungiit's been in place since june00:40
fungisorry, catching up from having stepped away from the keyboard for a bit00:40
fungioh, nevermind, you said "logging the apache source port on the connection to gitea"00:41
clarkbfungi: oh yup, the problem is we have a second proxy involved now00:41
fungiyeah i don't think we do that yet00:41
clarkbwe are good between haproxy and apache00:41
clarkbnot apache and gitea, though apache and gitea log roughly the same stuff so its probably not an urgent need00:41
openstackgerritMerged opendev/system-config master: Update airship mirror address
fungiianw: oh! thanks for spotting that, i missed it when updating dns00:47
openstackgerritClark Boylan proposed opendev/system-config master: Preliminary haproxy rate limiting ideas
clarkbI'm going to mark that workinprogress because I'm almost certain the rules in that change are wrong00:58
clarkbbut I think having the framework in place will help us talk about what might be right if we want to take it further00:58
clarkbalso I think that haproxy role might be trying to be generic and I've just shoved some super site specific info in its config. I'll have to work on making it configurable00:59
ianwi have which makes the haproxy stuff more generic01:05
ianwthat was from when we were thinking of putting static behind it01:05
openstackgerritJeremy Stanley proposed opendev/bindep master: Move all jobs in-repo
openstackgerritJeremy Stanley proposed opendev/bindep master: Build docs for OpenDev
openstackgerritJeremy Stanley proposed opendev/bindep master: Remove release note about rpm path references
openstackgerritJeremy Stanley proposed opendev/bindep master: Move all jobs in-repo
openstackgerritJeremy Stanley proposed opendev/bindep master: Build docs for OpenDev
openstackgerritJeremy Stanley proposed opendev/bindep master: Remove release note about rpm path references
*** dviroel has quit IRC02:13
*** DSpider has quit IRC02:13
*** hamalq has quit IRC02:45
openstackgerritMerged opendev/system-config master: Remove AFS puppet
*** ysandeep|away is now known as ysandeep04:31
*** ysandeep is now known as ysandeep|ruck04:31
*** ykarel has joined #opendev04:49
*** auristor has quit IRC05:07
*** hemanth_n has joined #opendev05:16
*** ykarel has quit IRC05:17
*** ykarel has joined #opendev05:18
*** whoami-rajat__ has joined #opendev05:23
*** auristor has joined #opendev05:36
*** ykarel_ has joined #opendev05:51
*** ykarel has quit IRC05:53
*** ykarel_ is now known as ykarel06:00
*** marios has joined #opendev06:04
openstackgerritOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml
openstackgerritMerged openstack/project-config master: Normalize projects.yaml
*** slaweq has joined #opendev07:15
fricklermnaser: any update on the IPv6 issue? let me know if you need help with that. I also found which recommends filtering /48s, so I would strongly suggest to you to also announce and document your full /3207:23
*** sboyron_ has joined #opendev07:25
*** eolivare has joined #opendev07:31
*** ralonsoh has joined #opendev07:36
*** rpittau|afk is now known as rpittau07:39
*** jpena|off is now known as jpena07:51
*** fressi has joined #opendev07:59
*** jaicaa has quit IRC08:09
*** jaicaa has joined #opendev08:10
*** hashar has joined #opendev08:18
ykarelis there some known issue with limestome mirrors? i see [Errno 14] curl#7 - "Failed connect to; No route to host" in jobs08:21
*** cgoncalves has quit IRC08:31
*** cgoncalves has joined #opendev08:33
*** haleyb has quit IRC08:34
*** tosky has joined #opendev08:46
*** andrewbonney has joined #opendev08:50
*** fbo|off is now known as fbo08:55
*** valery_t has joined #opendev09:11
*** ykarel is now known as ykarel|lunch09:15
openstackgerritDmitriy Rabotyagov proposed openstack/project-config master: Add ansible-role-pki to zuul
*** DSpider has joined #opendev09:28
*** icey has quit IRC09:46
*** icey has joined #opendev09:50
*** eolivare_ has joined #opendev10:14
*** ykarel|lunch is now known as ykarel10:16
*** eolivare has quit IRC10:17
*** dviroel has joined #opendev10:47
*** hashar is now known as hasharLunch10:50
*** dtantsur|afk is now known as dtantsur11:01
ysandeep|ruckykarel, i also noticed issue with limestone mirrors for some jobs , I have also opened a bug for tracking
openstackLaunchpad bug 1914585 in tripleo "Content provider jobs failed after failing to connect to mirrors, Error: Failed to download metadata for repo 'delorean-current': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried" [High,Triaged]11:30
ykarelokk let's see if some one from infra can look into it11:31
*** auristor has quit IRC11:33
*** valery_t has quit IRC11:35
*** auristor has joined #opendev11:35
*** arxcruz|ruck has joined #opendev11:43
*** sshnaidm|afk is now known as sshnaidm11:53
*** hasharLunch has quit IRC11:58
*** auristor has quit IRC12:05
*** auristor has joined #opendev12:06
*** hemanth_n has quit IRC12:22
*** jpena is now known as jpena|lunch12:30
*** hasharLunch has joined #opendev12:46
*** mtreinish has joined #opendev13:06
*** dtantsur is now known as dtantsur|brb13:14
*** hasharLunch is now known as hashar13:19
*** jpena|lunch is now known as jpena13:33
*** roman_g has joined #opendev13:48
roman_gHello team. May I ask you for the OpenStack project name which is used to launch CI instances by Zuul on KNA1 (CityCloud)? There is one called 'Airship_Opendev_Mirror', but I suspect that this one is used for the package mirrors (permanent) VM instance. Thank you!13:51
*** JohnnyRainbow has quit IRC13:59
*** DSpider has quit IRC14:05
*** JohnnyRainbow has joined #opendev14:05
*** DSpider has joined #opendev14:05
*** dtantsur|brb is now known as dtantsur14:16
roman_gmordred thank you! (I've searched in a wrong place)14:32
mordredroman_g: there are many places one can search :)14:32
roman_gI've expected it to be under openstack configs, not opendev. What's the best way to search code in all* projects?14:33
roman_ggrep -Rni whatever .14:34
roman_gjrosser bookmarked. Thank you!14:35
*** fressi has quit IRC15:09
fungiykarel: ysandeep|ruck: i wonder if there are networking problems in there. i can get to that url just fine, but sounds like maybe some nodes we boot in there can't?15:17
*** hashar has quit IRC15:19
fungiroman_g: are you having trouble with the nodes in kna1 again? we had to stop using that provider over the weekend because some network incident there caused the mirror server's network to no longer have any connectivity at all. we finally got it added back on tuesday after confirming things were reachable again15:20
roman_gfungi , we have noticed that. Thank you for your work.15:20
roman_gWe found issues with mirror VM, then they disappeared (thanks to you).15:21
fungiokay, just making sure you were aware it was off for a few days15:21
roman_gAnd then we got a number of NODE_FAILURE's15:21
roman_gNow I'm working with CityCloud to check on quotes and their nova failures to schedule new instances.15:22
fungiunfortunately our only listed contact for that environment is someone at ericsson (i guess that's who pays the bill?) but noonedeadpunk was able to get the right folks in citycloud looking into it for us15:22
roman_gI'm in contact with CityCloud. We are fine here.15:22
roman_gHowever I would appreciate to know financial contact name from Ericsson (because quota affects the bill).15:23
roman_gTo private msgs, please.15:23
fungiroman_g: sure, just a sec and i'll look it up15:25
*** sshnaidm is now known as sshnaidm|afk15:48
openstackgerritMerged opendev/system-config master: Remove karbor channel from bot list
ysandeep|ruckroman_g, fungi, thanks! is it okay to recheck the jobs now, if issue with mirror cleared?15:55
roman_gysandeep|ruck sure.15:56
fungiysandeep|ruck: roman_g: i think you're talking about different providers15:56
fungiysandeep|ruck was seeing intermittent issues with nodes in limestone trying to reach the mirror in that environment15:57
roman_gI'm about kna1, CityCloud.15:57
fungiyeah, so unrelated15:57
roman_gAhha. another one. OK.15:57
fungiysandeep|ruck probably we need to check logstash to see what the hit rate for those errors is and whether we can put together more detail for logan- (who takes care of that environment)15:58
fungiysandeep|ruck: for example, if we can correlate failures to a specific host-id, then that might help him pinpoint where in the environment a network problem has arisen15:59
fungiysandeep|ruck: also if they're particularly frequent, we can temporarily stop booting new nodes there while we investigate further16:00
ysandeep|ruckfungi, i am seeing green run since ~9:00 gmt , faced majority of issue around ~07:3016:09
*** mlavalle has quit IRC16:09
clarkbas a heads up I'm going to try and continue reviews (I started trying to catch up on them yesterday before gitea had a sad), then I awnt to look at some of those gerrit accounts and see if I can run another group through the "retire these accounts without preferred email settings" step16:09
clarkbI think there may be 10-15 out of the remaining 28 in that situation that we can safely turn off without much fuss16:10
ysandeep|ruckfungi, fyi.. I have filtered on one a of job for which i noticed this error, other jobs could be impacted as well16:10
clarkbysandeep|ruck: fungi: you can use logstash to look at all jobs and filter by cloud and error message16:11
*** artom has quit IRC16:12
fungiyeah, i've been in a meeting tunnel all morning, which continues for the next few hours still16:13
fungiso trying to poke at stuff as i can16:13
clarkbnode to mirror traffic in limestone should happen over ipv6 (because we have a AAAA record in dns for the mirror and linux will prefer ipv6 over ipv4 if available)16:14
clarkbI wonder if there is some internal ipv6 routing issue?16:14
*** ysandeep|ruck is now known as ysandeep|away16:15
*** mlavalle has joined #opendev16:15
*** ykarel has quit IRC16:33
*** ykarel has joined #opendev16:33
fungior if it was just a brief network incident there16:35
openstackgerritClark Boylan proposed opendev/system-config master: Preliminary haproxy rate limiting ideas
clarkbcool my workinprogress was preserved on that16:39
clarkbalso yay testing saying I got haproxy configs wrong :)16:45
*** jpena is now known as jpena|off17:01
*** marios is now known as marios|out17:15
openstackgerritJeremy Stanley proposed opendev/bindep master: Overhaul Python package metadata
openstackgerritJeremy Stanley proposed opendev/bindep master: Update contributor doc and readme
*** openstack has joined #opendev17:23
*** ChanServ sets mode: +o openstack17:23
*** marios|out has quit IRC17:28
*** ykarel has quit IRC17:28
*** bodgix has quit IRC17:35
*** bodgix has joined #opendev17:36
*** artom has joined #opendev17:43
fungiweird, seeing some very large outbound network traffic spikes for review.o.o and they don't correspond with our backup cronjobs either:
clarkbfungi: looking at the longer historical trends that is in line with what we had seen before17:56
clarkbfungi: if I had to guess git protocol v2 helped a lot with bw usage17:56
clarkbso maybe some v1 client was doing updates for ci or something?17:56
fungicould be17:57
clarkbI assume that has been applied to all our debuntu hosts by now which would allow us to land however we did switch to preferring https with our mirrors so I'll do some checking on that before approving 71678917:58
*** snbuback_ has joined #opendev17:59
clarkbI think the thing to check there is grepping the image build logs to see that the dib side is trying to add it in and our element noops?18:00
clarkb that log lgtm. I can see the baseline-tools script pull it in then later when our element runs package installs it says it is laready at the latest. Now to double check bionic and focal, but I think we can approve that change18:03
*** snbuback_ has quit IRC18:04
*** snbuback_ has joined #opendev18:05
*** snbuback_ is now known as snbubac18:06
clarkbfungi: xenial, bionic, focal, buster, and stretch all look good for apt-transport-https according to their image build logs. That should be everywhere we need that package right?18:07
*** hashar has joined #opendev18:07
*** snbubac is now known as snbuback18:08
*** sboyron_ has quit IRC18:08
*** snbuback is now known as Guest1789718:09
*** rpittau is now known as rpittau|afk18:10
* clarkb thinks that is it and goes for it.18:10
*** Guest17897 has quit IRC18:11
*** snbuback has joined #opendev18:11
*** dtantsur is now known as dtantsur|afk18:12
*** hamalq has joined #opendev18:14
*** ralonsoh has quit IRC18:18
openstackgerritMerged openstack/project-config master: zuul-worker: remove additional install of apt-transport-https
fungiyeah, sorry, got sidetracked by lunch but can look in a bit18:20
clarkbfungi: I managed to convince myself we were good18:22
clarkbthe change removed package listings for only debian and ubuntu which was the last thing I needed to check to confirm I had surveyed properly18:22
*** eolivare_ has quit IRC18:26
clarkbmordred: is still a thing now with collections and stuff? I'm happy to rebase that if so, but not sure if it even makes sense anymore18:30
*** roman_g has quit IRC18:52
*** sboyron has joined #opendev18:59
fungizbr: i've got a stack of small cleanup/update changes to complete the move of bindep to the opendev tenant in zuul and modernize package metadata, correct urls, et cetera in preparation for tagging 2.9...
fungiif those look good to you i'll approve them and tag the tip of master as 2.9.019:01
*** sboyron has quit IRC19:05
*** d34dh0r53 has quit IRC19:09
*** d34dh0r53 has joined #opendev19:12
*** d34dh0r53 has quit IRC19:13
*** d34dh0r53 has joined #opendev19:14
*** andrewbonney has quit IRC19:20
*** artom has quit IRC19:30
openstackgerritMartin Kopec proposed opendev/system-config master: Deploy refstack with ansible docker
*** artom has joined #opendev19:57
*** akrpan-pure has joined #opendev20:00
*** artom has quit IRC20:02
*** stand has quit IRC20:12
openstackgerritMartin Kopec proposed opendev/system-config master: Deploy refstack with ansible docker
*** hashar has quit IRC21:07
*** whoami-rajat__ has quit IRC21:10
ianw       350 home/gerrit2/review_site/git/openstack/cinder.git/objects/pack/pack-6580ae06ed541e3a180c5600d01d79f4ed40f07a.pack21:20
ianw       379 home/gerrit2/review_site/git/openstack/neutron.git/objects/pack/pack-b5ac46fe2863a138711c6ae695491f257d090811.pack21:20
ianw       701 home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-8675125f931d721e1aff139d2134a83cb081db91.pack21:20
ianwin case you're wondering where the space goes in incremental backups of review ...21:21
ianwi guess .pack files have the same problem as us backing up the db .gzip files ... we didn't have 700mb of changes in nova but just pushing a few changes will completely update the .pack21:21
clarkbthough I think it is our repacking which does it21:32
clarkb(which is necessary for other efficiencies)21:32
ianwyep; i guess the flip side is that these objects are all unique, so will be pruned21:33
ianwi assume openstackwatch is dead?  it's cron job is still firing (and writing out a log file -- i noticed from going through the non-zero delta file list in backup)21:37
clarkbthats a name I haven't heard in a while. I wonder if anyone was using it until it stopped working (thats the thing that makes rss/atomic feeds?021:39
clarkbmy strong hunch is we can turn it off21:39
ianwit's just in a failing loop; it won't be deployed on the new server due to it only being setup by puppet anyway22:00
openstackgerritMaksim Malchuk proposed openstack/diskimage-builder master: Fix hooks order for CentOS/Fedora when mirror used
*** cap has quit IRC22:00
ianwok, i've removed the cronjob for it22:08
clarkbI've got an inmotion cloud spinning up now. Their dashboard says it should take an hour or so22:12
fungiianw: yes, we ripped out openstackwatch ages ago but apparently never cleaned up the cronjob for it on the server22:16
*** lbragstad_ has joined #opendev22:22
*** lbragstad has quit IRC22:24
clarkbI'm up to 10 more accounts that I think we can just retire to fix some of these consistency problems22:28
clarkbbut trying to find as many as possible before batching them up and updating them22:28
*** tkajinam has quit IRC22:59
*** tkajinam has joined #opendev22:59
clarkbanyone know what gerrit's behavior is if you remove the preferred email address for an account that otherwise has external ids for openid, email, and ssh username?23:00
clarkbthere is at least one account where they don't seem to have duplicates they just seem to have orphaned their preferred account and i'm wondering if unsetting preferred is reasonable23:00
fungiat a minimum, they will cease getting e-mail notifications from gerrit23:01
fungibut beyond that, i have no idea if newer gerrit considers that a consistency problem23:01
clarkbok I may just file that one away for now. Looking at the emails invovled they seem to have a personal one and an employer one. The employer one is orphaned so I suspect they removed it at some point but it remained preferred?23:02
clarkbin that case not spamming the employer email may actually be a good thing23:02
clarkbthough maybe it is already not sending email since no external id exists for it?23:03
*** slaweq has quit IRC23:03
fungior do you mean what happens if you delete your preferred address in the preferences ui? i don't think it will allow that, requires you to pick a secondary as preferred first, and then you can delete the address when it's a secondary one23:03
*** lbragstad_ is now known as lbragstad23:03
clarkbfungi: no I mean directly in All-Users23:03
clarkbbasiclly the preferred email has no external ids. The emails in enxternal ids match no other accounts. The preferred email matches no other accounts23:03
fungiyeah, fundamentally it means no e-mail notifications can be sent, but beyond that i don't know whether gerrit will care23:04
clarkbI'm wondering if we can just unset their preferred email and then they can fix it later if they wish23:04
fungii think they also can't be selected to add to groups or add as requested reviewers when there's no e-mail address23:04
fungisimilar to when the address is in conflict with another account23:04
clarkbwell there aer other emails23:04
clarkb(that is part of what makes this confusing and also potentailly inconsistentable, preferred email and external ids are stored in different refs so you can't atomically update them)23:05
fungijust thinking about the impact we had when not setting a preferred e-mail for our admin accounts, you couldn't update the accounts through the cli, gerrit would error23:07
clarkbya might have to set a value via git directly?23:10
clarkboh right any other settings wouldn't be settable23:10
clarkbso if we remove ap referred email we should also deactivate the account23:10
*** lbragstad has quit IRC23:21
*** lbragstad has joined #opendev23:23
clarkbok review-test:~clarkb/gerrit-consistency-notes/further-preferred-email-cleanups has more notes on things I think we can just retire. We're down to just a small number of accounts that we may actually need to fix assuming others are comfortable with my analysis and choices in that doc23:51
clarkbmy inmotion cloud deployment succeeded. Looks like the way you bootstrap things is when you build the cloud you give them an ssh public key. Once it is built that key can ssh in and get some info which allows you to horizon (and presumably keystone directly)23:59
clarkbI'm going to check basic functionality of things but then likely need to pick up actual resource provisioning tomorrow (in this case users in keystone I think and then we can use cloud launcher for everything else?)23:59

Generated by 2.17.2 by Marius Gedminas - find it at!