Tuesday, 2020-10-27

ianwhttps://gerrit-review.googlesource.com/Documentation/rest-api-accounts.html#get-account-external-ids ... in theory i guess00:07
fungiyeah, i linked the response data structure doc section in the etherpad00:25
fungiin the ptg etherpad i mean00:25
fungihttps://simplesamlphp.org/ is the base component knikolla suggested in our earlier discussion, linked in the spec00:30
*** hamalq has quit IRC00:30
*** qchris has quit IRC00:42
clarkbmnaser: we've discovered it would be good to talk to you about running k8s for infra things. We are about at time today and our block "tomorrow" likely isn't in a good spot for your timezone. Is there some time tomorrow when you might be available to jump on meetpad and talk about that?00:43
*** qchris has joined #opendev00:56
*** mugsie has quit IRC01:00
ianwfungi: for when you have time, i think the reprepro mirrors to ansible is gtg : https://review.opendev.org/#/c/757660/  ... all i've done is turn off the cron job as suggested, and add some basic testinfra since you last looked01:02
fungiclarkb: so just to confirm, i'm showing it's 05:00-07:00 utc wednesday, which will be 9-11pm tuesday pacific daylight time01:03
fungi(1-3am wednesday eastern daylight for me)01:03
fungioh, sorry, 10pm-midnight pacific01:04
*** mugsie has joined #opendev01:04
fungiianw: awesome, i'll try to take a look before i pass out01:05
clarkbfungi: thabks01:06
fungiianw: i guess it also got rebased at some point in the last six patch sets too01:31
fungiin ps24 apparently01:34
fungiianw: lgtm! are you okay if i approve it now or do you think it needs more reviews first? i think it should be safe since it's just deploying unused files initially and not adding the cronjob01:38
ianwfungi: sorry, getting lunch.  i can approve and watch it, and do a bit of testing01:52
ianwthe other thing i might just do now as well is cleanup the debian-ceph mirrors https://review.opendev.org/#/c/758517/01:53
ianwlooking at the abuse response for google cloud it seems like you get a warning; https://support.google.com/cloud/answer/7002354?hl=en and also the abuse form has space to put free-form text in01:56
ianwi think i might just report the two IP's as we have pretty specific info; with the request that they get the account owner to contact us to work something out01:56
ianwi don't expect them to get blocked ... but if it continues, we can look at a layer 7 approach of blocking the queries01:57
fungii went ahead and approved it02:02
*** DSpider has quit IRC02:07
*** zbr has quit IRC02:51
openstackgerritMerged opendev/system-config master: reprepro: convert to Ansible  https://review.opendev.org/75766002:57
openstackgerritMerged opendev/system-config master: Remove old debian-ceph mirrors  https://review.opendev.org/75851702:57
openstackgerritIan Wienand proposed opendev/system-config master: ARM64 : run base test on Focal too  https://review.opendev.org/75662902:58
*** zbr has joined #opendev03:12
openstackgerritlikui proposed openstack/diskimage-builder master: Remove the unused coding style modules  https://review.opendev.org/75979103:38
*** auristor has quit IRC03:43
*** auristor has joined #opendev03:47
ianw#status log removed ceph h/j/l/m AFS volumes and mirroring jobs04:02
openstackstatusianw: finished logging04:02
openstackgerritIan Wienand proposed opendev/system-config master: reprepro: fixup script name  https://review.opendev.org/75980304:10
openstackgerritIan Wienand proposed opendev/system-config master: reprepro: install keytab  https://review.opendev.org/75980404:17
*** ykarel has joined #opendev04:31
*** ykarel has quit IRC04:35
*** ykarel has joined #opendev04:36
*** ykarel has quit IRC05:10
*** ykarel has joined #opendev05:16
*** ykarel_ has joined #opendev05:18
openstackgerritMerged opendev/system-config master: reprepro: fixup script name  https://review.opendev.org/75980305:19
*** ykarel has quit IRC05:21
openstackgerritMerged opendev/system-config master: reprepro: install keytab  https://review.opendev.org/75980405:22
openstackgerritIan Wienand proposed opendev/system-config master: reprepro: run deploy job on role changes  https://review.opendev.org/75981005:30
*** ysandeep|holiday is now known as ysandeep|ruck05:42
openstackgerritIan Wienand proposed opendev/system-config master: mirror-update: publish reprepro logs  https://review.opendev.org/75981105:45
*** zbr1 has joined #opendev06:03
*** marios has joined #opendev06:03
*** zbr has quit IRC06:06
*** zbr1 is now known as zbr06:06
*** marios has quit IRC06:31
*** fressi has quit IRC06:32
*** marios has joined #opendev06:44
openstackgerritRico Lin proposed opendev/system-config master: Remove TW User Group ML  https://review.opendev.org/58403507:03
ttxianw: re: that crawler, I'll need a bit more information to narrow it down. Is it walking through all changes, or just a period of time / specific repositories? Also is there a pattern in the timing?07:04
ianwttx: it's pretty much going as fast as it can07:05
ttxRe: Bitergia they are only gathering stats on StarlingX and Zuul07:05
ttxianw: is it more like a daily cronjob?07:05
ianwthe requests all look like "GET /changes/?q=status:merged&q=status:abandoned&o=ALL_REVISIONS&o=ALL_FILES&o=ALL_COMMITS&o=MESSAGES&o=DETAILED_ACCOUNTS&n=100&S=297400 HTTP/1.1" 200 209096 "-" "python-requests/2.23.0"07:05
ttxah, that's a pretty wide net indeed07:06
ianwit walks every change07:06
*** ralonsoh has joined #opendev07:07
ttxOK, that's probably not them, if only because they have been around for a long time and we would have noticed earlier, but also because they should only walk very specific repos for very specific periods of time07:07
openstackgerritMerged opendev/system-config master: reprepro: run deploy job on role changes  https://review.opendev.org/75981007:07
ttxBut I'll ask, could be a bug07:07
ttxianw: does it happen every day at the same time? If yes, what time is that?07:08
*** sboyron has joined #opendev07:08
ianwttx: no, not regular, but the requests always look the same07:12
openstackgerritMerged opendev/system-config master: mirror-update: publish reprepro logs  https://review.opendev.org/75981107:13
*** marios has quit IRC07:19
*** eolivare has joined #opendev07:28
*** rpittau|afk is now known as rpittau07:42
*** ysandeep|ruck is now known as ysandeep|lunch07:46
*** ykarel_ is now known as ykarel07:56
*** andrewbonney has joined #opendev08:10
*** slaweq has joined #opendev08:11
*** lpetrut has joined #opendev08:11
*** ralonsoh has quit IRC08:13
*** ykarel has quit IRC08:13
*** ykarel has joined #opendev08:16
*** tosky has joined #opendev08:36
*** ralonsoh has joined #opendev08:37
*** DSpider has joined #opendev08:37
*** marios has joined #opendev08:43
*** manpreet has quit IRC09:01
*** ysandeep|lunch is now known as ysandeep|ruck09:01
*** fressi has joined #opendev09:04
*** webmariner has quit IRC09:06
*** hashar has joined #opendev09:31
fricklerinfra-root: does anyone know about a recent change in focal images that could result in localhost getting resolved to ::1 now instead of
* frickler is seeing designate jobs being broken because of that. there is an entry in /etc/hosts with "::1 localhost" but I'm not sure whether that might be new or why10:04
*** ykarel_ has joined #opendev10:06
*** ykarel has quit IRC10:09
*** ykarel_ is now known as ykarel11:17
*** sboyron has quit IRC11:45
fungido some jobs archive /etc/hosts? we could look back at earlier builds to compare them that way11:47
fungimy debian machines have included localhost as an alias on ::1 (in addition to for ages11:48
fungiout of curiosity, why does that break jobs?11:50
fungiis it just that they're hard-coded to expect or is there something only listening on instead of ::1?11:55
fricklerfungi: I think it is just this one https://review.opendev.org/759850 , memcached is listening on explicitly. but I'm not sure why this is breaking only now and why it doesn't seem to affect any other service12:01
*** lpetrut has quit IRC12:01
*** lpetrut has joined #opendev12:03
fricklerseems to have changed on 2020-10-23, not sure yet whether the change is in the focal image or in our setup phase12:07
sean-k-mooneyfungi: fedora also has ::1 as an aias for local hosts12:48
sean-k-mooneyi think its pretty common now12:48
sean-k-mooneyfrickler: the grenade job hit the current nova gate blocker12:50
sean-k-mooney TypeError: Parameterized generics cannot be used with class or instance checks12:51
sean-k-mooneyhttps://review.opendev.org/#/c/759831/ should fix that12:51
fricklersean-k-mooney: yes, I've seen that issue, but I guess we need a stable backport of that fix in order to get grenade to work again? or does it only affect master?13:04
fricklerI guess we'll see that on the grenade job for that patch itself13:06
sean-k-mooneyis grenade using victoria for the base version13:08
sean-k-mooneyi guess it is now13:08
sean-k-mooneyso ya13:08
fungimaster grenade will start with victoria13:08
sean-k-mooneyi think this only affect 20.04 based jobs but that includes victoria so ya this need to be merged there first or grenade need to be made non voting13:09
sean-k-mooneyi was going to say upper constiratign might help13:10
sean-k-mooneybut no13:10
sean-k-mooneysince its failing in that grenade job13:11
fricklerfungi: how about we clean up our akick lists a bit, too? 6y old IPs likely aren't relevant any longer. fwiw I also added that haunted spammer to some more channels where I saw them13:26
openstackgerritMichal Pryc proposed zuul/zuul-jobs master: Allow bindep role to install additional test requirements  https://review.opendev.org/75986813:48
fungifrickler: yeah, can't hurt to remove old entries there14:04
*** sshnaidm|rover has quit IRC14:05
*** sshnaidm has joined #opendev14:05
*** sshnaidm is now known as sshnaidm|rover14:06
*** mlavalle has joined #opendev14:07
*** slaweq is now known as slaweq|ptg14:07
openstackgerritMichal Pryc proposed zuul/zuul-jobs master: Allow bindep role to install additional test requirements  https://review.opendev.org/75986814:24
*** lpetrut has quit IRC14:32
*** lpetrut has joined #opendev14:33
*** Goneri has joined #opendev15:18
GoneriOpendev is in Korean and it's anoying because I don't speak the language15:20
Goneriis there a way to change that? https://i.imgur.com/CqirrRh.png15:20
fricklerGoneri: please try to clear your cookies. we have seen that before, but don't know exactly when or why this happens15:21
clarkbGoneri: yes there is a setting bottom of the page next to a glob looking icon15:21
Gonerioh it's indeed much better if I clean my cookies... :-)15:22
Gonerithe settings button was in Korean too, so it don't really help.15:23
Gonerieheh :-)15:23
frickleris gerrit getting slowed down again? /me is in ptg session and can't check right now15:23
*** lpetrut has quit IRC15:23
clarkbfungi: ^ ? I'm not quite to a computer yet either.15:23
fungii saw reports of it, but was bogged down on other things, will see if i can identify the new address15:29
*** ykarel has quit IRC15:40
louroto/ just checking if I can get another +2 on this simple project-config change, thanks! https://review.opendev.org/#/c/758429/15:43
openstackgerritMasayuki Igawa proposed opendev/irc-meetings master: Update QA office hour  https://review.opendev.org/75989015:45
*** sboyron has joined #opendev15:46
*** tosky_ has joined #opendev16:07
sshnaidm|roverhi, folks, if someone is not in the middle of ptg meeting, can you please take a look why zuul can't "freeze" the graph in this patch: https://review.opendev.org/#/c/759892 ? I'd appreciate any ideas there16:07
*** tosky has quit IRC16:08
*** ysandeep|ruck is now known as ysandeep|away16:08
*** slaweq|ptg is now known as slaweq16:08
fungithat usually implies a configuration problem/conflict, but i'll see if i can spot it16:09
*** fressi has quit IRC16:09
fungithis'll take a bit. zuum is dragging my workstation to its knees16:11
sshnaidm|roverfungi, yeah, it's frustrating not to have detailed info, usually we try to play around by "try and see" effort16:11
sshnaidm|roverfungi, yeah, mine too :)16:11
fungisshnaidm|rover: it says "Job tripleo-ci-centos-8-undercloud-containers depends on tripleo-ci-centos-8-content-provider which was not run." so i guess the question is why did it decide tripleo-ci-centos-8-undercloud-containers needed to run but not tripleo-ci-centos-8-content-provider16:17
clarkbhttps://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml#L369 is why16:17
fungiaha, yep16:18
clarkbinfra-root I'm going to try and refine https://etherpad.opendev.org/p/lAv5xqj0oNUjZKeV4vxg a bit more today so that that can go out16:18
clarkbhaving a slow start today which is probably good since it will be a late night :)16:19
fungisshnaidm|rover: so the short story there is that having giant tangles of file exclusions in interdependent jobs quickly becomes impossible to reason about16:20
sshnaidm|roverfungi, I see, thanks, will try to fix it16:20
fungiit's one of the reasons we try to avoid irrelevant-files in our jobs16:20
openstackgerritMerged opendev/irc-meetings master: Update QA office hour  https://review.opendev.org/75989016:24
openstackgerritMerged openstack/project-config master: Mirror charm-neutron-api-plugin-ironic to GitHub  https://review.opendev.org/75842916:25
clarkbok did some editing of https://etherpad.opendev.org/p/lAv5xqj0oNUjZKeV4vxg to capture thoughts during discussions yesterday16:36
fungiclarkb: you mention the summary table going away, but probably equally important to some users is the additional ci comment toggle16:39
fungithat goes away too, right?16:39
clarkboh ya I'll add that16:40
clarkbit sort of goes away16:40
clarkbyou can filter by vote category16:40
clarkbwhich gives a similar but not quite the same experience16:40
fungioh, that's a good point, also you can filter by comment category right?16:41
clarkbnot sure about comment category. Pretty sure I did vote and confirmed that worked at least16:41
fungithe comments can be flagged as added by automation, which zuul started doing sometime early in 3.x16:42
fungibut jenkins or older zuul-based 3pci comments likely won't do that16:42
clarkboh ya there are bot comments but not sure if gerrit does anything with them yet16:42
clarkbI'm sure a plugin we write could if gerrit itself currently ignores them though16:43
fungithe "autogenerated" message tag16:43
fungiahh, okay, so no current pg feature to filter on autogenerated16:43
*** tosky_ is now known as tosky16:56
*** odyssey4me is now known as odyssey4me|PTO17:04
*** marios is now known as marios|out17:05
*** ricolin has quit IRC17:10
*** marios|out has quit IRC17:14
*** ralonsoh has quit IRC17:25
*** olaph has joined #opendev17:30
*** rpittau is now known as rpittau|afk17:34
clarkbcorvus: frickler ianw if you get a chance to look at https://etherpad.opendev.org/p/lAv5xqj0oNUjZKeV4vxg today I would love feedback. I'll try to send that out once ianw's day has started and has had a chance to look at it17:43
*** eolivare has quit IRC17:49
fungiclarkb: on the bit where it mentions the git v2 requirement, would it help to point out that centos/rhel 7 have a default git version which won't work with it?17:56
clarkbya, though I'm trying to double check where I saw taht in the release notes and am not finding it so maybe this isn't true?17:58
*** hashar has quit IRC17:59
clarkbhttps://www.gerritcodereview.com/2.16.html#git-clients-older-than-2x-are-not-supported-anymore there it is17:59
fungithe oldest debian and ubuntu releases which still have packages listed on their respective package sites are all 2.x at least, so seems this will mostly be a problem for centos/rhel 7 users18:00
clarkbok added a little note.18:01
*** hashar has joined #opendev18:02
fungiwould it help to mention trying things out on revire-test? maybe that should be a followup once we've got it upgraded again18:05
fungier, review-test18:05
clarkbya I didn't want to mention that until we have a review-test up and running again18:05
fungimakes sense18:08
corvusclarkb: made some small changes at the top (mostly to get the most actionable info in the 1st pgraph)18:08
*** mlavalle has quit IRC18:11
*** mlavalle has joined #opendev18:14
zbris this git v2 requirement specific to gerrit only or gitea will continue to work?18:20
clarkbgerrit only and really only for the commit message hook18:21
fungigerrit only18:21
zbra requirement on git v2 would render maintenance branches useless for ancient clients18:21
zbrin that case, go for it!18:21
*** andrewbonney has quit IRC18:21
fungiyeah, you could roll your own commit hook or just manually generate commit ids18:21
zbri doubt any developer is using a centos-7 box18:21
fungior even reuse the old commit hook probably?18:21
clarkbyes the release notes suggest using the old hook as a workaround18:22
zbrif someone is affected, let them dig a way out of that pit ;)18:22
fungiright, if the question is "do we make fedora 33 users adjust their ssh configs or make centos 7 users install a third-party git package?" i think the latter wins18:22
zbrmodern platforms should always take priority18:23
*** Green_Bird has joined #opendev18:23
fungiclarkb: draft message lgtm18:29
*** hashar has quit IRC18:29
fricklerclarkb: fungi: how about the two workarounds mentioned in https://www.gerritcodereview.com/2.16.html#git-clients-older-than-2x-are-not-supported-anymore ? IIUC the requirement for new git is only via the commit-hook script, we could either keep serving the old script for some time, or at least test whether using the legacy script (option 2) does work for centos 718:50
frickleralso I wonder whether the mail should somewhere mention the year 2020, just to be sure18:51
clarkbfrickler: ++ to adding the year and maybe mention it is for the commit hook and if people complain we can host a version somewhere for them?18:53
clarkbI dont want to host it if no one needs it18:53
clarkb(maybe it can go inti git review packaging)18:54
fungii've been tempted to integrate something like that into git-review anyway, since without ssh access people can't fetch the one gerrit serves, and we do support folks with only https client access after all18:55
fungigit review could ni theory embed both variants and install the correct one for the user's git version18:58
clarkbfrickler: how about that? then if people ask about how they can use the commit hook on centos 7 we can figure out embedding it in git review or something18:59
*** Green_Bird has quit IRC19:01
fungion the other hand, we'll likely want to drop python 2.7 support in git-review sometime soon as well, so folks on centos/rhel 7 would need to install the nonstandard python 3.6 which got added there when we do19:01
clarkbfungi: its standard now aiui19:01
clarkbit just doesn't come with a full set of libs (not really a problem for git review being pip installed)19:02
fungi"nonstandard" as in they need to expressly install it rather than it coming preinstalled like 2.7 does19:02
fricklerclarkb: that sounds o.k. to me, waiting for feedback on how many people would actually be affected19:11
* frickler heads away now and will try to be back at 5, might be half an hour late or so19:13
clarkbno worries see you then19:14
fungistill not sure if i'll be awake19:17
clarkbthe intent with the variety of times was that it would enable more people to interact and not to force people to stay up at odd hours :P19:22
clarkbits ok if you would rather sleep19:22
fungihuh, the magic "gerrit code review" account can't set group membership, but it can suexec as, for example, the openstack-project-creator account which can set group membership19:25
fungii guess that's a reasonable workaround19:25
fungii feel like i've already discovered this fact more than once19:27
fungialso gerrit doesn't allow / in usernames. it's (case-insensitive) alphanumeric plus ._-19:29
*** webmariner has joined #opendev19:30
fungiso i'm going with fungi.admin as my admin username19:30
clarkbthat seems reasonable19:30
fungianyway, turns out it can all be done in a single command on review.o.o:19:31
fungisudo -u gerrit2 ssh -i ~gerrit2/review_site/etc/ssh_host_rsa_key -p 29418 -l 'Gerrit Code Review' localhost "suexec --as openstack-project-creator -- gerrit create-account --group 'Administrators' --ssh-key 'ssh-rsa AAAA...THBj fungi@bridge' fungi.admin"19:32
fungii'm still trying to decide how i feel about hopping through bridge.o.o to access the account19:35
fungivs directly from my workstation19:36
fungiultimately i'll wind up securing the ssh key i use to log into bridge the same as i do the ssh key used to log into that gerrit account (and they could also just be the same key, i don't think that'e necessarily any less secure?)19:36
clarkbya not sure about that one either19:38
clarkbI think it was corvus who mentioned it as an option?19:38
fungieither account could be used to gain control of the other anyway19:39
corvusyeah, just brainstorming.19:40
fungino worries, just trying to think through whether it adds tangible security or merely complexity19:53
*** melwitt has joined #opendev20:07
*** slaweq has quit IRC20:16
*** rpittau|afk is now known as rpittau20:18
*** slaweq has joined #opendev20:19
ianwclarkb: thanks for writing; i put one suggestion there on the intro paragraph, feel free to take or leave, but otherwise LGTM20:38
clarkbianw: that looks good I'll swap the content out for your suggestion20:57
clarkband I'll send this email out shortly20:57
*** gouthamr has quit IRC20:58
*** logan- has quit IRC20:58
*** gouthamr has joined #opendev20:58
*** gouthamr has quit IRC20:59
*** qchris has quit IRC20:59
*** gouthamr has joined #opendev20:59
*** logan- has joined #opendev21:01
*** rpittau is now known as rpittau|afk21:03
*** slaweq has quit IRC21:04
clarkbalright, the email is queued up in my mua. I'll give it a few more minutes for last second edits otherwise sending it out21:06
clarkbI had on the etherpad that should go to service-discuss but I'm switching that to announce21:16
clarkband sending now21:16
fungiahh, yeah i wondered21:17
fungiwe can still do followup minor additions to -discuss, like when review-test is up and available21:18
*** hamalq has joined #opendev21:20
*** fressi has joined #opendev21:21
*** slaweq has joined #opendev21:22
melwittclarkb: have you gotten a chance to try out the streaming log processing? I notice e-r indexing is behind 10 hrs at present21:27
clarkbmelwitt: sorry no there are very many distractions right now. That may be something I can look at tomorrow since my main ptg commitments will be done by then21:28
ianwfungi: if you'd like to check anything on mirror-update.opendev.org, i think we're ready to cut-over reprepro to there.  i've been running the deb-docker mirror overnight, exporting logs @ https://static.opendev.org/mirror/logs/reprepro/ and it's working21:28
melwittclarkb: no worries, was just curious21:28
TheJulia are pep8 jobs timing out expected these days?21:45
*** Goneri has quit IRC21:48
clarkbTheJulia: I looked at that a few days ago and it appeared that our pip installs were really slow, but tox/pip don't record timestamps for installation times. More recently fungi pointed out that we've got reduced capacity due to taking a clodu out of rotation after ip address conflicts and arp fights. That may lead to more noisy neighbor problems?21:49
*** sshnaidm|rover is now known as sshnaidm|afk21:50
clarkblooking at https://420bbb86771d88900a0a-0bd2acfdbec8f6316ef91de37dd7cea6.ssl.cf5.rackcdn.com/701410/1/gate/openstack-tox-pep8/9ef06ca/job-output.txt that appears to have had a slow apt-get installation21:51
TheJuliaNo idea, just seen it a couple times in the last few days on the pep8 jobs and that is understandably not great all things considered. :(21:51
TheJuliaFun :(21:51
clarkbthen the job times out as it is installing pip things21:51
fungiianw: oh! thanks, i meant to check it out earlier today but have been sidetracked. i should have some time between the oilabs and opendev ptg sessions though21:51
clarkbthe mirror server there seems happy21:51
clarkbhttps://mirror.ord.rax.opendev.org/ubuntu/dists/focal/main/ is navigable, server load is low and there is plenty of free memory21:52
clarkbhttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=68289&rra_id=all is curious though21:53
clarkbcould we be maximizing network bw?21:53
openstackgerritwes hayutin proposed zuul/zuul-jobs master: Allow rdo repos to be turned off for openvswitch install  https://review.opendev.org/75910721:54
TheJuliathat is oddly consistent if jobs are evenly distributed21:59
clarkbapache logs look the way I'd expect. A lot of rackspace ips pulling pacakges that things like openstack need21:59
clarkbone oddity I notice (likely unrelated to the issue) is it seems some centos 8 pulls are via http and others are via https22:01
clarkbfungi: did we add latex back as a dep ?22:01
clarkb(seems like that may cause similar issues and figure we should rule it out quickly)22:01
*** fressi has quit IRC22:05
*** sboyron has quit IRC22:06
clarkbour cache miss to hit rate for pypi looks decent22:13
openstackgerritIan Wienand proposed opendev/system-config master: reprepo: enable cron jobs  https://review.opendev.org/75996522:15
clarkbianw: fungi I notice we've got some html.tmp files in our wheel mirror that apache seems to complain about. I don't think that is related to the slowness in this region but it does fill our apache error logs and is distracting. https://mirror.ord.rax.opendev.org/wheel/ubuntu-18.04-x86_64/asyncio/22:17
clarkbis that a side effect of when we were trying to write our own html files? we may want to clean those up?22:18
ianwhrm yes i think so.  i can clear them out22:18
*** Green_Bird has joined #opendev22:19
clarkbrxtx factor that flavor is 1600.0 and I believe we're supposed to have a theoretical max for any one connection of ~1/2 that value22:23
clarkbwe seem to be well below that currently22:23
clarkbhttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=68280&rra_id=all seems to show the backup22:24
clarkbthat looks a lot to me like we're getting throttled and so the requests are piling up22:25
clarkbI guess the next thing to check is if apache has free slots22:25
fungi42gb used in /var/cache/openafs so i don't think we're overrunning our afs cache limit22:25
clarkbapache shows plenty of open slots22:28
TheJuliacould it just be variable performance on rax with our timers being too low to account for it?22:28
TheJuliaI ask because we've actually had to have our devstack plugin auto-extend some in-job timeouts on rax because of performance differences and variability22:29
clarkbit could be, though it seems network related as the other resources we're measuring don't seem to be having trouble22:29
clarkbpossible we're just overwhelming a switch or router22:30
clarkbfetching https://mirror.ord.rax.opendev.org/fedora/atomic/stable/Fedora-29-updates-20190820.0/AtomicHost/x86_64/images/Fedora-AtomicHost-29-20190820.0.x86_64.qcow2 (a large file) is indeed very slow to my desktop22:31
ianwclarkb: ok, tmp files gone at least22:31
clarkblike KBps measurement slow22:31
clarkbbut the server itself has plenty of memory to service connections as well as cpu which is making me suspect a network throttle somewhere22:32
ianwagree, same here22:32
clarkbpulling from dfw is much quicker22:33
fungicould just be there's a party going on in chicago22:33
fungiour ord max-servers is higher right?22:34
clarkbits about the same as iad22:34
clarkbthen dfw is a bit lower22:34
ianwlocally on the mirror itself : 2020-10-27 22:34:09 (39.7 MB/s) - ‘Fedora-AtomicHost-29-20190820.0.x86_64.qcow2’ saved [717020672/717020672]22:34
fungiord is a good 33% higher than dfw/iad22:35
clarkbfungi: not for actual utilziation though22:35
clarkbI think the quotas may haev been tweaked down?22:35
fungioh interesting22:35
fungiyeah i was merely looking at our config22:35
clarkbianw: I think that observation points to a throttle off host22:36
ianwyou wouldn't think it would throttle to internal rax ord hosts though?22:37
clarkbwe use the public address not the internal one though, but also it could be an unintentional throttle22:37
clarkbsome switch or router having a hard time for $reason22:37
ianwcan they see the 10.29 address?22:37
clarkbI think so? It would depend on how glean configures it22:38
clarkbwe could try it to see if it is faster via the 10 net22:38
ianwat about 150k to the backup server in rax ord22:39
clarkbmight also want to try ipv6 vs ipv4, though I think the failed job example I had would've used ipv622:39
openstackgerritmelanie witt proposed opendev/elastic-recheck master: Add query for bug 1901739  https://review.opendev.org/75996722:39
openstackbug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [Undecided,New] https://launchpad.net/bugs/190173922:39
clarkbianw: yay it is consistent :)22:40
ianw12mb/s when using "wget"22:40
ianwfrom same mirror22:40
ianws/mirror/backup server/ to mirror22:40
ianwsame speed to public address with -4 & -622:41
clarkbnot sure I got all that. The backup server to mirror is slow when and fast when?22:42
clarkboh I think I get it. public v4 and v6 are both slow. private v4 is good22:43
ianwsorry; yeah stream of conciousness22:45
ianwfrom the backup server in the same region, it is fast to grab the iso over the private v4 network (10.209 address), but slow to get it from the public ip via both the server's public ipv4 and ipv6 address22:46
ianwwe could probably setup /etc/hosts on test nodes to use the internal address for rax hosts?22:47
ianwas part of the mirror configuration22:47
clarkbor create a mirror-int record in dns and use that?22:49
clarkbthat will likely be easier to understand 6 months from now22:49
clarkbbut we have to check glean configures that interface and routes properly (I think it does)22:50
ianwclarkb: the thing with that is that i think we'll be back to https issues getting a cert for that?22:50
clarkboh ya hrm22:51
ianwi jumped on a focal node currently doing something, and it can access the mirror on it's private address22:52
ianwand it's pulling that iso at 12mb/s22:52
clarkbwe can add mirror-int to the altnames then reissue certs22:54
ianwumm, i guess we do dns validation so yes22:57
clarkbfungi: ^ thoughts?22:58
fungiseems like a fine experiment22:59
ianwit doesn't have a internal ipv6 address though?23:00
ianwi can add the records, just a tick23:00
clarkbya just ipv4 for internal23:01
openstackgerritIan Wienand proposed opendev/zone-opendev.org master: RAX ord mirror : add internal address  https://review.opendev.org/75997023:03
clarkbthat looks fine to me but helpign with kids so not in a spot to leave avote23:04
openstackgerritIan Wienand proposed opendev/system-config master: Generate internal certs for RAX ORD mirror  https://review.opendev.org/75997123:05
*** slaweq has quit IRC23:10
openstackgerritIan Wienand proposed openstack/project-config master: Use internal address for RAX ORD  https://review.opendev.org/75997223:14
ianwi think that's the chain then.  i'm pretty confident ^ works because it was what we used to use to swtich between openstack.org/opendev.org servers when that was a thing23:15
johnsomIt seems the StoryBoard/Gerrit bot is still not functional.23:19
*** mlavalle has quit IRC23:20
fungioh, yep, we never did restart gerrit, thanks for the reminder23:25
fungimaybe before the 05:00 opendev ptg session would be a good opportunity, if i manage to be awake for it23:26
ianwfungi: i think i missed that?  happy to manage if there's a tl;dr23:30
ianwi'll get the dns entries and certs for ord internal mirror, then we can test from a node23:31
fungiianw: oh, so as part of the breach cleanup i blew away all the session keys and api tokens for production sb after deescalating an account and disabling it, but that all happened after gerrit was brought back up so the new api token i generated for the its-storyboard plugin to use won't be read from configuration until the next gerrit restart23:34
fungiwhich i meant to take a moment for over the weekend23:35
ianwahh.  yeah a few more hours is probably a good time for minimal impact23:36
*** Goneri has joined #opendev23:38
openstackgerritMerged opendev/zone-opendev.org master: RAX ord mirror : add internal address  https://review.opendev.org/75997023:46
*** tosky has quit IRC23:52
*** Green_Bird has quit IRC23:58
*** DSpider has quit IRC23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!