Wednesday, 2023-09-13

Clark[m]Ya I doubt they are updating that quickly but they will probably tell us00:53
opendevreviewMerged openstack/project-config master: Retire os-win: end project gating
opendevreviewMerged openstack/project-config master: Retire compute-hyperv: end project gating
opendevreviewMerged openstack/project-config master: Retire networking-hyperv: end project gating
opendevreviewMerged openstack/project-config master: Retire oswin-tempest-plugin: end project gating
opendevreviewBernhard Berg proposed zuul/zuul-jobs master: prepare-workspace-git: Add ability to define synced pojects
*** amoralej is now known as amoralej|lunch11:03
opendevreviewPiotr Parczewski proposed openstack/diskimage-builder master: Include linux-firmware in Rocky linux element
opendevreviewAlex Kavanagh proposed openstack/project-config master: Add the Pure Storage Flashblade charm-manila subordinate
opendevreviewMayank Patel proposed openstack/project-config master: Add node-interface-metrics-exporter app to StarlingX
*** amoralej|lunch is now known as amoralej12:31
opendevreviewMatthias Runge proposed openstack/project-config master: Add python-observabilityclient
opendevreviewAlex Kavanagh proposed openstack/project-config master: Add the Pure Storage Flashblade charm-manila subordinate
PerduHi, I'm following the Getting Started page to send change to a project. When running the 'git review -s' command for the first time, ssh prompts me to validate the fingerprint for '[]:29418. Is this fingerprint written somewhere so that I can cross-check for security?14:16
fungiPerdu: excellent question! yes, we should add that to the documentation but it can be found in gerrit's webui. let me get you a direct link14:18
fungimmm, or it used to be. just a sec14:20
SvenKieskePerdu: glad you found your way here :) I linked you to the getting started guide on github (@artificial-intelligence, over there)14:22
SvenKieskefwiw, that is what I got in my known_hosts file: []:29418 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIncSZ8k+c0YJTO6Wux/Kpu2jnktFkepoU3jBkbIzXve14:23
fungiyeah, we've got it published in git, but i'm trying to figure out what happened to the part of the gerrit webui that used to display it14:24
SvenKieskemhm just has the port number..14:26
fungiyeah, the settings page where you upload your ssh keys used to show the host key fingerprint right there too14:27
fungii think that must have disappeared in a recent upgrade, so i'll probably need to hound gerrit upstream devs in their matrix/discord channel to get clarification. in the meantime i'll add something to the infra-manual14:28
PerduAnother missing step in the documentation is the necessity to select a contributor agreement (and what are these)14:29
fungiopenstack's onboarding documentation covers that (not all projects hosted in opendev require an individual contributor license agreement, but official openinfra projects do for legal reasons, basically asserting that the patches you upload are your own work or you have permission of the copyright holder, e.g. your employer)14:30
fungiguilhermesp_____: have you heard anything about the routing loop in ca-ymq-1?14:31
fungiPerdu: for the cla reference, looks like maybe we could do with referring to the developer's guide chapter earlier in the quick setup guide, but this is the relevant bit:
fungitrying to find a good balance between "your setup documentation has too many steps, i'm not going to bother" and "your setup documentation omits important details i wanted more information on"14:36
fungiand also not duplicating too much information between chapters14:37
SvenKieskewell, it clearly has too many steps :D I remember myself reading it the first time, some years ago and I needed multiple attempts to resolve everything :D14:37
SvenKieskebut it _is_ fairly complete, tbf. and many steps would not get that simpler, even if we'd switch to different infra, like sometimes is suggested.14:38
guilhermesp_____fungi: not yet sorry, im trying to find a slot to take a look at it today. Is this a blocker for you so we may raise the priority? 14:42
fungiguilhermesp_____: nah, i went ahead and added a /etc/hosts entry on my workstation so that everything will just resolve exclusively ipv4 for the gerrit server so i don't have to wait for v6 connections to it to time out14:43
fungii don't know how many other people are using my isp's ipv6 support (it's a fairly popular broadband isp in the usa, but what percentage of traffic that amounts to i couldn't guess)14:43
fungii haven't heard from any of our users about problems connecting to our servers in ca-ymq-1 so it may be a very small number (perhaps just me) impacted14:44
guilhermesp_____yeah ideally would be good for me to try and reproduce with my isp here 14:45
fungiguilhermesp_____: probably you won't see any problems unless there are more routes missing than just the one for my provider14:45
Perdufungi: You get an error when you run 'git review -s' and you haven't selected the CLA in gerrit's setting, so there should be a line about this14:46
fungiPerdu: right, it's worth thinking about how to cover that in the documentation since you'll only get that if you're trying to set up to contribute to a project that has cla enforcement (not all projects hosted in opendev do)14:46
fungievery additional "you might see this and if so do this or look here" we add to the quick start guide makes it more cumbersome for people that doesn't apply to14:47
guilhermesp_____so fungi the idea here is that you are trying to reach the review02 and ipv6 addrs from your local computer right? ping6 <ipv6> which is never giving you a reply? 14:47
Perdufungi: Oh, I see14:48
fungiguilhermesp_____: correct, and in the other direction (trying to reach my home address from the server) i see what looks like a routing loop between your gateways14:48
PerduI think any step that leads to an error when you're trying to set things up is pretty frustrating and should be handled in some way14:49
fungiPerdu: the way we've tried to approach it is to suggest that projects have their own getting started guides, and are free to copy from or refer to ours however helps (it's published under an open license so can be freely reused and remixed_14:49
fungiso for example, trying to follow the opendev manual for contributing to openstack may miss some key details that you would find in the openstack contributor guide here:
fungithere they cover the cla up front, because all their projects require it14:50
fungithe idea is that new contributors to, say, kolla-ansible would spot and follow the link to the openstack contributors guide first14:51
fungithe "get started" link up in the top navbar at is a bit of a nuisance in that users are likely to spot it first and that will take them to our more generic manual for opendev's infrastructure14:52
fungimight be worth a feature request to gitea to allow the get started link to vary by project being browsed (like the "issues" and "proposed changes" links) rather than being global for the entire site14:53
fungiclarkb: ^ what do you think?14:53
guilhermesp_____hum yeah ok from a vm inside ca-ymq-1 dc i can ping6 ... however from my home14:53
fungiguilhermesp_____: trying to ping6 my home address from one of our servers, i eventually (after seeing a very long wait) get "From 2604:e100:1::2 icmp_seq=12 Time exceeded: Hop limit" which is indicative of a routing loop14:57
fungiguilhermesp_____: and attempting to traceroute6 to anything in 2600:6c60::/24 just bounces hops between 2604:e100:1::1 and 2604:e100:1::214:58
fungiguilhermesp_____: it's hard to hazard a solid guess since you don't operate a public bgp looking glass that i'm aware of, but it's acting just like i would expect if the announcement for 2600:6c60::/24 was being filtered or otherwise not making it into your tables14:59
guilhermesp_____yeah sadly i dont have access to our core network infra... im trying to get some confirmations with mnaser  to see if thats in our domain 15:00
fungialso i put most of this detail into the e-mail i sent to the support address last week15:00
guilhermesp_____yeah i saw it.. i have an internal chat with mnaser  to discuss that but he is not available until thuersday since he is traveling  15:02
*** blarnath is now known as d34dh0r5315:02
guilhermesp_____so trying to do the checks as far as i can go 15:02
fungino worries, i have workarounds, i just don't know how many of our (or your) users are impacted. a /24 is pretty huge from ipv6 standards ;)15:03
fungiif you have any questions or anything you want me to test, let me know15:03
fungiif that route isn't in your tables, then there's a good chance there are others missing too (maybe need to increase the memory you've allowed for v6 tables, for example)15:04
guilhermesp_____sure yeah, we havent heard back from other customers but what you showed us is in fact something we should keep an eye to validate 15:04
guilhermesp_____but this is in my radar fungi i will try to move forward with him and update you asap 15:06
fungii was running the external networking group for a large isp back in the days when v4 exhaustion was nearing criticality and the global v6 route count started to explode, we had to fiddle with our memory configs multiple times to accommodate the ever growing table sizes (also because of v4 fragmentation)15:06
fungii saw precisely these sorts of misbehaviors when routers would run out of memory15:07
fungibut of course, there are countless possible causes for what i'm observing15:08
guilhermesp_____thats helpful from the man that has authority to :) 15:08
guilhermesp_____i appending the suggestions here into our internal discussion 15:08
opendevreviewJeremy Stanley proposed opendev/infra-manual master: Document Gerrit service SSH host key fingerprints
fungiPerdu: that ^ will add gerrit's ssh host key fingerprints to the infra manual once it gets approved, but you can see a rendered preview here in the meantime:
fungii've also asked in about the bit in the webui that used to display the fingerprints15:38
fungiand i just remembered that clarkb is gone fishing, so i'll pick his brain tomorrow about the gitea feature request for differentiating the "get started" link by project15:39
fungithanks for bringing these things up. it's the sort of stuff that we can do better, but established users tend not to notice, so it needs a new user who's willing to spend time pointing them out15:41
fungilike the ssh host keys... i have no clue how many gerrit upgrades ago those might have disappeared. could have been years, but nobody brought it up15:42
fungichatter in the gerrit channel confirms it disappeared sometime between 2.11 and 3.5 (we're on 3.7), best guess is it never got added to the new polygerrit interface that showed up in 3.015:43
PerduYea, that what my guess: no established developer is going to check the 'Getting started' page as part of their daily routine15:47
fungiheaded out to run some quick errands, shouldn't hopefully be more than an hour16:34
opendevreviewMichal Nasiadka proposed opendev/irc-meetings master: Rename Containers Team meeting to Magnum
fungilooks like nothing fell apart in that hour17:35
opendevreviewMerged opendev/irc-meetings master: Rename Containers Team meeting to Magnum
fricklernot sure whether that is some issue with my browser or my connections, but I've had multiple times now where a comment in gerrit is only shown as "[object Object]". after a reload the real comment is shown19:51
fungisounds like it might be some lag on the backend, or yeah maybe failed rest api queries20:14
TheJuliais Gerrit okay? page loads seem to be stalling out from my end21:17
fungilooking. i wonder if frickler's earlier observations were a precursor21:52
fungiTheJulia: is it sometimes or always? random? comes and goes?21:53
fungii just pulled up and it displayed near-instantly for me21:53
funginoting that i'm connecting over ipv4 for the moment since there are problems with my isp's v6 space being unreachable in vexxhost ca-ymq-121:54
fungiTheJulia: do you see similar timeouts/stalling trying to browse ?21:55
TheJuliait was pretty much consistent and caused me to stop and shift what I was working on21:56
TheJuliabut it now loads21:56
fungiokay, so whatever it was may have stopped. i'll check cacti graphs21:56
funginothing on the system resource graphs jumps out at me around the time you reported problems21:58
fungimight have been something going on somewhere on the internet or inside that vexxhost region maybe21:58
fungibut whatever it was doesn't seem to have either prevented snmp queries nor shown up as a spike in cpu/memory/io/bandwidth/packetrate21:59

Generated by 2.17.3 by Marius Gedminas - find it at!