Friday, 2024-01-12

clarkbtonyb: was it a rax node and added the 10 net address?00:06
clarkbI don't think we haev any floating IP clouds anymore but that can happe nthere too potentially00:06
tonybclarkb: Correct.00:06
clarkbwe probably need to work backwards from the zuul inventory to zuul to nodepool to openstack sdk to the cloud to figure that one out00:06
clarkbtonyb: that said I would expect the meetpad server to also be using the 10 net address in the configs if it was applied consistently then it should work because we'd connect from 10 net to 10 net00:08
tonybI don't really want to change the rules template but, can we use `ansible_host` instead of `public_v4` ?00:08
clarkbpotentially. Looking at the public address is the public non 10 net address for all nodes00:08
clarkbis our translation from zuul inventory to system-config bridge inventory at fault maybe?00:09
tonybwhere does that happen?00:10
clarkbtonyb: playbooks/zuul/run-base.yaml runs write-inventory00:11
clarkband that role lives in zuul-jobs00:12
clarkbok that explains why we use that value. It does not explain why network connections weren't made over that network. Or maybe they are and it isn't routable?00:14
clarkboh no its because we configure the xmpp target to be the public address00:14
clarkb there00:15
tonybI can update that to a better value00:15
clarkbok cool we understand now. I think the fix here is to set meetpad_jvb_xmpp_server in the test group vars file00:15
clarkbthat way we can address my comment there and also select public_v4 rather than ansible_host00:15
clarkbthe prod value will continue as is unchagned pointed at meetpad01 (until we have a meetpad02) and the test side can talk to meetpad99 on the correct addr00:16
clarkbthis is good I feel like I'm learning things.00:17
tonybNext question .... I created an autohold with the webUI and set the trigger count to 5 ... because I figured I'd need it a more than once.  So how do I release the autohold without deleteing the entry?00:18
tonybor is that not what the trigger count is for?00:18
clarkbyou know thats a really good question. I think the trigger count might be more applicable to casting a wide net to gather more datapoints for a single failure rather than iterative one after another captures00:19
clarkbyou can delete the does in nodepool directly and leave the hold in place00:19
clarkbbut I don't think you can do the cleanup you want via zuul directly00:19
clarkbcorvus: ^ fyi on zuul functionality00:19
clarkb* you can delete the nodes00:20
tonybclarkb: Thanks.  I'll work on updateing the chnage and making a coffee to give corvus time to reply.00:20
clarkbI could go for a warm beverage.00:21
clarkbwe had to take the long way around to get home from school today beacuse the kids don't have boots on and everything is swampy00:22
tonybclarkb: so the crux of the update would be:
clarkbyes I think that should do it00:27
corvusyeah i think there's a gap there.  :)  i agree you should be able to do that thru the ui but can't.  so what clarkb said.00:30
corvusalso, as long as opendev isn't at capacity, we're not going to notice a few held nodes sitting around a bit too long00:31
corvusso for times like right now, probably okay to just ignore it00:31
opendevreviewTony Breeds proposed opendev/system-config master: Switch meetpad servers to jammy
opendevreviewTony Breeds proposed opendev/system-config master: DNM: force jitsi_meet node failure for testing
opendevreviewTony Breeds proposed opendev/system-config master: Switch meetpad servers to jammy
opendevreviewTony Breeds proposed opendev/system-config master: DNM: force jitsi_meet node failure for testing
tonybIs there a better way to achieve:
Clark[m]tonyb wrap the section in raw tags02:09
Clark[m]Section == line in this case02:10
tonybI searched and search for that02:11
opendevreviewTony Breeds proposed opendev/system-config master: Switch meetpad servers to jammy
opendevreviewTony Breeds proposed opendev/system-config master: DNM: force jitsi_meet node failure for testing
tonybThat's much neater02:19
tonybOkay, the servers are up and held.  Things seem to work, but there are worrying outputs in the container logs.05:29
tonybI'm going to finish up for today05:29
tonybclarkb: are you able to look at the held node with me tomorrow to work out if the things I'm seeing are problems?05:31
Clark[m]tonyb: I should be able to06:05
*** mmalchuk_ is now known as mmalchuk06:13
*** noonedeadpunk_ is now known as noonedeadpunk08:14
fricklerguilhermesp: mnaser: I'm having no connectivity to (review.) via IPv6 once again. global reachability seems to be only via cogent, which is ... a questionable design choice. anything you could do about this? (both short and long term) cf.
SvenKieskeme too10:34
SvenKieskeat first I was wondering if was down oO10:35
fricklerSvenKieske: you are also behind AS3320 iirc?10:36
SvenKieskeat least it works once the IPv4 Version is cached.10:46
SvenKieskemhm, the WebUI is rather unresponsive and slow.. :/10:50
ihalomihey, not sure if this is right channel but i have problem with my account at opendev. I changed prefered email in my ubuntu one account and now I cannot login into opendev account. It sents me to this page,SIGN_IN,Contact+site+administrator who should I contact about it? I think its because there is already account in opendev using my new prefered email but ubuntu one accout used to create it no 10:50
SvenKieske"Error submitting review TypeError: NetworkError when attempting to fetch resource."10:56
fricklerihalomi: seems your message was cut off at the end after "... but ubuntu one accout used to create it no". I can't check right now, please wait for some other admin to show up11:51
ihalomi it was endding no longer exists, okay i will wait thanks for reply12:04
fungiihalomi: changing your ubuntuone e-mail address creates a new ubuntuone id, so you end up trying to log into gerrit as a new id which creates a new gerrit account. if the address for the new account was also an address associated with your old gerrit account, gerrit will refuse to create the new account because multiple accounts aren't allowed to have any of the same e-mail addresses12:35
ihalomifungi: okay but what can i do about it? 12:36
fungiyou'll need one of our sysadmins to deactivate your old account and clear the addresses out of it, which is a complicated and time-consuming process. i'm just waking up and don't have time to work on it right now, but may be able to later today12:37
ihalomifungi: okay so there is nothing I can do otherwise than remind myself to sysadmins, please try to do it I would like to push some changes but i cant do it under old client address 12:39
fungiihalomi: if you can /msg me the e-mail address you're trying to use 12:47
fungithat should help me find the old and new accounts12:47
SvenKieskemhm, seems there is some general trouble with the largest german ISP :/ routing problems after a botched network upgrade13:28
opendevreviewElod Illes proposed openstack/project-config master: Adapt make_branch script to new 'unmaintained/<series>' branch
dpanechHi, this review is stuck. It has all the votes, but is not merging: . Could someone help me investigate why?15:00
fungidpanech: gerrit says the parent of that change got revised and merged but the change was never rebased onto that new state15:15
fungidpanech: the parent commit of 904421 is patchset #1 of 904420, but patchset #2 of 904420 is what ended up merging15:16
fungidpanech: if you rebase 904421 onto the current state of the master branch it should work15:16
opendevreviewJeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container
fungiClark[m]: what's the order of operations for cleaning external-ids out of an account and retiring it? i do this so infrequently i can't recall which comes first, and the comments in those scripts don't say15:50
fungialso, what's the easiest way to look up the id if all i know is an e-mail address? put together a rest api query or is there an easier route?15:51
Clark[m]fungi: I believe it is removal of the external ids first. Then removal of preferred email and disabling the account happen together in the second step15:52
Clark[m]We should communicate that the user will end up with a new account15:53
Clark[m]Not sure if we did that. Currently on a school run15:53
fungiyeah, i did (see scrollback)15:53
dpanechfungi: ok thanks15:59
fungiinfra-root: update on the matrix homeserver hosting situation... i got budget approval for openinfra foundation to cover the account upgrade cost and reached out to ems yesterday asking them to proceed. they responded today stating we're in the upgrade queue now and they'll get back to us with more details at the end of the month16:00
clarkbfungi: ack thanks for the update16:11
fungii'm headed out to meet some friends for lunch on an adjacent island and run a quick errand, but should be able to get to ihalomi's old account retirement when i get back around 18:30 utc16:14
clarkbfungi: enjoy!16:14
clarkbservice coordinator election plan email sent making it official17:06
fungiclarkb: what's the easiest way to look up the id if all i know is an e-mail address? put together a rest api query or is there an easier route?19:08
clarkbfungi: one sec19:09
clarkbfungi: In the web ui do a owner:emailaddr query. On the resulting page will be a view dashboard link. That link has the id in it19:10
clarkbthats the hack way I do it19:10
fungioh, good trick19:11
clarkbfungi: I think the error log will also record the conflicting id for the email address on login attempts19:13
clarkbsomething liek couldn't create id foo because email addr xyz@com conflicts with id bar19:13
clarkbit might be a good idea to double check hyou see that error message before making the account changes19:13
clarkbjust to confirm this is the issue19:13
fungigood call. it took about three clicks from the query to get to a version of the owner dashboard that embedded the id but i did find it. grepping the error log for that id number also turns up the "cannot create external id" error19:25
fungiwould adding that new external id to the existing account be a cleaner solution than killing the old account?19:25
fungithat was what we mostly did in the pre-notedb days, but not sure if there's an equivalent solution nopw19:26
clarkbyes, but the only way to do that is to push the update under gerrit which can only safely be done while offline19:27
clarkbbecuse gerrit will validate the entire db then complain about those unrelated errors we still haven't fixed and prevent this update from happening19:27
clarkbit is possible we might eb able to get away with an out of band push while gerrit is running and then do a reindex but I don't want to risk it given its the account database19:27
clarkbif we fix all the errors then we could do this properly. But I have a hard tiem prioritizing it because its pretty annoying drudge work that really only helps a few people19:28
clarkbwe could also potentially declare bankruptcy and break all the remaining accounts (I think there are like 30 of them) on the assumption that most (all?) are not active users and if/when they do become active we can sort them out later19:29
fungioh, got it. i keep forgetting we can't push updates for the all-users repo into running gerrit currently19:30
mbeierlHello. I'm following this and when it comes to the `git review -s`, it fails due to 404 on "GET"19:39
clarkbmbeierl: it should be fetching that commit-msg hook from gerrit not gitea (gerrit is at and gitea runs
clarkbmbeierl: did you manually set your git remote for 'gerrit' ?19:41
opendevreviewJeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container
mbeierlNo.  I did the clone like so: "git clone<namespace>/<projectname"  Should that be from instead?19:42
clarkbmbeierl: no, you should clone from to keep non code review load off of the code review server.19:43
clarkbmbeierl: so you cloned then ran `git review -s` without changing any git remotes? Can you share the project so that we can check the project config?19:44
fungiso it's definitely odd that git review -s isn't finding the .gitreview file in the repo19:44
mbeierlSo I have a `origin` fetch and push like so:
clarkbfungi: and that it is using https instead of ssh19:44
fungioh, yes that's also strange. maybe a global git environment setting?19:44
clarkb that config looks fine to me19:44
clarkbit should be using ssh too19:44
clarkbmaybe run git review with the verbose flag so that we can see the decisions it is making and paste that?19:45
fungito (not directly in the channel, it will be a lot of text)19:46
clarkbthats the -v or --verbose flag19:46
clarkbmbeierl: ya the origin remote should point there. Then when you run git review -s it should create a new 'gerrit' remote by default using ssh not https that points to gerrit for pushing of code changes. It also uses this ssh remote to scp the commit msg hook by default. I think what we're confused about is that it is choosing the wrong protocol and the wrong host19:48
mbeierlaha.  I found I had an old .gitconfig [gitreview] entry in my home directory, with remote=origin in it.  That certainly can mess things up19:49
fungisounds like you'd configured something to try to avoid using ssh with a gerrit (somewhere else maybe?)19:50
clarkbmaybe we should go ahead and vendor the hook in git-review19:52
clarkbadd a flag to fetch it from gerrit using the existing code path that is disabled by default. Though that wouldn't help here because the next step would be pushing and gitea would reject that19:53
fungi#status log Retired Gerrit account 35105 at the request of ihalomi who is has changed their UbuntuOne login E-mail resulting in a new OpenID20:06
opendevstatusfungi: finished logging20:06
opendevreviewJeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container
opendevreviewJeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container
tonybclarkb: Are you around to look at meetpad99?21:11
tonybmbeierl: I *think* you're using the packaged version of git-review and you need the version from pypi21:17
clarkbtonyb: yup!21:21
clarkbsorry just finished up lunch21:21
clarkbdo you want ot use meetpad prod and voice chat or just use irc?21:21
tonybLet's stick with IRC21:22
clarkbok I'm looking up held nodes now21:23
tonybthat way we can use the test meetpad21:23
tonyb# Temp hosts for OpenDev testing21:23
tonyb23.253.166.123  bridge99.opendev.org21:23
tonyb23.253.166.151  jvb99.opendev.org21:23
tonyb23.253.166.131  meetpad99.opendev.org21:23
clarkbok I'm sshing into meetpad and jvb21:23
clarkbare you saying you want to try and do voice chat on them too?21:24
tonybWe will need some load21:24
clarkback let me update /etc/hosts21:24
tonybfor my testing I did just have a muted call running21:25
clarkb I'm there and it seems to have loaded the etherpad successfully too21:26
tonybWhat I was seeing is a) meetpad isn't scheduling callin on jvb99, I don't know if we can force that21:27
tonyb... I guess we could down the jvb containre on meetpad9921:27
clarkbI don't know how to force it either. But usually I think it just round robins or maybe one after hte other21:27
clarkboh ya that would be the way to try and force it21:27
clarkbI'll stop my call then we do that on meetpad99 and try again21:27
clarkbok I stopped jvb on meetpad9921:28
clarkbas soon as you joined the meeting jvb99's log exploded with content21:29
clarkbso I think it is working but I'll unmute to triple check21:29
tonybjvb_1  | JVB 2024-01-12 21:29:31.551 SEVERE: [24] [confId=750e571cd2cb8973 conf_name=isitbroken@muc.localhost meeting_id=44f07352 epId=b2c414b3 stats_id=Odie-8XV] Endpoint.scheduleEndpointMessageTransportTimeout$lambda$30#702: EndpointMessageTransport still not connected.21:30
clarkbtonyb: ha this doesn't work beacuse chrome is verifying the https cert issuer21:44
* clarkb tries firefox21:44
clarkbor maybe it is trying to drop to http beacuse of some redirect in any case both ff and chrome don't like it when I edit /etc/hosts to make meetpad99 also meetpad21:45
clarkb" has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site."21:45
tonybYeah, and as there is an invalid cert it's refusing21:46
clarkbI guess this is about not downgrading to http21:46
tonybOkay I'll rebuild and set the hostnames everywhere to meetpad9921:46
clarkbI don't know why it would downgrade to http though21:46
tonybThat was super helpful21:47
clarkbI don't see my requests hitting it. Really wish browsers had a "I'm ok with shooting myself in the foot with certs and names" mode for testing21:49
clarkbintsead of "you cannot do this"21:49
clarkbanyway I think if we get the names right it should work21:49
tonybYeah me too21:49
clarkbfor anyone at home wondering the web client was trying to connect to to set up a websocket to the jvb server. Problem is != meetpad99 which we used to set up the call so it failed21:52
clarkbneed to get the test env configured to set the names properly21:52
opendevreviewTony Breeds proposed opendev/system-config master: Switch meetpad test servers to jammy
opendevreviewTony Breeds proposed opendev/system-config master: DNM: force jitsi_meet node failure for testing
opendevreviewTony Breeds proposed opendev/system-config master: Allow overriding of meetpad PUBLIC_URL
tonybI don't know if 905510 is correct, it feels "hinky" because of mixing role and group vars22:02
opendevreviewJeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container
fungiclarkb: i think we saw that same websocket issue testing with held nodes in the past22:18
clarkbtonyb: I think that should work though but ya its a lot of plumbing in part because we have somewhat duplicated config between the jvb and meetpad proper22:22
tonybthanks.  I broke it somewhere.  I might pick this back up on Monday 22:23
opendevreviewMerged opendev/irc-meetings master: Revitalize I18n meeting slot

Generated by 2.17.3 by Marius Gedminas - find it at!