| *** sshnaidm|ruck is now known as sshnaidm|afk | 00:08 | |
| *** tosky has quit IRC | 00:09 | |
| *** brinzhang_ has joined #opendev | 00:34 | |
| *** brinzhang has quit IRC | 00:37 | |
| *** lbragstad_ has joined #opendev | 00:38 | |
| *** lbragstad has quit IRC | 00:40 | |
| openstackgerrit | Ghanshyam proposed opendev/subunit2sql master: Fix compatibility with latest oslo.config https://review.opendev.org/c/opendev/subunit2sql/+/764832 | 00:41 |
|---|---|---|
| *** _mlavalle_1 has quit IRC | 01:09 | |
| *** lbragstad_ is now known as lbragstad | 01:37 | |
| openstackgerrit | Ghanshyam proposed opendev/subunit2sql master: Fix compatibility with latest oslo.config https://review.opendev.org/c/opendev/subunit2sql/+/764832 | 01:45 |
| *** lbragstad has quit IRC | 01:46 | |
| openstackgerrit | Merged openstack/project-config master: Set up access for #openinfra channel https://review.opendev.org/c/openstack/project-config/+/771073 | 01:57 |
| *** hemanth_n has joined #opendev | 02:05 | |
| openstackgerrit | Merged opendev/system-config master: Set stop_grace_period on nodepool-builder containers https://review.opendev.org/c/opendev/system-config/+/771899 | 02:20 |
| *** hamalq has quit IRC | 02:24 | |
| *** ysandeep|away is now known as ysandeep | 02:43 | |
| *** brinzhang_ has quit IRC | 02:44 | |
| *** brinzhang_ has joined #opendev | 02:45 | |
| *** ianw is now known as ianw_pto | 03:12 | |
| *** lbragstad has joined #opendev | 03:25 | |
| *** Mister_Q has joined #opendev | 03:53 | |
| *** Mister_Q has quit IRC | 04:28 | |
| *** ykarel has joined #opendev | 04:59 | |
| *** lbragstad has quit IRC | 05:11 | |
| *** marios has joined #opendev | 06:16 | |
| openstackgerrit | Rico Lin proposed openstack/project-config master: Mark min-ready for ubuntu-focal-arm64 https://review.opendev.org/c/openstack/project-config/+/771912 | 06:30 |
| *** sboyron has joined #opendev | 06:32 | |
| *** rpittau|afk is now known as rpittau | 07:34 | |
| *** hashar has joined #opendev | 07:41 | |
| *** ralonsoh has joined #opendev | 07:48 | |
| *** eolivare has joined #opendev | 07:50 | |
| *** slaweq has joined #opendev | 07:54 | |
| *** ysandeep is now known as ysandeep|lunch | 08:00 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 08:02 |
| *** andrewbonney has joined #opendev | 08:13 | |
| frickler | mnaser: I finally got feedback from AS3320 regarding IPv6 from vexxhost. seems their main complaint is that there are no IRR records for your prefixes https://bgp.he.net/net/2604:e100:3::/48#_irr | 08:15 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename config repos file config for CentOS Stream https://review.opendev.org/c/zuul/zuul-jobs/+/770815 | 08:15 |
| frickler | mnaser: you should get that fixed, it is likely to affect more connectivity in the future, ping me if you need help with that | 08:16 |
| *** fressi has joined #opendev | 08:16 | |
| *** tosky has joined #opendev | 08:44 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** klonn has joined #opendev | 09:08 | |
| *** zigo has quit IRC | 09:14 | |
| *** zigo has joined #opendev | 09:15 | |
| *** mordred has quit IRC | 09:21 | |
| *** Eighth_Doctor has quit IRC | 09:21 | |
| *** ysandeep|lunch is now known as ysandeep | 09:33 | |
| openstackgerrit | Pranali Deore proposed openstack/project-config master: Add official-openstack-repo-jobs for openstack/glance-tempest-plugin https://review.opendev.org/c/openstack/project-config/+/771954 | 09:39 |
| *** DSpider has joined #opendev | 09:40 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 09:45 |
| *** dtantsur|afk is now known as dtantsur | 09:59 | |
| *** Eighth_Doctor has joined #opendev | 10:08 | |
| *** hemanth_n has quit IRC | 10:13 | |
| *** hemanth_n has joined #opendev | 10:13 | |
| *** mordred has joined #opendev | 10:27 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 10:33 |
| *** hashar is now known as hasharAway | 10:44 | |
| *** hemanth_n has quit IRC | 10:57 | |
| *** hemanth_n has joined #opendev | 10:58 | |
| *** ykarel_ has joined #opendev | 11:08 | |
| *** ykarel has quit IRC | 11:08 | |
| *** ykarel__ has joined #opendev | 11:16 | |
| *** ykarel_ has quit IRC | 11:19 | |
| *** calcmandan has quit IRC | 11:22 | |
| *** calcmandan has joined #opendev | 11:22 | |
| *** ysandeep is now known as ysandeep|afk | 12:15 | |
| *** klonn has quit IRC | 12:21 | |
| *** iurygregory_ has joined #opendev | 12:28 | |
| *** iurygregory has quit IRC | 12:28 | |
| *** iurygregory_ is now known as iurygregory | 12:29 | |
| *** jpena is now known as jpena|lunch | 12:35 | |
| *** hasharAway is now known as hashar | 12:52 | |
| *** AJaeger has joined #opendev | 12:54 | |
| *** ysandeep|afk is now known as ysandeep | 12:56 | |
| *** hemanth_n has quit IRC | 13:11 | |
| *** fressi has quit IRC | 13:12 | |
| *** tkajinam_ has quit IRC | 13:16 | |
| *** sboyron has quit IRC | 13:16 | |
| *** sboyron has joined #opendev | 13:27 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| openstackgerrit | Maksim Malchuk proposed openstack/diskimage-builder master: Don't use hardcode while override base image file https://review.opendev.org/c/openstack/diskimage-builder/+/771978 | 13:37 |
| *** lbragstad has joined #opendev | 13:42 | |
| *** ykarel__ is now known as ykarel | 13:44 | |
| *** AJaeger has quit IRC | 13:53 | |
| openstackgerrit | Javier Peña proposed openstack/diskimage-builder master: Fix CentOS Stream 8 base repo in centos element https://review.opendev.org/c/openstack/diskimage-builder/+/771979 | 13:54 |
| openstackgerrit | Javier Peña proposed zuul/zuul-jobs master: Fix BaseOS repo file name for CentOS Stream https://review.opendev.org/c/zuul/zuul-jobs/+/771980 | 13:54 |
| *** mlavalle has joined #opendev | 13:58 | |
| *** klonn has joined #opendev | 14:00 | |
| *** ysandeep is now known as ysandeep|away | 14:06 | |
| *** klonn has quit IRC | 14:18 | |
| *** klonn has joined #opendev | 14:18 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 14:22 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 14:22 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 14:24 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 14:25 |
| openstackgerrit | Sorin Sbârnea proposed opendev/git-review master: Allow choosing field for author in named branch https://review.opendev.org/c/opendev/git-review/+/444574 | 14:25 |
| *** rpittau is now known as rpittau|afk | 14:58 | |
| openstackgerrit | Merged opendev/git-review master: Allow choosing field for author in named branch https://review.opendev.org/c/opendev/git-review/+/444574 | 14:59 |
| *** mugsie has joined #opendev | 15:09 | |
| *** mugsie has quit IRC | 15:18 | |
| *** zbr3 has joined #opendev | 15:22 | |
| *** zbr has quit IRC | 15:24 | |
| *** zbr3 is now known as zbr | 15:24 | |
| *** mugsie has joined #opendev | 15:51 | |
| *** lpetrut has joined #opendev | 16:10 | |
| *** zul has quit IRC | 16:11 | |
| *** lpetrut has quit IRC | 16:14 | |
| openstackgerrit | Javier Peña proposed zuul/zuul-jobs master: Fix repo files for CentOS Stream https://review.opendev.org/c/zuul/zuul-jobs/+/771980 | 16:22 |
| *** ykarel has quit IRC | 16:47 | |
| openstackgerrit | Javier Peña proposed zuul/zuul-jobs master: Fix repo files for CentOS Stream https://review.opendev.org/c/zuul/zuul-jobs/+/771980 | 16:47 |
| *** marios has quit IRC | 17:15 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 17:16 |
| *** eolivare has quit IRC | 17:18 | |
| fungi | paladox: tristanC: do either of you have a basic understanding of how notedb functions? we've been trying to work out how to manually update an external-id in modern gerrit but questions to the ml have been met with silence | 17:20 |
| clarkb | in particular All-Users:refs/meta/external-ids doesn't appear to be a notes thing if you `git fetch origin refs/meta/external-ids:refs/meta/external-ids && git notes list refs/meta/external-ids`. So it isn't clear how one is meant to update the external ids notes in that ref | 17:21 |
| clarkb | if you git log refs/meta/external-ids you get what appears to be a normal git tree history | 17:22 |
| fungi | short story is a user's openid seems to have changed, we need to associate it with their prior account. in earlier gerrit we'd just update the row in the external_ids table, but with notedb it seems like we need to commit files in refs/meta/external-ids and then push (potentially racing other updates to external ids when doing so) | 17:22 |
| fungi | there's no rest api method we can find documented for adding or changing external ids, only fetching and deleting | 17:23 |
| clarkb | https://phabricator.wikimedia.org/T197192 has some interesting related info (though not the answer to our questions as far as I can tell) | 17:24 |
| clarkb | someone found a way to do it then didn't write it down :( | 17:24 |
| *** hashar has quit IRC | 17:25 | |
| corvus | clarkb: i ran into a problem recently using a (non-opendev) current gerrit with openid (not sso, any openid) where i set a username for my account, then removed the original email address, and ended up with a new user on next login. the only way i could correct the situation was to shut gerrit down and rewind all-users. | 17:26 |
| corvus | that sounds similar (but maybe not exactly) like what your msg describes. | 17:26 |
| fungi | yeah, this could be similar | 17:27 |
| corvus | to be fair, i did think that there might be a way to correct it with the aid of a second admin account and three different transitions, but that sounded way harder than rewinding, and i didn't want to go make another account | 17:27 |
| fungi | also that wmf phab entry frightens me. i think once we do try to push this we're going to hit the same validation errors because i know for a fact that we have accounts with duplicate e-mail addresses | 17:27 |
| clarkb | fungi: ya | 17:28 |
| fungi | seems like it indicates we'll have to go through and clean up (likely hundreds) of old accounts which previous gerrit allowed to have duplicate addresses | 17:28 |
| clarkb | what is really confusing to me is that refs/meta/external-ids doesn't seem to be notes but a proper git tree | 17:28 |
| clarkb | however the contents of that tree look note like so I'm all kinds of confused | 17:29 |
| corvus | clarkb, fungi: the main thing i think was the issue in my case was the inequivalency of different external ids. there seems to be a 'primary' id when you look at all-users, and doing certain operations changes that. to the point where the actual openid didn't matter for the account anymore, it was the email address gerrit cared about. | 17:29 |
| corvus | clarkb: there's 2 things, let me dig them up. 1 sec | 17:29 |
| clarkb | corvus: yes in this case we just want to change the account id in the note for the openid url I think | 17:29 |
| clarkb | we're poking at it on review-test | 17:30 |
| corvus | clarkb: refs/meta/external-ids is a tree with each file the sha of the external id | 17:31 |
| clarkb | corvus: yup | 17:31 |
| corvus | so you look up the external id "username:corvus" by sha-ing that and looking up the file with that name | 17:32 |
| clarkb | yup where we've gotten is roughly git fetch origin refs/meta/external-ids && git checkout FETCH_HEAD | 17:32 |
| clarkb | then I can vim the files and edit the account id for the openid file I want | 17:32 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 17:32 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 17:32 |
| clarkb | but thats a very normal git tree appraoch but gerrit implies these are git notes instead. I think the distinction matters because if I do a git commit after vim'ing some files and push that to refs/meta/external-ids I'm likely to conflict with any updates that happened between when I fetched and when I pushed? | 17:33 |
| clarkb | or at leasat the risk of racing is there | 17:33 |
| corvus | ah right, then there's refs/users which has the account info (including preferred email) | 17:33 |
| corvus | i think they mostly wrote-off the idea of actually using git notes | 17:34 |
| corvus | clarkb: so i think you understand that file correctly. | 17:34 |
| clarkb | corvus: hrm that would match what I observe except for the magical tree sharding which is supposedly something ti does automatically for objects? | 17:34 |
| corvus | i assumed that was just convention -- like, it's a ref pointing to a tree with a bunch of files and they didn't want a directory with millions of entries | 17:35 |
| corvus | but i don't know for sure | 17:35 |
| clarkb | corvus: well I pushed a docs update for it beacuse it was super confusing when I first ran into this and they told me no git does that automatically for notes | 17:35 |
| fungi | but apparently it's variable-depth, based on some algorithm git also uses for note storage | 17:35 |
| clarkb | and the level varies based on total numbers of objects | 17:35 |
| corvus | clarkb: if you do decide to update the repo, remember the cache will need to be cleared if you do it while server is up. | 17:36 |
| clarkb | corvus: ok, do you know if accounts need to be reindexed too? | 17:36 |
| corvus | clarkb: weird. maybe they're using the underlying implementation for git notes just not actual notes. | 17:36 |
| fungi | which is not too dissimilar from needing to clear the cache after we updated tables in sql previously | 17:36 |
| clarkb | (that was another question on my thread that has been ignored) | 17:36 |
| paladox | fungi you can git clone the repo and make changes and git push i think (then reindex). | 17:37 |
| corvus | clarkb: i don't know. i would think that would be a good idea though. | 17:37 |
| corvus | (ie, i would guess 'yes' it needs to be reindexed) | 17:37 |
| clarkb | paladox: do we make those changes with git notes add/edit or vim and then git commit? | 17:37 |
| paladox | no | 17:37 |
| clarkb | if we do it via git commit then the process is inherently racy I believe | 17:38 |
| fungi | paladox: do you know how (or if) it avoids undoing changes which occur in the seconds between pull/edit/commit and push? | 17:38 |
| clarkb | but if you want the gerrit checks to run you have to push to gerrit and not directly when gerrit is off | 17:38 |
| fungi | though i suppose pushing to the filesystem with gerrit stopped is how we'd avoid needing to immediately clean up the mess of duplicate e-mail addresses alluded to at the beginning of https://phabricator.wikimedia.org/T197192 | 17:39 |
| clarkb | fungi: unless it somehow decides to not start up at all | 17:39 |
| clarkb | I feel like that is more risky | 17:39 |
| corvus | clarkb: why would there be duplicates? | 17:40 |
| clarkb | paladox: maybe you can give us a high level overview from git fetch to checkout to edit to commit to push? | 17:40 |
| clarkb | corvus: see https://phabricator.wikimedia.org/T197192 | 17:40 |
| clarkb | corvus: fungi expects we're in the same situation | 17:40 |
| corvus | clarkb: i mean in our case | 17:40 |
| clarkb | corvus: I think its a hunch based on the number of accounts we've got and general user ability to break things that are breakable :) | 17:40 |
| corvus | why would fixing this cause a startup error? | 17:41 |
| clarkb | corvus: if we bypass gerrit's checking and push to git directly as a way to workaround conflicts then we potentially add other problems to the "db" that gerrit will refuse to start on | 17:42 |
| paladox | clarkb i just git clone the repo, edit .git/config (replacing refs/heads/ with refs/*) then i checkout external-id and then i edit the file. Then i push back to the branch. | 17:42 |
| clarkb | gerrit checks more than just email addr conflicts. I'm not sure if any of those additioanl issues it checks for would be fatal if we introduced one via a side load | 17:42 |
| paladox | you can edit via the disk too | 17:42 |
| corvus | i guess i'm missing why changing one account is going to cause hundreds of new errors to appear on startup | 17:43 |
| clarkb | corvus: no I'm suggesting that if we get the one account change wrong then perhaps gerrit will not start | 17:43 |
| corvus | or are you saying you're concerned that the "push to gerrit" path will fail, not the push to disk | 17:43 |
| clarkb | right if we pushed to gerrit gerrit is going to validate a number of potential issues with the edit | 17:43 |
| clarkb | if we push directly that does not happen and we could potentially break the whole thing (I don't have evidence that is a real concern, just that it seems possible) | 17:44 |
| corvus | clarkb: if the supposition is that we have 200 existing errors, i don't see how having 201 would cause it to fail to start. | 17:44 |
| clarkb | corvus: because there are ~6 errors checekd for and the ~200 is only one of those 6 | 17:44 |
| clarkb | paladox: ok, isn't that racy? if you fetch, then a user updates their rest api token, then you edit and push you'll overwrite their changes | 17:45 |
| corvus | there's probably too many unknowns here. seems like testing on review-test is warranted either way. | 17:45 |
| clarkb | paladox: this was one of my questions on the mailing list from a couple of days ago | 17:45 |
| clarkb | corvus: yup that was the paln all along. fungi I guess we do the edit, commit and push on review-test like paladox suggests and see what we learn? | 17:45 |
| paladox | as long as you don't force git push, it *should* ask you to git pull if someone updated it. | 17:45 |
| clarkb | paladox: ya I mentioend that you could git rebase and push in a loop until it goes through in my email too | 17:46 |
| fungi | corvus: sorry, troubleshooting something unrelated at the same time, but catching up the reason we have duplicate e-mail addresses is that the openid plugin used to create them, in newer gerrit it will properly refuse to, but under older gerrit it did not have that | 17:46 |
| clarkb | paladox: mostly I was trying to call out that this process seems incredibly complicated compared to the old system and it isn't well documented and help would be appreciated :) | 17:46 |
| paladox | Yeh, agreed. | 17:46 |
| clarkb | paladox: in particular I think that races like that should be called out in the docs | 17:47 |
| clarkb | because for many git users I bet git push -f is their first reaction :/ | 17:47 |
| paladox | i think that some tool should be created that allows you to do this without worry. | 17:47 |
| clarkb | anyway, fungi I think our next step is backup All-Users on review-test, commit then push and see what happens? | 17:47 |
| fungi | not just a hunch that we have duplicate e-mail addresses, i know for a fact that there are a lot because i have workarounds in the openstack election tooling to work around it and combine/deduplicate account records when two or more accounts have the same e-mail addresses | 17:47 |
| clarkb | fungi: got it | 17:48 |
| fungi | basically normal account creation in gerrit would refuse to add an account with the same address as an existing account, or would refuse to let you add an address to an account if it was already set on another account, but the openid plugin didn't go through those checks when autocreating new accounts and would happily create a new account with the same address as an existing one (for example, if the openid | 17:49 |
| fungi | changed but the openid provider continued to assert the same e-mail address for it) | 17:49 |
| fungi | also new gerrit has gotten stricter it seems, i've seen evidence that it will now refuse to add an address to an account even if that address is only already used by a disabled (inactive) account | 17:51 |
| *** ralonsoh has quit IRC | 17:51 | |
| fungi | so our old method of just setting duplicate accounts inactive without deleting the addresses from them will likely provide even more duplicates | 17:51 |
| clarkb | fungi: for backing up All-Users I figure we stop Gerrit, then rsync the All-Users repo to another location, then restart Gerrit? | 17:51 |
| clarkb | just to avoid any potential updates there too? | 17:51 |
| clarkb | paladox: ya another thing I pointed out on the mailing list is you can get and delete external ids but not edit them via the rest api | 17:52 |
| clarkb | paladox: having an edit api would be great then tools could use that | 17:52 |
| fungi | paladox: probably the "some tool" would be just extending the rest api to add methods for creating/updating external ids similar to the existing methods for getting and deleting them | 17:52 |
| paladox | yeh | 17:52 |
| fungi | the complexity there is likely in makign sure those go through the right validation codepaths | 17:53 |
| clarkb | and avoiding races | 17:53 |
| clarkb | gerrit could sequence them internally if it was handling them all | 17:53 |
| *** klonn has quit IRC | 17:54 | |
| *** jpena is now known as jpena|off | 17:55 | |
| clarkb | fungi: I've gone ahead and committed my edits in my cloned All-Users | 17:58 |
| clarkb | will stop gerrit on review-test and work to make a copy of prod All-Users next | 17:59 |
| clarkb | then we can psuh and see what happens | 17:59 |
| fungi | yeah, as far as races go, presumably it already know how to avoid them when handling its own updates to the refs/meta/external-ids so that bit is presumably solved | 18:01 |
| fungi | as far as the rest api would be concerned | 18:01 |
| clarkb | fungi: how does that rsync command in screen look? | 18:01 |
| fungi | looks legit. you've correctly omitted the trailing / on the source path so it should create /home/gerrit2/backups/All-Users.git | 18:04 |
| fungi | i assume /home/gerrit2/backups/All-Users.git doesn't currently exist? | 18:04 |
| fungi | if it does you'll probably want to add --delete | 18:04 |
| fungi | so you don't end up with extra files in there | 18:04 |
| clarkb | its empty | 18:05 |
| clarkb | I'll run that comamdn now for real | 18:05 |
| clarkb | ok starting gerrit back up again | 18:06 |
| *** slaweq has quit IRC | 18:12 | |
| *** lbragstad has quit IRC | 18:16 | |
| *** dtantsur is now known as dtantsur|afk | 18:17 | |
| TheJulia | is gerrit running a bit slow today? | 18:19 |
| *** klonn has joined #opendev | 18:19 | |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=26&rra_id=all | 18:19 |
| fungi | load average was really high a little while ago | 18:19 |
| fungi | >40 on a 16-cpu vm | 18:20 |
| TheJulia | hmm, okay | 18:20 |
| fungi | it's not exactly low now either, but it's not as high as it was around 16:30z | 18:20 |
| TheJulia | yeah, I think it was a little before then I uploaded a batch of patches and was like "wow, this is taking a while" | 18:21 |
| *** lbragstad has joined #opendev | 18:22 | |
| corvus | TheJulia: you wrote so much code you really made the gerrit server work :) | 18:25 |
| * TheJulia blushes | 18:26 | |
| *** klonn has quit IRC | 18:33 | |
| *** klonn has joined #opendev | 18:33 | |
| clarkb | fwiw it is a fairly regular load spike around then | 18:36 |
| clarkb | at one time I thought it might be related to backups but we run backups ~12 hours later without the same issues | 18:36 |
| clarkb | possibly some ci system doing periodic jobs or thats when we've got the most user overlap and we just hit the server harder then | 18:36 |
| clarkb | will likely need diggign in logs to characterize ebtter | 18:36 |
| fungi | user overlap has been my assumption. we get pacific coast of the americas getting to their desks around then, and europe hasn't quite finished for the day yet | 18:37 |
| clarkb | corvus: paladox: ok fungi and I have discovered some new things. Gerrit will reject the push if it isn't fast forwardable so my hunch that we rebase and retry in a loop when pushing to a live server seems to eb a good one. We have 642 email address errors | 18:37 |
| fungi | glad to know that my estimate was on the low side... :/ | 18:38 |
| clarkb | these 642 email conflict errors seem to come in two versions, first is two different oepnid external ids sharing an email. The other is mailto and openid external ids sharing an email | 18:38 |
| fungi | still for different account ids in the latter case right? | 18:38 |
| clarkb | I'm going to pause here and write a response to myself on repo-discuss explaining what the process we'ev used is and what we've learned before it escapes my head | 18:39 |
| fungi | i assume a mailto and an openid having the same addy is fine if they're for the same account id | 18:39 |
| clarkb | fungi: oh tahts a good question I can check that | 18:39 |
| fungi | clarkb: awesome, then you'll no longer be able to say nobody has responded to your questions! ;) | 18:39 |
| clarkb | but then I think our process might become: push edits to live server to ensure that gerrit doesn't complain about the changes we are making (does it report all validations or just the first set it checks when failing). Once gerrit doesn't complain about our changes we can stop gerrit, rebase to ensure we are fast forwardable, then push into git directly under gerrit. Then start gerrit, reindex and clear | 18:40 |
| clarkb | caches | 18:40 |
| clarkb | I'll suggest that as a workaroudn on the list too to see if anyone complains | 18:40 |
| clarkb | in the meantime we might want to give DannyMassa the go ahead to use the new account or something? I don't know what the right answer is for that, but I don't think we'll have this sorted today | 18:41 |
| *** bodgix has quit IRC | 18:41 | |
| *** bodgix_ has joined #opendev | 18:41 | |
| fungi | longer term we'll probably want to work through the conflicts and try to clean them up, but in many cases that's likely to involve reaching out to users who are unaware they're using two accounts because their openid split from their ssh login at some point years ago and they've been pushing changes with one account and reviewing with a different one | 18:41 |
| fungi | in the past we really only ever found out if someone had trouble adding them to a review group or as a requested reviewer on a change | 18:42 |
| clarkb | fungi: first example I've checked is two different accounts | 18:43 |
| mordred | fungi: that sounds like a "fun" group of conflicts to rectify | 18:43 |
| fungi | yeah, that's what i'm expecting (or at least hoping) they'll all be | 18:43 |
| clarkb | fungi: that example is now in the screen if you want to check it | 18:43 |
| fungi | yeah, that example looks exactly like what i expected | 18:44 |
| clarkb | I'm going to put that list of conflicts in my homedir on review-test too | 18:44 |
| clarkb | then I'm making tea | 18:44 |
| fungi | if we cross-reference against inactive accounts we can probably clean up a bunch, but i'm not sure if it will be a majority | 18:44 |
| clarkb | ~clarkb/conflicting_emails if you want to check them | 18:45 |
| clarkb | paladox: do you know how wikimedia worked around these conflicts? | 18:46 |
| clarkb | paladox: the bug mentions it was worked around, but doesn't say how | 18:46 |
| fungi | we might also just send a form letter to the preferred addresses for the accounts in that state letting them know we need to deactivate one of their accounts and if we don't hear back from them then deactivate both or pick one at random or something | 18:52 |
| fungi | where "deactivate" would imply marking them inactive and removing the conflicting external-ids | 18:52 |
| *** andrewbonney has quit IRC | 18:56 | |
| fungi | okay, my dance card for the day is topped up, i've got to get through stuff i've already committed to, so probably going relatively silent to try to get most of it donw | 18:57 |
| fungi | done | 18:57 |
| paladox | clarkb i think they just edited on disk | 18:59 |
| fungi | so basically what we're looking at doing | 19:00 |
| paladox | yeh | 19:01 |
| fungi | at least that's confirmation that it's probably the easiest interim solution | 19:01 |
| clarkb | fungi: thank you for all the help on this | 19:01 |
| paladox | you push then reindex | 19:01 |
| fungi | paladox: figured. possibly also flush caches since accounts may be persisted in the disk cache between restarts? | 19:02 |
| paladox | yeh | 19:02 |
| fungi | not all that different from what we used to have to do when editing the db behind gerrit's back | 19:03 |
| paladox | yeh | 19:04 |
| clarkb | paladox: do you know if that was done with or without gerrit online? | 19:04 |
| paladox | it was done with gerrit online | 19:04 |
| paladox | since the only difference is you are editing directly skipping gerrit | 19:05 |
| clarkb | ok | 19:05 |
| paladox | so you just git clone <path to gerrit repo on disk> | 19:05 |
| fungi | i suppose as long as gerrit checks the state of the repo on disk every time it commits to refs/meta/external-ids it won't get flustered. if it assumes nothing besides it is touching that ref, then it might throw errors until reindexed? | 19:06 |
| fungi | or maybe it does the same thing we would do: try to push and if it fails rebase onto what's on disk and try again | 19:07 |
| clarkb | https://bugs.chromium.org/p/gerrit/issues/detail?id=9256#c16 aha | 19:09 |
| clarkb | that says offline though | 19:09 |
| fungi | TheJulia: here you can see it wasn't just gerrit that got busy around 16:30, we apparently topped out the zuul executors to the point they stopped accepting build requests, node requests shot up from almost nothing to nearly 1k, and so on... https://grafana.opendev.org/d/9XCNuphGk/zuul-status?orgId=1&from=now-12h&to=now&kiosk=tv | 19:24 |
| fungi | so my guess is a large dump of changes entered gerrit around that time | 19:24 |
| clarkb | alright email sent to repo-discuss. I also sent email to DannyMassa to give an update as well | 19:37 |
| *** slaweq has joined #opendev | 19:46 | |
| fungi | saw it, thanks! | 19:49 |
| *** slaweq has quit IRC | 20:01 | |
| openstackgerrit | Merged opendev/subunit2sql master: Fix compatibility with latest oslo.config https://review.opendev.org/c/opendev/subunit2sql/+/764832 | 20:19 |
| *** auristor has quit IRC | 20:20 | |
| *** auristor has joined #opendev | 20:23 | |
| openstackgerrit | Merged opendev/bindep master: Tolerate missing bindep.txt when using --profiles https://review.opendev.org/c/opendev/bindep/+/734719 | 20:58 |
| prometheanfire | should gean work when distro is defined as networkd on debian? | 21:26 |
| prometheanfire | I imagine the answer is yes | 21:26 |
| clarkb | I don't know that anyone has ever tested it | 21:26 |
| clarkb | we use it with the e/n/i support | 21:26 |
| prometheanfire | probably I'm having is when two interfaces come up both with default gateways... debian network scripts go boom | 21:27 |
| prometheanfire | eni being networkmanager? | 21:27 |
| fungi | basically the difference between "should work" (we consider it a bug if not) and "does work" (we actually test and use it that way) | 21:39 |
| openstackgerrit | Artom Lifshitz proposed zuul/zuul-jobs master: WIP: Generate Zuul temp key in PEM format https://review.opendev.org/c/zuul/zuul-jobs/+/772122 | 21:52 |
| prometheanfire | fungi: yep | 21:56 |
| prometheanfire | on debian glean outputs dhcp for static IP networkd, so gonna look at that | 22:18 |
| *** CWalker has quit IRC | 22:20 | |
| *** klonn has quit IRC | 22:59 | |
| *** paladox has quit IRC | 23:14 | |
| *** paladox has joined #opendev | 23:17 | |
| clarkb | #status log Removed email from DannyMassa's stale gerrit account so that a new account can be created with that email. This is admittedly a workaround but we need to figure out external-id edits more properly to properly solve this. | 23:39 |
| openstackstatus | clarkb: finished logging | 23:39 |
| *** paladox has quit IRC | 23:40 | |
| clarkb | this was suggested by the user fwiw | 23:51 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove old service cleanups from zuul https://review.opendev.org/c/opendev/system-config/+/772133 | 23:51 |
| clarkb | we might also consider disabling the preexisting accounts | 23:53 |
| clarkb | but I think that can wait for when we're "happy" with the running state | 23:53 |
| *** paladox has joined #opendev | 23:55 | |
| artom | Is Depends-on: <zuul-jobs patch> allowed? Will it use the patch when deploying? | 23:57 |
| artom | I've Depends-on https://review.opendev.org/c/zuul/zuul-jobs/+/772122, but it doesn't look like it took hold... | 23:57 |
| clarkb | artom: it depends on whether or not that contents of the zuul-jobs cahnge are executed by a trusted or untrusted portion of a job | 23:58 |
| artom | Or I've grievously misunderstood something | 23:58 |
| clarkb | if they are part of a trusted job portion then they have to be landed first | 23:58 |
| artom | clarkb, setting up the ssh build keys? | 23:58 |
| clarkb | ya thats part of opendev/base-jobs/playbooks/pre.yaml which is trusted | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!