| *** DSpider has quit IRC | 00:00 | |
| clarkb | fungi: looking at the wheel cache stuff we run release-wheel-cache after all the other cache builds run but we skip it if any fail | 00:02 |
|---|---|---|
| clarkb | fungi: I think that between inap ip conflicts and linaro arm64 boot issues we haven't had a recent situation where all of them run together happily | 00:03 |
| clarkb | maybe we can cehck it tomorrow and see if arm is any happier and work from there? | 00:04 |
| *** brinzhang has quit IRC | 00:05 | |
| fungi | sure thing | 00:11 |
| *** ysandeep|away is now known as ysandeep | 00:17 | |
| *** brinzhang has joined #opendev | 00:17 | |
| kevinz | fungi: Alex_Gaynor: Node failuer again? | 00:50 |
| fungi | kevinz: yeah, it looks like nodepool is seeing timeouts waiting for booted instances to become active or reachable | 00:52 |
| fungi | it eventually gives up waiting and tries a couple more times before zuul reports a node_failure result | 00:53 |
| *** mlavalle has quit IRC | 01:04 | |
| kevinz | fungi: looks it is still the rabbitmq error block the cluster. I will figure out why it always happens | 01:10 |
| fungi | thanks kevinz! | 01:10 |
| kevinz | fungi: np, will let you know when it ready | 01:11 |
| kevinz | fungi: network issue caused Rabbitmq break into several partition, so block the cluster. Now recovered | 02:02 |
| kevinz | I will also write a script to monitor it | 02:02 |
| fungi | thanks again for taking care of it kevinz! | 02:13 |
| *** brinzhang has quit IRC | 05:20 | |
| *** sgw has quit IRC | 05:21 | |
| *** sgw has joined #opendev | 05:21 | |
| *** brinzhang has joined #opendev | 05:21 | |
| *** marios has joined #opendev | 06:24 | |
| *** spotz has quit IRC | 06:54 | |
| *** redrobot has quit IRC | 06:56 | |
| *** redrobot has joined #opendev | 06:56 | |
| *** eolivare has joined #opendev | 07:37 | |
| *** ralonsoh has joined #opendev | 07:52 | |
| *** zoharm has joined #opendev | 07:55 | |
| *** rpittau|afk_ is now known as rpittau | 07:57 | |
| *** slaweq has joined #opendev | 07:59 | |
| *** sboyron has joined #opendev | 08:05 | |
| *** andrewbonney has joined #opendev | 08:19 | |
| *** klonn has joined #opendev | 08:27 | |
| *** klonn has quit IRC | 08:29 | |
| *** tosky has joined #opendev | 08:45 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** DSpider has joined #opendev | 09:00 | |
| openstackgerrit | Riccardo Pittau proposed openstack/project-config master: Add key editHashtags to normalize_acl script https://review.opendev.org/c/openstack/project-config/+/772484 | 09:25 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 10:12 |
| *** brinzhang has quit IRC | 10:14 | |
| *** dtantsur|afk is now known as dtantsur | 10:18 | |
| priteau | Hello. We have kayobe stable/train jobs often failing on fetch data from one of the mirrors: Timeout on https://mirror-int.iad.rax.opendev.org/epel/7/x86_64/repodata/repomd.xml | 10:38 |
| priteau | Is there a known issue with it? | 10:38 |
| frickler | priteau: do you have a sample job log? no known issue as far as I can tell | 10:44 |
| priteau | At the end of https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_838/771246/1/check/kayobe-seed-upgrade-centos7/8383419/primary/ansible/seed-deploy-pre-upgrade | 10:46 |
| priteau | But if you think it works fine, it could be some issues with our test environment | 10:46 |
| priteau | Nevermind, this is probably a test environment issue | 10:54 |
| *** brinzhang has joined #opendev | 11:04 | |
| *** rpittau is now known as rpittau|bbl | 11:09 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 11:15 |
| *** Eighth_Doctor has quit IRC | 11:19 | |
| *** mordred has quit IRC | 11:19 | |
| *** Eighth_Doctor has joined #opendev | 11:30 | |
| openstackgerrit | Aleksey proposed openstack/diskimage-builder master: Extend ubuntu-minimal element https://review.opendev.org/c/openstack/diskimage-builder/+/772502 | 11:32 |
| *** mordred has joined #opendev | 11:48 | |
| openstackgerrit | Aleksey proposed openstack/diskimage-builder master: Extend ubuntu-minimal element https://review.opendev.org/c/openstack/diskimage-builder/+/772502 | 11:48 |
| *** klonn has joined #opendev | 12:05 | |
| noonedeadpunk | hi! Sorry, I totally missed that a while back - but can you kindly remind me if there are any plans to renew integration of gerrit with launchpad? | 12:16 |
| *** eolivare has quit IRC | 12:16 | |
| *** klonn has quit IRC | 12:22 | |
| *** jpena is now known as jpena|lunch | 12:25 | |
| *** rpittau|bbl is now known as rpittau | 12:59 | |
| frickler | noonedeadpunk: afaict the "plan" is to wait for someone to re-implement it, haven't seen anything more specific yet | 13:04 |
| *** jpena|lunch is now known as jpena | 13:17 | |
| *** eolivare has joined #opendev | 13:22 | |
| *** hashar has joined #opendev | 13:27 | |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add zuul-storage-proxy in zuul namespace https://review.opendev.org/c/openstack/project-config/+/772364 | 13:54 |
| *** mlavalle has joined #opendev | 13:57 | |
| openstackgerrit | Merged opendev/system-config master: OpenstackId v3.0.17 https://review.opendev.org/c/opendev/system-config/+/772348 | 14:08 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 14:20 | |
| *** stand has quit IRC | 14:24 | |
| *** brinzhang has quit IRC | 14:25 | |
| *** brinzhang has joined #opendev | 14:25 | |
| *** stand has joined #opendev | 14:26 | |
| *** stand has quit IRC | 14:43 | |
| *** brinzhang has quit IRC | 14:48 | |
| *** sgw has left #opendev | 14:48 | |
| *** brinzhang has joined #opendev | 14:51 | |
| *** brinzhang has quit IRC | 14:51 | |
| *** brinzhang has joined #opendev | 14:52 | |
| clarkb | noonedeadpunk: fungi has indicated a desire to work on it, but unfortunately higher priority issues keep cropping up | 14:57 |
| noonedeadpunk | ok, thanks for providing info) | 14:57 |
| *** ralonsoh has quit IRC | 15:04 | |
| *** ralonsoh has joined #opendev | 15:04 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 15:07 | |
| *** hashar has quit IRC | 15:10 | |
| *** fressi has quit IRC | 15:19 | |
| openstackgerrit | Aleksey proposed openstack/diskimage-builder master: Extend ubuntu-minimal element https://review.opendev.org/c/openstack/diskimage-builder/+/772502 | 15:41 |
| fungi | noonedeadpunk: clarkb: yeah, help welcome, i'm starting to think it may be better if we look at doing it with a nodeless zuul job instead of fixing the hook script in jeepyb | 15:42 |
| fungi | mainly because fixing what's in jeepyb will be a not-insignificant rewrite to swap out database queries with rest api calls | 15:43 |
| fungi | and that in turn means more credentials to manage | 15:43 |
| clarkb | ya I think that would work. The odwnside to it is a busy zuul may not get to updates until later (but maybe a supercedent queue where it does as much as possible when it runs makes sense?) | 15:44 |
| clarkb | fungi: fwiw I've started on a script on review-test to dump a bunch of relevant data for the 642 conflicting emails in external ids accounts | 15:44 |
| clarkb | treating account ids as strings is great until you hit prefix collisions :) | 15:45 |
| fungi | the gerrit hook script wasn't robust anyway, so often failed to comment for a number of reasons and the only logs were locally on the gerrit server | 15:48 |
| fungi | whereas if it's in zuul, hopefully we can safely make it so that folks can see errors (so long as we make sure the build log won't leak gerrit or launchpad credentials) | 15:49 |
| clarkb | at first glance the output data is telling me "this is not going to be fun" | 15:50 |
| clarkb | there are a small number of inactive accounts where we might be able to brute force a cleanup, but many of these accounts have different openids and different ssh usernames implying that both or either could be in use | 15:50 |
| clarkb | the output is still running through (and not terribly quickly as its doing a fair bit of disk io scanning info | 15:51 |
| clarkb | oh hey I see a smcginnis :) | 15:51 |
| smcginnis | :) | 15:52 |
| clarkb | lol stackalytics is in that list too | 15:52 |
| clarkb | smcginnis: we might bug you later to udnerstand how to best untangle this gerrit account situation using you as an example | 15:52 |
| clarkb | (for now I'm still trying to collect the data we need to begin to understand anything) | 15:52 |
| smcginnis | I haven't been following the scrollback, but just let me know if you need me to do anything on this side. | 15:52 |
| clarkb | smcginnis: ya ignore me for now, you can pay attention to the board meeting. I'll reach out if we need your input later | 15:53 |
| smcginnis | I have a reviewerstats script that runs every four hours that pulls some data via SSH. That should be the only thing coming from me. | 15:53 |
| smcginnis | Looks like the code does at least attempt to close the connection - https://opendev.org/openstack/reviewstats/src/branch/master/reviewstats/utils.py#L301 | 15:55 |
| fungi | smcginnis: but the summary is that gerrit now strongly wants all email addresses to be uniquely associated with a single account (can't even conflict with an inactive account), and that's blocking us from making further changes to gerrit accounts | 15:55 |
| smcginnis | Ah, because I had two accounts at some point? | 15:55 |
| clarkb | ya its not about activity, its about account config state | 15:55 |
| fungi | yeah | 15:55 |
| clarkb | smcginnis: yes | 15:55 |
| fungi | so, like, we may need to delete addresses from your old inactive account or something | 15:56 |
| smcginnis | If y'all have the ability, you could update the email associated with that old inactive account. | 15:56 |
| smcginnis | That's fine. I don't think anything with that old account matters to me. | 15:56 |
| fungi | right, we're still trying to work out what the effective approaches could be for resolving the various sorts of conflicts gerrit is unhappy about | 15:56 |
| clarkb | I also think that one method of resolution is going to create a new config state error which means we'll end up two passing things | 15:57 |
| clarkb | (ugh) | 15:57 |
| fungi | smcginnis: but stop letting us distract you from the board meeting | 15:57 |
| dpawlik | fungi, clarkb: hi. Could you check the "fix" for get-pip.py for python2 for Diskimage Builder please? https://review.opendev.org/c/openstack/diskimage-builder/+/772254 | 15:58 |
| clarkb | dpawlik: thinking out loud here: why not use the stable get-pip always? | 16:01 |
| clarkb | in particular I think your fix is incomplete for python 3.5 distros (like xenial) | 16:01 |
| dpawlik | clarkb: I was mostly focusing for Redhat distros. In that case, I can drop change related to the Xenial | 16:03 |
| clarkb | oof I see a transitive conflict now too in the account state | 16:03 |
| clarkb | dpawlik: I don't think you need to drop a change? I'm just thinking out loud here but I believe the version that is python2.7 compat should work everywhere right now (until possibly python 3.9/3.10 starts to show up on distros?) | 16:04 |
| dpawlik | clarkb: the version for 3.4 is working well on all distros, someone comment that on Github in the issue topic | 16:05 |
| dpawlik | clarkb: but still, using 3.4 for 2.7 is a workarround | 16:06 |
| clarkb | dpawlik: right but the version for python2.7 also works for everything I think? | 16:07 |
| clarkb | I'm suggesting that you don't use two different get-pip scripts and instead use the one that works with all pythons | 16:07 |
| clarkb | it simplifies the code and ensures we aren't leaving python3.5 distros behind | 16:07 |
| clarkb | another fun one: a third party CI conflicts with a human beacuse they share an email address :( | 16:08 |
| clarkb | one person has 6 accounts that conflict with each other. several of them are consecutive account ids | 16:09 |
| clarkb | the number of people that asked for our help in resolving these issues is significantly less than the number of people that have run itno these problems | 16:10 |
| dpawlik | clarkb: TBH I really would like to use python 2 get-pip that will work well on py2 and hope that pip community will not raise any problems in the future instead of searching universal solution, that will work everywhere. Here IMHO the pip community should also care to keep one main get-pip script that works on all distro | 16:11 |
| dpawlik | clarkb: I will compare 2.7, 3.4 and the newest one what is the difference that it breaks DIB | 16:12 |
| fungi | clarkb: wrt "why not use the stable get-pip always?" that would result in installing old pip even in python3 based jobs right? or an i misunderstanding the suggestion | 16:13 |
| clarkb | dpawlik: ok, if we want to be more cautious like that then we need to use an appropriate get-pip.py that works with python2.7, 3.5, 3.6, 3.7, and 3.8 due to various distros dib attempts to support depending on the context | 16:14 |
| clarkb | fungi: get-pip.py is a zipped up all in one pip that then installs the latest pip | 16:14 |
| clarkb | fungi: the version of get-pip.py shouldn't matter much as long as it can continue to talk to pypi properly | 16:14 |
| fungi | ahh, i guess "installs latest pip which your interpreter version can support"? | 16:14 |
| clarkb | I'm not even sure if it does that, but maybe | 16:15 |
| clarkb | but it talks to pypi to then do an sintall of what is on pypi. It doesn't do a direct extraction aiui | 16:15 |
| fungi | because the second order problem hiding behind latest get-pip.py not working on python 2.7 is that latest pip also does not work on python 2.7 | 16:15 |
| clarkb | have they not annotated the pypi packages with version support? | 16:17 |
| clarkb | they have so that should solve that problem | 16:17 |
| clarkb | you'll only have this problem if your starting pip is too old to look for those python requires tags. get-pip.py (including the one for 2.7) should be up to date enough to support them | 16:18 |
| *** artom has quit IRC | 16:28 | |
| *** artom has joined #opendev | 16:29 | |
| *** klonn has joined #opendev | 16:34 | |
| *** klonn has quit IRC | 16:34 | |
| fungi | yeah, that's why i suggested that maybe it will correctly work out "latest pip which your interpreter version can support" | 16:35 |
| dpawlik | clarkb: I see in the new comments that also python3.5 can be broken with new version O_O | 16:36 |
| dpawlik | fungi good idea, but I see that pip community does not want to play with that | 16:36 |
| dpawlik | there are at least 5 issues with same topic | 16:36 |
| clarkb | dpawlik: yes, that is why I brought this up :) your change is not complete because it leaves python 3.5 distro releases still broken | 16:37 |
| fungi | dpawlik: right, i'm only guessing the "stable" get-pip.py might get that behavior as a side effect | 16:37 |
| clarkb | I think if you just use the "python 2.7" version then all python will work | 16:37 |
| dpawlik | clarkb: but I'm calling get-pip exactly with python 2 interpreter, but I guess it replace the main binary to pip, which will base on 20.4.3. Changing the order that py2 is installed first, then 3 will raise other issues | 16:39 |
| guillaumec | dpawlik, https://review.opendev.org/c/opendev/system-config/+/772369 I reused part of your commit message :D , though it filers against OS/distro and not python version | 16:39 |
| dpawlik | guillaumec: hmm, so now centos will be broken | 16:40 |
| dpawlik | guillaumec: but I understand that patch set is mostly focused on Ubuntu | 16:41 |
| dpawlik | guillaumec: So it's ok. Additional patchset will be required for centos/fedora/suse | 16:42 |
| clarkb | dpawlik: run get-pip-python2.7.py with python3, I think that will work | 16:42 |
| guillaumec | dpawlik, yes system-config-run-review is using xenial and bionic | 16:42 |
| clarkb | at least until python3.10 releases and is no longer compatiblewith python2 compatible syntax | 16:42 |
| clarkb | guillaumec: and focal (though not much focal yet) | 16:42 |
| guillaumec | which is the job that was failing | 16:42 |
| clarkb | fungi: ~42 accounts have inactive portions in the email conflict list so we can solve those fairly easily I bet. Essentially just retire those accounts like we're retiring the lack of preferred email address external ids | 16:43 |
| clarkb | fungi: but that is < 10 % of the total :( | 16:43 |
| *** mlavalle has quit IRC | 16:44 | |
| dpawlik | clarkb: so I will replace the script to use just 2.7 until it breaks our CI so next patch will be done later :P | 16:44 |
| clarkb | dpawlik: ya I mean there is some risk with doing it the way I suggest. The other option is to use the 2.7 version for 3.5 too | 16:45 |
| *** mlavalle has joined #opendev | 16:45 | |
| clarkb | I just want 3.5 to work, your change doesn't do taht if I read it correctly | 16:45 |
| dpawlik | clarkb: in that case I prefer to use 3.5 get-pip or 3.6 that is also working with 2.7 and maybe it will also work with 3.1X instead of using old one that can raise problems soon | 16:47 |
| clarkb | infra-root: review-test:~clarkb/gerrit-consistency-notes/conflicting_email_user_info I've tried to collect the refs/users/XY/ABXY and all external ids for each account with email conflicts in external ids there. I'm still just trying to get a high level overview feel for what these look like myself, but getting more thoughts on that would likely be good (since this is a bit complicated) | 16:48 |
| clarkb | dpawlik: I think <= 3.5 and >=3.6 is the split pip has made | 16:49 |
| clarkb | so you shouldn't use 3.6 on 3.5 and older | 16:49 |
| clarkb | but ya that probably works too | 16:49 |
| dpawlik | ups,url with 3.6 does not work so 3.5 as you said | 16:50 |
| dpawlik | thanks for advice clarkb++ | 16:50 |
| clarkb | you're welcome | 16:52 |
| openstackgerrit | daniel.pawlik proposed openstack/diskimage-builder master: Install last stable version of get-pip.py script https://review.opendev.org/c/openstack/diskimage-builder/+/772254 | 16:55 |
| zbr | clarkb: for 42 accounts, can you do something like converting email to `original+tmp@example.com`? most domains support parameters. | 16:55 |
| fungi | those 42 are the ones we're sure are safe to just clean up | 16:56 |
| fungi | they're also a tiny fraction of the overall problem space, which is mostly full of far more complicated conflicts | 16:56 |
| fungi | zbr: but yes, if we decide to replace addresses on any accounts rather than removing them, perhaps localpart extensions are a reasonable solution there, good suggestion | 16:57 |
| zbr | at least it should allow them recover their accounts, if needed. | 16:58 |
| fungi | seems more reasonable than replacing with ${RANDOM}@example.org | 16:58 |
| clarkb | I think the problem with that is these emails are associated with an openid | 17:07 |
| clarkb | and I think if the email provided by the openid and the openid external id email in gerrit don't match then gerrit will try to create a new exteranl id that will conflict and the user won't be able to log in | 17:08 |
| clarkb | right now I believe they are able to login because gerrit isn't checking for conflict during the normal login process, only when it htinks it needs to create a new external id | 17:08 |
| clarkb | but yes we could make that change. I just don't believe it will get us to a btter place | 17:09 |
| clarkb | I'm beginning to think that for the other ~600 accounts we should mark them all inactive, see if anyone complains over a period of time, then fix accounts for the complainers and retire all the rest | 17:12 |
| clarkb | the problem is that I don't think we can mechanically apply judgement to how to merge these accounts properly without understanding the user's desired end state | 17:13 |
| clarkb | because users may be pushing code via ssh and reviewing with https. Some of those users may prefer to merge to the ssh side so they don't change their development process and others may prefer to merge to the https side to keep a better review history | 17:13 |
| clarkb | we could apply our own valuation methods to ^ and make a decision for people | 17:14 |
| fungi | well, just making them inactive doesn't help though right? the inactive accounts will still conflict | 17:14 |
| *** marios is now known as marios|out | 17:15 | |
| clarkb | correct, making them inactive is so that people will notice. I'm calling the cleanup "retire" in this case | 17:15 |
| clarkb | retiring is where we delete the external ids and remove the preferred email from the account and set it inactive | 17:15 |
| clarkb | we'll be in the inconsistent state until we fix those who notice and retire the remaining accounts | 17:15 |
| fungi | got it, so just marking them all inactive is the first step | 17:16 |
| clarkb | ya | 17:16 |
| fungi | should we try to reach out to the addresses associated with those accounts first before we take that step? | 17:16 |
| clarkb | we could do that too. | 17:16 |
| clarkb | we also need some form of bookkeeping | 17:16 |
| fungi | just wondering if there is a way for people to find out their accounts are conflicting so they can contact us to resolve the problem before we lock them out | 17:17 |
| clarkb | so maybe the rough plan should be: email the ~600 and ask if they are still using the account and if so to reach out to us. After a week or two set everyone else to inactive. See who complains. Then from that we can do a massive update to external ids that fixes all the issues | 17:18 |
| clarkb | (that way we avoid adowntime) | 17:18 |
| clarkb | when we email people the info we'll want back is the ssh username used to push code and the account id listed in the web ui | 17:20 |
| clarkb | so that we can check if they use a single account or multiple | 17:20 |
| clarkb | donnyd: your account is actually one that fits into the "potentially using two different gerrit accounts one for ssh and one for https" scenario. If you have time I would be curious to know what your ssh username for pushing to gerrit is and what your account id number is at https://review.opendev.org/settings/ feel free to pm them to me | 17:23 |
| clarkb | I don't think that will tell us anything definitive but will help give me more sense of what users are doing | 17:23 |
| *** marios|out has quit IRC | 17:27 | |
| *** rpittau is now known as rpittau|afk | 17:27 | |
| clarkb | fungi: do you know if there is a way to lookup user's last login? | 17:38 |
| clarkb | we could check for most recent code push and review, but that may lag other logins | 17:39 |
| clarkb | (just trying to brainstorm ways of determining activity from these users) | 17:39 |
| fungi | i don't know that gerrit tracks that, other than in the logs we collect | 17:39 |
| clarkb | ya I see registered date in account details | 17:39 |
| fungi | the account fields in the old db were only for creation and update | 17:39 |
| clarkb | oh and ya we could check when they were last updated | 17:40 |
| clarkb | which is likely to significantly lag | 17:40 |
| fungi | we could up our log retention and try to see what's actively in use | 17:40 |
| fungi | get openid auth from apache access logs and ssh logins from gerrit's sshd log | 17:41 |
| fungi | our current retention isn't great for that, so would imply some delay if we wanted a longer sample | 17:41 |
| * fungi looks to see how much we keep now | 17:42 | |
| fungi | a month of apache access logs | 17:43 |
| fungi | and roughly a month of sshd logs as well | 17:43 |
| clarkb | the web ui search helpfully replaces my username and account id searches with email addresses which completely breaks what I'm trying to do :) I should just use the api I guess | 17:43 |
| fungi | so at least as of this moment we could determine which accounts have authenticated via openid, rest api or cli in the past month | 17:44 |
| fungi | after working out how to filter and massage the right loglines/fields out of them | 17:45 |
| fungi | does that seem like it would help? | 17:45 |
| clarkb | what it could help us do is identify people using two different accounts or a single account consistently. We probably still want to reach out to them to help merge things correctly but maybe we'll eb able to give them a better educated guess at what would work for them? | 17:46 |
| clarkb | I was really hoping for a way to say "account xyz has basically never been used/logged in, but account abc is used weekly" | 17:46 |
| clarkb | and we can't make the first bit of that statement from those logs but could do the second part if the data shows up | 17:47 |
| fungi | yeah, we could at least tell which accounts are used ~weekly and which accounts not | 17:47 |
| fungi | another tactic could be to query for footprints they've left behind (uploaded changes/revisions, review comments, label votes...) | 17:48 |
| fungi | generally people don't authenticate and then do nothing | 17:48 |
| fungi | they authenticate and then create something, which will have a timestamp and be associated with the account | 17:49 |
| clarkb | ya I think a query for owner:xyz OR commentby:xyz sort of thing may help us approximate | 17:49 |
| *** zoharm has quit IRC | 17:49 | |
| fungi | if people are performing write operations, some evidence and a timestamp will be left behind. if they're not performing write operations, they generally don't need to authenticate since we don't have private/access-restricted content | 17:50 |
| fungi | owner:xyz and reviewer:xyz most likely | 17:51 |
| fungi | reviewer should show up if they commented or voted on a change, or were added as a requested reviewer | 17:51 |
| fungi | i think pushing a revision for an existing change may also make you a reviewer of that cange | 17:52 |
| fungi | change | 17:52 |
| *** zul has quit IRC | 17:52 | |
| clarkb | ah neat | 17:53 |
| clarkb | there is a built in from: predicate too | 17:54 |
| fungi | so, yeah, given we have a limited set of accounts we need to evaluate, the queries shouldn't take all that long to perform | 17:55 |
| fungi | that seems like the most effective way to gauge whether an account has been used and what it's done, as opposed to just trying to figure out who's authenticated when | 17:56 |
| clarkb | we can also limit:1 them since we only care about the most recent activity | 17:57 |
| *** jpena is now known as jpena|off | 17:57 | |
| fungi | that will sort by most recent? or is that what the from predicate is for? | 17:58 |
| clarkb | all gerrit queries revert sort now by default iirc | 17:59 |
| clarkb | *reverse | 17:59 |
| clarkb | just poking at it a bit I'm reasonably confident that in donnyd's example the smaller account number is the one used. The bigger account number has been usedp reviously but not for a couple of years | 17:59 |
| clarkb | in donnyd's case we might use info like that to suggest we merge the external ids for the higher account number into the lower account number. Then we retire the higher account number by setting it inactive and removing its preferred email | 18:00 |
| fungi | yeah, that seems like an efficient approach | 18:04 |
| clarkb | one issue I'm beginning to realize exists here is that since we can't do htis little by little, we have to do significant planning up front which isn't the worst thing. The problems arise when if we've got somethign wrong now everyone is unhappy rather than one or two people at a time | 18:06 |
| clarkb | but I guess we just need to accept that as reality | 18:06 |
| clarkb | fungi: hrm one thing that makes this type of search more complicated is it sorts by update at the change level | 18:14 |
| clarkb | so you could have an old change that is abanonded recently sort newer | 18:14 |
| fungi | right | 18:16 |
| clarkb | however setting limit > 1 and looking at a couple of accounts it seems that this approximation is still liekyl good enough for many cases | 18:17 |
| fungi | the way i tackle that in election and engagement tools is to query with no limit other than a reference after date, and then iterate over all changes and all comments and revisions for the returned changes looking for the account in question and updating the last seen timestamp if it's newer that what's already in memory | 18:18 |
| fungi | it goes fairly quickly, the most fiddly bit is handling pagination | 18:19 |
| clarkb | I'm doing a small "random" sampling and it seems to largely be "either there is stuff on this account or there really isn't" which makes the approximation work well enough | 18:19 |
| fungi | yeah, certainly if the query for owner:xyz or reviewer:xyz is null, that's easy ;) | 18:20 |
| fungi | clarkb: also i've updated the paragraphs at lines 10 and 12 in https://etherpad.opendev.org/p/tact-sig-2021-rfh integrating your suggestion and overall improving flow | 18:21 |
| clarkb | oh I'll take a look. I need to stop staring at gerrit accuonts for a bit anyway to prep for the meeting | 18:25 |
| clarkb | fungi: that lgtm. We can probably send it out? | 18:27 |
| fungi | mnaser: ^ this bubbled up from your request and discussion in the tc meeting, have you had a chance to see if that addresses what you're interested in seeing? | 18:29 |
| clarkb | fungi: re accounts not logging in unless they do writes, stackalytics is in the list :) | 18:32 |
| clarkb | interestingly, it almost looks like we created the original stackalytics account as a system level account? one of them ahs no openids | 18:33 |
| fungi | clarkb: great point, the major exception to accounts authenticating and not writing is those which need access to stream-events | 18:36 |
| fungi | and even then, most are ci systems which at least leave comments on changes | 18:37 |
| *** eolivare has quit IRC | 18:41 | |
| *** ianw_pto is now known as ianw | 18:59 | |
| *** andrewbonney has quit IRC | 19:16 | |
| *** cloudnull has quit IRC | 19:26 | |
| *** cloudnull has joined #opendev | 19:26 | |
| *** hamalq has joined #opendev | 19:36 | |
| kopecmartin | clarkb: hi, can you retake a look at https://review.opendev.org/c/opendev/system-config/+/705258 ? In patchset 31 it was failing on not finding podman (if i got it right) and now it's failing on installing it (I reused install-podman role), i feel i might have gone a wrong direction | 19:38 |
| clarkb | kopecmartin: ya we dno't use podman, that shouldn't be required | 19:38 |
| clarkb | the log grabbing looks for docker and podman logs but when one is missing it isn't an error | 19:38 |
| kopecmartin | aha .. ok, then you can ignore last two pathsets :D .. i got stuck at this error then https://e091b1f78fc30984e3e8-38748499bf4992b1c907bbe2ddad6f91.ssl.cf2.rackcdn.com/705258/31/check/system-config-run-refstack/59f0b51/job-output.txt | 19:39 |
| clarkb | kopecmartin: that looks to be an error with rsync losing its network connection. i would just try a rerun of the jobs from that patchset | 19:41 |
| kopecmartin | clarkb: ok, i'm gonna revert it back and let's see | 19:42 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 19:43 |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Test/assert Python 3.9 support https://review.opendev.org/c/opendev/git-review/+/772589 | 19:49 |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Add release notes in preparation for next release https://review.opendev.org/c/opendev/git-review/+/772590 | 19:49 |
| openstackgerrit | Merged opendev/system-config master: Install get-pip.py for python3.5/xenial with specific url https://review.opendev.org/c/opendev/system-config/+/772369 | 19:52 |
| *** Alex_Gaynor has left #opendev | 19:52 | |
| fungi | zbr: if 772589 and 772590, look okay i'd like to include them in git-review 2.0.0 before tagging. infra-root: any opinion on whether we should do a 2.0.0.0rc1 before we make 2.0.0? we've done prereleases for the past couple of releases, but no idea if that helped anyone or not... | 20:00 |
| clarkb | fungi: I think the last time we did a release I Just installed it from git and didn't rely on the sdist rc | 20:00 |
| clarkb | fungi: maybe get someone to do that on mac, osx, and windows if possible and call it good? | 20:01 |
| fungi | and before anyone gets scared, the major version bymp is to signal dropping python 2.7 support | 20:01 |
| clarkb | corvus: for your user account changes did you make them pushing an update to account.config in refs/users/us/corvus ? | 20:01 |
| clarkb | corvus: or was it a change you made in the web ui? | 20:02 |
| clarkb | I guess it could've been the rest api too | 20:02 |
| corvus | clarkb: web ui. added new email address. removed original openid email address. logged in again, got new account. | 20:02 |
| corvus | clarkb: was not expecting a new account, because the identifier in openid should be, well, the openid | 20:02 |
| fungi | now that i've done a pass at adding missing release notes for git-review, i'll do the same for bindep hopefully tomorrow or thursday for a new release of it as well | 20:02 |
| fungi | since it's been a while and users have asked for a release | 20:03 |
| clarkb | corvus: oh yup, that is what happened to this other user too | 20:03 |
| corvus | clarkb: i did not change the email address in launchpad | 20:03 |
| clarkb | corvus: ya they started by changing the email address in lp, but then updated the email address to revert it in lp. But by that point the email was no longer ingerrit so it created a new account and moved the openid to it | 20:03 |
| corvus | okay, then yeah, i did only "step 2" of their process of breaking it :) | 20:04 |
| clarkb | step 1 for them was changing the emails in gerrit and lp which produced a new openid which tried to create a new account but failed due to an email address conflict ( an existing account already had the email set on it) | 20:05 |
| clarkb | they then reverted the email changes on the lp side and that created a new account and did the step 2 you ran into | 20:05 |
| clarkb | fun | 20:05 |
| * clarkb finds lunch then back to fix our group inconsistency and do a prod consistency check | 20:06 | |
| clarkb | corvus: fwiw I do think that is a bug in the openid implementation, it seems that an openid shouldn't be able to move like that | 20:06 |
| clarkb | I called that out on my thread upstream but haven't had much input on that aspect of my email | 20:07 |
| zbr | clarkb: i am using git-review on mac daily, installed as editable, works fine. | 20:07 |
| zbr | imho, almost nobody uses pre-releses, those that are interested can also install from master. | 20:09 |
| fungi | looks like i need to get a tox-py39 job added to zuul-jobs | 20:09 |
| fungi | will poke at that real quick | 20:09 |
| zbr | that is based on my experience releasing molecule and ansible-lint, i did pre-release a lot, but not many bothered with them. | 20:09 |
| zbr | yep py39 would be very useful, that is what i use locally but I do not count as a "CI tested" | 20:10 |
| fungi | zbr: looks like you proposed it already, just haven't responded to my review comment from november | 20:11 |
| fungi | zbr: https://review.opendev.org/762192 to be precise | 20:12 |
| zbr | fungi: feel free to edit it. is almost 9pm,... | 20:13 |
| fungi | zbr: will do, thanks! | 20:13 |
| zbr | that is valid for any CR i raise, i don't mind when some does fixes on them. | 20:13 |
| zbr | even of git-review, if there is a minor issue, sometimes I fix it myself intead of adding a comment, especially on old reviews. | 20:14 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Add tox-py39 job https://review.opendev.org/c/zuul/zuul-jobs/+/762192 | 20:16 |
| fungi | zbr: ^ also yep, i agree that's a good policy | 20:16 |
| openstackgerrit | sebastian marcet proposed opendev/system-config master: OpenstackId v3.0.18 https://review.opendev.org/c/opendev/system-config/+/772594 | 20:17 |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Test/assert Python 3.9 support https://review.opendev.org/c/opendev/git-review/+/772589 | 20:17 |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Add release notes in preparation for next release https://review.opendev.org/c/opendev/git-review/+/772590 | 20:17 |
| openstackgerrit | Brian Rosmaita proposed openstack/project-config master: Add rbd-iscsi-client to cinder project https://review.opendev.org/c/openstack/project-config/+/772596 | 20:36 |
| *** ralonsoh has quit IRC | 20:42 | |
| clarkb | I have removed networking-ibm-release from group networking-ibm-release | 20:42 |
| clarkb | I will run the consistency checks in just a few minutes | 20:42 |
| clarkb | against prod I mean. Going to rerun against test first and make sure I've got the command doing what I want | 20:42 |
| *** brinzhang_ has joined #opendev | 20:44 | |
| *** brinzhang has quit IRC | 20:47 | |
| clarkb | prod consistency check is running now. test took 2 minutes and 37 seconds | 20:50 |
| clarkb | prod is taking longer (almost up to 4 minutes) | 20:54 |
| *** dtantsur is now known as dtantsur|afk | 20:55 | |
| clarkb | its done, now just cleaning up the results (I need to pass it through python -m json.tool after removing the "header") | 20:57 |
| clarkb | and I should be able to compare between prod and -test | 20:57 |
| clarkb | groups show clean on prod now too | 20:57 |
| clarkb | so networking-ibm-release was the only sad grop after johnsom fixed octavia-core's loop | 20:57 |
| clarkb | there is only one new inconsistency between prod and test | 21:01 |
| clarkb | which is good news because it means we can continue to use test predominantly for figuring things out | 21:01 |
| clarkb | (and none have disappeared | 21:01 |
| clarkb | I've copied both the old review-test consistency results and the current prod ones onto review-test so others can see that delta | 21:03 |
| openstackgerrit | Merged opendev/system-config master: OpenstackId v3.0.18 https://review.opendev.org/c/opendev/system-config/+/772594 | 21:06 |
| clarkb | kopecmartin: I'm looking at your latest failure now | 21:07 |
| clarkb | kopecmartin: the issue appears related to letsencrypt self signed cert stand in provisioning. I'll try to figure it out and leave you a comment | 21:08 |
| *** hamalq has quit IRC | 21:21 | |
| *** hamalq has joined #opendev | 21:21 | |
| clarkb | kopecmartin: I left a comment but I thik I'll also push up a patch to reduce the rtt | 21:24 |
| clarkb | hrm I Think I just realized what the underlying issue is, its wanting to do an opendev.org LE cert | 21:27 |
| clarkb | er openstack.org LE cert | 21:27 |
| clarkb | I don't think we want a refstack.opendev.org hostname anywhere since refstack is a very openstack specific application | 21:28 |
| clarkb | infra-root ^ does that make sense? if so I wonder how we should do the testing, maybe just use snakeoil certs rather than le at all? | 21:28 |
| clarkb | then for prod we'll point at some purchased cert? | 21:28 |
| clarkb | or are we doing openstack.org LE certs somehow? | 21:29 |
| clarkb | oh wait we delegated the domain from openstack.org to opendev.org? is that right? | 21:29 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 21:30 |
| clarkb | kopecmartin: ^ ok, let's see how that does | 21:31 |
| fungi | clarkb: yeah, we le openstack.org certs using cnames in the openstack.org domain to the acme.opendev.org | 21:32 |
| clarkb | cool then I think the latest patchset is correct for le things | 21:35 |
| ianw | clarkb: we can do openstack.org certs, just have to make sure we have the _acme-challenge pointing to acme.opendev.org | 21:44 |
| ianw | for testing though, that shouldn't matter? | 21:44 |
| clarkb | ianw: ya for testing it shouldn't matter. I was just thinking that maybe we shouldn't use LE there at all for half a second | 21:46 |
| clarkb | but its fine we do the delegation thing and ya would need to update dns first | 21:46 |
| clarkb | ianw: also not sure if you've seen https://review.opendev.org/c/openstack/diskimage-builder/+/772254 to deal with get-pip.py problems in dib now that get-pip.py is >= python3.6 unless you do some gymnastics | 22:03 |
| *** sboyron has quit IRC | 22:08 | |
| *** slaweq has quit IRC | 22:08 | |
| ianw | sigh, no, i'll pull it up | 22:24 |
| fungi | diablo_rojo_phon: once you've got consensus on the desired end state for the foundation irc channels, let's sync up and draft a maintenance plan in an etherpad with the steps we'll need to take and the timeline, since there's some initial setup work which needs to be performed for the new channels, but also our usual process for forwarding channels involves waiting days between some steps, so it'll be easier | 22:26 |
| fungi | to work into our schedules if we have a timeline | 22:26 |
| fungi | and there's cleanup to do later as well | 22:27 |
| kopecmartin | clarkb: thank you for that patch, you made it work! the only thing left is a test of a submission, it's failing, i'm gonna edit that | 22:51 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 22:51 |
| clarkb | kopecmartin: that is great news, thank you for helping with that | 22:51 |
| diablo_rojo_phon | Sounds good fungi. I figure we give the thread a couple days to languish and then move forward. | 23:06 |
| *** tosky has quit IRC | 23:18 | |
| fungi | awesome, great plan | 23:21 |
| clarkb | kopecmartin: left a comment on the latest patchset (I think we may want the anonymous upload thing to be test only), otherwise if that passes testing I think we're looking good | 23:35 |
| kopecmartin | clarkb: that's true, is there a specific var set for distinguishing zuul/testing run? | 23:38 |
| clarkb | kopecmartin: no, instead what you can do is make a testing specific host_vars file that is only included in testing then set the var to enable anonymous users there | 23:39 |
| clarkb | let me find some links | 23:39 |
| clarkb | it acts like a variable mix in | 23:39 |
| kopecmartin | oh, i see, yeah, that's a cleaner way to do it | 23:39 |
| clarkb | kopecmartin: https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/templates/host_vars you put the file there then add it to https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/run-base.yaml#L74 | 23:40 |
| kopecmartin | clarkb: ok, thanks .. let's see now | 23:53 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 23:53 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!