Tuesday, 2021-01-26

*** DSpider has quit IRC00:00
clarkbfungi: looking at the wheel cache stuff we run release-wheel-cache after all the other cache builds run but we skip it if any fail00:02
clarkbfungi: I think that between inap ip conflicts and linaro arm64 boot issues we haven't had a recent situation where all of them run together happily00:03
clarkbmaybe we can cehck it tomorrow and see if arm is any happier and work from there?00:04
*** brinzhang has quit IRC00:05
fungisure thing00:11
*** ysandeep|away is now known as ysandeep00:17
*** brinzhang has joined #opendev00:17
kevinzfungi: Alex_Gaynor: Node failuer again?00:50
fungikevinz: yeah, it looks like nodepool is seeing timeouts waiting for booted instances to become active or reachable00:52
fungiit eventually gives up waiting and tries a couple more times before zuul reports a node_failure result00:53
*** mlavalle has quit IRC01:04
kevinzfungi: looks it is still the rabbitmq error block the cluster.  I will figure out why it always happens01:10
fungithanks kevinz!01:10
kevinzfungi: np, will let you know when it ready01:11
kevinzfungi: network issue caused Rabbitmq break into several partition, so block the cluster. Now recovered02:02
kevinzI will also write a script to monitor it02:02
fungithanks again for taking care of it kevinz!02:13
*** brinzhang has quit IRC05:20
*** sgw has quit IRC05:21
*** sgw has joined #opendev05:21
*** brinzhang has joined #opendev05:21
*** marios has joined #opendev06:24
*** spotz has quit IRC06:54
*** redrobot has quit IRC06:56
*** redrobot has joined #opendev06:56
*** eolivare has joined #opendev07:37
*** ralonsoh has joined #opendev07:52
*** zoharm has joined #opendev07:55
*** rpittau|afk_ is now known as rpittau07:57
*** slaweq has joined #opendev07:59
*** sboyron has joined #opendev08:05
*** andrewbonney has joined #opendev08:19
*** klonn has joined #opendev08:27
*** klonn has quit IRC08:29
*** tosky has joined #opendev08:45
*** jpena|off is now known as jpena08:57
*** DSpider has joined #opendev09:00
openstackgerritRiccardo Pittau proposed openstack/project-config master: Add key editHashtags to normalize_acl script  https://review.opendev.org/c/openstack/project-config/+/77248409:25
openstackgerritMartin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker  https://review.opendev.org/c/opendev/system-config/+/70525810:12
*** brinzhang has quit IRC10:14
*** dtantsur|afk is now known as dtantsur10:18
priteauHello. We have kayobe stable/train jobs often failing on fetch data from one of the mirrors: Timeout on https://mirror-int.iad.rax.opendev.org/epel/7/x86_64/repodata/repomd.xml10:38
priteauIs there a known issue with it?10:38
fricklerpriteau: do you have a sample job log? no known issue as far as I can tell10:44
priteauAt the end of https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_838/771246/1/check/kayobe-seed-upgrade-centos7/8383419/primary/ansible/seed-deploy-pre-upgrade10:46
priteauBut if you think it works fine, it could be some issues with our test environment10:46
priteauNevermind, this is probably a test environment issue10:54
*** brinzhang has joined #opendev11:04
*** rpittau is now known as rpittau|bbl11:09
openstackgerritMartin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker  https://review.opendev.org/c/opendev/system-config/+/70525811:15
*** Eighth_Doctor has quit IRC11:19
*** mordred has quit IRC11:19
*** Eighth_Doctor has joined #opendev11:30
openstackgerritAleksey proposed openstack/diskimage-builder master: Extend ubuntu-minimal element  https://review.opendev.org/c/openstack/diskimage-builder/+/77250211:32
*** mordred has joined #opendev11:48
openstackgerritAleksey proposed openstack/diskimage-builder master: Extend ubuntu-minimal element  https://review.opendev.org/c/openstack/diskimage-builder/+/77250211:48
*** klonn has joined #opendev12:05
noonedeadpunkhi! Sorry, I totally missed that a while back - but can you kindly remind me if there are any plans to renew integration of gerrit with launchpad?12:16
*** eolivare has quit IRC12:16
*** klonn has quit IRC12:22
*** jpena is now known as jpena|lunch12:25
*** rpittau|bbl is now known as rpittau12:59
fricklernoonedeadpunk: afaict the "plan" is to wait for someone to re-implement it, haven't seen anything more specific yet13:04
*** jpena|lunch is now known as jpena13:17
*** eolivare has joined #opendev13:22
*** hashar has joined #opendev13:27
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add zuul-storage-proxy in zuul namespace  https://review.opendev.org/c/openstack/project-config/+/77236413:54
*** mlavalle has joined #opendev13:57
openstackgerritMerged opendev/system-config master: OpenstackId v3.0.17  https://review.opendev.org/c/opendev/system-config/+/77234814:08
*** sshnaidm|ruck is now known as sshnaidm|afk14:20
*** stand has quit IRC14:24
*** brinzhang has quit IRC14:25
*** brinzhang has joined #opendev14:25
*** stand has joined #opendev14:26
*** stand has quit IRC14:43
*** brinzhang has quit IRC14:48
*** sgw has left #opendev14:48
*** brinzhang has joined #opendev14:51
*** brinzhang has quit IRC14:51
*** brinzhang has joined #opendev14:52
clarkbnoonedeadpunk: fungi has indicated a desire to work on it, but unfortunately higher priority issues keep cropping up14:57
noonedeadpunkok, thanks for providing info)14:57
*** ralonsoh has quit IRC15:04
*** ralonsoh has joined #opendev15:04
*** sshnaidm|afk is now known as sshnaidm|ruck15:07
*** hashar has quit IRC15:10
*** fressi has quit IRC15:19
openstackgerritAleksey proposed openstack/diskimage-builder master: Extend ubuntu-minimal element  https://review.opendev.org/c/openstack/diskimage-builder/+/77250215:41
funginoonedeadpunk: clarkb: yeah, help welcome, i'm starting to think it may be better if we look at doing it with a nodeless zuul job instead of fixing the hook script in jeepyb15:42
fungimainly because fixing what's in jeepyb will be a not-insignificant rewrite to swap out database queries with rest api calls15:43
fungiand that in turn means more credentials to manage15:43
clarkbya I think that would work. The odwnside to it is a busy zuul may not get to updates until later (but maybe a supercedent queue where it does as much as possible when it runs makes sense?)15:44
clarkbfungi: fwiw I've started on a script on review-test to dump a bunch of relevant data for the 642 conflicting emails in external ids accounts15:44
clarkbtreating account ids as strings is great until you hit prefix collisions :)15:45
fungithe gerrit hook script wasn't robust anyway, so often failed to comment for a number of reasons and the only logs were locally on the gerrit server15:48
fungiwhereas if it's in zuul, hopefully we can safely make it so that folks can see errors (so long as we make sure the build log won't leak gerrit or launchpad credentials)15:49
clarkbat first glance the output data is telling me "this is not going to be fun"15:50
clarkbthere are a small number of inactive accounts where we might be able to brute force a cleanup, but many of these accounts have different openids and different ssh usernames implying that both or either could be in use15:50
clarkbthe output is still running through (and not terribly quickly as its doing a fair bit of disk io scanning info15:51
clarkboh hey I see a smcginnis :)15:51
clarkblol stackalytics is in that list too15:52
clarkbsmcginnis: we might bug you later to udnerstand how to best untangle this gerrit account situation using you as an example15:52
clarkb(for now I'm still trying to collect the data we need to begin to understand anything)15:52
smcginnisI haven't been following the scrollback, but just let me know if you need me to do anything on this side.15:52
clarkbsmcginnis: ya ignore me for now, you can pay attention to the board meeting. I'll reach out if we need your input later15:53
smcginnisI have a reviewerstats script that runs every four hours that pulls some data via SSH. That should be the only thing coming from me.15:53
smcginnisLooks like the code does at least attempt to close the connection - https://opendev.org/openstack/reviewstats/src/branch/master/reviewstats/utils.py#L30115:55
fungismcginnis: but the summary is that gerrit now strongly wants all email addresses to be uniquely associated with a single account (can't even conflict with an inactive account), and that's blocking us from making further changes to gerrit accounts15:55
smcginnisAh, because I had two accounts at some point?15:55
clarkbya its not about activity, its about account config state15:55
clarkbsmcginnis: yes15:55
fungiso, like, we may need to delete addresses from your old inactive account or something15:56
smcginnisIf y'all have the ability, you could update the email associated with that old inactive account.15:56
smcginnisThat's fine. I don't think anything with that old account matters to me.15:56
fungiright, we're still trying to work out what the effective approaches could be for resolving the various sorts of conflicts gerrit is unhappy about15:56
clarkbI also think that one method of resolution is going to create a new config state error which means we'll end up two passing things15:57
fungismcginnis: but stop letting us distract you from the board meeting15:57
dpawlikfungi, clarkb: hi. Could you check the "fix" for get-pip.py for python2 for Diskimage Builder please? https://review.opendev.org/c/openstack/diskimage-builder/+/77225415:58
clarkbdpawlik: thinking out loud here: why not use the stable get-pip always?16:01
clarkbin particular I think your fix is incomplete for python 3.5 distros (like xenial)16:01
dpawlikclarkb: I was mostly focusing for Redhat distros. In that case, I can drop change related to the Xenial16:03
clarkboof I see a transitive conflict now too in the account state16:03
clarkbdpawlik: I don't think you need to drop a change? I'm just thinking out loud here but I believe the version that is python2.7 compat should work everywhere right now (until possibly python 3.9/3.10 starts to show up on distros?)16:04
dpawlikclarkb: the version for 3.4 is working well on all distros, someone comment that on Github in the issue topic16:05
dpawlikclarkb: but still, using 3.4 for 2.7 is a workarround16:06
clarkbdpawlik: right but the version for python2.7 also works for everything I think?16:07
clarkbI'm suggesting that you don't use two different get-pip scripts and instead use the one that works with all pythons16:07
clarkbit simplifies the code and ensures we aren't leaving python3.5 distros behind16:07
clarkbanother fun one: a third party CI conflicts with a human beacuse they share an email address :(16:08
clarkbone person has 6 accounts that conflict with each other. several of them are consecutive account ids16:09
clarkbthe number of people that asked for our help in resolving these issues is significantly less than the number of people that have run itno these problems16:10
dpawlikclarkb: TBH I really would like to use python 2 get-pip that will work well on py2 and hope that pip community will not raise any problems in the future instead of searching universal solution, that will work everywhere. Here IMHO the pip community should also care to keep one main get-pip script that works on all distro16:11
dpawlikclarkb: I will compare 2.7, 3.4 and the newest one what is the difference that it breaks DIB16:12
fungiclarkb: wrt "why not use the stable get-pip always?" that would result in installing old pip even in python3 based jobs right? or an i misunderstanding the suggestion16:13
clarkbdpawlik: ok, if we want to be more cautious like that then we need to use an appropriate get-pip.py that works with python2.7, 3.5, 3.6, 3.7, and 3.8  due to various distros dib attempts to support depending on the context16:14
clarkbfungi: get-pip.py is a zipped up all in one pip that then installs the latest pip16:14
clarkbfungi: the version of get-pip.py shouldn't matter much as long as it can continue to talk to pypi properly16:14
fungiahh, i guess "installs latest pip which your interpreter version can support"?16:14
clarkbI'm not even sure if it does that, but maybe16:15
clarkbbut it talks to pypi to then do an sintall of what is on pypi. It doesn't do a direct extraction aiui16:15
fungibecause the second order problem hiding behind latest get-pip.py not working on python 2.7 is that latest pip also does not work on python 2.716:15
clarkbhave they not annotated the pypi packages with version support?16:17
clarkbthey have so that should solve that problem16:17
clarkbyou'll only have this problem if your starting pip is too old to look for those python requires tags. get-pip.py (including the one for 2.7) should be up to date enough to support them16:18
*** artom has quit IRC16:28
*** artom has joined #opendev16:29
*** klonn has joined #opendev16:34
*** klonn has quit IRC16:34
fungiyeah, that's why i suggested that maybe it will correctly work out "latest pip which your interpreter version can support"16:35
dpawlikclarkb: I see in the new comments that also python3.5 can be broken with new version O_O16:36
dpawlikfungi good idea, but I see that pip community does not want to play with that16:36
dpawlikthere are at least 5 issues with same topic16:36
clarkbdpawlik: yes, that is why I brought this up :) your change is not complete because it leaves python 3.5 distro releases still broken16:37
fungidpawlik: right, i'm only guessing the "stable" get-pip.py might get that behavior as a side effect16:37
clarkbI think if you just use the "python 2.7" version then all python will work16:37
dpawlikclarkb: but I'm calling get-pip exactly with python 2 interpreter, but I guess it replace the main binary to pip, which will base on 20.4.3. Changing the order that py2 is installed first, then 3 will raise other issues16:39
guillaumecdpawlik, https://review.opendev.org/c/opendev/system-config/+/772369 I reused part of your commit message :D   , though it filers against OS/distro and not python version16:39
dpawlikguillaumec: hmm, so now centos will be broken16:40
dpawlikguillaumec: but I understand that patch set is mostly focused on Ubuntu16:41
dpawlikguillaumec: So it's ok. Additional patchset will be required for centos/fedora/suse16:42
clarkbdpawlik: run get-pip-python2.7.py with python3, I think that will work16:42
guillaumecdpawlik, yes system-config-run-review is using xenial and bionic16:42
clarkbat least until python3.10 releases and is no longer compatiblewith python2 compatible syntax16:42
clarkbguillaumec: and focal (though not much focal yet)16:42
guillaumecwhich is the job that was failing16:42
clarkbfungi: ~42 accounts have inactive portions in the email conflict list so we can solve those fairly easily I bet. Essentially just retire those accounts like we're retiring the lack of preferred email address external ids16:43
clarkbfungi: but that is < 10 % of the total :(16:43
*** mlavalle has quit IRC16:44
dpawlikclarkb: so I will replace the script to use just 2.7 until it breaks our CI so next patch will be done later :P16:44
clarkbdpawlik: ya I mean there is some risk with doing it the way I suggest. The other option is to use the 2.7 version for 3.5 too16:45
*** mlavalle has joined #opendev16:45
clarkbI just want 3.5 to work, your change doesn't do taht if I read it correctly16:45
dpawlikclarkb: in that case I prefer to use 3.5 get-pip or 3.6 that is also working with 2.7 and maybe it will also work with 3.1X instead of using old one that can raise problems soon16:47
clarkbinfra-root: review-test:~clarkb/gerrit-consistency-notes/conflicting_email_user_info I've tried to collect the refs/users/XY/ABXY and all external ids for each account with email conflicts in external ids there. I'm still just trying to get a high level overview feel for what these look like myself, but getting more thoughts on that would likely be good (since this is a bit complicated)16:48
clarkbdpawlik: I think <= 3.5 and >=3.6 is the split pip has made16:49
clarkbso you shouldn't use 3.6 on 3.5 and older16:49
clarkbbut ya that probably works too16:49
dpawlikups,url with  3.6 does not work  so 3.5 as you said16:50
dpawlikthanks for advice clarkb++16:50
clarkbyou're welcome16:52
openstackgerritdaniel.pawlik proposed openstack/diskimage-builder master: Install last stable version of get-pip.py script  https://review.opendev.org/c/openstack/diskimage-builder/+/77225416:55
zbrclarkb: for 42 accounts, can you do something like converting email to `original+tmp@example.com`? most domains support parameters.16:55
fungithose 42 are the ones we're sure are safe to just clean up16:56
fungithey're also a tiny fraction of the overall problem space, which is mostly full of far more complicated conflicts16:56
fungizbr: but yes, if we decide to replace addresses on any accounts rather than removing them, perhaps localpart extensions are a reasonable solution there, good suggestion16:57
zbrat least it should allow them recover their accounts, if needed.16:58
fungiseems more reasonable than replacing with ${RANDOM}@example.org16:58
clarkbI think the problem with that is these emails are associated with an openid17:07
clarkband I think if the email provided by the openid and the openid external id email in gerrit don't match then gerrit will try to create a new exteranl id that will conflict and the user won't be able to log in17:08
clarkbright now I believe they are able to login because gerrit isn't checking for conflict during the normal login process, only when it htinks it needs to create a new external id17:08
clarkbbut yes we could make that change. I just don't believe it will get us to a btter place17:09
clarkbI'm beginning to think that for the other ~600 accounts we should mark them all inactive, see if anyone complains over a period of time, then fix accounts for the complainers and retire all the rest17:12
clarkbthe problem is that I don't think we can mechanically apply judgement to how to merge these accounts properly without understanding the user's desired end state17:13
clarkbbecause users may be pushing code via ssh and reviewing with https. Some of those users may prefer to merge to the ssh side so they don't change their development process and others may prefer to merge to the https side to keep a better review history17:13
clarkbwe could apply our own valuation methods to ^ and make a decision for people17:14
fungiwell, just making them inactive doesn't help though right? the inactive accounts will still conflict17:14
*** marios is now known as marios|out17:15
clarkbcorrect, making them inactive is so that people will notice. I'm calling the cleanup "retire" in this case17:15
clarkbretiring is where we delete the external ids and remove the preferred email from the account and set it inactive17:15
clarkbwe'll be in the inconsistent state until we fix those who notice and retire the remaining accounts17:15
fungigot it, so just marking them all inactive is the first step17:16
fungishould we try to reach out to the addresses associated with those accounts first before we take that step?17:16
clarkbwe could do that too.17:16
clarkbwe also need some form of bookkeeping17:16
fungijust wondering if there is a way for people to find out their accounts are conflicting so they can contact us to resolve the problem before we lock them out17:17
clarkbso maybe the rough plan should be: email the ~600 and ask if they are still using the account and if so to reach out to us. After a week or two set everyone else to inactive. See who complains. Then from that we can do a massive update to external ids that fixes all the issues17:18
clarkb(that way we avoid adowntime)17:18
clarkbwhen we email people the info we'll want back is the ssh username used to push code and the account id listed in the web ui17:20
clarkbso that we can check if they use a single account or multiple17:20
clarkbdonnyd: your account is actually one that fits into the "potentially using two different gerrit accounts one for ssh and one for https" scenario. If you have time I would be curious to know what your ssh username for pushing to gerrit is and what your account id number is at https://review.opendev.org/settings/ feel free to pm them to me17:23
clarkbI don't think that will tell us anything definitive but will help give me more sense of what users are doing17:23
*** marios|out has quit IRC17:27
*** rpittau is now known as rpittau|afk17:27
clarkbfungi: do you know if there is a way to lookup user's last login?17:38
clarkbwe could check for most recent code push and review, but that may lag other logins17:39
clarkb(just trying to brainstorm ways of determining activity from these users)17:39
fungii don't know that gerrit tracks that, other than in the logs we collect17:39
clarkbya I see registered date in account details17:39
fungithe account fields in the old db were only for creation and update17:39
clarkboh and ya we could check when they were last updated17:40
clarkbwhich is likely to significantly lag17:40
fungiwe could up our log retention and try to see what's actively in use17:40
fungiget openid auth from apache access logs and ssh logins from gerrit's sshd log17:41
fungiour current retention isn't great for that, so would imply some delay if we wanted a longer sample17:41
* fungi looks to see how much we keep now17:42
fungia month of apache access logs17:43
fungiand roughly a month of sshd logs as well17:43
clarkbthe web ui search helpfully replaces my username and account id searches with email addresses which completely breaks what I'm trying to do :) I should just use the api I guess17:43
fungiso at least as of this moment we could determine which accounts have authenticated via openid, rest api or cli in the past month17:44
fungiafter working out how to filter and massage the right loglines/fields out of them17:45
fungidoes that seem like it would help?17:45
clarkbwhat it could help us do is identify people using two different accounts or a single account consistently. We probably still want to reach out to them to help merge things correctly but maybe we'll eb able to give them a better educated guess at what would work for them?17:46
clarkbI was really hoping for a way to say "account xyz has basically never been used/logged in, but account abc is used weekly"17:46
clarkband we can't make the first bit of that statement from those logs but could do the second part if the data shows up17:47
fungiyeah, we could at least tell which accounts are used ~weekly and which accounts not17:47
fungianother tactic could be to query for footprints they've left behind (uploaded changes/revisions, review comments, label votes...)17:48
fungigenerally people don't authenticate and then do nothing17:48
fungithey authenticate and then create something, which will have a timestamp and be associated with the account17:49
clarkbya I think a query for owner:xyz OR commentby:xyz sort of thing may help us approximate17:49
*** zoharm has quit IRC17:49
fungiif people are performing write operations, some evidence and a timestamp will be left behind. if they're not performing write operations, they generally don't need to authenticate since we don't have private/access-restricted content17:50
fungiowner:xyz and reviewer:xyz most likely17:51
fungireviewer should show up if they commented or voted on a change, or were added as a requested reviewer17:51
fungii think pushing a revision for an existing change may also make you a reviewer of that cange17:52
*** zul has quit IRC17:52
clarkbah neat17:53
clarkbthere is a built in from: predicate too17:54
fungiso, yeah, given we have a limited set of accounts we need to evaluate, the queries shouldn't take all that long to perform17:55
fungithat seems like the most effective way to gauge whether an account has been used and what it's done, as opposed to just trying to figure out who's authenticated when17:56
clarkbwe can also limit:1 them since we only care about the most recent activity17:57
*** jpena is now known as jpena|off17:57
fungithat will sort by most recent? or is that what the from predicate is for?17:58
clarkball gerrit queries revert sort now by default iirc17:59
clarkbjust poking at it a bit I'm reasonably confident that in donnyd's example the smaller account number is the one used. The bigger account number has been usedp reviously but not for a couple of years17:59
clarkbin donnyd's case we might use info like that to suggest we merge the external ids for the higher account number into the lower account number. Then we retire the higher account number by setting it inactive and removing its preferred email18:00
fungiyeah, that seems like an efficient approach18:04
clarkbone issue I'm beginning to realize exists here is that since we can't do htis little by little, we have to do significant planning up front which isn't the worst thing. The problems arise when if we've got somethign wrong now everyone is unhappy rather than one or two people at a time18:06
clarkbbut I guess we just need to accept that as reality18:06
clarkbfungi: hrm one thing that makes this type of search more complicated is it sorts by update at the change level18:14
clarkbso you could have an old change that is abanonded recently sort newer18:14
clarkbhowever setting limit > 1 and looking at a couple of accounts it seems that this approximation is still liekyl good enough for many cases18:17
fungithe way i tackle that in election and engagement tools is to query with no limit other than a reference after date, and then iterate over all changes and all comments and revisions for the returned changes looking for the account in question and updating the last seen timestamp if it's newer that what's already in memory18:18
fungiit goes fairly quickly, the most fiddly bit is handling pagination18:19
clarkbI'm doing a small "random" sampling and it seems to largely be "either there is stuff on this account or there really isn't" which makes the approximation work well enough18:19
fungiyeah, certainly if the query for owner:xyz or reviewer:xyz is null, that's easy ;)18:20
fungiclarkb: also i've updated the paragraphs at lines 10 and 12 in https://etherpad.opendev.org/p/tact-sig-2021-rfh integrating your suggestion and overall improving flow18:21
clarkboh I'll take a look. I need to stop staring at gerrit accuonts for a bit anyway to prep for the meeting18:25
clarkbfungi: that lgtm. We can probably send it out?18:27
fungimnaser: ^ this bubbled up from your request and discussion in the tc meeting, have you had a chance to see if that addresses what you're interested in seeing?18:29
clarkbfungi: re accounts not logging in unless they do writes, stackalytics is in the list :)18:32
clarkbinterestingly, it almost looks like we created the original stackalytics account as a system level account? one of them ahs no openids18:33
fungiclarkb: great point, the major exception to accounts authenticating and not writing is those which need access to stream-events18:36
fungiand even then, most are ci systems which at least leave comments on changes18:37
*** eolivare has quit IRC18:41
*** ianw_pto is now known as ianw18:59
*** andrewbonney has quit IRC19:16
*** cloudnull has quit IRC19:26
*** cloudnull has joined #opendev19:26
*** hamalq has joined #opendev19:36
kopecmartinclarkb: hi, can you retake a look at https://review.opendev.org/c/opendev/system-config/+/705258 ? In patchset 31 it was failing on not finding podman (if i got it right) and now it's failing on installing it (I reused install-podman role), i feel i might have gone a wrong direction19:38
clarkbkopecmartin: ya we dno't use podman, that shouldn't be required19:38
clarkbthe log grabbing looks for docker and podman logs but when one is missing it isn't an error19:38
kopecmartinaha .. ok, then you can ignore last two pathsets :D .. i got stuck at this error then https://e091b1f78fc30984e3e8-38748499bf4992b1c907bbe2ddad6f91.ssl.cf2.rackcdn.com/705258/31/check/system-config-run-refstack/59f0b51/job-output.txt19:39
clarkbkopecmartin: that looks to be an error with rsync losing its network connection. i would just try a rerun of the jobs from that patchset19:41
kopecmartinclarkb: ok, i'm gonna revert it back and let's see19:42
openstackgerritMartin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker  https://review.opendev.org/c/opendev/system-config/+/70525819:43
openstackgerritJeremy Stanley proposed opendev/git-review master: Test/assert Python 3.9 support  https://review.opendev.org/c/opendev/git-review/+/77258919:49
openstackgerritJeremy Stanley proposed opendev/git-review master: Add release notes in preparation for next release  https://review.opendev.org/c/opendev/git-review/+/77259019:49
openstackgerritMerged opendev/system-config master: Install get-pip.py for python3.5/xenial with specific url  https://review.opendev.org/c/opendev/system-config/+/77236919:52
*** Alex_Gaynor has left #opendev19:52
fungizbr: if 772589 and 772590, look okay i'd like to include them in git-review 2.0.0 before tagging. infra-root: any opinion on whether we should do a before we make 2.0.0? we've done prereleases for the past couple of releases, but no idea if that helped anyone or not...20:00
clarkbfungi: I think the last time we did a release I Just installed it from git and didn't rely on the sdist rc20:00
clarkbfungi: maybe get someone to do that on mac, osx, and windows if possible and call it good?20:01
fungiand before anyone gets scared, the major version bymp is to signal dropping python 2.7 support20:01
clarkbcorvus: for your user account changes did you make them pushing an update to account.config in refs/users/us/corvus ?20:01
clarkbcorvus: or was it a change you made in the web ui?20:02
clarkbI guess it could've been the rest api too20:02
corvusclarkb: web ui.  added new email address.  removed original openid email address.  logged in again, got new account.20:02
corvusclarkb: was not expecting a new account, because the identifier in openid should be, well, the openid20:02
funginow that i've done a pass at adding missing release notes for git-review, i'll do the same for bindep hopefully tomorrow or thursday for a new release of it as well20:02
fungisince it's been a while and users have asked for a release20:03
clarkbcorvus: oh yup, that is what happened to this other user too20:03
corvusclarkb: i did not change the email address in launchpad20:03
clarkbcorvus: ya they started by changing the email address in lp, but then updated the email address to revert it in lp. But by that point the email was no longer ingerrit so it created a new account and moved the openid to it20:03
corvusokay, then yeah, i did only "step 2" of their process of breaking it :)20:04
clarkbstep 1 for them was changing the emails in gerrit and lp which produced a new openid which tried to create a new account but failed due to an email address conflict ( an existing account already had the email set on it)20:05
clarkbthey then reverted the email changes on the lp side and that created a new account and did the step 2 you ran into20:05
* clarkb finds lunch then back to fix our group inconsistency and do a prod consistency check20:06
clarkbcorvus: fwiw I do think that is a bug in the openid implementation, it seems that an openid shouldn't be able to move like that20:06
clarkbI called that out on my thread upstream but haven't had much input on that aspect of my email20:07
zbrclarkb: i am using git-review on mac daily, installed as editable, works fine.20:07
zbrimho, almost nobody uses pre-releses, those that are interested can also install from master.20:09
fungilooks like i need to get a tox-py39 job added to zuul-jobs20:09
fungiwill poke at that real quick20:09
zbrthat is based on my experience releasing molecule and ansible-lint, i did pre-release a lot, but not many bothered with them.20:09
zbryep py39 would be very useful, that is what i use locally but I do not count as a "CI tested"20:10
fungizbr: looks like you proposed it already, just haven't responded to my review comment from november20:11
fungizbr: https://review.opendev.org/762192 to be precise20:12
zbrfungi: feel free to edit it. is almost 9pm,...20:13
fungizbr: will do, thanks!20:13
zbrthat is valid for any CR i raise, i don't mind when some does fixes on them.20:13
zbreven of git-review, if there is a minor issue, sometimes I fix it myself intead of adding a comment, especially on old reviews.20:14
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: Add tox-py39 job  https://review.opendev.org/c/zuul/zuul-jobs/+/76219220:16
fungizbr: ^ also yep, i agree that's a good policy20:16
openstackgerritsebastian marcet proposed opendev/system-config master: OpenstackId v3.0.18  https://review.opendev.org/c/opendev/system-config/+/77259420:17
openstackgerritJeremy Stanley proposed opendev/git-review master: Test/assert Python 3.9 support  https://review.opendev.org/c/opendev/git-review/+/77258920:17
openstackgerritJeremy Stanley proposed opendev/git-review master: Add release notes in preparation for next release  https://review.opendev.org/c/opendev/git-review/+/77259020:17
openstackgerritBrian Rosmaita proposed openstack/project-config master: Add rbd-iscsi-client to cinder project  https://review.opendev.org/c/openstack/project-config/+/77259620:36
*** ralonsoh has quit IRC20:42
clarkbI have removed networking-ibm-release from group networking-ibm-release20:42
clarkbI will run the consistency checks in just a few minutes20:42
clarkbagainst prod I mean. Going to rerun against test first and make sure I've got the command doing what I want20:42
*** brinzhang_ has joined #opendev20:44
*** brinzhang has quit IRC20:47
clarkbprod consistency check is running now. test took 2 minutes and 37 seconds20:50
clarkbprod is taking longer (almost up to 4 minutes)20:54
*** dtantsur is now known as dtantsur|afk20:55
clarkbits done, now just cleaning up the results (I need to pass it through python -m json.tool after removing the "header")20:57
clarkband I should be able to compare between prod and -test20:57
clarkbgroups show clean on prod now too20:57
clarkbso networking-ibm-release was the only sad grop after johnsom fixed octavia-core's loop20:57
clarkbthere is only one new inconsistency between prod and test21:01
clarkbwhich is good news because it means we can continue to use test predominantly for figuring things out21:01
clarkb(and none have disappeared21:01
clarkbI've copied both the old review-test consistency results and the current prod ones onto review-test so others can see that delta21:03
openstackgerritMerged opendev/system-config master: OpenstackId v3.0.18  https://review.opendev.org/c/opendev/system-config/+/77259421:06
clarkbkopecmartin: I'm looking at your latest failure now21:07
clarkbkopecmartin: the issue appears related to letsencrypt self signed cert stand in provisioning. I'll try to figure it out and leave you a comment21:08
*** hamalq has quit IRC21:21
*** hamalq has joined #opendev21:21
clarkbkopecmartin: I left a comment but I thik I'll also push up a patch to reduce the rtt21:24
clarkbhrm I Think I just realized what the underlying issue is, its wanting to do an opendev.org LE cert21:27
clarkber openstack.org LE cert21:27
clarkbI don't think we want a refstack.opendev.org hostname anywhere since refstack is a very openstack specific application21:28
clarkbinfra-root ^ does that make sense? if so I wonder how we should do the testing, maybe just use snakeoil certs rather than le at all?21:28
clarkbthen for prod we'll point at some purchased cert?21:28
clarkbor are we doing openstack.org LE certs somehow?21:29
clarkboh wait we delegated the domain from openstack.org to opendev.org? is that right?21:29
openstackgerritClark Boylan proposed opendev/system-config master: WIP Deploy refstack with ansible docker  https://review.opendev.org/c/opendev/system-config/+/70525821:30
clarkbkopecmartin: ^ ok, let's see how that does21:31
fungiclarkb: yeah, we le openstack.org certs using cnames in the openstack.org domain to the acme.opendev.org21:32
clarkbcool then I think the latest patchset is correct for le things21:35
ianwclarkb: we can do openstack.org certs, just have to make sure we have the _acme-challenge pointing to acme.opendev.org21:44
ianwfor testing though, that shouldn't matter?21:44
clarkbianw: ya for testing it shouldn't matter. I was just thinking that maybe we shouldn't use LE there at all for half a second21:46
clarkbbut its fine we do the delegation thing and ya would need to update dns first21:46
clarkbianw: also not sure if you've seen https://review.opendev.org/c/openstack/diskimage-builder/+/772254 to deal with get-pip.py problems in dib now that get-pip.py is >= python3.6 unless you do some gymnastics22:03
*** sboyron has quit IRC22:08
*** slaweq has quit IRC22:08
ianwsigh, no, i'll pull it up22:24
fungidiablo_rojo_phon: once you've got consensus on the desired end state for the foundation irc channels, let's sync up and draft a maintenance plan in an etherpad with the steps we'll need to take and the timeline, since there's some initial setup work which needs to be performed for the new channels, but also our usual process for forwarding channels involves waiting days between some steps, so it'll be easier22:26
fungito work into our schedules if we have a timeline22:26
fungiand there's cleanup to do later as well22:27
kopecmartinclarkb: thank you for that patch, you made it work! the only thing left is a test of a submission, it's failing, i'm gonna edit that22:51
openstackgerritMartin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker  https://review.opendev.org/c/opendev/system-config/+/70525822:51
clarkbkopecmartin: that is great news, thank you for helping with that22:51
diablo_rojo_phonSounds good fungi. I figure we give the thread a couple days to languish and then move forward.23:06
*** tosky has quit IRC23:18
fungiawesome, great plan23:21
clarkbkopecmartin: left a comment on the latest patchset (I think we may want the anonymous upload thing to be test only), otherwise if that passes testing I think we're looking good23:35
kopecmartinclarkb: that's true, is there a specific var set for distinguishing zuul/testing run?23:38
clarkbkopecmartin: no, instead what you can do is make a testing specific host_vars file that is only included in testing then set the var to enable anonymous users there23:39
clarkblet me find some links23:39
clarkbit acts like a variable mix in23:39
kopecmartinoh, i see, yeah, that's a cleaner way to do it23:39
clarkbkopecmartin: https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/templates/host_vars you put the file there then add it to https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/run-base.yaml#L7423:40
kopecmartinclarkb: ok, thanks .. let's see now23:53
openstackgerritMartin Kopec proposed opendev/system-config master: Deploy refstack with ansible docker  https://review.opendev.org/c/opendev/system-config/+/70525823:53

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!