Monday, 2021-01-25

*** zaro has quit IRC00:17
*** zaro has joined #opendev00:18
*** DSpider has quit IRC00:36
*** ysandeep|away is now known as ysandeep03:30
*** raukadah is now known as chandankumar04:07
*** ykarel has joined #opendev04:39
*** marios has joined #opendev06:11
*** hemanth_n has joined #opendev06:49
*** lpetrut has joined #opendev07:08
*** eolivare has joined #opendev07:11
fricklerwe should recycle #openstack-chef, it has become very quiet in there otherwise07:24
fricklerand can't be too much offtopic, either, I regulary get job offers as pizza baker or similar, just for mentioning "experience with Chef" in my resume07:25
openstackgerritdaniel.pawlik proposed openstack/diskimage-builder master: Install last stable version of script
*** ralonsoh has joined #opendev07:55
dpawlikfrickler: hey, hope this will fix issue with building new images that uses python 207:57
*** fressi has joined #opendev07:57
*** andrewbonney has joined #opendev08:19
*** rpittau|afk is now known as rpittau08:35
*** ysandeep is now known as ysandeep|lunch08:39
*** sboyron has joined #opendev08:39
*** DSpider has joined #opendev08:56
*** jpena|off is now known as jpena08:56
*** zul has joined #opendev09:08
*** ysandeep|lunch is now known as ysandeep09:23
*** tosky has joined #opendev09:23
*** marios has quit IRC09:26
openstackgerritGuillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap
openstackgerritGuillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag
openstackgerritdaniel.pawlik proposed openstack/diskimage-builder master: Install last stable version of script
*** marios has joined #opendev10:12
openstackgerritGuillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap
openstackgerritGuillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag
*** sshnaidm is now known as sshnaidm|ruck10:39
*** hashar has joined #opendev11:03
openstackgerritdaniel.pawlik proposed openstack/diskimage-builder master: Install last stable version of script
*** dtantsur|afk is now known as dtantsur11:11
*** zbr is now known as notify11:41
*** notify has quit IRC11:46
*** notify has joined #opendev11:46
*** ysandeep is now known as ysandeep|afk11:46
*** sboyron has quit IRC12:16
*** sboyron has joined #opendev12:17
*** bodgix_ has quit IRC12:19
*** bodgix has joined #opendev12:19
*** dirk has quit IRC12:21
*** dirk has joined #opendev12:21
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add swift-proxy repo in vexxhost tenant
*** jpena is now known as jpena|lunch12:25
fricklerdid someone change colors on the gerrit diff view or is it just me suddenly seeing something like yellow/purple instead of red/green for removed/added?12:32
frickleroh, wait, this likely isn't new and only seems to happen when comparing different PSs ... very weird still12:37
*** hemanth_n has quit IRC12:38
*** ysandeep|afk is now known as ysandeep12:44
*** notify is now known as zbr13:16
*** jpena|lunch is now known as jpena13:31
dpawlikfrickler: I understand that support python2 is difficult, but it is still available and images are using it so maybe we could fix the the DIB for now: ?13:34
dpawlikcc fungi ^^ (thanks in advance for the review)13:35
*** dirk has quit IRC13:51
*** dirk has joined #opendev13:51
*** iurygregory_ has joined #opendev13:59
*** zbr4 has joined #opendev13:59
*** iurygregory has quit IRC14:00
*** iurygregory_ is now known as iurygregory14:00
*** Alex_Gaynor has joined #opendev14:00
Alex_GaynorI'm seeing ARM64 jobs not starting.14:00
*** zbr has quit IRC14:01
*** zbr4 is now known as zbr14:01
fricklerkevinz: still lots of error node attempts and next to no nodes in use
fricklerAlex_Gaynor: yes, seems to be an ongoing issue with linaro14:08
*** ykarel_ has joined #opendev14:09
*** ykarel has quit IRC14:11
*** lbragstad has joined #opendev14:18
*** ykarel__ has joined #opendev14:19
*** ykarel_ has quit IRC14:21
*** lbragstad has quit IRC14:34
*** lbragstad has joined #opendev14:37
*** ykarel__ is now known as ykarel14:40
openstackgerritsebastian marcet proposed opendev/system-config master: OpenstackId v3.0.17
openstackgerritMaksim Malchuk proposed openstack/diskimage-builder master: Fix hooks order for CentOS/Fedora when mirror used
*** d34dh0r53 has quit IRC14:48
*** d34dh0r53 has joined #opendev14:54
*** artom has joined #opendev14:57
sshnaidm|ruckhi, is problem with retry_limits known? like,814:58
sshnaidm|ruckproblem with some of clouds?14:58
*** hashar has quit IRC14:59
fungisshnaidm|ruck: we turned inap back on again, so we should probably look into whether it's back to the same old problem we unhooked it for the last several times (ip address conflicts from rogue vms)15:06
sshnaidm|ruckfungi, ack15:07
sshnaidm|ruckfungi, is it possible to see from web interface which cloud retry limits are on?15:07
fungiif they uploaded logs, then yes15:08
sshnaidm|ruckfungi, maybe worth to publish here:
fungiif they didn't, we likely have to go hunting in service debugging logs15:09
fungisshnaidm|ruck: also the cloud name is embedded in the hostname15:09
fungibut yeah, there's no node detail recorded in the database which that dashboard pulls from, i don't think15:10
*** lpetrut has quit IRC15:10
fungiwe normally rely on checking the zuul inventory which gets included with the log upload15:10
fungii'll see what nodes got used by that build id15:11
funginode name was centos-8-inap-mtl01-002270316415:13
fungiso yep, it ran there15:13
fungiterminated with "Ansible complete, result RESULT_UNREACHABLE code None"15:14
fungirough estimate we've logged ~2k RESULT_UNREACHABLE states since executor logs rotated at 06:25 utc today15:18
openstackgerritJeremy Stanley proposed openstack/project-config master: Revert "Revert "Revert "Revert "Temporarily stop booting nodes in inap-mtl01""""
fungiconfig-core: ^ let's stop using inap again while we see what additional details we can collect from our logs this time15:20
mnaserfungi: +215:21
*** ysandeep is now known as ysandeep|dinner15:24
*** ykarel has quit IRC15:27
Alex_Gaynorfrickler: thanks. Is there anyway to track the status of the incident? (I guess that's really a question for the folks at linaro)15:37
fungiAlex_Gaynor: kevinz is really our only contact there as far as i'm aware. usually we try to reach him in here and by e-mail to get him to look into whatever's going on in there. i can also try to see if we get any useful details out of their api with the boot failure messages15:41
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add swift-proxy in vexxhost tenant
fungiinfra-root: processing the executor debug logs to map build results to providers, i see that 80% of the RESULT_UNREACHABLE states logged on ze01 since 06:25 have been for inap-mtl0115:43
fungii haven't checked the other 11 executors for comparison, but expect it's probably a representative sample15:43
clarkbfungi: hrw as well iirc as a linaro contact (though not on the cloud side directly)15:44
clarkbfungi: disabling inap seems fine then and maybe we get dansmith and melwitt in contact with them?15:44
clarkbfungi: btw I've learned about via discussion on my gerrit repo-discuss thread15:46
clarkbI'm going to fiddle with that on review-test, and try and get a sense for what sort of changes we need to make15:46
fungioh neat15:46
openstackgerritMerged openstack/project-config master: Revert "Revert "Revert "Revert "Temporarily stop booting nodes in inap-mtl01""""
fungiAlex_Gaynor: kevinz: yeah, the most we get from nodepool is that it gives up waiting for the boot request to return a reachable instance and tries again, then after the third time of the same gives up and rejects the node request, and since there is no other provider to fulfil a request for that node type zuul returns NODE_ERROR15:53
openstackgerritGuillaume Chauvel proposed opendev/system-config master: Install for python3.5/xenial with specific url
openstackgerritMartin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker
*** brinzhang_ has quit IRC16:04
Open10K8SHi team. Can you check this PS please? other stuff are waiting this :)16:09
*** mlavalle has joined #opendev16:10
openstackgerritGuillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap
openstackgerritGuillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag
fungiOpen10K8S: how does that ref in the commit message relate to it? i'm a bit confused16:14
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add swift-proxy in vexxhost tenant
Open10K8Sfungi: sorry, i fixed16:16
*** brinzhang has joined #opendev16:20
clarkbok consistency results for the test server have been generated and are in my homedir there in a file called consistency.results16:22
*** ykarel has joined #opendev16:23
clarkbwe've got three types of issues: accounts without an externalid for their preferred email address, conflicting email addresses in external ids, and groups that loop their membership. This last class of problem is only affecting a small number of gruops so I think I'll start digging in there to keep scope small to start16:23
*** brinzhang has quit IRC16:24
*** ysandeep|dinner is now known as ysandeep|away16:33
jrossercan i ask about ipv6 addresses, i've got a focal node with an ipv6 address and a buster node without one, ( vs.
jrosserwhat would you expect to see here?16:35
clarkbjrosser: they ran in different clouds. One the cloud provides ipv6 and the other doesn't16:37
clarkb(technically bhs1 does provide ipv6 but not via ways our VMs can consume and configure it so its not really there from the test node perspective)16:37
jrosseroooh ok16:37
openstackgerritAkihiro Motoki proposed openstack/project-config master: Preparation for tempest-horizon retirement
clarkbrm_work: johnsom: it appears the octavia-core and neutron-lbaas-core include each other whcih is a gerrit config consistency warning. Looking at the two groups I think the proper fix is to remove neutron-lbaas-core from octavia-core. neutron-lbaas-core has no members so including it in octavia-core doesn't do much. The other way around does make sure octavia-core members are in neutron-lbaas-core though16:39
clarkbrm_work: johnsom: any chance you can take a look at that and make that change if this is correct?16:39
* clarkb makes this change on the test server16:41
johnsomclarkb Agreed, octavia core as a member of neutron-lbaas-core is the correct answer.16:41
clarkbthanks for checking, if you make the change on the prod side as a group manager that woudl be great. I'll sync up my testing to ensure the complaints go away16:41
johnsomclarkb Done. Took a minute as I haven't had to do this since the gerrit change. lol16:43
clarkbjohnsom: thanks!16:43
*** ykarel has quit IRC16:46
fungizbr: looking back through the git-review changelog for things which have landed since the last release, it seems like we've missed adding release notes for some important things (particularly dropping support for python 2.7). i'm happy to push a quick set of updates to release notes but what else do you think should be incorporated?16:50
clarkbcool the other group that is an issue is networking-ibm-release including itself. I just removed it from itself and now check_groups consistency reports no errors on the test server16:50
zbrrelease notes are the only thing i know16:50
clarkbI have not modified ^ on the prod side, but can do that next if people think that is a reasonable thing to do?16:50
zbrthere was a feature i was considering: filtering output from remote16:51
fungizbr: er, i meant what other release notes do you think we need to add16:51
fungi`git log --oneline --no-merges 1.28.0..origin/master` shows a fair amount of stuff, some of which might warrant a note16:51
* zbr digs into git log16:51
fungiso far we just have a release note for the --notify option16:51
clarkbthe other two classes of errors are going to be more difficult to address as they involve users and their usage of gerrit in a more "personal" way (email; login openids)16:52
*** marios is now known as marios|out16:53
fungizbr: certainly dropping python 2.7 support, but also maybe notopic configuration, newer python (up to 3.8) testing, dropping draft support, commit hooks in submodules, push --no-follow-tags16:53
clarkbdoing a quick skim of the other issues in some cases they are related because you'll have multiple accounts that don't have external ids for their preferred email and they are all the same email16:54
clarkbwhich means if we "fix" the lack of external id naively we'll create more of the email address conflicts16:54
zbrfungi: are you adding one, or should do it?16:55
fungizbr: well, i'm asking whether you think any/all of those things warrant release notes. i can try to add notes for them but don't want to waste time on any which are not important enough to announce, nor do i want to miss important ones we want included16:57
fungiso sanity checking the list of things i mentioned against the 1.28.0..origin/master log would be appreciated16:58
clarkbI'm going to try disabling an account to see if gerrit ignores it for further consistency checking16:59
*** marios|out has quit IRC16:59
clarkbif that works one approach we may be able to take is simply disable the accounts, send email saying if your account stopped working please talk to us?16:59
clarkbanyway need to test that first (chances are it won't help based on previous observations)16:59
zbri am doing it now. which versioning is going be next? 1.29.0, 2.0, or 21.0 (aka pip style) ?16:59
clarkbhuh the rest api may not even have a method for disabling an account?17:00
fungizbr: i was planning to do 2.0.0 or maybe if you think announcing a prerelease on the ml first is a good idea17:01
fungidropping python 2.7 support seems to me to be significant enough to warrant a major version increase17:01
clarkboh nevermind tehy call it "active"17:01
fungiclarkb: yeah, active/inactive is what it's always been17:01
fungiclarkb: however, be forewarned that from what i've witnessed gerrit will still check inactive accounts for conflicts, so merely making conflicting accounts inactive likely doesn't help17:02
fungithough filtering for inactive accounts probably gives us an easy list of conflicts we can safely delete17:03
clarkbfungi: yup, I just want to double check it in the "no externalid id for this preferred address" case17:04
clarkbwe know it doesn't help in the conflicts for external ids case17:04
openstackgerritGuillaume Chauvel proposed opendev/system-config master: Install for python3.5/xenial with specific url
clarkbonce I've confirmed it doesn't help, I'm going to dig into a couple of example and see if I can sort out if there is a pattern or not. I don't expect this to be quick and easy but probably our only real shot and sorting it out is to dig in :(17:05
zbrtbh, i do find the generated ChangeLog file good enough and I find no joy into expanding it into fragments.17:06
clarkbconfirmed setting an account to inactive does not change the issue with missing externalid for preferred email address17:07
fungizbr: the reason i asked is currently we seem to only publish the release notes not the commit-level changelog:
zbrcannot we change that to lower the amount of reno bureaucracy needed? -- that is a small project.17:12
fungizbr: i'd be okay taking the use of reno back out, though maybe we should see if anyone else has found it especially useful. looks like mordred added the first ones in late 2018:
fungiwe've only got a total of 8 release notes in there17:14
fungithough the missed-notes one contains four entries actually17:15
fungiso maybe ~10 notes17:15
fungias a quick alternative, i can add a note about dropping python 2.7 support and call it a day. the other changes are not as significant for potential impact17:16
mordredzbr, fungi it's a small project, but it has a LOT of users, and it is typically slow to change significantly. that's why I thought doing explicit reno notes would be a good idea - having a solid way to communicate when something actually _does_ change seems important17:18
fungiyeah, i think if we want to keep relying on release notes there, we need to be more diligent about asking change submitters to also include release notes in their changes17:19
fungii see probably 6 things since the last release which could have release notes but are missing them17:20
mordred++ ... that said - if we're not using it, then maybe it doesn't make sense to keep it. but - in general, the change rate is slow enough maybe it's a good habit to develop?17:21
clarkbfungi: looking at the "preferred email ahs no external id case" since there are fewer of them than the externalid conflicts first. It almost looks like this is the set of users we have mabye merged into another user in the past. The ~4 I've checked so far have no external ids at all but have a user record with a preferred email set17:22
clarkband some of them are inactive17:22
clarkbthats the long way of saying "setting these accounts to inactive seems to be the proper fix here"17:23
fungiclarkb: yes, at one time our process was to just move the external ids from the old account to the new account, but then we started marking the old account inactive later17:24
clarkbthey are account records that we don't want to just "pollute" with fake data to make the checker happy, but we also need to indicate that they are not to be used17:24
clarkbbut gerrit isn't happy with that :/17:24
clarkbI wonder if we can unset the preferred email address directly17:24
clarkbthat keeps the account number and name present17:25
fungii expect we'll need to unset preferred_email in those cases, or may need to set it to randomly generated garbage if gerrit doesn't like accounts having no address at all17:25
clarkbjust thinking out loud here another option would be to add external ids for bogus email addrs and set that as the preffered email then set teh account inactive17:26
fungiright, that... but it does sound painful17:26
clarkbya let me write some notes down, then I'll look into whether or not we can unset preferred email addresses I guess17:27
*** ykarel has joined #opendev17:29
clarkbusing the rest api to delete an email address does not work. It is an error ebacuse the email address can't be found :17:33
*** eolivare has quit IRC17:34
* clarkb looks for docs17:34
clarkb"Since all data in the account.config file is optional the account.config file may be absent from some user branches." but also "When users update their account properties by pushing to the user branch, it is verified that the preferred email exists in the external IDs."17:35
clarkbI think that maybe we can just delete the account.config file and push that in?17:36
clarkboh except that the active state goes in that file. Instead maybe we can remove the preferredEmail line and set active = false and see if it verified that ok?17:36
clarkbI think ^ is the next thing to try17:37
clarkbbut I can't do that online with review test because it won't verify the other 750 errors :/17:37
iurygregoryhey opendev folks, does anyone know if only the owners of the patch can add hashtags to it? =)17:38
clarkbI'm going to take a break to check that I don't need to catch up on anything else this morning then I ugess stop gerrit, make that edit for one of the accounts I've identified, push it directly, restart gerrit, reindex accounts and clear caches, then rerun consistency checks?17:38
clarkbiurygregory: I think zbr said our acls are currently set up that way17:38
clarkbI don't know if they need to be set up that way or not, I'm not looked at the privilege implications of using hashtags17:38
iurygregoryclarkb, in ironic we wanted to add hashtags to have a better visibility of priorities... so if we can add hashtags to other people patches that would help17:39
clarkbright, someone needs to look at the privilege implications of that then potentially update tehe acls17:39
iurygregoryat least if cores could update other ppl patches..17:40
clarkbit seems the default is to be a bit more restricted, we just need to udnerstand why and if it is safe to open it up further17:40
iurygregoryclarkb, this would need a discussion on openstack-discuss or something?17:40
clarkbno it needs someone to read teh gerrit docs and udnerstand why gerrit defaults to restricted access to that, then propose a change if it is safe (ideally with some pointing at relevant documentation)17:41
clarkbwe don't need discussion as much as understanding17:41
iurygregoryby any chance you know the gerrit docs ? =)17:41
iurygregoryI can take a look and see if I can understand =)17:42
clarkbiurygregory: there is a link to them in the top banner of our gerrit pages17:42
iurygregoryclarkb, tks!17:42
clarkb is likely to have relevant info17:42
iurygregoryThe change owner, branch owners, project owners, and site administrators can always edit or remove hashtags (even without having the Edit Hashtags access right assigned).17:43
iurygregorywe can probably edit...17:43
clarkbI bet the reason it isn't default enabled for all is the "remove" ability17:44
clarkbI guess edit is potentially destructive too if you replace tag foo with tag bar17:44
clarkbin that cse allowing cores to do it seems fine17:45
clarkbthe downside to doing it that way is we'd have to update project acls one by one rather than a global edit for registered uses17:46
iurygregorywell, we want that in ironic hehe, not sure about other projects XD17:47
iurygregoryI can ofc push the patches (just need to know what should be added...)17:48
iurygregorywe are trying to improve the reviews so a search using "hashtag:ironic-prio" would give the list of all patches that need attention or something =)17:48
clarkbiurygregory: I think you add al ink roughly at that says 'Edit Hashtags = group ironic-core"17:50
iurygregoryclarkb, ack o/ I will push a patch and add ironic-cores in the review17:51
*** ralonsoh has quit IRC17:56
*** sboyron has quit IRC17:56
*** ralonsoh has joined #opendev17:57
*** sboyron has joined #opendev17:57
*** ralonsoh has quit IRC17:58
fungii think adding core reviewer access to use hashtags on a per-project basis initially would be a fairly safe way to gauge whether it can just be allowed for all users18:00
fungilike we do with allowing +1/-1 code review votes and comments18:00
fungiease into it18:00
*** rpittau is now known as rpittau|afk18:10
*** ykarel has quit IRC18:12
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add swift-proxy in zuul namespace
*** jpena is now known as jpena|off18:22
clarkbon review-test I did: stopped gerrit, backed up all-users, cloned all-users, fetched refs/userse/XY/ABXY, checked out FETCH_HEAD, deleted preferredEmail in account.config, added active = false in account.config. committed then did git push origin refs/users/XY/ABXY. Ran an offline reindex of accounts and groups. Started gerrit thene flushed accounts and groups caches18:24
clarkbconsistency check no longer complains about account ABXY18:24
clarkbthe tricky thing about doing this in a downtime is we can't do it in a single atomic commit since all the users have different refs18:25
clarkbbut maybe we schedule an hour or two downtime. Script up the fixes for ^ and whatever we come up with for the external id conflicts, make all those changes in one go, do an offline reindex, and then clear caches on startup?18:26
clarkbnot ideal, but that should get us to a point where we can push these changes to gerrit online and have it verify them for us18:26
clarkbwhcih means it should be a one time thing18:26
*** andrewbonney has quit IRC18:28
clarkbI'm going to take a break but then will look at the external id conflicts shortly18:28
clarkbactually I should do a better audit of the preferred email issue. I've only looked at 4 accounts so far and checking all of them shouldn't take too long if I write a script to print some stuff out18:29
clarkbhrm ya doing an audit I've already found an example that didn't match my first three. In this case it is an active account with a secondary email that mathces an external id18:45
clarkbfor those we might be able to tell gerrit to use teh secondary id as primary email?18:46
clarkbthat can be done online via the api I think18:46
clarkbI think no matter what we do we're going to be making compromises workign with incomplete data (and as a result some users may be left in a weird spot)18:47
clarkbI'm going to keep looking at this preferred id issue list and see if there are other scenarios18:47
*** dtantsur is now known as dtantsur|afk18:50
openstackgerritGuillaume Chauvel proposed opendev/system-config master: Install for python3.5/xenial with specific url
openstackgerritGuillaume Chauvel proposed opendev/system-config master: Move xenial distutils check in OS specific files.
clarkbwe can also set a bunch of these accounts to inactive nowish via online updates. Then see if there is screaming to resolve18:52
clarkbthat way when we get to taking a downtime we'll hopefully have more complete information18:53
* clarkb writes down more notes18:53
fungiright, we also have e-mail addresses for most of them, so like i suggested last week we could just send a form letter saying that on some specific date we'll be taking action on those accounts to resolve the conflict, and asking for users to reach out to us if the action we're proposing is the wrong one for them18:54
clarkbyes, though I expect for many users they won't actually know18:55
clarkband do we do the same thing for all 750 issues or spend all day investigating each situation to prescribe a solution?18:55
fungii expect for most users those addresses are going to /dev/null and we can just do whatever we want18:55
clarkbI'ev just found an account that seems to have been merged except for the ssh side, so it is possible the person is always reviewing as user foo but pushing code as user bar :/18:56
fungiright, those are the specific cases i worry about (users actually using two separate accounts and not realizing it)18:56
clarkbthe bit I'm struggling with most here is we seem to have many different situations we need to resolve and so fixing this isn't going to be quick but we also need to take a downtime.18:58
clarkbmakes scripting it more difficult as well as communicating it18:59
clarkbit wouldn't be so bad if we could just do all the easy ones first without a downtime18:59
fungii think the first step is exactly what you're doing though: trying to classify different types of situations18:59
clarkbthe first bucket is accounts that are already inactive: we can just remove their preferred email address and move on18:59
fungiit's hard to know what to do next without a fairly complete picture of the various problems18:59
clarkblet me do a better job of pulling all those out and setting themaside19:00
clarkbthen the next set is active accounts that have no external ids set (we can just remove their preferred email addr and set them inactive)19:00
clarkbno additional external ids is functioanlly equivalent to being inactive, they cannot log in19:01
clarkbthen we're left with the weird ones which I'm only just starting to dig into19:01
clarkbalso, I'm still workign against review-test so at some point we'll need to generate the data from prod19:01
clarkbbut that shouldn't be too bad once we've sorted things out on the test side19:01
fungithe situation with ssh keys will get hairy too, for sure, because before 2.13 ssh keys were managed in the db and we could recombine the records into other accounts easily when merging them. in 2.13 they'd already moved ssh keys into the all-users repo, so that was no longer reasonable to fix manually19:03
fungiso there will certainly be accounts with nothing other than ssh keys for access, because in 2.13 we left them alone19:04
fungithough hopefully we were consistent about marking them inactive in those cases19:04
clarkbalso note I've not yet started looking at accounts with external id conflicts19:10
clarkbbut as you say step by step building a bigger picture seems like what we need to do here19:10
clarkbalso, maybe we take multiple downtimes to spread this out a bit and see progress. In particular that reduces the number of things we might be modifying which reduces the risk we'll get something wrong19:13
clarkbjust under half seem to be inactive accounts19:16
clarkbnow to find the set that have no externald ids19:16
fungiinfra-root: i'm unlikely to get to it today (or probably even tomorrow), but looks like our "wheel mirrors" haven't updated in two weeks. likely something broke with the zuul jobs which update them, i haven't looked19:33
*** brinzhang has joined #opendev19:37
iurygregoryclarkb, something like ?19:41
clarkbiurygregory: yes I suspect that is correct, but I'm not in a great spot to verify it right now19:47
clarkbI'm trying to run down a bunch of consistency issues in our gerrit accounts db19:47
clarkbfungi: that might make for a good distraction once I've spent most of the day looking at accounts. We'll see if i find time to dig in later today (re wheels)19:48
iurygregoryclarkb, no worries =) I will push the patch to see how it goes19:48
openstackgerritIury Gregory Melo Ferreira proposed openstack/project-config master: Update ACLs of Ironic Projects to allow Edit Hashtags
corvusfungi, mordred: did we document the "git config gitreview.*" options?  i don't see them in
corvusbasically, it looks like you can run "git config --global gitreview.rebase false" but i can't find a doc telling me that should work20:02
clarkbfungi: it just occured to me that since refs/users/XY/ABXY are their own refs we may be able to push to them individually without needing a downtime. I will have to test that20:02
fungicorvus: mmm, you know, i'm not entirely sure20:03
fungii thought we had incorporated them in the docs but maybe not, checking20:03
fungiclarkb: yes for anything not involving external-ids that might be doable, but unfortunately not for external-ids as those are all in the same ref20:04
clarkbfungi: ya, but maybe that means we can tackle this in a couple of iterations and get to where we need to take a downtime last20:04
clarkbanyway I'm still classifying things under refs/users/ its not a quick process20:05
fungicorvus: the configuration section of the git-review(1) manpage maybe?20:06
fungiit does have a gitreview.rebase entry at least20:07
fungii'm not sure if it covers everything, but hopefully it does20:07
fungi"This setting determines whether changes submitted will be rebased to the newest state of the branch." [and then delves into details of what true and false do]20:08
fungiso yeah, maybe the problem is that the sphinx docs and manpage are out of sync?20:08
fungialso it looks like the last time anyone added anything to the manpage was 4 years ago, so changes since then aren't reflected there i guess20:09
fungii have a feeling we started out with the manpage as the source of truth, added very incomplete sphinx docs, and then at some point people started documenting things in sphinx and not in the manpage20:10
fungiprobably what we need is to merge the manpage content into the sphinx source, and then use a sphinx extension to generate the manpage and run that at package build time?20:11
fungizbr: ^ you've been reviewing git-review changes lately too, what do you think?20:12
fungigit-review also has its own --help output, which you'll only see when running `git-review --help` because `git review --help` wants to show the manpage (and if you've installed with pip, that won't exist in your manpath because pip can't install files there)20:13
corvusfungi: oh yep, i see it there, thanks!20:14
fungicorvus: opinions on how to improve that?20:14
corvusfungi: (to clarify, i see it in the manpage now)20:15
corvusand yeah, i use --help a lot, but that doesn't mention the rebase afaict20:15
corvusfungi: i agree that we should move docs to sphinx and generate manpage from that if possible20:15
corvusfrom the looks of it, manpage is still way ahead of sphinx docs20:16
fungii think that's what we do for bindep, will double-check20:16
corvusthe current state is not ideal because one could (as i did) find the sphinx docs, assume they are complete, and then give up20:16
fungiyep, totally agree20:17
corvusi think it'd be great if zbr is interested in taking that on :)20:17
openstackgerritMatthew Treinish proposed opendev/subunit2sql master: Add release notes to prepare for release
fungimmm, i'm not finding generated manpages in bindep either. i know i've seen it in use somewhere in our tools20:24
fungioh, i bet i know at least one place it's in use20:33
funginope, thought pbr at least was using it, since it's what does it20:42
mordredI believe we had generated manpages back in the day20:43
mordredbut with the shift to wheel-based pip stuff we gave up20:44
fungifunctionality is still in pbr, and i could swear i did something recently on a project generating them20:44
clarkbout of 109 accounts with preferred emails that don't have external ids 12 are in the "complicated case" where they have ssh usernames and no other similar external id bits to tie to. 16 have other email addresses that we could possibly add as externalids then set as preferred email (though this is less clear to me). The rest are either inactive or functionality inactive because they have no external ids20:46
clarkbthat still doesn't touch any of the email address conflicts20:46
corvusbtw, i've been testing git-review with onlykey; seems to work so far, but due to invoking ssh multiple times (for the rebase) it can be awkward.  onlykey normally prompts for a pin so you push 3 buttons on the device, and you do that twice for each of the ssh connections, but it has a setting where you can hit any button, and that makes it easier (more like a yubikey or signet).  or you could turn off the20:47
clarkbbut I think we can probably address the 109 - 12 - 16 remaining preferred email errors as step 120:47
mtreinishfungi: pbr supports man page generation? For stestr I just wrote a bash script to do man page generation from the sphinx docs:
clarkbI'm going to test if I can update those while gerrit is online next20:47
fungimtreinish: heh, wfm ;)20:48
clarkbiirc the issue pbr ran into is that you can build the manpages no problem but you can't install them20:48
clarkbbeacuse MANPATH is typically privilegd space and you don't want an install going to a venv to write there20:48
fungiwell, can't install them with a python package into anything manpathish20:48
fungiyou can still generate manpages to be used in distro packaging and the like20:49
clarkbyup and sphinx supports that just fine20:49
clarkbbut pbr specifically has trouble with that beacuse it often writes to venvs without root20:49
fungioh, maybe i'm thinking of an existing sphinx feature for generating manpages20:49
corvus++ and that's the value i see here (git-review should mostly be installed by most people via distro packages these days)20:49
mtreinishheh, well for stestr I haven't been able to convince any packagers to include a manpage with it. Despite putting it in the readme20:49
clarkbfungi: `make manpage` or whatever the target is should just do it20:50
mtreinishwhen I used to pacakge it in arch I did it, but they deleted my aur package when someone on the core team did their own stestr package and it doesn't have a manpage...20:50
fungiclarkb: that needs a makefile right?20:50
fungimtreinish: so many distros want to package python modules by pulling from pypi and unpacking the python packages that i can see where that would happen20:51
fungifor libraries it makes sense. for utilities which just happen to be written in python, less so in my opinion20:52
clarkbfungi: yes, sphinx should write out a make file in your sphinx dir stuff20:53
clarkbwhen you do the sphinx canned setup it writes that for you20:53
fungiahh, right20:53
clarkbinfra-root review-test:~clarkb/gerrit-consistency-notes/preferred-email-classifications has the classifications of the first 109 problems detected on review-test20:54
fungimordred: okay, i was confused, the manpages support in pbr is just for packing available manpages into the dist20:54
funginot generating them20:54
clarkbwe still need to rerun consistency checking against prod, but I'm getting orientated on the test server20:54
clarkbfungi: correct20:54
clarkband we removed that from pbr because it only caused problems, but you can still build manpages if you want them20:54
clarkbnow to test if I can push an update to refs/users/XY/ABXY while it is running to fix a problem20:55
fungiclarkb: well, manpages support is still present in pbr master, maybe some of it was removed? it's still in the commands and files hooks, and the packaging module20:56
fungiand apparently it's `make man` in the sphinx tree20:58
*** slaweq has quit IRC21:02
clarkbfungi: oh interesting, I recall having turned ti off in a number of places because we turned it on for like debian and then broke everyone21:04
clarkbsomething like that21:04
fungialso some projects including git-review are still including a pbr.manpages list in their setup.cfg21:06
clarkbok I think I have confirmed that for the 109 - 12 - 16 = whatever accounts we can fix those while gerrit is online pushing directly to their refs/users/XY/ABXY ref21:09
clarkbthen we need to sort out the 12 and 16 groups of users. Then figure out conflicting external ids21:10
clarkbI'm going to take a break now as its not raining or snowing and I'm due for a bike ride. When I get back I'll probably look at wheel mirrors? Then we can pick up gerrit things tomorrow21:11
clarkbinfra-root: thinking out loud here, maybe we can do paired ops to fix the bits identified as fixable so far some time tomorrow as the impact should be low?21:11
clarkbthat will make the inconsistencies list small and make it easier to focus on the other bits as we go along21:12
fungiwednesday might be a little doable for me, but if someone else wants to join you on it tomorrow that's cool21:12
fungier, wednesday might be a little more doable for me21:13
openstackgerritIury Gregory Melo Ferreira proposed openstack/project-config master: Update ACLs of Ironic Projects to allow Edit Hashtags
clarkbfungi: ok, we can probably plan for wednesday too21:13
clarkboh and I'll get the meeting agenda sent out when I return and try and put as much of the info I've learned so far in there21:14
clarkbthinking out loud here, we can probably fix the external ids in aggregate too (a single large commit) and push those fixes without a downtime21:14
clarkbwhere that will potentially get complicated is rebasing to keep up with the changes happening in gerrit. If we find we can't keep up quickly enough then we can take a downtime?21:14
fungiyeah, *if* we can work it all out in a single commit that should be doable21:14
clarkbI'll continue to ponder on that aspect of the problem21:15
clarkbya also that21:15
fungiif we want to make changes in the meantime, that would need downtime21:15
clarkbin any case I think I've figured out fixing about 10% of the errors so progress ! :)21:15
fungiprogress fore sure21:15
fungier, for sure21:15
*** tosky has quit IRC21:35
*** Alex_Gaynor has quit IRC21:36
*** tosky has joined #opendev21:36
*** TheJulia has quit IRC21:36
*** TheJulia has joined #opendev21:37
*** rpittau|afk has quit IRC21:37
*** johnsom has quit IRC21:37
*** rpittau|afk_ has joined #opendev21:38
*** johnsom has joined #opendev21:38
*** Alex_Gaynor has joined #opendev21:38
*** artom has quit IRC21:41
*** artom has joined #opendev21:47
*** slaweq has joined #opendev21:48
*** stand has joined #opendev21:49
*** sboyron has quit IRC21:52
*** stand has quit IRC21:56
*** stand has joined #opendev21:56
*** dmsimard has quit IRC22:50
*** dmsimard has joined #opendev22:51
*** slaweq has quit IRC23:01
openstackgerritGhanshyam proposed opendev/subunit2sql master: Add release notes to prepare for release
clarkbfungi: mnaser: shoudl I keep the infra-core (really config-core) discussion topic on the meeting agenda?23:27
clarkbI think we're pretty set with the plan we made last week and just need to get the ball rolling?23:27
fungiclarkb: if folks have a chance to look over the draft rfh i wrote up and linked in #openjstack-infra that would be swell23:30
fungiother than that i don't really have any updates for the topic23:30
fungier, #openstack-infra i mean, not the java one23:30
clarkbcool I'll take it off of the agenda and review the draft once I send the agenda out23:31
fungii can always link it during the action items topic23:31
mordredfungi: wow, I haven't thought about openjstack in forever years23:31
fungiwas it even a thing? i was just making fun of my typo23:36
fungioh, right, oracle has jstack23:37
mordredoh - no, I was also just making fun of your typo23:38
fungicool, appreciated23:38
mordredalthough I do think at one point that one guy did an java reimpl sortof but then gave up23:38
*** tosky has quit IRC23:47
clarkbok I took a quick look at the conflicting emails in external ids and I think the vast majority of these will need us to merge accounts or pick a side to "retire"23:56
clarkbalso note that when we merge the external ids I believe we'll create a bunch of refs/users/XY/ABXY preferred email addr orphans similar to the accounts above23:57
clarkbso we'll have a asecond step that requires us to retire those as I've proposed for the already inactive accounts above23:57
clarkbthnking out loud here maybe we set all of the side we're going to retire to inactive as a second step (first step is doing an audit and deciding which side of the two is merging into the other)23:58
clarkbthen if people notice and complain we can talk to them about the best way forward?23:58
clarkbI dunno this needs more thinking.23:59

Generated by 2.17.2 by Marius Gedminas - find it at!