clarkb | I do think we've mentioned a decent plan for addressing those now. Just need to figure out if it is viable when cross checking real data | 00:00 |
---|---|---|
clarkb | does that seem reasonable to other infra-root and stewie925 ? | 00:01 |
clarkb | Unrelated: ze01.openstack.org's executor did restart a few minuets ago. I have asked it to pause so that I can docker-compose down it | 00:03 |
fungi | the suggestion for dealing with duplicate accounts in that scenario, while not ideal, sounds reasonable | 00:03 |
stewie925 | clarkb: so if those reviews are orphaned they would still be there, just curious as to how I can filter for them? | 00:04 |
clarkb | fungi: which suggestion for which scenario? Sorry there are a cuple things in play there | 00:04 |
clarkb | stewie925: you'd do a reviewedby:accountid or reviewedby:email | 00:05 |
fungi | sorry, "I do think we've mentioned a decent plan for addressing those now" -> "the third option" -> "more quickly fix the account inconsistencies due to external id email conflicts then revisit this with stewie925 in a week or two" | 00:06 |
clarkb | stewie925: basically we're in a situation where you can keep the change association or your review/comment association | 00:06 |
clarkb | stewie925: because we have to decide if the new or old account is what ends up persisting into the future. If you choose to persist the old account then any activity on the new account gets orphaned (since you're not pushing code with the new account it would just be review comments via the web ui). | 00:06 |
clarkb | if instead you decide to simply move forward with the new account then the changes pushed under the old account would be orphaned. They are all still there in both cases just harder to associate with the current (in the future) account | 00:07 |
stewie925 | clarkb: ahh ok, what if I stay with the old account, and just purge the new account? | 00:11 |
*** tosky has quit IRC | 00:12 | |
clarkb | stewie925: thats the thing that requires us to make the gerrit notedb consistent to do without a dowtime. In that case any reviews/comments you made with the new account will be harder to associate but the changes you've pushed should all still be associated to the active account | 00:13 |
kopecmartin | ianw: hi, shouldn't this be set by a var to either refstack01 or refstack depending whether it's a production deployment or not? https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/refstack/templates/refstack.vhost.j2#L11 | 00:17 |
clarkb | you know what there is something I haven't considered yet (but is dangerous for others reasons) | 00:18 |
clarkb | I wonder if we force push will gerrit bypass verification of the accounts/ | 00:18 |
fungi | whee! | 00:18 |
clarkb | this is dangerous because if we race someone updating their emails ro whatever we'll undo that for them | 00:18 |
fungi | yep | 00:19 |
clarkb | but maybe we should test that on review-test and if it works intstead of takign a downtime we can take a virtual downtime where we disable gerrit access | 00:19 |
clarkb | I dunno just trying to think beyond previous considerations | 00:19 |
clarkb | stewie925: I'm happy to think on this a few days if you are. And I'll start prioritizing looking at fixing the conflicts in other accounts so we can push updates sanely next week | 00:21 |
clarkb | definitely seems like we won't have an answer today and maybe sleeping on it will produce some great ideas :) | 00:21 |
fungi | or at least produce better-rested people | 00:22 |
clarkb | stewie925: does that work for you? feel free to ask questions as you have them and we'll do our best | 00:22 |
stewie925 | clarkb: yes, no rush - take your time | 00:22 |
stewie925 | its not a priority or rush for me | 00:22 |
clarkb | great. The first step was in understanding the account situation (which I think we have done now). Next we need to let brains do what they do | 00:23 |
stewie925 | clarkb: fungi: appreciate your help | 00:23 |
clarkb | as part of doing debugging for ^ I made my http able account admin in gerrit. I have now removed it from that group | 00:28 |
clarkb | the force push idea is so tempting if it weren't so risky :) | 00:28 |
clarkb | I'll work on filing this bug with migrating openids tomorrow morning too | 00:30 |
clarkb | I don't expect they will fix it quickly or at all, but we may get suggestions on how to address that and we could write a patch or something | 00:30 |
clarkb | I'll also catch up with ze01.openstack.org tomorrow morning as it is still pausing | 00:36 |
*** mlavalle has quit IRC | 01:08 | |
*** hamalq has quit IRC | 01:10 | |
*** stewie925 has quit IRC | 01:17 | |
*** jhesketh_ is now known as jhesketh | 01:49 | |
*** zbr has joined #opendev | 03:50 | |
*** dmsimard has quit IRC | 04:08 | |
*** dmsimard has joined #opendev | 04:09 | |
*** brinzhang has quit IRC | 04:17 | |
*** marios has joined #opendev | 05:35 | |
*** marios has quit IRC | 06:28 | |
*** lpetrut has joined #opendev | 07:29 | |
*** icey_ is now known as icey | 07:43 | |
*** elod has quit IRC | 07:45 | |
*** elod has joined #opendev | 07:49 | |
*** slaweq_ has joined #opendev | 07:50 | |
*** gibi has joined #opendev | 07:53 | |
*** slaweq_ has quit IRC | 08:22 | |
*** andrewbonney has joined #opendev | 08:24 | |
*** yoctozepto5 has joined #opendev | 08:26 | |
*** yoctozepto has quit IRC | 08:28 | |
*** yoctozepto5 is now known as yoctozepto | 08:28 | |
*** yoctozepto6 has joined #opendev | 08:32 | |
*** yoctozepto has quit IRC | 08:33 | |
*** yoctozepto6 is now known as yoctozepto | 08:33 | |
*** fressi has joined #opendev | 08:47 | |
*** fressi has left #opendev | 08:47 | |
*** roman_g has joined #opendev | 08:59 | |
*** DSpider has joined #opendev | 09:31 | |
*** mugsie_ is now known as mugsie | 10:16 | |
*** yoctozepto8 has joined #opendev | 10:47 | |
*** yoctozepto has quit IRC | 10:47 | |
*** yoctozepto8 is now known as yoctozepto | 10:47 | |
*** yoctozepto1 has joined #opendev | 10:54 | |
*** yoctozepto has quit IRC | 10:55 | |
*** yoctozepto1 is now known as yoctozepto | 10:55 | |
*** iurygregory has quit IRC | 11:13 | |
*** marios has joined #opendev | 11:23 | |
*** tosky has joined #opendev | 11:25 | |
*** yoctozepto2 has joined #opendev | 11:31 | |
*** yoctozepto has quit IRC | 11:33 | |
*** yoctozepto2 is now known as yoctozepto | 11:33 | |
*** roman_g has quit IRC | 11:57 | |
*** yoctozepto4 has joined #opendev | 12:45 | |
*** yoctozepto has quit IRC | 12:46 | |
*** yoctozepto4 is now known as yoctozepto | 12:46 | |
*** slaweq_ has joined #opendev | 12:49 | |
*** zbr9 has joined #opendev | 13:20 | |
*** zigo has joined #opendev | 13:20 | |
*** zbr has quit IRC | 13:22 | |
*** zbr9 is now known as zbr | 13:22 | |
*** slaweq_ has quit IRC | 13:31 | |
*** stephenfin has joined #opendev | 13:38 | |
*** zbr0 has joined #opendev | 13:43 | |
*** zbr has quit IRC | 13:46 | |
*** zbr0 is now known as zbr | 13:46 | |
*** iurygregory has joined #opendev | 13:49 | |
*** zbr3 has joined #opendev | 14:03 | |
*** zbr has quit IRC | 14:05 | |
*** zbr3 is now known as zbr | 14:05 | |
*** zbr7 has joined #opendev | 14:08 | |
*** zbr has quit IRC | 14:10 | |
*** zbr7 is now known as zbr | 14:10 | |
*** zbr4 has joined #opendev | 14:33 | |
*** zbr has quit IRC | 14:35 | |
*** zbr4 is now known as zbr | 14:35 | |
openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 14:36 |
*** fressi has joined #opendev | 14:51 | |
*** zbr4 has joined #opendev | 14:52 | |
*** fressi has left #opendev | 14:52 | |
*** zbr has quit IRC | 14:53 | |
*** zbr4 is now known as zbr | 14:53 | |
*** marios has quit IRC | 15:00 | |
*** zbr5 has joined #opendev | 15:12 | |
*** zbr has quit IRC | 15:14 | |
*** zbr5 is now known as zbr | 15:14 | |
*** lpetrut has quit IRC | 15:15 | |
*** tosky has quit IRC | 15:19 | |
*** tosky has joined #opendev | 15:20 | |
*** zbr8 has joined #opendev | 15:30 | |
*** zbr has quit IRC | 15:33 | |
*** zbr8 is now known as zbr | 15:33 | |
clarkb | ze01.openstack.org seems to have paused, going to down it now | 16:01 |
clarkb | #status log Added new focal ze01.opendev.org and stopped zuul-executor on ze01.openstack.org | 16:02 |
openstackstatus | clarkb: finished logging | 16:02 |
*** lpetrut has joined #opendev | 16:05 | |
*** Dmitrii-Sh has quit IRC | 16:11 | |
*** zbr2 has joined #opendev | 16:11 | |
*** Dmitrii-Sh has joined #opendev | 16:11 | |
*** zbr has quit IRC | 16:13 | |
*** zbr2 is now known as zbr | 16:13 | |
openstackgerrit | Clark Boylan proposed opendev/system-config master: Update zuul-executor shutdown handling https://review.opendev.org/c/opendev/system-config/+/777768 | 16:14 |
*** zbr6 has joined #opendev | 16:15 | |
fungi | rackspace e-mailed us to say that the refstack.openstack.org instance had to be rebooted | 16:17 |
*** mlavalle has joined #opendev | 16:17 | |
*** zbr has quit IRC | 16:17 | |
*** zbr6 is now known as zbr | 16:17 | |
fungi | server uptime is ~3 hours and the site seems to be up | 16:18 |
clarkb | corvus: ^ that change is the result of our conversation yesterday. It doesn't look like sighup is used by the executor for anything (and the scheduler deprecated its use for reloading configs). Also the executor seems to be the only service that supports graceful | 16:18 |
clarkb | fungi: thanks for checking on it | 16:18 |
corvus | clarkb: lgtm | 16:22 |
clarkb | fungi: stewie925 is not here but I realzied something about the account fixing. We'll also need to reassociate the openid email address with the account to avoid gerrit thinking it should create a new accoutn again. That should also be doable via external id edits so not a huge deal. But it occurred to me that would be an extra step | 16:23 |
clarkb | fungi: also separately it occurred to me if we're just going to declare bankruptcy on the inactive accounts we can probably use the gerrit rest api to delete their conflicting external ids | 16:24 |
clarkb | the upside to this is we should be able to do that one at a time and check on things as we go to ensure we are improving the situation | 16:24 |
clarkb | the downside is we won't get nice commit messages | 16:24 |
fungi | #status log The refstack.openstack.org service was offline 12:57-13:35 UTC due to a localized outage in the cloud provider where it's hosted | 16:24 |
openstackstatus | fungi: finished logging | 16:24 |
clarkb | but I think if we can use that to whittle down the list to the active set then do nicer commit messages for them that may be a win | 16:24 |
clarkb | as we'll end up with a much easier commit to understand for the accounts that are more important | 16:25 |
fungi | i think we can assume any rest api changes made by one of our administrative accounts during this timeframe is related to cleanup | 16:25 |
clarkb | ya | 16:25 |
clarkb | the history won't be impossible to understand, just more robotic | 16:25 |
openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Add test helpers for unstaged/uncommitted changes https://review.opendev.org/c/opendev/git-review/+/777687 | 16:26 |
openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Don't test rebasing with unstaged changes https://review.opendev.org/c/opendev/git-review/+/777456 | 16:26 |
clarkb | I mention that because I'm thinking the process might look something like: generate lists of active and inactive accounts, retire all inactive accounts, remove conflicting external ids via rest api for all inactive accounts, rerun consistency checks and confirm the situation is better. Then figure out what is hopefully a much smaller set of active accounts by hand with actual fixes | 16:27 |
clarkb | the key bit being we can actuall check things are improving as we go rather than needing to do a giant commit that is hard to test and may not actually help | 16:28 |
fungi | retiring inactive accounts with conflicts, not all inactive accounts, yeah? | 16:29 |
clarkb | right only looking at the subset of accounts that have problems | 16:31 |
clarkb | active and inactive lists are subsets of the "these accounts are inconsistenct" set | 16:31 |
*** icey has quit IRC | 16:33 | |
*** icey has joined #opendev | 16:34 | |
fungi | ianw: your rackspace ticket about the error deleting volumes got resolved, so i closed it | 16:36 |
*** zbr has quit IRC | 16:38 | |
clarkb | any idea why cacti doesn't have zm01-08.opendev.org in it? Also I expected ze01.opendev.org in it too. The cacti hosts were updated in system-config and puppet ran successfully against cacti | 16:40 |
clarkb | fungi: this is a fun one, I've just discovered that querying accounts by email over the rest api doesn't always return all results | 17:01 |
clarkb | was trying to do that to fully automate the list creation (the gerrit inconsistency error only gives us emails and openids so have to map to accounts) | 17:01 |
clarkb | we may need to do an out of band lookup of the account ids, then automate from there :( | 17:02 |
clarkb | or as another option just do our best with what the api gives us, address those, then figure out what is left after | 17:02 |
*** slaweq_ has joined #opendev | 17:08 | |
fungi | clarkb: interesting, i wonder if the rest api is assuming there wouldn't be conflicting accounts with the same address and optimizing by returning the first one it finds | 17:09 |
clarkb | fungi: well it seems to work in other cases and return the duplicates. I'm looking at the one that doesn't and comparing it to others to see what may be different and not really noticing anything | 17:10 |
clarkb | one thought was maybe the other account is inactive, but it isn't | 17:10 |
fungi | yeah, that would have been my next guess too. oh well | 17:12 |
fungi | we should probably just assume that the rest api exhibits undefined behavior where e-mail address conflicts are concerned | 17:13 |
clarkb | ya it seems to be good enough in most cases and if we get back < 2 results we can set that account aside for now | 17:13 |
clarkb | or generate the list externally | 17:14 |
*** hamalq has joined #opendev | 17:16 | |
clarkb | fungi: another thing to test is I'm doing all this unauthenticated. Maybe switching to authenticated will give more complete results | 17:18 |
fungi | oh! yes recent gerrit got a lot stricter about user details and particularly e-mail addresses | 17:25 |
fungi | you'll basically need to be an admin to fetch lists of e-mail addresses for any account besides your own now | 17:26 |
fungi | but there may be other related implications | 17:26 |
openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Move git-review to opendev tenant https://review.opendev.org/c/openstack/project-config/+/777774 | 17:36 |
*** lpetrut has quit IRC | 17:40 | |
openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Update jobs for opendev tenant https://review.opendev.org/c/opendev/git-review/+/777799 | 17:54 |
*** slaweq_ has quit IRC | 17:54 | |
*** andrewbonney has quit IRC | 18:15 | |
fungi | config-core: if someone has a moment to look over https://review.opendev.org/777774 i'd be most grateful, that way i can proceed with pending docs and python packaging polish before trying to tag a git-review release candidate | 18:19 |
mnaser | fungi: lgtm, do we need to create a change to add project-template 'publish-opendev-tox-docs' ? | 18:20 |
fungi | mnaser: shouldn't need to, it's defined in the tenant we're moving to | 18:22 |
mnaser | fungi: oh i see, that's why its complaining | 18:23 |
fungi | i'll recheck 777799 once 777774 deploys | 18:23 |
fungi | i'm working on other changes i plan to stack up behind that, but don't want to waste node cycles on them until i know jobs are passing in the new tenant | 18:24 |
*** slaweq_ has joined #opendev | 18:33 | |
clarkb | fungi: approved | 18:38 |
fungi | thanks! | 18:38 |
*** slaweq_ has quit IRC | 18:44 | |
openstackgerrit | Merged openstack/project-config master: Move git-review to opendev tenant https://review.opendev.org/c/openstack/project-config/+/777774 | 18:50 |
clarkb | ok the numebr of accounts that don't return results as expected through the rest api is a lot higher than I had hoped :( | 19:01 |
clarkb | though at least some of them do appear to be due to inactive accounts | 19:02 |
clarkb | I wonder if there is an option we can pass to say give me the inactive ones too | 19:02 |
clarkb | there is! | 19:05 |
fungi | magic | 19:08 |
clarkb | ya now I just have to figure out how to express +(is:inactive+OR+is:active) in python requests such taht gerrit will accept it | 19:12 |
clarkb | sending it through curl works just fine :/ | 19:12 |
clarkb | I wanted spaces not +'s | 19:27 |
clarkb | no idea why curl manages to make that work though | 19:28 |
clarkb | there are apparently still many that the rest api will only return oen account for, interesting | 19:30 |
clarkb | that must mean there is more than one reason for this behavior :( | 19:30 |
clarkb | fungi: ok. If I authenticate as a normal user I still only see the one account. If I authenticate as an admin I see both | 19:41 |
clarkb | I think I'm going to need to put this down for the day though and make sure other work is getting done too | 19:41 |
clarkb | but on monday I can update my script to authenticate | 19:41 |
clarkb | and rerun it and hopeflly get a complete picture | 19:41 |
fungi | yeah, strange but not especially surprising | 19:42 |
*** slaweq_ has joined #opendev | 19:44 | |
*** slaweq_ has quit IRC | 20:22 | |
openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Switch to default Sphinx theme https://review.opendev.org/c/opendev/git-review/+/777825 | 20:46 |
openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Overhaul Python package metadata and OpenDev URLs https://review.opendev.org/c/opendev/git-review/+/777826 | 20:46 |
clarkb | fungi: ^ those lgtm though the overhaul change failed python 3.9 testing (everything else was fine though) | 21:52 |
clarkb | I +2'd it as I figure that may be a recheck situation and improve the testing as we go item | 21:52 |
clarkb | infra-root I've found that the create_graphs.sh script seems to have run successfully for ze01.opendev.org as well as zm01.opendev.org but they don't show up in the lsit of hosts on the left side of the web ui | 22:01 |
clarkb | is there another step that needs to be take? | 22:01 |
clarkb | oh interesting if i go to the list view they seem to be there | 22:02 |
clarkb | of course now I can't figure out how to get the graphs for ze01.openstack.org on previous days to compare | 22:05 |
clarkb | looks like memory and cpu use may be better though we're using slight more swap | 22:09 |
clarkb | not seeing any crazy deltas which is good | 22:10 |
clarkb | I'm thinking the last thing to check up on is if afs using jobs are still happy | 22:10 |
clarkb | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_46b/05249aec2333ccad1a83d53cee7295a0784fb028/post/publish-openstack-python-branch-tarball/46b413b/job-output.txt that looks like one happy afs job | 22:12 |
clarkb | fungi: are there specific jobs that use afs in ways that are different enough we should check them or is one tarball job being happy sufficient evidence you think? | 22:12 |
clarkb | cacti for the mergers also looks fine in comparison | 22:16 |
fungi | clarkb: the failure is probably test timeouts, seems to have been hitting that randomly on some nodes even before we added the 3.9 job | 22:43 |
fungi | will take a look, thanks | 22:44 |
fungi | clarkb: i expect a tarball job and a docs publish to be relatively similar, nothing out of the ordinary in afs use between jobs afaik | 22:45 |
clarkb | cool in that case I think ze01 looks good | 22:49 |
clarkb | we can probably continue to replace servers next week in a rolling fashion. | 22:49 |
fungi | clarkb: yep, as suspected, TimeoutException: https://8f63d1ba12262ef08ef5-137d8ab1f6e795378a0b5ffa2abe6af6.ssl.cf5.rackcdn.com/777826/1/check/tox-py39/59b685c/testr_results.html | 23:05 |
fungi | clarkb: does this apply to all tests? https://opendev.org/opendev/git-review/src/branch/master/git_review/tests/__init__.py#L200 | 23:08 |
fungi | like can i just bump that from 120 to 300 and increase the timeout for all tests that way? | 23:08 |
fungi | most of git-review's functional tests share the same basic workflow, and so are equally prone to hit the timeout | 23:09 |
openstackgerrit | Clark Boylan proposed opendev/system-config master: Add tools being used to make sense of gerrit account inconsistencies https://review.opendev.org/c/opendev/system-config/+/777846 | 23:09 |
clarkb | fungi: it will apply to all tests that inherit from BaseGitReviewTestCase | 23:10 |
fungi | okay, i think that's all the ones which time out | 23:10 |
fungi | thanks | 23:10 |
clarkb | fungi: ^ if you get a chance can you look over that change? I've largely got things put together now I think and can run that as admin next week | 23:10 |
clarkb | would be curious to have input on whether or not that appraoch makes sense to you for identifying accounts that are completely dead | 23:11 |
fungi | looks like that timeout was added by https://review.openstack.org/71223 in 2014, so it's lasted a while | 23:11 |
openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Increase test timeout to 5 minutes https://review.opendev.org/c/opendev/git-review/+/777848 | 23:14 |
fungi | clarkb: first thought, we shouldn't care about inactive accounts which are in conflict... we should just strip out their ids, right? | 23:22 |
fungi | if the account is already inactive, then we clean those up first and see what's still conflicting afterward | 23:23 |
clarkb | fungi: ya thats sort of what I'm thinking. So my audit script should produce the list of those accounts, then we can write a separate script that deletes the external ids for each of those accounts | 23:23 |
clarkb | we can do that via the api | 23:24 |
fungi | we only care to try to fix accounts which aren't set inactive in gerrit, so if they're only conflicting with accounts set inactive then retiring the inactive ones may reduce the set dramatically | 23:24 |
clarkb | I have a todo in the audit script that it needs to distinguish between gerrit inactive and my concept of inactive due to idleness because I think that is a good first pass | 23:24 |
fungi | here i'm referring to "inactive" in the gerrit sense, not simply stale/unused for a while | 23:24 |
clarkb | yup | 23:24 |
clarkb | I don't know that it will be a dramatic reduction but it is a non zero set | 23:25 |
clarkb | I agree though that that would be a good first pass and updating the script to call out the gerrit concept of inactive is in there as a todo | 23:25 |
clarkb | basically we retire and delete external ids for any accounts that are gerrit inactive already | 23:26 |
clarkb | that should be entirely safe and can be done without gerrit downtime or mega push | 23:26 |
fungi | as for the query for push/review activity, why not use an "after" constraint in the changes query? | 23:26 |
clarkb | fungi: mostly that I didn't consider it :) I think that is a reasonable idea. I would probably still do the math directly though to double check gerrit isn't doing something weird | 23:27 |
fungi | you can say after:2020-02-26 or whatever and then you won't get any results for changes touched older that that | 23:27 |
clarkb | but that is a good way to simplify the input | 23:27 |
fungi | so much less to iterate through | 23:27 |
fungi | of course, just because the change was touched more recently than the after date doesn't mean that's activity from our candidate, but it will result in a lot less data to churn through | 23:28 |
clarkb | yes, that is actually an issue with the current setup. But I think it will get us close enough for now | 23:28 |
clarkb | "at least a year old" is what that means | 23:28 |
clarkb | we shouldn't end up being greedy | 23:28 |
clarkb | fungi: there are ~51 accounts that gerrit sees as properly inactive that we can just go and remove external ids for | 23:31 |
clarkb | which is just under 10% of the total number of conflicts | 23:31 |
clarkb | a decent chunk but not going to solve all our problems :) still I think that is a good suggestion for making progress and reducing the problem set so I'll get that done as the next thing | 23:32 |
fungi | yeah, so not substantial, but at lest it's that many fewer to have to ponder | 23:32 |
clarkb | I also mean to make this audit script do this activity checking on the preferred email has no external id cases to see if we can just turn them all off due to inactivity | 23:34 |
clarkb | thats another 20 or so | 23:34 |
fungi | sure | 23:39 |
clarkb | fungi: do you think ~1year is a decent metric for identifying aged out accounts? | 23:39 |
*** hamalq has quit IRC | 23:40 | |
fungi | it's a good starting point | 23:40 |
fungi | like i said in review comments. we can run it with different timeframes and compare the number | 23:40 |
clarkb | oh I haven't seen the comments yet /me refreshes | 23:41 |
fungi | it's possible most of these are inactive >4yr or something and so a longer period doesn't mean that many more we have to sort out | 23:41 |
clarkb | yup good point | 23:42 |
*** gothicserpent has joined #opendev | 23:50 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!