| clarkb | I've identified a setup where the user has one active accoutn and one inactive accoutn and I know they are not using either. I've selected this user to test retiring the inactive account, then delete the externalids from that account as it should be safe to do so | 00:03 |
|---|---|---|
| clarkb | then tomorrow maybe we can have someone like smcginnis volunteer to be a test case where one of the accounts is used | 00:03 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] handle zuul-summary-results as .jar / per-project config https://review.opendev.org/c/opendev/system-config/+/778116 | 00:10 |
| *** osmanlicilegi has joined #opendev | 00:15 | |
| *** osmanlicilegi has quit IRC | 00:15 | |
| *** osmanlicilegi has joined #opendev | 00:16 | |
| *** gibi has joined #opendev | 00:17 | |
| *** tosky has quit IRC | 00:17 | |
| clarkb | alright the two step cleanup of first removing the preferred email from refs/users/xy/abxy:account.config then doing a curl to post a delete for the corresponding external id has been completed. I'm going to run a consistency check again prod now to see if that makes the consistency issue go away ( we expect it to, this should confirm it ) | 00:19 |
| clarkb | now i just have to figure out how to run the consistency check again | 00:19 |
| clarkb | the consistency check shows that account is no longer present in the conflicts list | 00:28 |
| clarkb | I also had it check for preferred emails missing just in case and that looks clean too | 00:28 |
| clarkb | I think that means the process of "retiring" accounts then removing their external ids should be viable here | 00:29 |
| clarkb | I need to switch gears to getting tomorrow's meeting agenda out then start dinner, but maybe if smcginnis sees this note we can test this same process on accounts where at least one is being used | 00:29 |
| ianw | it looks like writing out /etc/openafs/server/rxkad.keytab is breaking with some sort of interaction between ansible/!!binary in yaml/focal/??? ... :/ | 00:33 |
| *** hamalq has quit IRC | 00:41 | |
| clarkb | there are about 35 accounts that we can cleanup directlyin this manner I think | 00:42 |
| clarkb | and then another few hundred that we'll probably clean up in this way due to inactivity | 00:42 |
| clarkb | then we can improve our analysis to handle the resulting set | 00:42 |
| clarkb | avass: you are also in this bucket so maybe we will use you as the test scenario :) | 00:43 |
| ianw | ok, we've clearly never tried writing out the rxkad keys; i guess we've never deployed a new server. i think i just need to remove the !!binary, that was from before we base64 encoded them | 00:50 |
| ianw | (or realised that !!binary got turned into a utf-8 string IIRC) | 00:50 |
| clarkb | and agenda is sent. Sorry for getting that out a bit late today | 00:56 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove afs-admin group https://review.opendev.org/c/opendev/system-config/+/778120 | 00:57 |
| *** iurygregory has quit IRC | 01:07 | |
| *** mlavalle has quit IRC | 01:14 | |
| *** hemanth_n has joined #opendev | 01:25 | |
| *** iurygregory has joined #opendev | 01:30 | |
| auristor | three vlservers running. afsdb02 is the coordinator | 01:32 |
| auristor | same for ptservers | 01:33 |
| ianw | auristor: oh good ... it looked like it was working :) | 01:34 |
| ianw | #status log afsdb03.openstack.org now online, SRV records added | 01:35 |
| openstackstatus | ianw: finished logging | 01:35 |
| ianw | i think it bodes well for upgrade-in-place for the afs01/afs02 servers | 01:37 |
| auristor | now that you have HA, you can take the risk | 01:38 |
| auristor | I'm not seeing the SRV record changes yet. | 01:39 |
| auristor | are openstack.org clients configured with dbservers from a local CellServDB or via DNS? | 01:39 |
| auristor | if CellServDB, the clients will not learn about afsdb03 until a new version is distributed and the clients are restarted | 01:42 |
| *** LowKey has quit IRC | 01:44 | |
| *** LowKey has joined #opendev | 01:45 | |
| ianw | auristor: yeah, i've corrected the -/_ typo in the srv records now :) | 02:47 |
| ianw | i guess we ship a full CellServDB | 02:55 |
| auristor | did you add a PTR record for 104.130.158.72 to afsdb03.openstack.org ? | 03:09 |
| auristor | I see the SRV records | 03:10 |
| ianw | hrm i have to log in separately to add reverse dns | 03:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: trim CellServDB https://review.opendev.org/c/opendev/system-config/+/778127 | 03:37 |
| auristor | PTR record is visible | 03:58 |
| *** ykarel|away has joined #opendev | 04:09 | |
| *** ykarel|away is now known as ykarel | 04:14 | |
| *** whoami-rajat_ has joined #opendev | 05:25 | |
| *** auristor has quit IRC | 05:35 | |
| *** brinzhang_ has quit IRC | 06:01 | |
| *** brinzhang_ has joined #opendev | 06:01 | |
| *** lbragstad_ has joined #opendev | 06:03 | |
| *** auristor has joined #opendev | 06:05 | |
| *** marios has joined #opendev | 06:05 | |
| *** lbragstad has quit IRC | 06:06 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: SSH access to refstack for kopecmartin https://review.opendev.org/c/opendev/system-config/+/778090 | 06:44 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 06:47 |
| *** whoami-rajat_ is now known as whoami-rajat | 06:54 | |
| *** sboyron has joined #opendev | 06:54 | |
| *** ralonsoh has joined #opendev | 06:55 | |
| *** slaweq has joined #opendev | 07:08 | |
| *** lpetrut has joined #opendev | 07:27 | |
| *** ykarel_ has joined #opendev | 07:27 | |
| *** ykarel has quit IRC | 07:30 | |
| *** eolivare has joined #opendev | 07:44 | |
| *** ykarel_ is now known as ykarel | 07:51 | |
| *** andrewbonney has joined #opendev | 07:58 | |
| *** fressi has joined #opendev | 08:13 | |
| *** rpittau|afk is now known as rpittau | 08:22 | |
| avass | clarkb: heh, yeah sure go ahead :) | 08:24 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 08:32 |
| *** tosky has joined #opendev | 08:35 | |
| *** zimmerry has quit IRC | 08:52 | |
| *** hashar has joined #opendev | 08:56 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** toomer has joined #opendev | 08:59 | |
| *** brinzhang0 has joined #opendev | 09:15 | |
| *** brinzhang_ has quit IRC | 09:18 | |
| openstackgerrit | Merged opendev/irc-meetings master: Remove usused Public Cloud SIG meeting slot https://review.opendev.org/c/opendev/irc-meetings/+/777191 | 09:18 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 09:22 |
| *** zoharm has joined #opendev | 10:06 | |
| *** LowKey has quit IRC | 10:51 | |
| *** klonn has joined #opendev | 12:26 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** klonn has quit IRC | 12:46 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: trigger image upload https://review.opendev.org/c/opendev/system-config/+/778187 | 12:53 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 12:54 |
| sshnaidm | is gerrit slow today for me only? | 13:22 |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** lbragstad_ is now known as lbragstad | 13:41 | |
| *** mlavalle has joined #opendev | 13:59 | |
| *** hemanth_n has quit IRC | 14:04 | |
| *** fdegir has joined #opendev | 14:19 | |
| ykarel | fungi, hi | 14:50 |
| ykarel | fungi, can u check lists.openstack.org/pipermail/openstack-discuss/2021-March/020804.html | 14:51 |
| ykarel | i missed ur mail long ago :) | 14:51 |
| fungi | sure, just a few, getting ready to dial into the board meeting | 14:51 |
| ykarel | ack take ur time | 14:51 |
| ykarel | i cleaned up open reviews on that, there are still some open but those can also be closed if we get confirmation on branch deletion | 14:52 |
| ykarel | like https://review.opendev.org/c/openstack/instack-undercloud/+/777370 | 14:52 |
| smcginnis | clarkb: Still need to test something? | 14:57 |
| *** roman_g has joined #opendev | 15:04 | |
| roman_g | Hello team. Sorry to bother you. Still with CityCloud/Kna1. Is there any chance you keep NODE_FAILURE logs from 2021-02-16 - 2021-02-25? Could have a look? | 15:09 |
| fungi | roman_g: i'll see what i can put together | 15:09 |
| fungi | unfortunately the backtraces are multiline and not prefixed, so if we want those included too it's not just a simple grep | 15:10 |
| roman_g | The CityCloud also says that they saw "Image is not raw format" errors, probably from us. Image ID is c0e9524d-506c-467b-bed5-5fcdec309280. | 15:11 |
| roman_g | Might be also something to look at. | 15:12 |
| roman_g | Thank you, fungi. | 15:12 |
| fungi | roman_g: oldest logs we still have are from 2021-02-20, will that suffice? | 15:13 |
| roman_g | Sure. | 15:14 |
| *** ykarel has quit IRC | 15:17 | |
| fungi | roman_g: interestingly i don't think we're seeing the "no valid host" errors we had been seeing (still trying to confirm i'm not just looking at it wrong), but seeing quite a few "Unable to find public IP of server" errors | 15:18 |
| openstackgerrit | Merged openstack/project-config master: Add tripleo-ci-health-queries project https://review.opendev.org/c/openstack/project-config/+/777991 | 15:18 |
| fungi | more on some days than others, something like 10-30 a day | 15:19 |
| fungi | depending on the day | 15:19 |
| roman_g | Does it seem to be the reason why instances were not launching? No more IPs left in pool/quota? | 15:20 |
| fungi | nothing so straightforward, the metadata returned from the api is missing a public ip address | 15:20 |
| fungi | also sorry i'm trying to pay attention to the foundation board of directors meeting | 15:21 |
| fungi | so slightly distracted | 15:21 |
| fungi | looks like we're also seeing errors when attempting to delete floating ips there | 15:22 |
| fungi | also timeouts waiting for server deletion calls to complete | 15:24 |
| fungi | the image uuid you mentioned earlier isn't in our image list currently, so may have been from a truncated upload or something. did they say when they saw that? | 15:27 |
| roman_g | 2021-02-16 23:11:47 | 15:27 |
| fungi | ahh, yeah that's too far back for us to still have logged | 15:27 |
| toomer | roman_g: It's good to know that CityCloud come back to you on those issues. | 15:28 |
| roman_g | fungi list of NODE_FAILUREs https://paste.ubuntu.com/p/9MHZ4Wx7PC/ | 15:30 |
| roman_g | UTC time | 15:30 |
| roman_g | That's from Zuul | 15:30 |
| clarkb | smcginnis: yes, you have two gerrit accounts. One is inactive and I'd like to make it even more inactive then ahve you confirm your active (and currently used) account remains happy | 15:31 |
| fungi | roman_g: thanks, i can try to trace some of those back but it will take a bit, since i'll need to map them to node requests and from there to actual api calls | 15:31 |
| clarkb | smcginnis: I don't expect any issues since the other account has been marked inactive and shouldn't be used, but if we can have you confirm that after we do some work that would be excellent | 15:31 |
| smcginnis | clarkb: Sure, that should be fine. Board meeting right now, but I can verify my good account is still good afterwards. | 15:32 |
| roman_g | fungi thank you. Can I get back to you with this tomorrow? Save logs somewhere, please. | 15:32 |
| clarkb | smcginnis: ya I've got to finish booting my morning too so will be a bit before I get to it | 15:32 |
| smcginnis | +1 | 15:32 |
| fungi | roman_g: yes, i'm working on filtering the logs down first, the launcher which handles that cloud also connects to lots of others so i need to untangle the debug log entries from it | 15:33 |
| roman_g | OK. | 15:33 |
| clarkb | roman_g: fungi note https://opendev.org/opendev/system-config/src/branch/master/playbooks/templates/clouds/nodepool_clouds.yaml.j2#L158 is what will determine what file format we upload as | 15:33 |
| clarkb | I don't know if that profile sets it to raw or qcow2 etc. We can override it if necssary but should also fix that in sdk if necessary | 15:34 |
| *** lpetrut has quit IRC | 15:34 | |
| fungi | yeah, it will be buried somewhere in openstacksdk's defaults i think | 15:34 |
| fungi | but since they just mentioned one error about that 14 days ago, i expect it was a failed upload | 15:35 |
| clarkb | https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/config/vendors/citycloud.json doesn't seem to set it at all | 15:35 |
| *** openstackgerrit has quit IRC | 15:35 | |
| fungi | wow, one compressed scheduler debug log is going to take 15 minutes for me to retrieve to my workstation to analyze | 15:53 |
| clarkb | infra-root I'm going to run the "retire-user.sh" script against smcginnis' already inactive account | 16:00 |
| clarkb | then I'll remove the external ids for that account that conflict with the email address | 16:00 |
| smcginnis | Do I get a pension? | 16:00 |
| clarkb | smcginnis: unfortunately not | 16:01 |
| clarkb | smcginnis: fwiw I've called it "retire" because its more forceful that simply setting the account inactive. Apparently gerrit will still check all sorts of things on inactive accounts you'd prefer it didn't and just treat the entire account as inactive instead :( | 16:01 |
| smcginnis | :) | 16:02 |
| clarkb | cool that first step is done. Now to sort out the next step and remove the external ids associated with that account | 16:03 |
| clarkb | smcginnis: ok I'm all done on my side. If you want to double check that logging out and into gerrit etc are still working as normal that would be good | 16:08 |
| clarkb | I only modified the inactive account so don't expect any problems, but trying to be careful here | 16:08 |
| smcginnis | clarkb: ack, will test things out in a bit and report back. | 16:10 |
| fungi | ugh, the one day worth of scheduler debug log i'm trying to hunt in is ~80m lines long | 16:14 |
| fungi | even just slicing an hour out of it is taking a while | 16:17 |
| fungi | okay, that's better, only 5m lines | 16:20 |
| fungi | still kinda insane that we're logging 1k lines per second on average over a 24 hour period from just this one service | 16:21 |
| clarkb | infra-root there are about 35 more accounts similar to smcginnis' where at least one of the accounts is already marked inactive. I'd like to run the retire.sh script against those. Then figure out how to automate the external id deletion via the api (so far I've just done curl and manual json edits) | 16:29 |
| clarkb | any objections to me running retire-user.sh against those now then sorting out the external id cleanups? | 16:30 |
| fungi | no objection, sounds great--thanks! | 16:30 |
| clarkb | oh I should note that smcginnis' PM'd me where I gave some account details to check and said things looked fine | 16:31 |
| clarkb | great, I'll proceed then | 16:31 |
| fungi | yeah, i figured there had been some coordination | 16:31 |
| *** dviroel has quit IRC | 16:34 | |
| avass | fungi: yeah those are fun to dig through :) | 16:36 |
| fungi | trying to map a NODE_FAILURE result back to a node request | 16:39 |
| *** sshnaidm is now known as sshnaidm|afk | 17:00 | |
| *** marios is now known as marios|out | 17:01 | |
| *** zoharm has quit IRC | 17:04 | |
| *** marios|out has quit IRC | 17:06 | |
| roman_g | fungi Big Data processing :) | 17:13 |
| roman_g | egrep/sed/awk? | 17:14 |
| fungi | indeed. i mapped one node_failure report to an event id, now seeing if i can find the node requests associated with it | 17:14 |
| fungi | luckily the triggering event id only appears in a little over 10k lines, so shouldn't be too hard | 17:15 |
| *** openstackgerrit has joined #opendev | 17:16 | |
| openstackgerrit | Merged opendev/git-review master: Don't test rebasing with unstaged changes https://review.opendev.org/c/opendev/git-review/+/777456 | 17:16 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add tools being used to make sense of gerrit account inconsistencies https://review.opendev.org/c/opendev/system-config/+/777846 | 17:19 |
| clarkb | infra-root ^ that ps adds a script that reads a list of emails and accounts for which to delete external ids for that email under that account | 17:20 |
| clarkb | if that looks good to you I'll run it against the batch of accounts I retired | 17:20 |
| avass | fungi: regarding that, I'm not sure but I think the node request is part of the message and not the logged data like build and event_id is. Which I guess you don't really notice but when logging that as a key:value store to splunk or I guess elastic it would probably be nice if that was more searchable | 17:21 |
| avass | same problem with buildset in the executor I believe. might take a look at that some day | 17:21 |
| fungi | avass: yeah, i've found the list of node requests submitted for a particular event and am cross-checking against the list of node requests completed | 17:22 |
| fungi | then i'll at least know the node request to track down | 17:22 |
| *** lpetrut has joined #opendev | 17:28 | |
| openstackgerrit | Sorin Sbârnea proposed opendev/git-review master: Fixes test system dependencies https://review.opendev.org/c/opendev/git-review/+/772699 | 17:32 |
| fungi | huh, so the node request corresponding to the node_failure result does get logged as completed, but with "Node None" | 17:33 |
| fungi | so i guess that's the pattern to look for | 17:33 |
| avass | fungi: is that the problem that corvus was looking at with dependent changes or change queue a week or so ago? | 17:34 |
| fungi | also i now know the request id so can actually find that in the launcher log | 17:34 |
| fungi | avass: nah, i think maybe it's just how this gets logged. i'm still digging for a cause | 17:34 |
| fungi | not far enough along to have any theory as of yet | 17:35 |
| fungi | roman_g: okay, on the -2 for patchset 3 of https://review.opendev.org/776956 the node request was declined "because node type(s) [ubuntu-bionic-expanded] not available" | 17:37 |
| fungi | now to see if i can figure out why that was | 17:37 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add tools being used to make sense of gerrit account inconsistencies https://review.opendev.org/c/opendev/system-config/+/777846 | 17:38 |
| clarkb | infra-root ^ added another check to the bulk external id remover. Now it should only do that for accounts which are inactive in gerrit | 17:38 |
| roman_g | fungi "not available" is quite strange. It doesn't say why is it not available. | 17:40 |
| zbr | can gerrit 2.13 work with java 11 (at least for testing purposes) | 17:40 |
| fungi | roman_g: it probably does, but i haven't gotten that far yet | 17:40 |
| clarkb | zbr: I don't think so | 17:40 |
| clarkb | zbr: they only just added java 11 support in 3.2 aiui | 17:40 |
| zbr | thanks! i was trying to compute the test deps :p | 17:41 |
| clarkb | fungi: the kna1 provider is the only provider providing that label currently. If you grep the request id in nl02's launch debug logs it should tell you why | 17:41 |
| fungi | roman_g: nodepool.exceptions.ConnectionTimeoutException: Timeout waiting for connection to 188.95.227.31 on port 22 | 17:41 |
| clarkb | my hunch is it failed to boot 3 times in a row | 17:41 |
| clarkb | something something cloud redundancy and this is why we push so hard on the subject | 17:42 |
| fungi | roman_g: so looks like we gave up waiting for the booted node to respond on port 22 to gather the host keys from it | 17:42 |
| *** rpittau is now known as rpittau|afk | 17:42 | |
| roman_g | Ahha... Now we need to understand if it's the same problem with other launch failing attempts, right? | 17:43 |
| fungi | roman_g: and then tried two more times, with new nodes each time, before we gave up | 17:43 |
| roman_g | 3 times in a row, or with a delay between tries? | 17:43 |
| clarkb | roman_g: the "wait for boot/ssh/etc" is effectively a delay between tries | 17:43 |
| clarkb | but I Think that is the only real delay in play there | 17:44 |
| fungi | yeah, basically for this particular example, the one provider which offers that node type accepted the node request, tried three times to boot a node to fulfil it, each time it waited some minutes for ssh to become responsive, then gave up and booted another. once it failed that way three times in a row, it told zuul it could not fulfil the node request (by returning a "completed" result for the request with | 17:44 |
| fungi | a node of None) | 17:44 |
| fungi | once the server instance is reported ready by the nova api, the launcher starts trying to connect via ssh to gather host keys from it, and that's roughly 120 seconds | 17:46 |
| fungi | so i looks like the nodes weren't responding on their ssh ports for at least 120 seconds after they were reported to be in a ready state by the cloud. possible they were just very slow booting, i suppose, or that there was some network connectivity problem reaching them | 17:47 |
| clarkb | those servers use FIPs. Could be anything from neutron's NAT to glean | 17:47 |
| fungi | yeah, like their floating ips weren't forwarding, or maybe the hypervisor hosts to which they were scheduled weren't connected to the network | 17:48 |
| fungi | anyway, from this i can get a list of times and server instance uuids we can give to citynetwork | 17:48 |
| fungi | the fact that we try three times to get a reachable node before giving up suggests the ones reported as node_failure results are going to be the tip of the iceberg, since lots more requests probably succeeded on the second or third try | 17:49 |
| fungi | though parsing these out of the launcher log will still be a bit of a challenge since the data (instance uuid and failure exception) is logged on different lines without any common key to tie them together | 17:52 |
| openstackgerrit | Merged opendev/git-review master: Switch to default Sphinx theme https://review.opendev.org/c/opendev/git-review/+/777825 | 17:52 |
| openstackgerrit | Merged opendev/git-review master: Overhaul Python package metadata and OpenDev URLs https://review.opendev.org/c/opendev/git-review/+/777826 | 17:53 |
| *** jpena is now known as jpena|off | 18:02 | |
| clarkb | zbr: do you know what these tripleo ci ruck rover gerrit accounts are used for? they show up in our list of problem accounts | 18:02 |
| clarkb | they don't appear to have done any reviews or code pushes in the last year so are currently in our bucket of "disable and move on". But I wonder if there is some sort of interesting useage pattern there we need to account for instead | 18:03 |
| *** lpetrut has quit IRC | 18:07 | |
| clarkb | mwhahaha weshay|ruck ^ you may know as well | 18:08 |
| weshay|ruck | clarkb, hrm.. let me check our scripts... I suspect it's just pulling readonly info... that's all I can think of.. Mind if I go dig for a bit? | 18:10 |
| clarkb | weshay|ruck: ya it isn't a rush. We're currently dealing with the accounts that we're much more confident in modifying :) | 18:10 |
| clarkb | weshay|ruck: I'm just waiting for some reviews though so started skimming the list of ones we're less sure of and that stood out to me as a possible case where we aren't capturing a valid use case properly | 18:11 |
| weshay|ruck | k.. we may be using that account in more than the one place I'm thinking of | 18:12 |
| clarkb | if you can describe how it is used we can try to better account for it in our auditing of the accounts and subsequent classification of them | 18:13 |
| clarkb | currently we're looking at reviews and pushed changes in the last year as a starting point | 18:14 |
| roman_g | fungi, I would appreciate a list of times and server instance uuids, so that we can forward it to citynetwork | 18:19 |
| zbr | clarkb: sadly, even with latest changes running the test on fedora seems to generate lots of failure. looking at htop, i believe it attempts to run one gerrit for each test. | 18:19 |
| roman_g | We are getting issues quite often, and they are usually slow to respond. Still need to find a root cause, and get it resolved on their side. | 18:20 |
| fungi | roman_g: yep, nearly there. i've got ~700 in the past 10 days, just massaging the analysis to present the time the boot call returned, time we saw the instance go ready and time we gave up waiting for ssh to be reachable | 18:20 |
| zbr | weshay|ruck: ^ | 18:21 |
| *** hashar has quit IRC | 18:21 | |
| roman_g | fungi save your method to a script somewhere. Might be needed again later. | 18:23 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove ze01.openstack.org https://review.opendev.org/c/opendev/system-config/+/778227 | 18:23 |
| roman_g | fungi I'd love CityCloud to respond in realtime like you are doing, so that next time they could debug failing instance launch attempts on the go | 18:23 |
| clarkb | zbr: yes, the current git-review runs a gerrit per test, but out of a prepped work dir | 18:24 |
| clarkb | roman_g: you may want to consider using our normally sized instances then you can run across all the clouds | 18:24 |
| clarkb | roman_g: that will give you far more redundancy | 18:24 |
| clarkb | infra-root https://review.opendev.org/c/opendev/system-config/+/778227 is the next step in zuul-executor replacements. Basically want to get the now unused server complete out of the way then start replacing 2-3 at a time | 18:25 |
| roman_g | clarkb I know. I would love to. Devs say they strictly need possibility to launch a few bare nested VMs of a significant size and with a good performance, and boot them, and join into a cluster and execute workloads on those nested VMs. ¯\_(ツ)_/¯ | 18:27 |
| *** ralonsoh has quit IRC | 18:29 | |
| roman_g | clarkb we have moved all tests to standard 8GB RAM instances, and only have left one end-to-end test on Gate with 16GB instance with nested VMs. It gets executed on merge only. | 18:29 |
| clarkb | roman_g: sure, usually those tasks can be broken up such that you could for example run a cluster and boot a VM now you know that works. Next you test your workload that you want to run in VMs without the overhead of nested virt (or lack of virt). And that gives you very similar amounts of coverage | 18:30 |
| clarkb | (there are other factors to consider like how are devs expected to run or reproduce tests locally if the tests require significant resources, testable software tends to get tested more often and as a consequence is more robust, etc) | 18:35 |
| clarkb | I'm not going to claim we're perfect at it either :) just pointing out that as a goal it tends to be a good one | 18:36 |
| *** andrewbonney has quit IRC | 18:38 | |
| *** hamalq has joined #opendev | 18:38 | |
| fungi | roman_g: this is the basic list by the way, though only has the times the nova api responded with details for the instance it was booting: http://paste.openstack.org/show/803152/ | 18:42 |
| fungi | i'm still fumbling to get the times the api reported the instance was ready and teh times we gave up waiting to be able to ssh into it | 18:43 |
| fungi | er, fumbling to include them (i've got the patterns to grep on, just nitegrating them into the loop is finicky) | 18:43 |
| clarkb | fungi: the timeout value is a constant and should be pretty consistent | 18:44 |
| fungi | yep, just figured i'd include it because i can, the reported ready time is more important to include though | 18:45 |
| roman_g | fungi that is a list of instances which were not reachable up on launch? | 18:45 |
| fungi | roman_g: yes | 18:47 |
| fungi | and the dates/times the boot requests for them were made | 18:48 |
| roman_g | We don't get a lot of this kind of failures now, do we? https://grafana.opendev.org/d/QQzTp6EGz/nodepool-airship-citycloud?orgId=1 | 18:51 |
| fungi | roman_g: nope, last one i saw in the log was a few days ago | 18:52 |
| fungi | so maybe they've done something to address that issue | 18:52 |
| clarkb | note those graphs will be for all labels so can be hard to see something lable specific, but there are no errors marked there at all | 18:52 |
| fungi | the problem may not be label-specific either. the other labels can be tried in additional providers | 18:53 |
| fungi | this particular label though, if that provider can't boot it, then that's it | 18:53 |
| clarkb | good point | 18:54 |
| clarkb | it is a label specific user facing impact but may have happened for all the labels there | 18:54 |
| roman_g | From the logs here http://paste.openstack.org/show/803152/ I would guess that this is for all labels (i.e. including 8GB), as we don't launch that many 16/32GB instances. | 18:55 |
| roman_g | I'm quite surprised to see so many instances with problems with networking (instance launched but not accessible). | 18:56 |
| roman_g | Any additional Information you would want to pass to CityCloud? | 18:56 |
| fungi | roman_g: well, also remember we retry three times, so only a fraction of those are final tries | 18:57 |
| roman_g | What would help them to debug it on their side? | 18:57 |
| fungi | roman_g: that was just a preliminary list, i'm trying to combine the ready times and also when we gave up waiting | 18:57 |
| roman_g | fungi OK. | 18:57 |
| fungi | i doubt there's much more besides instance uuids they'd need, since their logs will tell them the hosts where those booted, the ip addresses assigned to them, et cetera | 18:58 |
| fungi | i'm including the times mostly to give them an idea for what vintage of logs they should be looking in | 18:59 |
| *** toomer has quit IRC | 19:08 | |
| *** eolivare has quit IRC | 19:33 | |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Adjust categories for some release notes https://review.opendev.org/c/opendev/git-review/+/778257 | 19:52 |
| clarkb | ianw: fungi: if you can look over the external id remove script in https://review.opendev.org/c/opendev/system-config/+/777846 that would be great. I think my greatest concern is somehow accidentally removing external ids from an active account if I get the input file wrong | 20:01 |
| clarkb | there are guards against that in the code, but let me know if you think tehy can be stronger | 20:02 |
| clarkb | I need to eat lunch now, but will try to sit back down again to work through those (and the script if if needs updates) | 20:03 |
| fungi | roman_g: http://paste.openstack.org/show/803159/ and http://paste.openstack.org/show/803160/ | 20:10 |
| fungi | i gave up trying to cram the detail into one line, and had to split it since the full set was more than the paste server would allow | 20:11 |
| fungi | roman_g: also that's strictly limited to nodes we made three attempts to boot. turns out the way we log nodes which succeed on the second or third try makes it hard to confirm with certainty that it was for that particular provider | 20:12 |
| fungi | roman_g: there are a handful where instead of timing out waiting for ssh, the boot request itself failed. there were only a handful, but also we don't get much useful info back when it happens either: | 20:16 |
| fungi | openstack.exceptions.SDKException: Error in creating the server (no further information available) | 20:16 |
| fungi | basically the nova api just reports an "error" status instead of the instance going to "active" | 20:17 |
| openstackgerrit | Merged opendev/system-config master: refstack: trigger image upload https://review.opendev.org/c/opendev/system-config/+/778187 | 20:17 |
| fungi | roman_g: those cases you can recognize because we never start gathering host keys, there's just a ~10-minute delay between waiting for server and failure | 20:18 |
| fungi | i'm guessing that's a timeout in nova on their end | 20:19 |
| fungi | or something else nova's waiting on | 20:19 |
| *** hashar has joined #opendev | 20:30 | |
| openstackgerrit | Merged opendev/git-review master: Adjust categories for some release notes https://review.opendev.org/c/opendev/git-review/+/778257 | 20:38 |
| *** dviroel has joined #opendev | 20:55 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use upstream jitsi-meet web image https://review.opendev.org/c/opendev/system-config/+/778308 | 21:00 |
| corvus | clarkb, fungi: ^ | 21:01 |
| fungi | thanks! | 21:04 |
| *** bhagyashri|rover has quit IRC | 21:06 | |
| *** bhagyashris has joined #opendev | 21:07 | |
| *** sboyron has quit IRC | 21:30 | |
| clarkb | ianw: thanks for the review. I think it was mostly beacuse handling the output for account details is a bit more consistent with the other output (its all json) | 21:33 |
| clarkb | ianw: but could update it to use that other url request instead. | 21:34 |
| ianw | that was my only thought, i mean otherwise it lgtm | 21:34 |
| clarkb | the current issue is I've discovered with the help of fungi that I think the : in external.ids:delete is being percent encoded by requests and gerrit wants a literal : | 21:34 |
| clarkb | so I'm about to go deep into the bowels of requests to see how I can have it stop doing that | 21:35 |
| clarkb | (is it bad that I considered forking out to curl?) | 21:35 |
| clarkb | corvus: thanks! looks like zuul isn't happy with it for some reason but the chagne on quick skim lgtm | 21:36 |
| clarkb | anyway, into the python requests depths | 21:36 |
| clarkb | ianw: fungi helped me find another bug too where we can have non unique keys as email or account id so I don't want to flatten them into a dict and instead just iterate over them one by one | 21:37 |
| clarkb | I'll get things pushed up once I figure out the : thing | 21:37 |
| clarkb | https://github.com/psf/requests/issues/1454#issuecomment-20832874 this comment says I get to create request objects from scratch :) | 21:44 |
| *** brinzhang0 has quit IRC | 21:47 | |
| clarkb | hrm doing testing against a local "server" maybe this isn't the issue. I see a literal : afterall | 21:51 |
| clarkb | now I'm really confused because apache logs for gerrit when we were trying this also seem to show we're getting the : as expected | 21:58 |
| clarkb | gerrit is returning "unauthorized" and I thought it was because the url I was hitting was just not valid | 21:58 |
| clarkb | earlier today I did smcginnis' external id removals using curl to the same POST location | 21:58 |
| clarkb | and I was authorized then | 21:58 |
| clarkb | I can also fetch external id info which has a similar level of authorization aiui | 21:59 |
| clarkb | that makes me think taht authentication isn't the problem | 21:59 |
| clarkb | I guess I'm going to test this via curl again to double check | 22:00 |
| clarkb | confirmed using curl continues to work | 22:06 |
| clarkb | hahahahaha I think I figured it out | 22:06 |
| clarkb | I wasn't passing auth info to the post :( | 22:06 |
| hashar | clarkb: but you figured it out! :-] | 22:09 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add tools being used to make sense of gerrit account inconsistencies https://review.opendev.org/c/opendev/system-config/+/777846 | 22:10 |
| clarkb | hashar: ya, but also I think that is a good indication I should pick this up tomorrow | 22:10 |
| clarkb | I've been working with fungi on this a bit and so fungi has bits paged in, I'll see if fungi wants to be an extension to my brain later and continue otherwise pick it up in the morning when hopeflly my brain is working better | 22:11 |
| hashar | oh fighting with Gerrit emails and external-ids :\ | 22:12 |
| hashar | I feel your pain, we had a few issues with it in a past version of Gerrit | 22:12 |
| clarkb | hashar: yup, we have ~640 accounts with email conflicts. Getting through the first set of 35 fixes now | 22:12 |
| clarkb | er its actually 640 emails with account conflicts | 22:13 |
| clarkb | the total number of accounts is at least twice that | 22:13 |
| hashar | ouch :-\ | 22:13 |
| clarkb | hashar: trying to whittle it down bit by bit | 22:13 |
| clarkb | anyway ianw fungi https://review.opendev.org/c/opendev/system-config/+/777846 has the auth passed in now as well as the other improvements I mentioned | 22:13 |
| clarkb | I'm going to switch gears shortly and get some other stuff done that having a tired brain is less important for | 22:14 |
| roman_g | fungi Thank you for additional logs! I will email them to CityCloud tomorrow. Will get back to you if I hear anything from them. | 22:14 |
| hashar | clarkb: +1 on using the Gerrit web API. We went messing up directly with the All-Users.git which was not that great. Then we only had to fix a few accounts | 22:17 |
| hashar | clarkb: good luck with less important stuff! | 22:17 |
| clarkb | hashar: ya the nice thing about the api is we can make changes a few accounts at a time without downtime | 22:18 |
| *** roman_g has quit IRC | 22:19 | |
| clarkb | hashar: which means we can make the changes in incremental steps as we are comfortable with them which I like | 22:19 |
| clarkb | hashar: we also think we've found a bug around openid and users changing their email addresses in gerrit. I still need to report that :/ | 22:21 |
| hashar | clarkb: :\ at least if you write a nicely detailed report upstream is usually fast to react. Though for openid I don't know how well maintained it is | 22:28 |
| openstackgerrit | Merged opendev/system-config master: Remove afs-admin group https://review.opendev.org/c/opendev/system-config/+/778120 | 22:40 |
| openstackgerrit | Merged opendev/system-config master: openafs-client: trim CellServDB https://review.opendev.org/c/opendev/system-config/+/778127 | 22:57 |
| clarkb | the stevedore caching is creating a sha256 hash of all the entrypoint files in egg info and distinfo dirs and their mtimes | 22:58 |
| clarkb | given the significant number of the resulting sha256 key'd cache files we haev I'm wondering if it isn't doing that in a stable manner | 22:58 |
| *** gnuoy has quit IRC | 23:04 | |
| ianw | clarkb: yeah i was just having a poke | 23:06 |
| ianw | if you run them through json_pp | 23:06 |
| ianw | "path_values" : [ | 23:06 |
| ianw | > [ | 23:06 |
| ianw | > "/tmp/ansible_os_keypair_payload_qstnh5m0/ansible_os | 23:06 |
| ianw | i think that's the reason the hash is unstable | 23:07 |
| clarkb | oh its not supposed to try and cache things in tmp bceause ansible does that | 23:07 |
| clarkb | maybe our stevedore is too old then? | 23:07 |
| ianw | yeah, but only on 3.3 which i doubt anything has pulled in | 23:07 |
| clarkb | fwiw the glob.iglob seems stable (though the sort order isn't alnum) | 23:07 |
| ianw | yep; it's definitely the cloud launcher bits. that's failing on openedge, i thought we had removed that? | 23:08 |
| clarkb | maybe we failed to remove it from the cloud launcher | 23:08 |
| *** hashar has quit IRC | 23:10 | |
| fungi | after going back over the release notes for git-review 2.0.0.0rc1 i don't think there's a significant need to announce the release candidate and ask people to test: https://docs.opendev.org/opendev/git-review/latest/releasenotes.html (i did also previously go through the list of changes and make sure everything relevant had a release note) | 23:16 |
| fungi | between having personally used it a bit, and the extensive functional regression testing we do for it in zuul, it should at least be nominally usable | 23:17 |
| fungi | the major version bump is mostly about things we removed (especially python 2.7 support) | 23:18 |
| clarkb | fungi: if you've been using it then ya I think you are probably good | 23:19 |
| clarkb | people can always downgrade if necessary | 23:19 |
| clarkb | and we can always do a 2.0.1 if necessary as well | 23:19 |
| fungi | yep, and probably will ;) | 23:20 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] handle zuul-summary-results as .jar / per-project config https://review.opendev.org/c/opendev/system-config/+/778116 | 23:39 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!