| ianw | clarkb: does https://zuul.opendev.org/t/openstack/build/d341689bf3ce49068c98a0ca2a2e8056/console all line in the results tab with your hires screen? | 00:00 |
|---|---|---|
| clarkb | ianw: right, just wondering if the plan is to force merge it. It did sound like fungi was ok with doing so and as we mentioned no artifacts should need promoting? | 00:00 |
| ianw | i think we might need to limit the width of the node name a bit more | 00:01 |
| clarkb | ianw: yes they all seem to be aligned | 00:01 |
| ianw | yeah i see https://imgur.com/a/SnCOWRC | 00:02 |
| corvus | clarkb: do you know why my messages are getting to irc (i assume they are) while yours aren't? | 00:05 |
| clarkb | corvus: I think because you are actually joined to the irc channel but I'm not | 00:06 |
| clarkb | I don't know why you are joined and I'm not though | 00:06 |
| clarkb | we both dropped out at roughly the same time | 00:07 |
| dasm | clarkb: corvus i can see your messages on irc (if that helps) | 00:07 |
| clarkb | then you rejoined at 15:58 but I haven't | 00:08 |
| clarkb | dasm: ya this is my IRC connection. My matrix one isn't in the channel nor do I see the messages I send to it via my irc connection | 00:08 |
| ianw | anyone mind if i restart the zuul-web container to pick up https://review.opendev.org/c/zuul/zuul/+/858250/ ? i'd like to play with it an the offsets from above | 00:09 |
| corvus | clarkb: apparently i reconnected just before 9am today | 00:10 |
| clarkb | ianw: I think that should be safe. I guess that landed after our restarts completed? | 00:10 |
| corvus | ianw: i think a rolling restart of both containers is fine; remember they take a few minutes to start. | 00:10 |
| corvus | (so be sure to check the component status page) | 00:10 |
| ianw | thanks | 00:11 |
| clarkb | corvus: I've tried to provide more details to #irc:matrix.org to see if they have any other insight | 00:12 |
| clarkb | I tried using !join #openstack-nova to join an entirely new channel with no luck (element gets the invite to the room from oftc-irc and I can join as far as element is concerned but I don't show up in the user list of the channel on the irc side of things | 00:15 |
| clarkb | I smell dinner. I need to pop out now. But I'll check to see if anything has changed in the morning | 00:16 |
| corvus | clarkb: maybe something with your nick? maybe you could change it and change back? just throwing out ideas | 00:20 |
| corvus | bon appetit | 00:20 |
| NeilHanlon | clarkb: it's a breakage on my side as I maintain the mirrors.. our mirrormanager software is supposed to cull mirrors which are serving bad content and/or are inaccessible, after failing some amount of syncs.. but for some reason it doesn't appear to be catching/fixing this situation | 00:42 |
| opendevreview | Merged opendev/system-config master: bootstrap-bridge: drop pip3 role, add venv https://review.opendev.org/c/opendev/system-config/+/856593 | 01:15 |
| *** rlandy is now known as rlandy|out | 01:25 | |
| ianw | so the bridge bootstrap failed -- "refusing to convert from file to symlink for /usr/local/bin/ansible" | 02:30 |
| ianw | however, it did redirect /usr/local/bin/ansible-playbook to the venv installed ansible | 02:30 |
| ianw | i think probably the clearest thing to do here is for me to manually run "pip uninstall ansible" on bridge; that will remove the global ansible pip install and the next bootstrap run should then be able to link it | 02:31 |
| ianw | the gate is ok, because it just calls "ansible-playbook" anyway | 02:32 |
| ianw | infra-prod-bootstrap-bridge should re-run out of the periodic jobs soon | 02:41 |
| *** diablo_rojo is now known as Guest2868 | 03:22 | |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: setup-keys: add bridge node to "bastion" group https://review.opendev.org/c/opendev/base-jobs/+/861026 | 03:23 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run jobs with a jammy bridge.openstack.org https://review.opendev.org/c/opendev/system-config/+/857799 | 03:57 |
| opendevreview | Ian Wienand proposed opendev/system-config master: testinfra: Update selenium calls https://review.opendev.org/c/opendev/system-config/+/858003 | 03:57 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Abstract name of bastion host for testing path https://review.opendev.org/c/opendev/system-config/+/858476 | 03:57 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Convert production playbooks to bastion host group https://review.opendev.org/c/opendev/system-config/+/858486 | 03:57 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run a base test against "old" bridge https://review.opendev.org/c/opendev/system-config/+/860802 | 03:57 |
| opendevreview | Ian Wienand proposed opendev/system-config master: bootstrap-bridge: use abstracted hostname https://review.opendev.org/c/opendev/system-config/+/861031 | 03:57 |
| ianw | ok, bootstrap bridge ran ok -> https://zuul.opendev.org/t/openstack/build/bf11d099adbb43039794ce84818d2759 | 04:01 |
| *** dasm is now known as dasm|off | 04:04 | |
| ianw | i think that stack should pass, and i'm now running out of places i think it might be broken too :) | 04:08 |
| opendevreview | Dr. Jens Harbott proposed opendev/base-jobs master: Drop ara related vars from the base jobs https://review.opendev.org/c/opendev/base-jobs/+/860693 | 04:41 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Switch the requirements-constraints job to py310 https://review.opendev.org/c/openstack/project-config/+/861035 | 05:25 |
| *** ysandeep|out is now known as ysandeep | 05:32 | |
| opendevreview | Tony Breeds proposed openstack/project-config master: Switch the requirements-constraints job to py310 https://review.opendev.org/c/openstack/project-config/+/861035 | 05:38 |
| *** luigi is now known as luigi-out | 05:42 | |
| *** pojadhav is now known as pojadhav|afk | 06:35 | |
| ramishra | bshephar: commented in the patch, I think there is one more place where it could be changed, though output_dir thing is kind of messy atm | 06:40 |
| ramishra | oops wrong channel:/ | 06:43 |
| *** pojadhav|afk is now known as pojadhav | 07:44 | |
| opendevreview | gnuoy proposed openstack/project-config master: Add project for managing zuul jobs for charms https://review.opendev.org/c/openstack/project-config/+/861046 | 08:51 |
| *** dasTor_ is now known as dasTor | 09:34 | |
| opendevreview | jayaditya gupta proposed openstack/diskimage-builder master: Fix issue in extract image https://review.opendev.org/c/openstack/diskimage-builder/+/850882 | 09:41 |
| *** marios is now known as marios|call | 10:00 | |
| *** marios|call is now known as marios | 10:04 | |
| *** ysandeep is now known as ysandeep|lunch | 10:12 | |
| *** rlandy|out is now known as rlandy | 10:30 | |
| *** ysandeep|lunch is now known as ysandeep | 11:21 | |
| *** bhagyashris_ is now known as bhagyashris | 11:33 | |
| opendevreview | Merged openstack/project-config master: Switch the requirements-constraints job to py310 https://review.opendev.org/c/openstack/project-config/+/861035 | 11:38 |
| *** ysandeep is now known as ysandeep|afk | 12:29 | |
| fungi | per-domain import logs for the latest mm3 migration test are available in 149.202.168.204:~fungi | 12:48 |
| fungi | i don't see any obvious new errors, and the ones about the fields which were too large for their db columns are now gone | 12:48 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 13:37 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Fork the maxking/docker-mailman images https://review.opendev.org/c/opendev/system-config/+/860157 | 13:37 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM force mm3 failure to hold the node https://review.opendev.org/c/opendev/system-config/+/855292 | 13:37 |
| fungi | clarkb: ^ i added redirects for the old list info page and the list index page urls | 13:37 |
| fungi | tested by adding manually on 149.202.168.204 first | 13:37 |
| *** ysandeep|afk is now known as ysandeep|out | 13:38 | |
| fungi | seems to work, though maybe i should add some testinfra checking on that now that i think about it | 13:38 |
| *** dasm|off is now known as dasm | 14:15 | |
| fungi | oh, but we're not actually testing the other redirects in the new deployment either | 14:24 |
| fungi | just checking for listening sockets and taking some screenshots | 14:24 |
| Clark[m] | Because we don't have the migration data to check with. I guess we could write a basic html file and use that though | 14:35 |
| fungi | right | 14:40 |
| fungi | well, we could test the redirects to the new interface since we pre-create the mailing lists in it, it's just the rewrites exposing the old archives we can't test without adding some content | 14:41 |
| Clark[m] | ++ | 14:43 |
| *** marios is now known as marios|out | 15:20 | |
| clarkb | https://zuul.opendev.org/t/openstack/build/d671978274324495b3ea163d3b6ad2a5 any idea what caused that to happen? Seems like our test static.o.o which loads up the afs ro content returned a 403 instead of 200 for starlingx content | 15:39 |
| clarkb | the prod content is available so not a systemic issue with their afs content and the testinfra tests for static lookup other data out of afs so not an afs specific issue | 15:40 |
| clarkb | I've rechecked to see if they are persistent issues | 15:40 |
| fungi | i looked at it briefly | 15:40 |
| fungi | pretty sure something happened and the test node had trouble reaching afs when apache wanted to read the .htaccess file | 15:41 |
| fungi | if you look at the error details from apache you get a little more insight | 15:41 |
| fungi | i didn't check syslog for actual afs errors, but wouldn't be surprised if there are some | 15:42 |
| fungi | assuming the job collected it | 15:42 |
| opendevreview | Merged openstack/project-config master: Add project for managing zuul jobs for charms https://review.opendev.org/c/openstack/project-config/+/861046 | 16:21 |
| jrosser_ | i've not received emails from review@openstack.org since the 9th - is it possible to tell if there have been delivery attempts or bounces? | 16:56 |
| clarkb | jrosser_: "SMTP error from remote mail server after end of data: 553-Message filtered." | 16:58 |
| jrosser_ | oh dear :( | 16:59 |
| clarkb | seems to be your mail servers are filtering it as spam | 16:59 |
| clarkb | we should double check on our end if the host ended up on any lists | 16:59 |
| clarkb | does sbl not take a full ipv4 address in their query form? | 17:02 |
| clarkb | fungi: ^ | 17:02 |
| clarkb | I found a different sbl query form and neither the ipv4 or ipv6 address is listed | 17:06 |
| clarkb | jrosser_: ^ its possible a different list has it listed, but sbl at least says we are good | 17:06 |
| jrosser_ | ok thanks - do you have a transcript with anything useful (like which server rejected it) as the mail routing i have to endure is terrible | 17:07 |
| clarkb | jrosser_: the IPs seems to vary but cluster1.eu.messagelabs.com appears to be the shared fqdn | 17:09 |
| jrosser_ | ok thats helpful | 17:10 |
| jrosser_ | i've added openstack.org as an allowed domain into my messagelabs portal | 17:13 |
| fungi | maybe add opendev.org too | 17:23 |
| fungi | since we will likely change that from address in the future if/when we set up an mta for the new domain | 17:24 |
| clarkb | Our rocky 9 image should try to rebuild again shortly. If it ends up on nb02 it will run without the mirrorlist change which would be a good check of that | 20:49 |
| *** timburke_ is now known as timburke | 20:59 | |
| ianw | clarkb: if you have time to loop back over https://review.opendev.org/q/topic:bridge-ansible-venv it should be ok. i did end up moving back to just using "bastion" as the group for testing and production as i think it's a bit easier. there's a new change to deal with base-jobs/bootstrap as you pointed out too | 21:10 |
| clarkb | ianw: oh ya | 21:12 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] switch testing bridge name to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 21:19 |
| clarkb | ianw: and I guess bootstrap bridge is weird due to its self referential nature? | 21:21 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/858476/6..10/playbooks/bootstrap-bridge.yaml | 21:21 |
| clarkb | oh I see the next chnage splits out the handling for that | 21:21 |
| ianw | https://review.opendev.org/c/opendev/system-config/+/861031/1/playbooks/bootstrap-bridge.yaml then updates that now ... hopefully the comment helps | 21:22 |
| clarkb | ianw: left a comment on that one. Apologies if my previous comments may have confused things. | 21:36 |
| clarkb | Google CLA issues sorted. https://gerrit-review.googlesource.com/c/gerrit/+/348194 that should fix ssh rsa problems with gerrit 3.5 | 21:40 |
| *** rlandy is now known as rlandy|bbl | 22:04 | |
| JayF | I feel like there might be something weird with the opendevreview bot -- https://review.opendev.org/c/openstack/ironic/+/860142 was posted as "verification failed" 3 minutes ago, but the V-2 was put on the patch at more like 20 minutes ago | 22:05 |
| JayF | I don't know if that's "normal" or what, but the latency surprised me so I thought I'd mention it in case it's evidence of some kind of service issue | 22:05 |
| clarkb | JayF: what do you mean by "was posted as verification failed" | 22:07 |
| JayF | > 22:01:40 opendevreview | Verification of a change to openstack/ironic stable/xena failed: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860142 | 22:07 |
| JayF | but on the patch itself, the V-2 was voted on by zuul at 21:38 | 22:08 |
| JayF | er, it's actually worse than that; at 20:38 | 22:08 |
| JayF | I do not care or am bothered by this latency; but noticed because I actioned the email notification, then saw an IRC notification and was like "oh no another one", but it was the same one | 22:09 |
| *** dasm is now known as dasm|off | 22:09 | |
| JayF | I just wanted to mention it because it's the kind of strange that you might wanna know about :) | 22:09 |
| clarkb | the bot got an event from gerrit at 20:38:59 which rules out gerrit emitting the event slowly | 22:12 |
| clarkb | and it says it sent the message at that time | 22:12 |
| JayF | I can guarantee it didn't hit my client at that time; and 22:01:40 is much too late for it to be like, client lag | 22:13 |
| clarkb | it then got a second event (the one generated by your comment) that it dcided it needed to post for as well | 22:14 |
| clarkb | and that one seems to be what generated the message you saw | 22:14 |
| JayF | Yep, and looking back | 22:14 |
| JayF | I see a 20:38:59 too | 22:14 |
| JayF | I did issue a recheck at 21:45 | 22:15 |
| JayF | between those events | 22:15 |
| clarkb | oh actually no I think it was the comment from the arm pipeline | 22:15 |
| clarkb | so ya I think this is fine other than it triggering off of any zuul comment and not necessarily the one that changes the state | 22:15 |
| JayF | okay; makes sense. extra notifications are not so bad just very, very confusingly timed there | 22:15 |
| clarkb | ya thats what it is. It posted at 20:38 when the -2 happened. Then at 22:01 it posts again in response to the arm64 pipeline comment | 22:15 |
| JayF | doesn't help that I missed that it notified at the right time as well | 22:15 |
| JayF | I may have about 40000 patches up with "Stable only: " or "CI: " prefix across multiple stable branches; it's all mixing together | 22:16 |
| ianw | clarkb: hrm, i guess you're right in that https://opendev.org/opendev/system-config/src/branch/master/zuul.d/infra-prod.yaml#L50 is using the zuul-run playbooks | 22:17 |
| fungi | JayF: clarkb: what triggered exactly? looking through the comments on that change i don't see anything amiss | 22:17 |
| JayF | fungi: tl;dr: a message popped at 20:38:59 that a change failed verification. This was accurate. Another identical message that it failed verification posted at 22:01:40, which appears to have been sprung by the ARM64 pipeline notification | 22:18 |
| fungi | zuul left the verified -2 result at 20:38, then the next comment i see from it is at 22:01 when it says the arm jobs passed | 22:18 |
| fungi | JayF: what does "popped up" mean in this contect? | 22:18 |
| fungi | context | 22:18 |
| JayF | IRC robot messages in #openstack-ironic | 22:18 |
| fungi | not the comment i guess? | 22:18 |
| JayF | from opendevreview | 22:18 |
| fungi | oh! i totally missed you were talking about irc there | 22:19 |
| fungi | got it. i think i've never noticed that behavior because none of the projects i work actively on have it set to do notifications on failure results | 22:19 |
| fungi | just new uploads and merges | 22:20 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] switch testing bridge name to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 22:27 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing an upstream gerrit change https://review.opendev.org/c/opendev/system-config/+/861117 | 22:34 |
| clarkb | oh I needed to force a failure too to hold the test nodes | 22:34 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing an upstream gerrit change https://review.opendev.org/c/opendev/system-config/+/861117 | 22:35 |
| clarkb | infra-root ^ I'm going to hold the gerrit 3.5 job for that and then use it to test that ssh looks happy with rsa keys | 22:35 |
| clarkb | if it does I'll submit the upstream fix and we can deploy that and everyone can use rsa again | 22:35 |
| fungi | awesome, thanks! | 22:36 |
| clarkb | in theory if I can ssh from my local machines to that held node using an rsa key it is working as my openssh is new enough | 22:37 |
| fungi | yeah, i have overrides in my .ssh/config for review.opendev.org | 22:38 |
| clarkb | oh heh they already submitted it upstream. Well we'll test it anyway :) | 22:38 |
| fungi | i could test with a non-overridden config | 22:39 |
| fungi | in fact, if i ssh by ip address, then my overrides won't be applied anyway | 22:39 |
| clarkb | ya I tested this pretty extensively when I fixed 3.6. So I'm 99% sure it will work | 22:40 |
| clarkb | but I figure being 100% sure is worthwhile | 22:40 |
| fungi | absolutely | 22:41 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] switch testing bridge name to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 22:57 |
| clarkb | I think the test gerrit instance is working | 23:33 |
| clarkb | fungi: 158.69.75.25 is the host if you want to test. I logged in via the web ui as the zuul user (you click on the button on the login page) and then added an rsa key | 23:34 |
| clarkb | when I run ssh -v -i throwaway_rsa_key I get debug1: kex_input_ext_info: server-sig-algs=<...,rsa-sha2-512,rsa-sha2-256,ssh-rsa> | 23:34 |
| clarkb | I'm going to update my change to be a rebuild gerrit change so that we can get new images and hopefully deploy that soon | 23:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update our Gerrit images https://review.opendev.org/c/opendev/system-config/+/861117 | 23:38 |
| clarkb | also I'm fairly certain I would've needed an override with my openssh client so the fact that it works at all is a good indication it is fixed | 23:40 |
| fungi | **** Welcome to Gerrit Code Review **** | 23:42 |
| fungi | Hi Zuul, you have successfully connected over SSH. | 23:42 |
| fungi | Connection to 158.69.75.25 closed. | 23:42 |
| clarkb | fungi: and that was port 29418 right? | 23:42 |
| clarkb | if so I'll go ahead and delete the autohold and I thin kwe can proceed with landing 861117 when we want to plan a gerrit restart | 23:43 |
| fungi | yeah | 23:44 |
| fungi | debug1: kex_input_ext_info: server-sig-algs=<...,rsa-sha2-512,rsa-sha2-256,ssh-rsa> | 23:44 |
| fungi | debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:... | 23:44 |
| fungi | also it would have to have been port 29418 for me to get the gerrit banner | 23:45 |
| clarkb | autohold deleted. Thank you for helping to test | 23:45 |
| fungi | any time! thanks for fixing it | 23:45 |
| clarkb | now I'm trying to remembre all the people who might've done an override. I guess we can send email to the mailing lists | 23:46 |
| fungi | yeah, just cast a wide net once we're upgraded | 23:50 |
| *** rlandy|bbl is now known as rlandy | 23:50 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!