| *** evrardjp has quit IRC | 00:05 | |
| *** tosky has quit IRC | 00:06 | |
| *** evrardjp has joined #openstack-release | 00:15 | |
| -openstackstatus- NOTICE: Our Gerrit upgrade maintenance has concluded successfully; please see the maintenance wrap-up announcement for additional details: http://lists.opendev.org/pipermail/service-announce/2020-November/000014.html | 00:54 | |
| *** ykarel has joined #openstack-release | 03:03 | |
| *** ricolin has quit IRC | 03:07 | |
| *** dave-mccowan has quit IRC | 03:17 | |
| *** ykarel has quit IRC | 04:09 | |
| *** ykarel has joined #openstack-release | 05:15 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-release | 05:33 | |
| *** ricolin has joined #openstack-release | 05:42 | |
| *** otherwiseguy has quit IRC | 06:24 | |
| *** otherwiseguy has joined #openstack-release | 06:24 | |
| *** whoami-rajat__ has joined #openstack-release | 06:46 | |
| *** sboyron has joined #openstack-release | 06:52 | |
| *** iurygregory has joined #openstack-release | 07:06 | |
| *** rpittau|afk is now known as rpittau | 07:29 | |
| *** slaweq has joined #openstack-release | 08:01 | |
| *** openstackgerrit has joined #openstack-release | 08:26 | |
| openstackgerrit | Abhishek Kekane proposed openstack/releases master: Release glance-store for wallaby-1 milestone https://review.opendev.org/c/openstack/releases/+/763748 | 08:26 |
|---|---|---|
| *** tosky has joined #openstack-release | 08:38 | |
| *** mgoddard has joined #openstack-release | 08:58 | |
| *** tinwood_ is now known as tinwood | 09:08 | |
| openstackgerrit | Merged openstack/releases master: [senlin] Transition Stein to EM https://review.opendev.org/c/openstack/releases/+/762421 | 09:14 |
| openstackgerrit | Merged openstack/releases master: [cloudkitty] Transition Stein to EM https://review.opendev.org/c/openstack/releases/+/762376 | 09:14 |
| openstackgerrit | Merged openstack/releases master: [tripleo] Transition Stein to EM https://review.opendev.org/c/openstack/releases/+/762411 | 09:16 |
| openstackgerrit | Merged openstack/releases master: [ironic] Transition Stein to EM https://review.opendev.org/c/openstack/releases/+/762369 | 09:16 |
| *** sboyron has quit IRC | 09:18 | |
| *** sboyron has joined #openstack-release | 09:19 | |
| *** e0ne has joined #openstack-release | 09:19 | |
| openstackgerrit | Merged openstack/releases master: [PowerVMStackers] Transition Stein to EM https://review.opendev.org/c/openstack/releases/+/762371 | 09:24 |
| ttx | hberaud: bunch of release jobs errors showed up | 09:27 |
| ttx | we should stop approvign | 09:27 |
| hberaud | ttx: Yep I just seen it | 09:27 |
| ttx | looking | 09:27 |
| ttx | "Could not connect to gerrit" | 09:28 |
| ttx | Host key verification failed | 09:28 |
| hberaud | ttx: nothing else was approved and is waiting for merging | 09:29 |
| hberaud | good catch | 09:29 |
| hberaud | looks like to misconfiguration | 09:30 |
| hberaud | I think it's related to the gerrit update | 09:31 |
| ttx | The script calls git review | 09:32 |
| ttx | trying manually | 09:33 |
| hberaud | ack | 09:33 |
| ttx | works for me with my username | 09:34 |
| ttx | maybe the release account does nto work | 09:34 |
| hberaud | yes | 09:35 |
| hberaud | how we can check that? | 09:35 |
| hberaud | can we try to use it manually? | 09:35 |
| ttx | not sure. Also the error points to host ssh key | 09:35 |
| hberaud | I think that we should involve the infra team on this point | 09:39 |
| ttx | it's a bit weird because the ID seems to be recognized when I try. So that would mean we have a stale host key on the machine the script is running on | 09:41 |
| hberaud | I see | 09:42 |
| ttx | but I don't think the host key changed, so that is weird | 09:43 |
| ttx | anyway, let's hold until fungi is up | 09:43 |
| hberaud | maybe they missed to sync the keys during the gerrit update for this part. If pub key is missing on the gerrit side it could explain that issue | 09:43 |
| hberaud | I agree | 09:44 |
| ttx | that would block everyone | 09:44 |
| ttx | I managed to run git review -s successfully, no host key update prompt | 09:44 |
| hberaud | I'll summarize it for the infra team | 09:44 |
| ttx | I did summarize it already | 09:44 |
| ttx | on https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes | 09:44 |
| hberaud | awesole | 09:45 |
| hberaud | awesome | 09:45 |
| ttx | I wish we had some way of setting a flag to RED on this channel so that we all know when it's safe to approve releases, safe to approve one but please watch it run, or not safe | 09:46 |
| ttx | Like a traffic light | 09:47 |
| hberaud | Maybe the openstack bot could help us on this point | 09:49 |
| ttx | we could use the topic | 09:50 |
| ttx | Like status: GREEN | 09:50 |
| *** dtantsur|afk is now known as dtantsur | 09:50 | |
| hberaud | You mean on the patches? | 09:50 |
| ttx | no on the IRC channel | 09:50 |
| hberaud | I see | 09:50 |
| hberaud | it could be a first step | 09:51 |
| hberaud | however, that implies to look on IRC before each approval | 09:51 |
| hberaud | #topic REDFLAG | 09:52 |
| openstackgerrit | Mark Goddard proposed openstack/releases master: Final Stein releases for Kolla projects https://review.opendev.org/c/openstack/releases/+/763758 | 09:52 |
| hberaud | openstack #topic redflag | 09:53 |
| *** ricolin has quit IRC | 09:53 | |
| ttx | yeah, was just a thought. I usually have a look on IRC before starting to approve things. | 09:53 |
| ttx | But sometimes it's hard to parse the discussion into a simple YES/NO/CAREFUL | 09:53 |
| openstackgerrit | Mark Goddard proposed openstack/releases master: [kolla] Transition Stein to EM https://review.opendev.org/c/openstack/releases/+/762389 | 09:55 |
| hberaud | You're right IRC redflags can't hurt | 09:56 |
| hberaud | it's just that we should not forget to take a look frequently, but we are all always active here so I think we will see them | 09:57 |
| hberaud | and yes #topic can help to highlight them | 09:58 |
| hberaud | to avoid to dive deep into previous discussions | 09:58 |
| ttx | we could discuss it in a future meeting | 10:01 |
| hberaud | also, depending on our IRC client, we might add notifications based on redflag keyword | 10:01 |
| hberaud | yes | 10:01 |
| hberaud | ttx: I added a related section to our next meeting agenda, feel free to add things there https://etherpad.opendev.org/p/wallaby-relmgt-tracking | 10:28 |
| *** iurygregory has quit IRC | 10:39 | |
| *** e0ne has quit IRC | 11:36 | |
| *** iurygregory has joined #openstack-release | 11:58 | |
| *** sboyron has quit IRC | 12:28 | |
| *** sboyron has joined #openstack-release | 12:37 | |
| *** sboyron has quit IRC | 12:49 | |
| *** sboyron has joined #openstack-release | 12:51 | |
| *** sboyron has quit IRC | 12:56 | |
| *** sboyron has joined #openstack-release | 13:01 | |
| *** sboyron_ has joined #openstack-release | 13:13 | |
| *** sboyron has quit IRC | 13:15 | |
| *** e0ne has joined #openstack-release | 13:18 | |
| *** sboyron_ has quit IRC | 13:21 | |
| *** sboyron_ has joined #openstack-release | 13:21 | |
| *** sboyron__ has joined #openstack-release | 13:23 | |
| *** sboyron_ has quit IRC | 13:25 | |
| *** sboyron__ has quit IRC | 13:28 | |
| *** sboyron__ has joined #openstack-release | 13:28 | |
| *** sboyron__ has quit IRC | 13:29 | |
| *** sboyron__ has joined #openstack-release | 13:33 | |
| *** sboyron__ is now known as sboyron | 13:33 | |
| fungi | i'll take a look at the host key mismatches as soon as i finish catching up and seeing what else might be broken | 13:50 |
| hberaud | ack thanks | 14:14 |
| *** dave-mccowan has joined #openstack-release | 14:20 | |
| *** sboyron has quit IRC | 14:28 | |
| *** sboyron has joined #openstack-release | 14:29 | |
| *** dave-mccowan has quit IRC | 14:35 | |
| *** rpittau is now known as rpittau|brb | 14:36 | |
| fungi | okay, i'm caught up and digging into the failure details | 14:49 |
| *** ykarel has quit IRC | 14:52 | |
| *** rpittau|brb is now known as rpittau | 15:01 | |
| fungi | i see the problem. it's running on ubuntu bionic and installing the distro package for git-review 1.26 which won't work with modern gerrit | 15:05 |
| fungi | if we make the tag-releases job run on an ubuntu-focal node it will get git-review 1.27 which should work fine | 15:05 |
| fungi | i'll start on the change for that momentarily | 15:05 |
| hberaud | awesome | 15:10 |
| *** elod has quit IRC | 15:27 | |
| *** elod has joined #openstack-release | 15:28 | |
| fungi | https://review.opendev.org/763797 should hopefully fix things | 15:31 |
| hberaud | fungi: Hopefully that it wouldn't raise new issues like the previous bindep errors. What do you think about this? | 15:38 |
| hberaud | it's unlikely that something similar will happen | 15:40 |
| hberaud | I think it's unlikely now that we cleaned our jobs dependencies | 15:41 |
| hberaud | Approved, thank you | 15:44 |
| fungi | i mean, it might turn up other issues, but ultimately we should be running jobs on focal instead of bionic anyway so whatever else it turns up we ought to also fix | 15:46 |
| hberaud | I see | 15:47 |
| fungi | most of openstack switched to running jobs on bionic before victoria released | 15:48 |
| fungi | er, switched to running jobs on focal i mean | 15:48 |
| hberaud | got it :) | 15:48 |
| *** elod has quit IRC | 15:54 | |
| *** armax has joined #openstack-release | 15:54 | |
| *** elod has joined #openstack-release | 15:54 | |
| *** mgoddard has quit IRC | 16:12 | |
| *** jbadiapa has joined #openstack-release | 16:13 | |
| *** mgoddard has joined #openstack-release | 16:15 | |
| fungi | it's merged, shall i try to reenqueue a failed build? | 16:18 |
| ttx | fungi: yes please | 16:26 |
| fungi | okay, will do when i get a sec | 16:26 |
| ttx | the failed builds actually if the first one works | 16:26 |
| ttx | there is like 5 | 16:26 |
| ttx | http://lists.openstack.org/pipermail/release-job-failures/2020-November/001487.html and next | 16:28 |
| *** EmilienM has quit IRC | 16:31 | |
| *** EmilienM has joined #openstack-release | 16:33 | |
| smcginnis | fungi: Would it be better long term if we change that git review install to "python -m pip install git-review" instead of using system packages? | 16:35 |
| fungi | smcginnis: maybe? but for now i expect this to help and we rarely have compatibility problems in the git-review codebase. this was a very long announced backward-incompatible api change in gerrit | 16:37 |
| smcginnis | Yeah, fair enough. | 16:38 |
| clarkb | smcginnis: unrelated to ^ you should test the removal of your fedora33 wroakround now | 16:44 |
| clarkb | it should work without that hack | 16:44 |
| openstackgerrit | Luigi Toscano proposed openstack/releases master: [sahara] Transition Stein to EM https://review.opendev.org/c/openstack/releases/+/762419 | 16:48 |
| openstackgerrit | Luigi Toscano proposed openstack/releases master: [sahara] last stein release for sahara-image-elements https://review.opendev.org/c/openstack/releases/+/763805 | 16:48 |
| smcginnis | clarkb: Oh right, thanks for the reminder. | 16:49 |
| *** rpittau is now known as rpittau|afk | 17:03 | |
| *** sboyron_ has joined #openstack-release | 17:13 | |
| *** sboyron has quit IRC | 17:16 | |
| *** armax has quit IRC | 17:20 | |
| *** sboyron__ has joined #openstack-release | 17:25 | |
| *** sboyron_ has quit IRC | 17:29 | |
| *** jbadiapa has quit IRC | 17:33 | |
| *** e0ne has quit IRC | 17:33 | |
| fungi | so before i reenqueue refs for any of these failed builds, is tag-releases idempotent (so i should just rerun the most recent ref) or does it need to be run for each ref which failed? | 17:33 |
| fungi | i'm separately investigating whether we should also update the known_hosts entries here to include all host key types gerrit's serving: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/secrets.yaml#L525-L526 | 17:40 |
| smcginnis | fungi: tag-releases should be able to just be rerun. It will check if the tag exists and only try to tag again if not. If I understand correctly, nothing was able to get a tag pushed, so it should just be like the initial post-release job. | 17:42 |
| fungi | smcginnis: several refs failed tag-releases, so asking whether i need to rerun them all or just the last one | 17:43 |
| fungi | sounds like just doing the last one will run everything which hasn't been tagged already? | 17:43 |
| smcginnis | Hmm, I think it may be each one, but we can start with just the last one and see. | 17:44 |
| *** mgoddard has quit IRC | 17:54 | |
| fungi | well, because the other worry i have is whether running them out of sequence would be a problem, assuming we discover that just running the last one isn't sufficient | 17:54 |
| fungi | but conversely, would the job misbehave if i enqueue a ref which is not the current state of the branch | 17:54 |
| fungi | i guess i'll try to see what the job does | 17:55 |
| *** weshay|ruck is now known as weshay|interview | 17:57 | |
| hberaud | fungi: I'm not sure how it will behave, so yes we try to reenqueue just one for now | 17:58 |
| hberaud | s/we try/we can try/ | 17:58 |
| hberaud | We can start with this one http://lists.openstack.org/pipermail/release-job-failures/2020-November/001487.html | 18:00 |
| fungi | so going to run this: | 18:04 |
| fungi | sudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release-post --project=openstack/releases --ref=refs/heads/master --newrev=2f6e3e30f67df4f60e0e2b386755118bb1338059 | 18:04 |
| fungi | and it's enqueued | 18:04 |
| hberaud | thanks | 18:04 |
| fungi | if we continue to see the same error, https://review.opendev.org/763830 may also be needed | 18:13 |
| hberaud | fungi: I seen passed a PGP error | 18:13 |
| hberaud | https://zuul.opendev.org/t/openstack/stream/60d1e591b30a442dad87bbd32300e707?logfile=console.log | 18:14 |
| hberaud | fungi: http://paste.openstack.org/show/800327/ | 18:14 |
| fungi | there may have been an earlier failure to create that directory, will want to look more closely once zuul reports the buildset | 18:15 |
| hberaud | ack | 18:15 |
| fungi | i need to step away for a few minutes, but will look at it as soon as i'm done doing morning things i should have done when i woke up 5 hours ago | 18:16 |
| hberaud | fungi: sure np, thanks | 18:16 |
| hberaud | Also I think that python-launchpadlib should be moved to python3-launchpadlib (c.f http://paste.openstack.org/show/800328/ and https://zuul.opendev.org/t/openstack/stream/60d1e591b30a442dad87bbd32300e707?logfile=console.log) ( https://ubuntu.pkgs.org/20.04/ubuntu-main-amd64/python3-launchpadlib_1.10.13-1_all.deb.html ) | 18:20 |
| *** armax has joined #openstack-release | 18:33 | |
| hberaud | the launchpadlib error come from https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/add-launchpad-credentials/tasks/main.yaml#L15 | 18:43 |
| hberaud | which is compatible with bionic only | 18:44 |
| hberaud | c.f https://zuul.opendev.org/t/openstack/build/044efccd8f22429792ca735b33bb3637/log/job-output.txt#380 for original error with reenqueued ref 2f6e3e30f67df4f60e0e2b386755118bb1338059 | 18:45 |
| *** sboyron__ has quit IRC | 18:49 | |
| *** sboyron__ has joined #openstack-release | 18:49 | |
| *** dtantsur is now known as dtantsur|afk | 18:49 | |
| *** sboyron__ has quit IRC | 19:10 | |
| *** sboyron__ has joined #openstack-release | 19:10 | |
| *** sboyron__ has quit IRC | 19:19 | |
| *** sboyron__ has joined #openstack-release | 19:19 | |
| *** sboyron__ has quit IRC | 19:25 | |
| *** sboyron__ has joined #openstack-release | 19:25 | |
| fungi | i've proposed https://review.opendev.org/763834 to switch to python3-launchpadlib in add-launchpad-credentials | 19:25 |
| *** mgoddard has joined #openstack-release | 19:26 | |
| *** mgoddard has quit IRC | 19:38 | |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is being restarted quickly to troubleshoot an unexpected error condition, downtime should be less than 5 minutes | 20:01 | |
| *** openstackgerrit has quit IRC | 20:02 | |
| *** sboyron__ has quit IRC | 20:13 | |
| *** sboyron__ has joined #openstack-release | 20:13 | |
| *** jhesketh has quit IRC | 20:16 | |
| *** jhesketh has joined #openstack-release | 20:23 | |
| *** weshay|interview is now known as weshay|ruck | 21:46 | |
| *** slaweq_ has joined #openstack-release | 21:49 | |
| *** slaweq has quit IRC | 21:52 | |
| *** sboyron__ has quit IRC | 22:03 | |
| *** iurygregory has quit IRC | 22:05 | |
| *** iurygregory has joined #openstack-release | 22:06 | |
| *** iurygregory has quit IRC | 22:20 | |
| *** armax has quit IRC | 22:22 | |
| *** armax has joined #openstack-release | 22:26 | |
| *** iurygregory has joined #openstack-release | 22:35 | |
| *** slaweq_ has quit IRC | 22:38 | |
| *** iurygregory_ has joined #openstack-release | 23:02 | |
| *** iurygregory has quit IRC | 23:04 | |
| *** iurygregory_ is now known as iurygregory | 23:08 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!