Monday, 2020-11-23

*** evrardjp has quit IRC00:05
*** tosky has quit IRC00:06
*** evrardjp has joined #openstack-release00:15
-openstackstatus- NOTICE: Our Gerrit upgrade maintenance has concluded successfully; please see the maintenance wrap-up announcement for additional details:
*** ykarel has joined #openstack-release03:03
*** ricolin has quit IRC03:07
*** dave-mccowan has quit IRC03:17
*** ykarel has quit IRC04:09
*** ykarel has joined #openstack-release05:15
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-release05:33
*** ricolin has joined #openstack-release05:42
*** otherwiseguy has quit IRC06:24
*** otherwiseguy has joined #openstack-release06:24
*** whoami-rajat__ has joined #openstack-release06:46
*** sboyron has joined #openstack-release06:52
*** iurygregory has joined #openstack-release07:06
*** rpittau|afk is now known as rpittau07:29
*** slaweq has joined #openstack-release08:01
*** openstackgerrit has joined #openstack-release08:26
openstackgerritAbhishek Kekane proposed openstack/releases master: Release glance-store for wallaby-1 milestone
*** tosky has joined #openstack-release08:38
*** mgoddard has joined #openstack-release08:58
*** tinwood_ is now known as tinwood09:08
openstackgerritMerged openstack/releases master: [senlin] Transition Stein to EM
openstackgerritMerged openstack/releases master: [cloudkitty] Transition Stein to EM
openstackgerritMerged openstack/releases master: [tripleo] Transition Stein to EM
openstackgerritMerged openstack/releases master: [ironic] Transition Stein to EM
*** sboyron has quit IRC09:18
*** sboyron has joined #openstack-release09:19
*** e0ne has joined #openstack-release09:19
openstackgerritMerged openstack/releases master: [PowerVMStackers] Transition Stein to EM
ttxhberaud: bunch of release jobs errors showed up09:27
ttxwe should stop approvign09:27
hberaudttx: Yep I just seen it09:27
ttx"Could not connect to gerrit"09:28
ttxHost key verification failed09:28
hberaudttx: nothing else was approved and is waiting for merging09:29
hberaudgood catch09:29
hberaudlooks like to misconfiguration09:30
hberaudI think it's related to the gerrit update09:31
ttxThe script calls git review09:32
ttxtrying manually09:33
ttxworks for me with my username09:34
ttxmaybe the release account does nto work09:34
hberaudhow we can check that?09:35
hberaudcan we try to use it manually?09:35
ttxnot sure. Also the error points to host ssh key09:35
hberaudI think that we should involve the infra team on this point09:39
ttxit's a bit weird because the ID seems to be recognized when I try. So that would mean we have a stale host key on the machine the script is running on09:41
hberaudI see09:42
ttxbut I don't think the host key changed, so that is weird09:43
ttxanyway, let's hold until fungi is up09:43
hberaudmaybe they missed to sync the keys during the gerrit update for this part. If pub key is missing on the gerrit side it could explain that issue09:43
hberaudI agree09:44
ttxthat would block everyone09:44
ttxI managed to run git review -s successfully, no host key update prompt09:44
hberaudI'll summarize it for the infra team09:44
ttxI did summarize it already09:44
ttxI wish we had some way of setting a flag to RED on this channel so that we all know when it's safe to approve releases, safe to approve one but please watch it run, or not safe09:46
ttxLike a traffic light09:47
hberaudMaybe the openstack bot could help us on this point09:49
ttxwe could use the topic09:50
ttxLike status: GREEN09:50
*** dtantsur|afk is now known as dtantsur09:50
hberaudYou mean on the patches?09:50
ttxno on the IRC channel09:50
hberaudI see09:50
hberaudit could be a first step09:51
hberaudhowever, that implies to look on IRC before each approval09:51
hberaud#topic REDFLAG09:52
openstackgerritMark Goddard proposed openstack/releases master: Final Stein releases for Kolla projects
hberaudopenstack #topic redflag09:53
*** ricolin has quit IRC09:53
ttxyeah, was just a thought. I usually have a look on IRC before starting to approve things.09:53
ttxBut sometimes it's hard to parse the discussion into a simple YES/NO/CAREFUL09:53
openstackgerritMark Goddard proposed openstack/releases master: [kolla] Transition Stein to EM
hberaudYou're right IRC redflags can't hurt09:56
hberaudit's just that we should not forget to take a look frequently, but we are all always active here so I think we will see them09:57
hberaudand yes #topic can help to highlight them09:58
hberaudto avoid to dive deep into previous discussions09:58
ttxwe could discuss it in a future meeting10:01
hberaudalso, depending on our IRC client, we might add notifications based on redflag keyword10:01
hberaudttx: I added a related section to our next meeting agenda, feel free to add things there
*** iurygregory has quit IRC10:39
*** e0ne has quit IRC11:36
*** iurygregory has joined #openstack-release11:58
*** sboyron has quit IRC12:28
*** sboyron has joined #openstack-release12:37
*** sboyron has quit IRC12:49
*** sboyron has joined #openstack-release12:51
*** sboyron has quit IRC12:56
*** sboyron has joined #openstack-release13:01
*** sboyron_ has joined #openstack-release13:13
*** sboyron has quit IRC13:15
*** e0ne has joined #openstack-release13:18
*** sboyron_ has quit IRC13:21
*** sboyron_ has joined #openstack-release13:21
*** sboyron__ has joined #openstack-release13:23
*** sboyron_ has quit IRC13:25
*** sboyron__ has quit IRC13:28
*** sboyron__ has joined #openstack-release13:28
*** sboyron__ has quit IRC13:29
*** sboyron__ has joined #openstack-release13:33
*** sboyron__ is now known as sboyron13:33
fungii'll take a look at the host key mismatches as soon as i finish catching up and seeing what else might be broken13:50
hberaudack thanks14:14
*** dave-mccowan has joined #openstack-release14:20
*** sboyron has quit IRC14:28
*** sboyron has joined #openstack-release14:29
*** dave-mccowan has quit IRC14:35
*** rpittau is now known as rpittau|brb14:36
fungiokay, i'm caught up and digging into the failure details14:49
*** ykarel has quit IRC14:52
*** rpittau|brb is now known as rpittau15:01
fungii see the problem. it's running on ubuntu bionic and installing the distro package for git-review 1.26 which won't work with modern gerrit15:05
fungiif we make the tag-releases job run on an ubuntu-focal node it will get git-review 1.27 which should work fine15:05
fungii'll start on the change for that momentarily15:05
*** elod has quit IRC15:27
*** elod has joined #openstack-release15:28
fungi should hopefully fix things15:31
hberaudfungi: Hopefully that it wouldn't raise new issues like the previous bindep errors. What do you think about this?15:38
hberaudit's unlikely that something similar will happen15:40
hberaudI think it's unlikely now that we cleaned our jobs dependencies15:41
hberaudApproved, thank you15:44
fungii mean, it might turn up other issues, but ultimately we should be running jobs on focal instead of bionic anyway so whatever else it turns up we ought to also fix15:46
hberaudI see15:47
fungimost of openstack switched to running jobs on bionic before victoria released15:48
fungier, switched to running jobs on focal i mean15:48
hberaudgot it :)15:48
*** elod has quit IRC15:54
*** armax has joined #openstack-release15:54
*** elod has joined #openstack-release15:54
*** mgoddard has quit IRC16:12
*** jbadiapa has joined #openstack-release16:13
*** mgoddard has joined #openstack-release16:15
fungiit's merged, shall i try to reenqueue a failed build?16:18
ttxfungi: yes please16:26
fungiokay, will do when i get a sec16:26
ttxthe failed builds actually if the first one works16:26
ttxthere is like 516:26
ttx and next16:28
*** EmilienM has quit IRC16:31
*** EmilienM has joined #openstack-release16:33
smcginnisfungi: Would it be better long term if we change that git review install to "python -m pip install git-review" instead of using system packages?16:35
fungismcginnis: maybe? but for now i expect this to help and we rarely have compatibility problems in the git-review codebase. this was a very long announced backward-incompatible api change in gerrit16:37
smcginnisYeah, fair enough.16:38
clarkbsmcginnis: unrelated to ^ you should test the removal of your fedora33 wroakround now16:44
clarkbit should work without that hack16:44
openstackgerritLuigi Toscano proposed openstack/releases master: [sahara] Transition Stein to EM
openstackgerritLuigi Toscano proposed openstack/releases master: [sahara] last stein release for sahara-image-elements
smcginnisclarkb: Oh right, thanks for the reminder.16:49
*** rpittau is now known as rpittau|afk17:03
*** sboyron_ has joined #openstack-release17:13
*** sboyron has quit IRC17:16
*** armax has quit IRC17:20
*** sboyron__ has joined #openstack-release17:25
*** sboyron_ has quit IRC17:29
*** jbadiapa has quit IRC17:33
*** e0ne has quit IRC17:33
fungiso before i reenqueue refs for any of these failed builds, is tag-releases idempotent (so i should just rerun the most recent ref) or does it need to be run for each ref which failed?17:33
fungii'm separately investigating whether we should also update the known_hosts entries here to include all host key types gerrit's serving:
smcginnisfungi: tag-releases should be able to just be rerun. It will check if the tag exists and only try to tag again if not. If I understand correctly, nothing was able to get a tag pushed, so it should just be like the initial post-release job.17:42
fungismcginnis: several refs failed tag-releases, so asking whether i need to rerun them all or just the last one17:43
fungisounds like just doing the last one will run everything which hasn't been tagged already?17:43
smcginnisHmm, I think it may be each one, but we can start with just the last one and see.17:44
*** mgoddard has quit IRC17:54
fungiwell, because the other worry i have is whether running them out of sequence would be a problem, assuming we discover that just running the last one isn't sufficient17:54
fungibut conversely, would the job misbehave if i enqueue a ref which is not the current state of the branch17:54
fungii guess i'll try to see what the job does17:55
*** weshay|ruck is now known as weshay|interview17:57
hberaudfungi: I'm not sure how it will behave, so yes we try to reenqueue just one for now17:58
hberauds/we try/we can try/17:58
hberaudWe can start with this one
fungiso going to run this:18:04
fungisudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release-post --project=openstack/releases --ref=refs/heads/master --newrev=2f6e3e30f67df4f60e0e2b386755118bb133805918:04
fungiand it's enqueued18:04
fungiif we continue to see the same error, may also be needed18:13
hberaudfungi: I seen passed a PGP error18:13
fungithere may have been an earlier failure to create that directory, will want to look more closely once zuul reports the buildset18:15
fungii need to step away for a few minutes, but will look at it as soon as i'm done doing morning things i should have done when i woke up 5 hours ago18:16
hberaudfungi: sure np, thanks18:16
hberaudAlso I think that python-launchpadlib should be moved to python3-launchpadlib (c.f and ( )18:20
*** armax has joined #openstack-release18:33
hberaudthe launchpadlib error come from
hberaudwhich is compatible with bionic only18:44
hberaudc.f for original error with reenqueued ref 2f6e3e30f67df4f60e0e2b386755118bb133805918:45
*** sboyron__ has quit IRC18:49
*** sboyron__ has joined #openstack-release18:49
*** dtantsur is now known as dtantsur|afk18:49
*** sboyron__ has quit IRC19:10
*** sboyron__ has joined #openstack-release19:10
*** sboyron__ has quit IRC19:19
*** sboyron__ has joined #openstack-release19:19
*** sboyron__ has quit IRC19:25
*** sboyron__ has joined #openstack-release19:25
fungii've proposed to switch to python3-launchpadlib in add-launchpad-credentials19:25
*** mgoddard has joined #openstack-release19:26
*** mgoddard has quit IRC19:38
-openstackstatus- NOTICE: The Gerrit service on is being restarted quickly to troubleshoot an unexpected error condition, downtime should be less than 5 minutes20:01
*** openstackgerrit has quit IRC20:02
*** sboyron__ has quit IRC20:13
*** sboyron__ has joined #openstack-release20:13
*** jhesketh has quit IRC20:16
*** jhesketh has joined #openstack-release20:23
*** weshay|interview is now known as weshay|ruck21:46
*** slaweq_ has joined #openstack-release21:49
*** slaweq has quit IRC21:52
*** sboyron__ has quit IRC22:03
*** iurygregory has quit IRC22:05
*** iurygregory has joined #openstack-release22:06
*** iurygregory has quit IRC22:20
*** armax has quit IRC22:22
*** armax has joined #openstack-release22:26
*** iurygregory has joined #openstack-release22:35
*** slaweq_ has quit IRC22:38
*** iurygregory_ has joined #openstack-release23:02
*** iurygregory has quit IRC23:04
*** iurygregory_ is now known as iurygregory23:08

Generated by 2.17.2 by Marius Gedminas - find it at!