Friday, 2020-12-18

*** brinzhang0 has quit IRC00:02
*** diablo_rojo has quit IRC00:05
*** brinzhang has joined #openstack-release00:43
*** armax has quit IRC00:58
*** armax has joined #openstack-release01:06
*** mgoddard has quit IRC01:55
*** armax has quit IRC02:31
*** armax has joined #openstack-release03:49
*** armax has quit IRC04:06
*** mgoddard has joined #openstack-release04:12
*** mgoddard has quit IRC04:53
*** ykarel has joined #openstack-release04:57
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-release05:33
*** mgoddard has joined #openstack-release06:11
*** vishalmanchanda has joined #openstack-release06:48
*** mgoddard has quit IRC06:53
*** slaweq has joined #openstack-release07:42
*** hberaud has quit IRC07:59
*** hberaud has joined #openstack-release08:00
*** mgoddard has joined #openstack-release08:05
*** sboyron has joined #openstack-release08:13
openstackgerritLucian Petrut proposed openstack/releases master: Release os-win 5.4.0 (Wallaby)
*** rpittau|afk is now known as rpittau08:30
*** e0ne has joined #openstack-release08:34
*** sboyron has quit IRC08:38
*** e0ne has quit IRC08:41
openstackgerritMerged openstack/releases master: Release Patrole 0.11.0
*** mgoddard has quit IRC08:53
*** elod is now known as elod_pto09:04
*** mgoddard has joined #openstack-release09:05
*** slaweq has quit IRC09:16
*** slaweq has joined #openstack-release09:18
*** slaweq has quit IRC09:20
*** slaweq has joined #openstack-release09:25
openstackgerritDmitriy Rabotyagov proposed openstack/releases master: Branch OpenStack-Ansible Roles
*** slaweq has quit IRC10:51
*** slaweq has joined #openstack-release10:55
openstackgerritDmitriy Rabotyagov proposed openstack/releases master: Branch OpenStack-Ansible Roles
*** e0ne has joined #openstack-release12:01
*** brinzhang has quit IRC12:08
*** sboyron has joined #openstack-release12:10
*** ykarel_ has joined #openstack-release12:14
*** ykarel has quit IRC12:17
*** ykarel_ is now known as ykarel12:18
*** sboyron has quit IRC12:33
*** tosky has joined #openstack-release12:58
*** e0ne has quit IRC14:09
*** e0ne has joined #openstack-release14:09
*** iurygregory has joined #openstack-release14:38
*** e0ne_ has joined #openstack-release15:02
*** e0ne has quit IRC15:05
*** armax has joined #openstack-release15:12
openstackgerritMerged openstack/releases master: Replace series name by template variable
openstackgerritMerged openstack/releases master: Branch OpenStack-Ansible Roles
*** rpittau is now known as rpittau|afk15:38
*** gibi is now known as gibi_pto16:35
hberaudA new release error to well start the hollidays =>
gmannhberaud: which comment you are referring?
hberaudgmann: our comment on PS1 (Dec 15)17:16
gmannah this one 'we need to cap constraints in tox also.' ?17:17
gmannhberaud: that is done, I updated comment.17:18
hberaudgmann: yes sorry I read wrong17:19
gmannhberaud: thanks17:19
*** vishalmanchanda has quit IRC17:19
hberaudyou're welcome17:19
openstackgerritMerged openstack/releases master: Release Tempest 26.0.0
*** dave-mccowan has quit IRC17:42
hberaudfungi: FYI we got an error [1] with during the tag-releases. The goal of this patch was branching OSA roles for victoria, I checked all the branching one by one and everything seems correct. I did some verifications in the logs [1] and I didn't noticed issues. The job exited with rc 56, I don't know what it is (exit code 56). I didn't find17:44
hberaudsignificant messages in the logs. Any ideas? [1]
hberaudI surely missed something during my investigation17:45
*** dave-mccowan has joined #openstack-release17:47
*** e0ne_ has quit IRC17:54
fungihberaud: a bit of looking around suggests that git will exit 56 on some network failures17:54
hberaudah ok17:56
hberaudhowever branching seems ok17:56
fungiseems to show up most frequently when a network connection initiated by git is prematurely terminated17:56
fungiwas openstack/openstack-ansible-tests the last repo it needed to branch?17:57
* hberaud looking17:59
hberaudfungi: so 59 repos asked for branching, 59 branching were successfully did, and apparently yes openstack/openstack-ansible-tests was the latest expected18:05
hberaudand the latest seems ok too =>
fungihberaud: aha, i think git exit codes were a distraction... looks like that exit code originated here:
fungithere are a few places in that function where error_count is incremented18:13
fungiand then the script returns the total tally as its exit code, which gets carried through by the calling scripts18:14
hberaudok, does this script contains some retry mechanismes which could explain why branches seems successfully created?18:14
fungii we're meant to read that as saying the job encountered 56 errors18:15
hberaudok I see18:15
fungiit looks like it's the total of the return codes of tag_release() and make_branch() function calls18:16
fungiwhich in turn is the number of subprocess.CalledProcessError exceptions raised in them18:17
fungilooks like it's supposed to spew on stderr, so maybe the job-output.json has them18:17
fungigrr, i can't transform that json file with flamel, "yaml.reader.ReaderError: unacceptable character #x001f: special characters are not allowed"18:18
fungioh, it's getting retrieved compressed18:19
hberaudas the branches were properly created do you think we need to reenqueue "publish-tox-docs-releases"18:19
fungino dice, the json says stderr from that task was an empty string, so if there was stderr emitted by it got swallowed or the fd wasn't correctly inherited by one of the calling layers18:22
fungihberaud: i'd be inclined to ignore it if everything the job needed to do looks done. but if you see that job continue to show nonzero exit codes we should probably add more debugging output in the script and/or look into why stderr isn't captured18:24
fungimy guess is something from is triggering a subprocess.CalledProcessError exception for most of those. could that happen if the branch already existed?18:25
fungievery so very many layers of scripts on more scripts18:26
hberaudfungi: +1 to ignore that, we'll watch if similar issues will happen, but for now I'm not aware of similar issues.18:27
fungireminds me of when i used to manage sco unix servers, and most of their userspace tooling was an endless maze of ksh scripts calling one another18:27
*** ykarel has quit IRC18:28
fungii see lots of commands in which could trigger errors under the right conditions18:29
fungithough it does seem to want to exit out cleanly without creating branches if they already exist18:30
fungidid the .gitreview changes get pushed for all of those branches?18:31
fungimaybe that's where the errors are stemming from18:31
hberaudI didn't verifying that point18:33
hberaudlemme check18:33
fungipushing changes to gerrit seems likely to be the most fragile part of make_branch.sh18:34
hberaudfungi: without checking all of them for now I can see => and
fungiif there are more than 3 then i think we can rule that out as the cause of the 56 failures out of 59 branches18:36
hberaudall seems ok there =>
hberaud9 patches18:38
fungi "Code needing to capture stdout or stderr should use run() instead"18:40
hberaudaround ~120 corresponding patches related seems created18:41
fungiso that's why we don't get the output from even though the comment here seems to indicate we should:
hberaudthat correspond to our 59 branching18:41
hberaudI see18:43
fungior maybe i'm misreading the implications there, but i guess if this continues to be a problem, we should rework to replace uses of check_call() with something which actually captures the stderr so we can directly emit it instead of relying on fd inheritance to take care of that18:44
hberaudso I think a fix is on the horizon18:44
hberaudyes I agree with you18:44
hberaudgood catch18:45
fungiit's entirely possible that when you run this locally your tty's stderr gets inherited by but when run under ansible that doesn't filter down18:45
hberaudI think it's depends on how we use requiretty18:47
hberaudwith ansible18:47
fungioh could be18:47
hberaudhowever I'm not an ansible expert18:48
fungisame here, i'm afraid18:49
fungii know just enough about ansible to be a danger to myself and others ;)18:49
hberaudThis is an ironical situation to face this kind of issue with OSA :)18:50
fungiexcellent point, maybe we can get them to help18:52
hberaudlet me ping them18:53
hberaudnoonedeadpunk o/, around by chance?18:55
hberaudnoonedeadpunk: fungi and myself need some help about Ansible's tty management18:56
hberaudnicolasbock: it's related to your patch who exited in error
hberaudnicolasbock: sorry wrong dest18:57
hberaudnoonedeadpunk: it's related to your patch who exited in error
nicolasbockNo worries hberaud :)18:58
hberaudnoonedeadpunk: indeed we faced an issue but we didn't get the stderr as expected in our script (c.f the previous discussion below)18:59
hberaudnoonedeadpunk: and we wonder if it could be related to ansible's tty management and by example to requiretty19:00
hberaudnoonedeadpunk: but we aren't both ansible expert19:01
hberaudfungi: if we didn't get response about this until tomorrow then I'll start a related thread on the ML to help us to track this and grab some experts feedbacks19:07
hberaudfungi: I suggest to move that in a more async mode for now, is it ok for you?19:09
fungiabsolutely! (i already have, also it's late for you i'm sure)19:14
noonedeadpunkhberaud: hey19:34
noonedeadpunkeventually ansible does not require tty unless it has pipelining enabled19:36
*** e0ne has joined #openstack-release19:36
noonedeadpunksince default behaviour is just to copy python modules to the destination and execute them locally from tmp dir with permissions of either connect user or become user19:37
*** dave-mccowan has quit IRC19:37
noonedeadpunktty becomes a requirement when pipelining is enabled but again I think it's related to become process only (since sudoers should not contain requiretty in such cases)19:38
noonedeadpunkisn't this an actual issue?19:38
noonedeadpunk`/home/zuul/scripts/release-tools/ line 51: python: command not found`?19:38
noonedeadpunkalso I'm wondering if that might be the case if let's say stable/ussuri has been branched from the same sha I'm trying to branch victoria now?19:40
noonedeadpunkI'm not sure if this exist since I haven't checked for that, but theoretically that might happen considering amount of roles (and that some of them do not change much)19:41
hberaudno idea19:42
noonedeadpunkbtw, in the meanwhile, things seemed to branch actually19:42
hberaudeverything seems ok, branches have been created and gerrit/victoria patches are on the rails for each repo19:43
hberaudfungi noticed that possibly a our error catching doesn't work as expected and the fd doesn't contains the stderr19:45
hberaudhowever it could come from different sides19:46
hberaudand one of them is how ansible manage the tty and the context of the execution19:47
hberaudnoonedeadpunk: thanks for feedback, I need to dash, during the week end I'll move the discussion into a ML thread19:49
noonedeadpunkoh, I think I know what got wrong19:50
noonedeadpunkbranches in gerrit were not created19:50
fungiright, this is ansible calling a shell script calling a python script calling a shell script... comments in the python script suggest the shell script it's calling should inherit the parent process's stderr, but it's not getting captured in the task json's stderr field19:50
noonedeadpunkI don't have gerrit/stable/victoria19:50
noonedeadpunkand my guess would be nobody has ever branched after gerrit upgrade?19:51
funginoonedeadpunk: which repo are you looking at?19:51
hberaudhm I think that yes19:52
fungishows it there19:52
noonedeadpunkeventually any19:52
fungii have a remotes/gerrit/stable/victoria19:53
noonedeadpunkI'm wondering why I have
fungigit clone && cd openstack-ansible-tests && git review -s && git remote update && git branch -a19:54
fungithat's what i just did anyway19:54
noonedeadpunkuh, ok, yes19:55
noonedeadpunkI did just git pull and thought that would be enough19:55
fungii can also `git review -d 767906` in there just fine19:55
noonedeadpunkgit remote update worked19:55
hberaudwe don't need frist ?19:56
fungiyeah, you may need to pull or remote update after branch creation before you can `git review -d ...` a change for that branch19:56
noonedeadpunkwe need to commit things but I think it shouldn't affect downloading of patches19:56
noonedeadpunkwell pull does not work for sure:) It just omit branch in case no new patches are there I guess19:56
noonedeadpunkok, sorry for false alarm19:57
fungino worries, i'd prefer the opportunity to double-check things than ignore a possible bug19:57
fungiwe tested what we could reasonably test on a copy of production before upgrading the service fo realz, but we know there are plenty of scenarios we simply couldn't or didn't have time to try19:59
noonedeadpunkthat's totally fair20:00
noonedeadpunkit's just me who should double checking stuff before spilling out the beans :)20:01
*** e0ne has quit IRC20:10
openstackgerritDmitriy Rabotyagov proposed openstack/releases master: Release OpenStack-Ansible Ussuri
openstackgerritDmitriy Rabotyagov proposed openstack/releases master: Release OpenStack-Ansible Train
*** e0ne has joined #openstack-release20:22
*** e0ne has quit IRC20:40
hberaudanyway it could be worth to get the opportunity to retry some isolated new branching creationns before the rush of wallaby's last weeks20:46
hberaudelse if something related to it goes wrong it could be really painful to manage it at `pow()`20:49
*** slaweq has quit IRC20:54
*** e0ne has joined #openstack-release21:36
*** e0ne has quit IRC21:41

Generated by 2.17.2 by Marius Gedminas - find it at!