Wednesday, 2021-04-14

« Tuesday, 2021-04-13 Index Thursday, 2021-04-15 »
*** ramishra has joined #openstack-qa00:01
*** ramishra has quit IRC00:06
*** ramishra has joined #openstack-qa00:12
*** dviroel has quit IRC00:32
*** tkajinam has quit IRC00:55
*** tkajinam has joined #openstack-qa00:56
*** ramishra has quit IRC01:08
*** ramishra has joined #openstack-qa01:14
*** artom has quit IRC01:56
*** paras333 has quit IRC02:14
*** paras333 has joined #openstack-qa02:25
*** hamalq has quit IRC02:29
*** paras333 has quit IRC02:30
*** enriquetaso has quit IRC02:37
*** ccamposr__ has quit IRC02:55
*** ccamposr__ has joined #openstack-qa02:55
*** rcernin has quit IRC03:54
*** rcernin has joined #openstack-qa04:00
*** psahoo has joined #openstack-qa04:16
*** vishalmanchanda has joined #openstack-qa04:45
*** gcheresh has joined #openstack-qa04:52
*** whoami-rajat_ has joined #openstack-qa05:20
*** ralonsoh has joined #openstack-qa05:21
*** vhari has quit IRC06:18
*** anandgvb has joined #openstack-qa06:19
*** hemanth_n has joined #openstack-qa06:24
*** ysandeep|holiday is now known as ysandeep06:24
*** zbr has quit IRC06:30
*** zbr has joined #openstack-qa06:32
*** eolivare has joined #openstack-qa06:32
*** csatari has quit IRC06:34
*** ajitha has quit IRC06:34
*** knikolla has quit IRC06:34
*** rpioso has quit IRC06:34
*** sboyron has joined #openstack-qa06:35
*** lxkong has quit IRC06:35
*** ajitha has joined #openstack-qa06:36
*** knikolla has joined #openstack-qa06:36
*** ajitha has quit IRC06:36
*** csatari has joined #openstack-qa06:36
*** lxkong has joined #openstack-qa06:37
*** rpioso has joined #openstack-qa06:37
*** anandgvb has joined #openstack-qa06:38
*** anandgvb has quit IRC06:39
*** vhari has joined #openstack-qa06:43
*** Yarboa has quit IRC07:04
*** Yarboa has joined #openstack-qa07:07
yoctozeptotbarron gouthamr vkmc gmann tosky kopecmartin : re: branching for ceph plugin - it aligns with my vision to branch all current stables; I replied to hberaud on the patch as well; I would use the stable/train branch to test all EMs people care about - it already is designed to be "branchless"; for future branches we will branch regularly07:32
* yoctozepto asked if we should branch tempest as well would say yes07:33
yoctozeptobut I'm not in the mood for fighting the tradition for now ;p07:34
yoctozeptoceph-plugin was different thought07:34
yoctozeptoI'm glad you liked my idea :D07:34
kopecmartinwell, branching would help in packaging, f.e. for packaging tempest for train we had to do quite a few workarounds (mainly reverts) in order to succeed07:46
kopecmartinhttps://review.rdoproject.org/r/c/openstack/tempest-distgit/+/3140807:46
kopecmartinand that was only 26.0.0 tag .. the newer ones are even more problematic07:46
kopecmartini would say the newer tags won't be delivered in a package for train , 27 not for sure, and 26.1.0 maybe07:47
kopecmartinon the other hand, i don't know if this benefit would overweight the disadvantages07:49
*** tosky has joined #openstack-qa07:50
yoctozeptokopecmartin: maybe we should prepare a SWOT for branching tempest and discuss this during the PTG then07:55
*** rpittau|afk is now known as rpittau07:55
*** jpena|off is now known as jpena07:57
kopecmartinyoctozepto: that we can do07:58
kopecmartinyoctozepto: i put the topic to the 3rd day - https://etherpad.opendev.org/p/qa-xena-ptg08:00
*** gfidente|afk is now known as gfidente08:01
*** lucasagomes has joined #openstack-qa08:01
yoctozeptokopecmartin: oh, cool, that's the best timing for me08:01
yoctozeptoyou are reading my mind08:01
kopecmartin::)08:01
yoctozeptoI will try to fill it in before then; I hope others join too (especially you, kopecmartin)08:02
kopecmartinsure, i'll write something down08:03
yoctozeptothanks :-)08:07
*** slaweq_ has joined #openstack-qa08:19
*** slaweq has quit IRC08:19
*** rcernin has quit IRC08:24
*** ricolin has quit IRC08:39
*** slaweq_ is now known as slaweq08:51
*** rcernin has joined #openstack-qa09:12
*** yamamoto has quit IRC09:28
yoctozeptodansmith: https://bugs.launchpad.net/devstack/+bug/192372809:43
openstackLaunchpad bug 1923728 in devstack "install_tempest randomly fails in CI" [High,Triaged]09:43
yoctozeptocc gmann, kopecmartin09:43
*** dtantsur|afk is now known as dtantsur09:45
*** yamamoto has joined #openstack-qa09:59
*** yamamoto has quit IRC09:59
*** yamamoto has joined #openstack-qa10:00
*** whoami-rajat_ is now known as whoami-rajat10:17
*** Luzi has joined #openstack-qa10:46
*** rcernin has quit IRC11:04
*** yamamoto has quit IRC11:06
*** paras333_ has joined #openstack-qa11:10
*** paras333_ has quit IRC11:11
*** eolivare_ has joined #openstack-qa11:12
*** tkajinam has quit IRC11:14
*** eolivare has quit IRC11:15
*** dviroel_ has joined #openstack-qa11:21
*** psahoo_ has joined #openstack-qa11:28
*** psahoo has quit IRC11:31
*** jpena is now known as jpena|lunch11:32
*** yamamoto has joined #openstack-qa11:32
*** dviroel_ is now known as dviroel__11:35
*** eolivare_ has quit IRC11:35
*** dviroel__ is now known as dviroel11:36
*** yamamoto has quit IRC11:39
*** rcernin has joined #openstack-qa11:45
openstackgerritOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/train: Update .gitreview for stable/train  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78620811:58
openstackgerritOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/train: Update TOX_CONSTRAINTS_FILE for stable/train  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78620911:58
openstackgerritOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/ussuri: Update .gitreview for stable/ussuri  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621011:59
openstackgerritOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/ussuri: Update TOX_CONSTRAINTS_FILE for stable/ussuri  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621111:59
openstackgerritOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621211:59
openstackgerritOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621311:59
*** rcernin has quit IRC11:59
openstackgerritOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/wallaby: Update .gitreview for stable/wallaby  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621411:59
openstackgerritOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/wallaby: Update TOX_CONSTRAINTS_FILE for stable/wallaby  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621511:59
*** vhari has quit IRC12:04
*** vhari has joined #openstack-qa12:10
yoctozeptoyay, branched12:12
*** eolivare_ has joined #openstack-qa12:16
*** hemanth_n has quit IRC12:24
*** Yarboa has quit IRC12:25
*** Yarboa has joined #openstack-qa12:26
*** nweinber has joined #openstack-qa12:27
*** jpena|lunch is now known as jpena12:29
*** Luzi has quit IRC12:44
*** ysandeep is now known as ysandeep|afk12:46
*** tkajinam has joined #openstack-qa13:14
dansmithyoctozepto: really hard to imagine what that could be, but I'll start looking yeah13:23
yoctozeptothanks dansmith13:23
*** ysandeep|afk is now known as ysandeep13:25
*** ricolin has joined #openstack-qa13:34
*** artom has joined #openstack-qa13:36
*** vhari has quit IRC13:37
*** anandgvb has joined #openstack-qa13:46
*** hemanth_n has joined #openstack-qa13:47
*** vhari has joined #openstack-qa13:50
*** hemanth_n has quit IRC13:52
gmannyoctozepto: kopecmartin mission for Tempest being branchless is to test all the supported stable branch with same set of tests to avoid the interop or backward compatibility issues. I do not think we need to go back to make it branched.13:53
rpittauhi all, it seems that ironic-grenade job in victoria is now fixed, in some way13:54
rpittaualthough we're still seeing an issue on master due to the cirros version13:54
rpittauor better, due to the CIRROS_VERSION variable in devstack13:54
yoctozeptoqa did not touch anything related afaik ;-)13:55
*** arxcruz has quit IRC13:57
rpittauyoctozepto: ok, then there's something wrong on how the value of CIRROS_VERSION gets set13:57
yoctozeptofor the record, I meant both the ironic-grenade and this :-)13:58
*** anandgvb has quit IRC13:59
*** arxcruz has joined #openstack-qa13:59
*** yamamoto has joined #openstack-qa14:01
rpittauyoctozepto: oh, I understand that, just wondering if I can get some help troubleshooting the issue14:04
rpittauthe only thing that I can see is that the value of CIRROS_VERSION from devstack is ignored by ironic plugin, but not by glance14:04
*** amodi has joined #openstack-qa14:06
yoctozeptorpittau: oh, sure; it's only on master now? this cirros14:06
rpittauyoctozepto: yeah14:06
yoctozeptohttps://opendev.org/openstack/ironic/src/commit/b4d8a493d9fdb693e64f3d0b11245523c2510665/devstack/common_settings#L1014:07
yoctozeptowell, you override this to 0.5.114:07
rpittauwe still have CIRROS_VERSION on 0.5.1 by default in ironic, I see that in devstack was moved to 0.5.2 in wallaby14:07
yoctozeptoyes14:07
rpittaummm ok but shouldn't that take 0.5.2 from devstack? Am I confusing the priority?14:08
dansmithyoctozepto: gmann: I'm just going to have to add some debug dumping for us to merge and run against the firehose I think14:10
rpittauI also tried updating that value to 0.5.2 in ironic but it still fails14:10
dansmiththe only thing I can think of that might have happened is if something in one of t hose job configs has forked the main shell and we arrived at the wait in a child instead of the parent14:10
yoctozeptorpittau: fails on what exactly?14:11
yoctozeptoI should have asked that first, shouldn't I? :D14:11
rpittauyoctozepto: glance reads the value from devstack and downloads 0.5.2, while ironic still reads 0.5.114:11
yoctozeptodansmith: can't it be done on affected projects?14:12
yoctozeptorpittau: you mean even after fixing it in ironic?14:12
yoctozeptoshow me14:12
rpittauyoctozepto: yeah14:12
dansmithyoctozepto: not sure what you mean14:12
rpittauyoctozepto: https://zuul.opendev.org/t/openstack/build/ef180709484d46c1b890e7a10000c30714:13
yoctozeptodansmith: I mean without merging debugging in the gate...14:13
yoctozeptocreate a debugging patch and depends-on in octavia and others14:13
yoctozeptorpittau: looking14:13
dansmithyoctozepto: with lots of rechecks you mean? sure, but the debug dump would be useful for the next time something happens, and will go quicker in the fire hose.. but whatever you want I guess14:13
gmanndansmith: yoctozepto its happening in many projects seems not just ocatvia http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22finished%20install_tempest%20with%20result%20127%5C%2214:14
yoctozeptohmm, since it's random14:14
yoctozeptodansmith might have a point here14:14
dansmithgmann: right14:14
yoctozeptothat it would be more efficient to just collect it14:14
yoctozeptoeh, eh14:14
yoctozeptowell, we released wallaby14:14
yoctozeptoso go ahead :D14:14
yoctozeptorpittau: I replied on the change14:16
*** slaweq has quit IRC14:16
yoctozeptorpittau: and one extra hint14:16
yoctozeptoideally, there would be no need to do these overrides14:17
yoctozeptoI see neutron has it overridden to 0.5.1 in a few places too, eh14:17
*** vhari has quit IRC14:17
rpittauyoctozepto: ok, I see the point, devstack should just take care of that14:18
yoctozeptoindeed14:19
yoctozeptobut then I see you are using this later for your own purposes14:19
rpittauyes, we're special :D14:19
yoctozeptoperhaps it's introduced because the value is not seen in the plugin14:19
yoctozeptoit was*14:19
rpittauI'll start to see if patching wallaby works fine14:19
*** slaweq has joined #openstack-qa14:20
rpittauthanks yoctozepto :)14:20
yoctozeptotry dropping this line altogether first, rpittau14:20
yoctozeptoyw14:20
rpittaualright, in case we want a specific version I guess we can set that as variable in the job14:21
rpittauI probably need to update wallaby first anyway14:21
yoctozeptoyes, that should be the way to go properly14:21
yoctozeptoyes, wallaby14:21
yoctozeptodrop the line in wallaby :D14:21
rpittauok, much clear now, thanks! :)14:21
yoctozeptothen drop the bass14:21
rpittau:D14:22
openstackgerritDan Smith proposed openstack/devstack master: Add some debug to async_wait failures  https://review.opendev.org/c/openstack/devstack/+/78625014:22
dansmithyoctozepto: gmann ^14:22
dansmithyoctozepto: gmann: looks like this: https://termbin.com/9kso14:23
dansmith(when forced to fail locally)14:23
yoctozeptoalmost like kernel panic14:23
yoctozeptodevstack panic14:24
dansmiththe failed ls is fine, it's just saying nothing else was running in the background.. else it will show those things14:24
yoctozeptodevstack screen of death14:24
dansmithyoctozepto: devstack guru meditation14:24
*** yamamoto has quit IRC14:24
yoctozeptolovin' it14:24
yoctozeptowe can keep this permanently14:25
yoctozeptoI was worried you had worse debugging in mind14:25
openstackgerritDan Smith proposed openstack/devstack master: Add some debug to async_wait failures  https://review.opendev.org/c/openstack/devstack/+/78625014:25
dansmith(small tweak)14:25
dansmithyoctozepto: right, I was expecting to just keep this14:26
yoctozeptodansmith: rightously14:26
yoctozepto+2 from me14:26
dansmiththanks14:26
yoctozeptoping gmann14:26
gmann+A14:33
gmannyoctozepto: these backports are ready https://review.opendev.org/q/Ic747ac9ddbb21a01e9dc18d8e8ad324d4d7d050d14:34
yoctozeptook14:34
*** dviroel is now known as dviroel|lunch15:07
*** hyang has joined #openstack-qa15:09
*** artom has quit IRC15:21
*** hyang has quit IRC15:21
*** hyang has joined #openstack-qa15:24
*** ysandeep is now known as ysandeep|away15:25
*** vhari has joined #openstack-qa15:31
*** rcernin has joined #openstack-qa15:39
*** vhari has quit IRC15:39
*** rcernin has quit IRC15:43
*** dviroel|lunch is now known as dviroel15:45
*** hyang has quit IRC15:59
*** gcheresh has quit IRC16:03
*** lucasagomes has quit IRC16:04
*** vhari has joined #openstack-qa16:06
openstackgerritMerged openstack/tempest master: Add live migration with trunk test  https://review.opendev.org/c/openstack/tempest/+/77468916:08
johnsomFYI, I am kicking the tires on the aync debug patch using the octavia tempest tests: https://review.opendev.org/78627516:15
*** rpittau is now known as rpittau|afk16:21
*** gfidente is now known as gfidente|afk16:25
*** hamalq has joined #openstack-qa16:28
*** psahoo_ has quit IRC16:45
*** ralonsoh has quit IRC16:45
*** eolivare_ has quit IRC16:48
yoctozepto++17:04
*** clarkb has quit IRC17:08
*** clarkb has joined #openstack-qa17:08
*** jpena is now known as jpena|off17:10
*** gcheresh has joined #openstack-qa17:11
johnsomLooks like that async debug patch is bad:17:13
johnsomhttps://www.irccloud.com/pastebin/skhViOKg/17:13
johnsomhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e33/786275/1/check/octavia-v2-dsvm-scenario/e33611c/job-output.txt17:13
dansmithjohnsom: no, that's expected17:13
dansmiththe ls thing I mean.. it means no other jobs were waiting17:13
dansmiththe cat failure means that the process we're waiting for is gone (i.e. already waiting)17:14
dansmither, already waited-for17:14
johnsomlol, ok... So we have some results! grin17:14
johnsomWhat am I looking for here. We have a number that passed and some that failed.17:15
johnsomLooks like the passing is stable branch, so probably doesn't have the async change that is failing17:15
johnsomHmm, that one I linked still says install failed with 127 though17:16
johnsom[2674 Async install_tempest:22726]: finished install_tempest with result 127 in 37 seconds17:16
dansmithjohnsom: the install is succeeding, as you can see. it's that our wait() on the child fails17:16
johnsom2021-04-14 16:37:38.438202 | controller | [2674 Async install_tempest:22726]: Waiting for completion of install_tempest running on PID 22726 (1 other jobs running)17:19
johnsom2021-04-14 16:37:38.440901 | controller | /opt/stack/devstack/inc/async: line 152: wait: pid 22726 is not a child of this shell17:19
dansmithunfortunately it seems like $$ isn't giving me what I want17:20
dansmiththis debug patch hasn't started running in the gate, so I'm going to tweak it and pull it out17:20
johnsomOk. Let me know if I can help in some way17:21
openstackgerritDan Smith proposed openstack/devstack master: Add some debug to async_wait failures  https://review.opendev.org/c/openstack/devstack/+/78625017:23
dansmithjohnsom: if you could recheck your DNM ^17:23
johnsomYep, NP17:23
johnsom786275 in zuul17:25
dansmithjohnsom: just to give you some context here, what we've done is run the install_tempest function with &, recorded its pid, and then later we call wait on that pid17:27
dansmithand bash is saying that that pid is not a child we can wait for17:27
dansmithit's clear that it's running because we see it finish well above the point at which we wait for it17:27
johnsomYeah, just coming up to speed on this now.17:28
dansmithso either we double-waited for it, or we're calling wait from not the original parent somehow17:28
johnsomWhere would the console log for the child end up (once this is resolved)?17:29
dansmithwe cat it when we do the wait (which is displayed), but it's also preserved in /opt/stack/async/install_tempest.log17:30
johnsomOk17:30
dansmithwe also shouldn't really be able to double-wait,17:31
dansmithbecause in order to wait, we cat the ini file for that task that we created, and then wait  on that pid17:31
dansmithso unless there's a crazy race in there, it surely seems like the task must be still ready for us to wait, but we're not the parent17:32
clarkband linux shouldn't be reusing pids until the parent waits and reaps them right?17:33
clarkb(maybe bash does some magic around that?)17:33
dansmithright, until we wait, the child is still "running" in Z state17:33
*** artom has joined #openstack-qa17:34
clarkbjust talking out loud here: if the child double forks and changes process groups it would reassocaite to init as its parent (daemonizing essentially) any chance something like that is happening?17:34
johnsomI would be super shocked if we are in a use case reusing PIDs.... lol17:34
johnsomWe have a lot of tests and such, but.... grin17:34
dansmithclarkb: a grandchild could do that, but we should still have the original child to wait for17:35
clarkbdansmith: ah right, we would never see the grandchild in taht case17:35
dansmithclarkb: but yeah I thought about that17:35
dansmithclarkb: right, so that could cause us to exit our wait before all the work has been done, but otherwise unrelated I think17:35
clarkb++17:35
dansmithjohnsom: you're loading a lot of plugins.. are you doing anything in the other stack phases like test_config  in any of those?17:38
*** rcernin has joined #openstack-qa17:39
johnsomI don't think so. Here is a successful run if you want to search: https://zuul.opendev.org/t/openstack/build/5d411e88a2394062994dbf3660172919/logs17:40
*** dtantsur is now known as dtantsur|afk17:44
*** rcernin has quit IRC17:44
dansmithjohnsom: and to be clear, it's not 100% fail right?17:44
dansmiththat kinda makes it even weirder17:44
johnsomYeah, that is absolutely the strange thing17:45
johnsomThe successful run I just linked was last night. Plus rechecks of the same patch will sometimes succeed.17:46
yoctozeptoeh the random issues17:46
dansmithokay... so...17:49
dansmithon a successful run it almost looks like install_tempest gets called twice17:49
dansmithwell, maybe not, maybe that's just plugin config after the install in the successful case17:54
*** gcheresh has quit IRC17:55
johnsomdansmith https://65eec222656bf012f685-8fedbdeaaaf080ca764e90bf11123056.ssl.cf5.rackcdn.com/786275/1/check/octavia-v2-dsvm-scenario/32e9a5f/job-output.txt18:05
johnsomFailure with the new patch18:05
yoctozeptowell, whatever you find out, I guess it might need fixing on both sides - for devstack to easy spot the oddity and octavia (and others) to avoid the oddity18:10
johnsomSo, looking closer at the logs. It's clear that job finished before the "wait" was called. Calling wait in bash on a non-existent PID will throw a 127 error. It's this just a bug that we are attempting to wait on a PID that already exited?18:10
yoctozeptoI might find some cycles later this week18:10
yoctozeptojohnsom: ooh, this!18:11
yoctozeptothat makes total sense18:11
clarkba child process shouldn't fully exit until its parent has waited on it and reaped it though18:11
yoctozeptobut why do we wait for it then?18:11
yoctozeptoclarkb: yes, but if we wait twice...18:11
yoctozeptonow though why do we? :D18:11
clarkbright that could explain it (the double wait theory)18:12
yoctozeptodansmith: ^^18:12
yoctozeptoDan knows his code so will find out18:13
* yoctozepto goes to his well-deserved rest18:13
dansmithjohnsom: no, the early "finish" message is printed by the child itself, basically saying "okay I'm done" and then later when we wait, we cat the console so you can see what the job was doinbg18:13
dansmiththere's only one wait, and that's where it's failing18:14
dansmithyeah, so that most recent fail seems to show that the pids are all correct.. i.e. it's the same parent waiting for the same child18:17
dansmithand this: cat: /proc/22719/status: No such file or directory18:17
dansmithreally means that the child is gone from the system18:17
dansmithand that pid does not get waited for multiple times (by us)18:20
jparkerartom: for https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/776112/7/whitebox_tempest_plugin/config.py#238 would I be updating zuul.yaml or the devstack settings?  I thought the zuul.yaml parameters were for how the environment was deployed.18:31
artomjparker, yeah, the devstack settings can work too18:32
jparkerartom: ack ty18:33
artomjparker, I agree the zuul/devstack settings thing can be confusing...18:33
artomDevstack can install whitebox (via the plugin) without Zuul18:33
artomAnd Zuul uses Devstack...18:33
artomSo it's a matter of "does the value that we're setting apply for *all* Devstack deployments, even those folks do manually on their own machines/VMs?"18:34
artom"Or is this something specific to our Zuul CI?"18:34
artomI realize it's *very* academic18:35
artomBecause in practice they're one and the same18:35
artomBut that's my reading of it18:35
jparkerartom: does it make more sense to just default to True?18:37
artomjparker, for the "allow_disabling" thing? Don't see why not18:38
*** whoami-rajat has quit IRC18:38
openstackgerritDan Smith proposed openstack/devstack master: Add some debug to async_wait failures  https://review.opendev.org/c/openstack/devstack/+/78625018:38
artomIt's True in upstream master, and is downstream OSP1618:38
artom*in18:38
jparkerartom: ack ok I'll just update the default18:39
jparkerartom: also not sure if we should set up some discussion about pinning whitebox for a py27 13 only deployment sometime soon18:39
*** jlvillal has joined #openstack-qa18:42
*** vishalmanchanda has quit IRC18:44
openstackgerritGhanshyam proposed openstack/devstack-plugin-ceph master: Remove the stable branch jobs from master gate  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78630718:53
openstackgerritGhanshyam proposed openstack/devstack-plugin-ceph stable/wallaby: Remove the stable branch jobs from stable/wallaby gate  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78630818:56
openstackgerritJames Parker proposed openstack/whitebox-tempest-plugin master: Test allow disabling CPU flags  https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/77611218:57
openstackgerritGhanshyam proposed openstack/devstack-plugin-ceph stable/victoria: Remove the stable branch jobs from stable/victoria gate  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78630919:08
openstackgerritGhanshyam proposed openstack/devstack-plugin-ceph stable/wallaby: Remove the stable branch jobs from stable/wallaby gate  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78630819:10
openstackgerritGhanshyam proposed openstack/devstack-plugin-ceph stable/victoria: Remove the stable branch jobs from stable/victoria gate  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78630919:12
dansmithjohnsom: clarkb: I dunno if you'd know, but any chance this job is running inside a container19:13
dansmith?19:13
johnsomShould not be, no.19:13
johnsomIt's just a standard dsvm job that runs tempest. We use the standard zuul templates.19:14
dansmithack, I wouldn't have thought, just looking for reasons why it would have been different19:15
dansmithbased on this:19:15
dansmithhttps://bugs.launchpad.net/ubuntu/+source/lxd/+bug/159000119:15
openstackLaunchpad bug 1590001 in lxd (Ubuntu) "bash complains wait: pid is not a child of this shell" [Undecided,Invalid]19:15
dansmithlooks like bash's wait() no longer translates directly to the waitpid() system call and it does its own processing of states, but that it can lose track19:16
dansmithand about 200 children seemed to trigger it in that bug19:16
dansmithwhich I'm sure we're near or above, so just wondering if that's what is happening19:17
openstackgerritGhanshyam proposed openstack/devstack-plugin-ceph stable/wallaby: Remove the stable branch jobs from stable/wallaby gate  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78630819:17
dansmithand,19:18
dansmiththat's by far the oldest child19:18
openstackgerritGhanshyam proposed openstack/devstack-plugin-ceph stable/ussuri: Remove the stable branch jobs from stable/ussuri gate  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78631019:20
openstackgerritGhanshyam proposed openstack/devstack-plugin-ceph stable/train: Remove the stable branch jobs from stable/train gate  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78631119:25
*** sboyron has quit IRC19:30
openstackgerritDan Smith proposed openstack/devstack master: Add some debug to async_wait failures  https://review.opendev.org/c/openstack/devstack/+/78625019:37
*** gcheresh has joined #openstack-qa19:52
openstackgerritMerged openstack/devstack-plugin-ceph stable/wallaby: Update .gitreview for stable/wallaby  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621419:54
openstackgerritMerged openstack/devstack-plugin-ceph stable/wallaby: Update TOX_CONSTRAINTS_FILE for stable/wallaby  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621519:54
*** nweinber has quit IRC20:00
*** gcheresh has quit IRC20:10
dansmithclarkb: you might be interested: https://github.com/mfragkoulis/bash/blob/master/jobs.c#L421320:12
dansmithCHILD_MAX defaults to 32, by the way20:13
dansmithjohnsom: latest patch passes your DNM, but we probably need more than one run to confirm a fix20:19
johnsomYep, it's intermittent for sure20:25
dansmithjohnsom: so if it's cool, I will recheck your patch when this is done, before I go clean up the devstack patch for actual merge if it succeeds again20:26
dansmithjohnsom: it sounded like it was pretty frequent and that two successive passes would be a good indication that it's better right?20:26
johnsomdansmith Be my guest. Yeah, it was repeatable, just not every time.20:27
dansmithjohnsom: what I meant was.. rechecking and then revising my patch to be cleaned up will take a little longer serial time,20:30
dansmithI dunno how panicked you are about your gate being unstable20:30
johnsomWe are not in “critical” mode at the moment20:32
johnsomThis week would be nice. Grin20:33
dansmithack, well, I'll recheck this in a bit and try to get it cleaned up this afternoon after that second run if things look good20:33
clarkbdansmith: ah so it is doing magic after all?20:38
dansmithclarkb: yeah, I think it's actually catching SIGCHLD, recording the status and then just making wait read from that status20:38
*** rcernin has joined #openstack-qa21:02
dansmithjohnsom: second run also worked21:25
dansmithjohnsom: agree that two in a row was unlikely before the fix?21:25
johnsomAre you watching the console on that DNM patch?21:26
johnsomAh, yeah, I see it started.21:26
johnsomI would lean towards three to be sure21:27
dansmithaight21:30
dansmithwell, I'll get the cleanup going and then we can recheck and get the third on top of that21:31
johnsomSounds like a pan21:31
johnsomplan21:31
openstackgerritMerged openstack/grenade stable/wallaby: Make heat and octavia grenade jobs as voting  https://review.opendev.org/c/openstack/grenade/+/78593021:35
openstackgerritMerged openstack/grenade stable/ussuri: Make heat and octavia grenade jobs as voting  https://review.opendev.org/c/openstack/grenade/+/78593221:35
openstackgerritDan Smith proposed openstack/devstack master: Add some debug to async_wait failures  https://review.opendev.org/c/openstack/devstack/+/78625021:36
openstackgerritDan Smith proposed openstack/devstack master: Work around CHILD_MAX bash limitation for async  https://review.opendev.org/c/openstack/devstack/+/78633021:36
dansmithjohnsom: ^21:37
johnsomNice, thanks~!21:37
*** rcernin has quit IRC21:45
*** rcernin has joined #openstack-qa21:46
*** rcernin has quit IRC22:20
*** yamamoto has joined #openstack-qa22:30
*** tkajinam has quit IRC22:43
*** tkajinam has joined #openstack-qa22:52
*** hamalq has quit IRC22:57
*** hamalq has joined #openstack-qa22:57
*** yamamoto has quit IRC22:59
*** yamamoto has joined #openstack-qa22:59
*** rcernin has joined #openstack-qa23:06
openstackgerritMerged openstack/devstack-plugin-ceph stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621223:10
openstackgerritMerged openstack/devstack-plugin-ceph stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78621323:10
*** tosky has quit IRC23:29
dansmithjohnsom: looks like we're good on #3 too23:49
johnsom+1, I have already given my +1 vote on your patch. Thanks for the work on this.23:49
dansmithjohnsom: ah I see now23:50
*** irclogbot_0 has quit IRC23:50
*** irclogbot_3 has joined #openstack-qa23:56
« Tuesday, 2021-04-13 Index Thursday, 2021-04-15 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!