Wednesday, 2019-10-16

*** apetrich has quit IRC02:08
*** brault has joined #oooq02:09
*** ykarel has joined #oooq02:50
*** dsneddon has quit IRC04:05
*** epoojad1 has joined #oooq04:06
*** dsneddon has joined #oooq04:09
*** dsneddon has quit IRC04:15
*** soniya29 has joined #oooq04:16
*** raukadah is now known as chandankumar04:29
*** skramaja has joined #oooq04:36
*** dsneddon has joined #oooq04:44
*** dsneddon has quit IRC04:49
*** dsneddon has joined #oooq05:01
*** marios has joined #oooq05:02
*** dsneddon has quit IRC05:05
*** surpatil has joined #oooq05:05
*** dsneddon has joined #oooq05:08
*** marios is now known as marios|rover05:08
*** dsneddon has quit IRC05:13
*** ratailor has joined #oooq05:23
*** ratailor has quit IRC05:29
*** udesale has joined #oooq05:44
*** dsneddon has joined #oooq05:49
*** matbu has joined #oooq05:51
*** saneax has joined #oooq05:52
*** ksambor has joined #oooq05:55
*** brault has quit IRC06:06
*** brault has joined #oooq06:08
*** dsneddon has quit IRC06:35
*** holser has joined #oooq06:42
*** brault has quit IRC06:49
*** brault has joined #oooq06:50
chandankumarmarios|rover: Good morning06:51
chandankumarmarios|rover: Did we get the fix for cache ?06:51
marios|roverchandankumar: yes merged little while ago06:51
*** apetrich has joined #oooq06:55
*** amoralej|off is now known as amoralej07:02
*** dsneddon has joined #oooq07:03
*** tesseract has joined #oooq07:12
*** tesseract has quit IRC07:14
*** tesseract has joined #oooq07:14
*** tosky has joined #oooq07:17
*** akahat has joined #oooq07:29
*** jpena|off is now known as jpena07:39
*** surpatil has quit IRC07:41
*** ykarel is now known as ykarel|lunch07:42
*** dsneddon has quit IRC07:49
*** dsneddon has joined #oooq07:51
zbrnobody doing promoter stuff this sprint?07:53
zbrpanda: morning! are you back?07:53
zbrdid you see #rdo discussions? we have molecule problems on ci-config.07:54
*** dsneddon has quit IRC07:56
*** jfrancoa has joined #oooq07:57
*** dsneddon has joined #oooq08:17
*** ykarel|lunch is now known as ykarel08:21
*** dsneddon has quit IRC08:26
arxcruzmarios|rover: 5 hours :/08:42
arxcruzchandankumar: ping08:42
marios|roverarxcruz: which bit08:42
arxcruzmarios|rover: https://review.rdoproject.org/r/#/c/23138/08:42
marios|roverarxcruz: yeah but the cache thing was reverted so we should be good now08:45
arxcruzmarios|rover: ack08:45
marios|roverarxcruz: see https://bugs.launchpad.net/tripleo/+bug/184758508:45
openstackLaunchpad bug 1847585 in tripleo "periodics centos-7-standalone-full-tempest-master + centos-7-ovb-1ctlr_2comp-featureset020-master timeout tempest tests" [Critical,Fix released] - Assigned to yatin (yatinkarel)08:45
*** surpatil has joined #oooq08:46
*** ykarel is now known as ykarel|away08:47
*** dsneddon has joined #oooq08:54
*** ykarel|away has quit IRC09:06
*** dtantsur|afk is now known as dtantsur09:07
zbrmarios|rover: arxcruz panda (others too), please answer on https://github.com/pycontribs/pytest-molecule/issues/4509:20
*** panda is now known as panda|drappt09:23
*** dsneddon has quit IRC09:40
*** dsneddon has joined #oooq09:46
*** dsneddon has quit IRC09:51
*** dsneddon has joined #oooq10:18
*** dsneddon has quit IRC10:23
marios|roverrfolco|ruck: filed that fyi the periodic master failed earlier cos the promote consistent->ci-testing timeout https://bugs.launchpad.net/tripleo/+bug/184832510:25
openstackLaunchpad bug 1848325 in tripleo "intermittent timeout in promote-consistent-to-tripleo-ci-testing job - "combination not found in RHEL"" [High,Triaged]10:25
marios|roverrfolco|ruck: seems quite rare (last time was on the 4th) so hopefully won't get it again on the next run... hoping master will be clear now we got unlucky with that ^10:26
rfolco|ruckmarios|rover, that is really weird10:26
rfolco|ruckmarios|rover, thx for opening the bug10:27
marios|roverrfolco|ruck: but we did well with the train push yesterday \o/10:27
rfolco|ruckmarios|rover, I still don't know how weshay fixed temporarily the report bit, it was reporting a different hash10:28
marios|roverrfolco|ruck: i think he included depends-on your fix? /me check10:30
marios|roverrfolco|ruck: heh no he hard coded it10:30
marios|roverrfolco|ruck: https://review.opendev.org/#/c/688765/2/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-train.yml10:30
marios|roverrfolco|ruck: he has depends on that ^ in https://review.rdoproject.org/r/#/c/23141/10:31
*** kopecmartin|off is now known as kopecmartin10:31
rfolco|ruckmarios|rover, specifying hash there wasn't enough. dlrnapi report result uses tripleo-ci-testing hash, which is not the same.10:34
rfolco|ruckmarios|rover, will ask him when he joins10:34
rfolco|ruckunless he made manual calls to report result for each job10:35
marios|roverrfolco|ruck: yeah i saw your patch.. i just thought he was overriding with that. ok weshay can confirm when he is in10:36
marios|roverrfolco|ruck: but either way nice to have a better report for today's call ;)10:36
rfolco|ruckmarios|rover, indeed10:37
*** skramaja has quit IRC10:46
*** dsneddon has joined #oooq10:53
*** rfolco|ruck is now known as rfolco|brb11:01
*** ratailor has joined #oooq11:20
weshaymarios|rover, rfolco|brb will explain in a bit.. have to setup for my wife's bday11:20
marios|roverweshay: ack11:21
*** jpena is now known as jpena|lunch11:27
*** ratailor has quit IRC11:30
marios|roverrfolco|brb: weshay panda|drappt chandankumar vote there please fixes nit when you have a minute thanks https://review.rdoproject.org/r/#/c/23135/11:30
chandankumarmarios|rover: do we still need it, since cache issue patch got reverted and merged11:31
marios|roverchandankumar: yes11:32
marios|roverchandankumar: its a nit11:32
marios|roverchandankumar: the timeout is in the wrong place11:32
marios|roverchandankumar: (its in vars, instead of at job variables)11:32
marios|roverchandankumar: the patch was oringally for time bump but instead fixing that there11:33
chandankumarmarios|rover: yes, thanks!11:33
marios|roverchandankumar: ack thanks11:33
*** dsneddon has quit IRC11:35
chandankumarmarios|rover: we also need a master promotion with cache issue reverted patch11:36
*** amoralej is now known as amoralej|lunch11:37
soniya29Hello everyone, I want to load data in mariadb in order to visualize it in cockpit. How can I do that, I mean, do I need to write separate sql file?11:41
marios|roversoniya29: if you run the cockpit locally, it will source data from the various zuul ... is that not enough? otherwise i don't know about doing what you want... like you want to load some custom data or something?11:44
soniya29marios|rover, Yeah I want other custom data to load.11:45
weshaymarios|rover, rfolco|brb back11:46
marios|roversoniya29: i think likely sshnaidm|pto is your best bet for something like that but looks like he is out11:46
marios|roverweshay: want to catch up after the prog call ?11:47
weshaymarios|rover, rfolco|brb I wrote an explanation regarding which patch got the promotion11:47
weshaymarios|rover, I'll try to.. lots-o-meetings11:48
marios|roverweshay: ack... no i thought you wanted to sync with 14:46 < weshay> marios|rover, rfolco|brb back11:48
weshayif panda|drappt is still at the doctor than sure11:48
weshaymarios|rover, will need to "resume build" on https://ci.centos.org/job/tripleo-quickstart-promote-train-current-tripleo-delorean-minimal/11:50
weshaymarios|rover, rfolco|brb /me was concerned w/ the gate late last night... did you see my notes on that?11:50
marios|roverweshay: yeah https://bugs.launchpad.net/tripleo/+bug/1848275 you mean?11:51
openstackLaunchpad bug 1848275 in tripleo "rpm DLRN builds failing due to ctlplane entry missing from /etc/hosts " [Critical,In progress] - Assigned to Harald Jensås (harald-jensas)11:51
weshayah.. harald has been awesome lately11:51
marios|roverweshay: looked to be recoverign this morning ... but then had some chat with harald we also need that https://review.opendev.org/#/c/688890/11:51
weshaynice11:52
marios|roverweshay: our % pass went down as far as 73 ... i thought it was going up up but it just went 73.4 -> 73.3 ...11:52
marios|roverweshay: problem was when i checked this morning all the fails in the list were from 15th, not 16th which is why i thought the problem was resolved11:53
soniya29marios|rover, If we see the directory structure, particularly, ci-config/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/ there are certain sql files. I roughly guess, something similar I need to do. But, I am not sure.11:54
marios|roverweshay: not sure what you mean by that bit 14:50 < weshay> marios|rover, will need to "resume build" on11:54
marios|rover                https://ci.centos.org/job/tripleo-quickstart-promote-train-current-tripleo-delorean-minimal/11:54
weshaymarios|rover, it would also be smart to run a testproject on master11:55
*** rfolco|brb is now known as rfolco|ruck11:55
weshayto get that updated11:55
marios|roverweshay: also we got very unlucky with master periodic earlier because of https://bugs.launchpad.net/tripleo/+bug/184832511:55
openstackLaunchpad bug 1848325 in tripleo "intermittent timeout in promote-consistent-to-tripleo-ci-testing job - "combination not found in RHEL"" [High,Triaged]11:55
rfolco|ruckmarios|rover, weshay will keep an eye on gate and lets try fixing master promotion. One thing that I did not understand is the linters failures on gate11:57
weshayrfolco|ruck, fyi.. rdo-tox-molecule is busted..https://review.rdoproject.org/r/#/c/23144/11:57
weshayshouldn't run on my cockpit change11:57
marios|roverweshay: https://review.rdoproject.org/r/#/c/23156/11:57
rfolco|ruckand should not fail11:58
weshayaye11:58
marios|roverrfolco|ruck: weshay zbr was checking there re tox-molecule... weshay i saw your note about "and need molecule and promotion tests NOT to run against everything in ci-config, just the promoter work" but     done already ? https://github.com/rdo-infra/ci-config/blob/6946229b2f55afab80a84aa7ca9440e3f5ed2073/zuul.d/layout.yaml#L14-L45 or you mean something else  ?11:58
chandankumarweshay: https://review.opendev.org/#/c/688860/111:59
weshaymarios|rover, it's blocking https://review.rdoproject.org/r/#/c/23144/11:59
marios|roverweshay: so you mean we should run all the master jobs in testproject ... should i worry about the periodic-master pipeline running (do i need to coordinate)11:59
*** dsneddon has joined #oooq12:00
marios|roverweshay: also not clear how you overcame the force_periodic and specifying your own hash issue... maybe i just need to depends-on rfolco fix https://review.rdoproject.org/r/#/c/2315012:00
weshaymarios|rover, just the failing ones12:00
marios|roverweshay: ack in which case will have to wait for next run and do it12:01
weshaymarios|rover, why wait?12:01
marios|roverweshay: i thought you mean the failing ones from the pipeline12:01
zbrmarios|rover: this means that we can fix rdo-tox-molecule by adding `PYTEST_ADDOPTS='-k "not delegated"'` to  it.12:01
marios|roverweshay: but it didn't run earlier because of the consistent->ci-testing  issue12:01
weshaymarios|rover, to push http://rhos-release.virt.bos.redhat.com:3030/rhosp master from red to green12:01
marios|roverweshay: so just fulltempest, fs20 (failing ones from yesterday ) k will post it and we can add there if needed ack12:02
weshaymarios|rover, check the previous run results.. perhaps we're only blocked on the tempest timeout12:02
marios|roverweshay: kk12:02
zbrwe could even add this to the template12:02
*** rlandy has joined #oooq12:14
rlandychandankumar: hi - you left two taiga task with a note for me to look at them12:19
rlandyI need to finish with the downstream/ internal stuff12:20
chandankumarrlandy: Hey, welcome back!12:24
rlandychandankumar: thanks ... I can look at those tasks once I am done with downstream12:24
rlandy2019-10-16 01:02:16.439609 | primary | TASK [Copy clouds.yaml file from user's home to /root directory] ***************12:25
rlandy2019-10-16 01:02:16.556271 | primary | Wednesday 16 October 2019  01:02:16 +0000 (0:00:01.748)       0:00:31.261 *****12:25
rlandy2019-10-16 01:02:18.393030 | primary | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the remote, see the remote_src option12:25
rlandy^^ still failing12:25
* rlandy wonders12:25
chandankumarrlandy: in downstream?12:26
rlandyyeo12:26
rlandyyep12:26
rlandythere was a patch for that12:26
* rlandy looks for it12:26
chandankumarrlandy: master one?12:26
rlandyyes12:26
chandankumarrlandy: please check whether clouds.yaml exists or not12:27
marios|roverrfolco|ruck: weshay https://review.rdoproject.org/r/#/c/23159/ for master full tempest/fs20 that were failing yesterday ... dind't specify a hash it will use tripleo-ci-testing k?12:27
rlandyarxcruz: ^^ didn't you have a patch to fix this12:28
rfolco|ruckmarios|rover, I think its ok for master12:28
weshaymarios|rover, I don't know.. I'd have to look at the results.. that's kind of like shooting w/o looking :)12:28
arxcruzrlandy: nope12:28
marios|roverweshay: ack so we should use same hash as the failing ones yesterday then sounds like12:28
rlandyweshay did12:29
marios|roverweshay: will update12:29
rlandylooking12:29
rlandyarxcruz: chandankumar: https://review.opendev.org/#/c/688148/12:30
*** jpena|lunch is now known as jpena12:31
* rlandy updates that12:32
marios|roverweshay: rfolco|ruck updated https://review.rdoproject.org/r/#/c/23159/2/.zuul.yaml12:33
*** amoralej|lunch is now known as amoralej12:42
rlandyweshay: marios|rover: chandankumar: https://review.opendev.org/#/c/688148/ updated - pls revote12:49
chandankumarrlandy: commit msg needs update12:52
chandankumarrlandy: for undercloud and standalone cloud.yaml got dump at two places12:53
chandankumaretc and user home dir .config/openstack12:53
rlandychandankumar: I just followed the comments12:53
chandankumarand for overcloud it is in user's home dir12:53
marios|roverrfolco|ruck: ack12:53
rlandypls comment and I will update12:53
marios|roverchandankumar: the original purpose changed i think ... it was originally for /etc12:54
marios|roverrlandy: i think you just need to update the title of the commit sec12:54
chandankumarmarios|rover: https://review.opendev.org/#/c/688148/3/playbooks/tempest.yml12:54
rlandyyeah ok12:54
chandankumarmarios|rover: did not understand12:54
marios|roverchandankumar: the title is wrong... it is no longer doing that12:54
chandankumarwhat is no longer applies after v212:54
*** soniya29 has quit IRC12:54
chandankumar?12:55
marios|roverchandankumar: it now is just about adding remote_src12:55
*** Goneri has joined #oooq12:55
marios|roverchandankumar: see comments on the review12:55
rlandymarios|rover: chandankumar: updated commit12:56
marios|roverack rlandy12:56
*** chem has quit IRC12:57
*** chem has joined #oooq12:58
rlandythanks all12:59
rlandybaremetal master job thanks you too13:00
rfolco|ruckweshay, rlandy chandankumar pls review this one https://review.rdoproject.org/r/#/c/23150/13:00
marios|roverzbr: tox linters broken? know anything about that? 2019-10-16 11:34:43.523401 | container |         AttributeError: 'SpecifierSet' object has no attribute 'split'13:01
marios|roverzbr: http://logs.rdoproject.org/35/23135/3/check/tox-linters/026d36a/job-output.txt13:01
*** SurajPatil has joined #oooq13:01
zbrmarios|rover: yes, i know. i failed bug moments ago. i was working on it.13:01
zbrquite complex issue but there is a workaround13:01
marios|roverzbr: cool thanks have the bug #?13:02
zbrmarios|rover: https://github.com/ansible/ansible-lint/issues/590 and workaround https://github.com/ansible/molecule/pull/2351/files13:02
*** epoojad1 has quit IRC13:02
marios|roverzbr: thank you13:02
zbrit was fixed in molecule, but not in ansible-lint, potential impact LOTS of projects.13:02
zbranother workaround is to *stop* adding workarounds to setup.py - i avoided it by dropping almost everything from setup.py on other projects.13:03
*** dsneddon has quit IRC13:03
*** surpatil has quit IRC13:03
zbransible-lint and ansible itself are two projects that have "nightmare setup.py examples"13:03
marios|roverzbr: well it blocks us on rdo-jobs gate e.g. there (where i saw it first) https://review.rdoproject.org/r/#/c/23135/13:03
marios|roverrfolco|ruck: tox-linters borked ^^^ (added note on the ethepad just now)13:04
zbrwell, is not my project....13:04
marios|roverzbr: ack :) yeah not blaming you just agreeing it has a big impact for us too13:04
marios|roverbut obviously we all know its your fault zbr13:04
zbryou could argue that setuptools broke it. but i bet setuptools guys have a different oppinion ;)13:04
marios|rovereven though no one is saying it13:04
zbri know, when lint strinkes, is must be zbr fault13:05
weshaymarios|rover, rfolco|ruck happy to sync now if you like13:06
rfolco|rucklets sing togetha13:06
marios|roverweshay: k13:06
weshayhttps://meet.google.com/hfu-icdo-kew13:07
*** derekh has joined #oooq13:09
chandankumarrlandy: did we have a fix for waiting on logger issue in reproducer https://bugs.launchpad.net/tripleo/+bug/1833465?13:13
openstackLaunchpad bug 1833465 in tripleo "tripleo reproducer fails w/ "waiting on logger"" [Critical,Triaged]13:13
chandankumarrlandy: please have a look at this one https://review.opendev.org/#/c/688860/ when free13:13
rlandychandankumar: on  https://bugs.launchpad.net/tripleo/+bug/1833465 - not that I know of13:14
openstackLaunchpad bug 1833465 in tripleo "tripleo reproducer fails w/ "waiting on logger"" [Critical,Triaged]13:14
chandankumarweshay: one question when using force_periodic in the job definition it overrides the docker registry na/13:14
chandankumar?13:14
rlandyI have a side project to look at my old reproducer work13:14
chandankumarrlandy: :-)13:15
chandankumarrlandy: weshay https://trunk.rdoproject.org/rhel8-train/ it got created13:18
rlandywoohoo13:19
chandankumarhehe more work13:19
rlandychandankumar: wrt downstream - to update you ... release file for rhos-16 is merged ... standalone job for rhos-16 ( with rhos-14 removed) is being merged now ... env C is jobs works except for one node - investigating that now . I still have to bring up a train job and add to cockpit - change logging13:21
chandankumarrlandy: ok, thanks :-)13:21
rlandyweshay: fyi - now I spewed my status ^^13:21
chandankumarrlandy: RHEL-8 train repo is still populating all the contents are not there13:28
chandankumarwe might work on that from next week13:28
rlandychandankumar: k - let's touch base then and we can divide the work13:29
chandankumarrlandy: sure13:29
rlandyI might be done downstream by then13:29
rlandyweshay: rfolco|ruck: this afternoon, can one of you guys fill me in on the whole '3rd Party CI and SRIOV' conversation?13:31
weshayrfolco|ruck, /me checks calendar13:32
weshayrlandy,  moved up your 10113:32
weshay1-113:32
rfolco|ruckok you meant rlandy13:32
weshayah ya13:32
rfolco|ruckrlandy, will try to register folco this time.... last time I tried I had exceeded my quota13:33
rfolco|ruckwill be a big win for both of us13:33
rfolco|rucklol13:33
chandankumarpanda|drappt: let me know when you are back.13:34
zbrmarios|rover: ansible-lint patch passed CI, now I am trying to poke ansible team to merge it and to make a new release. As you can image today they are busy... pressing the release button on ansible 2.913:35
zbrhttps://github.com/ansible/ansible-lint/pull/59113:35
*** dsneddon has joined #oooq13:36
marios|roverthanks zbr rfolco|ruck ^^^ fyi for that tox-linters bug 16:35 < zbr> https://github.com/ansible/ansible-lint/pull/59113:36
openstackbug 16 in Launchpad itself ""Swedish" and "Swedish (Sweden)" should be the same language" [Medium,Fix released] https://launchpad.net/bugs/16 - Assigned to Dafydd Harries (daf)13:36
marios|roverexception ^13:36
*** udesale has quit IRC13:37
*** udesale has joined #oooq13:37
rlandyrfolco|ruck: wait - your 1-1 or mine got moved up???13:39
rfolco|ruckrlandy, not mine13:39
rfolco|ruckrlandy, weshay was talking to you13:39
rlandyok13:40
*** saneax has quit IRC13:40
rlandyreally we only match irc nicks by one letter13:40
*** dsneddon has quit IRC13:41
*** surpatil has joined #oooq13:51
weshaylolz13:52
*** SurajPatil has quit IRC13:53
*** amoralej is now known as amoralej|afk14:06
*** jfrancoa has quit IRC14:08
*** jfrancoa has joined #oooq14:08
marios|roverpanda|drappt: zbr rfolco|ruck rlandy when you have a minute please limits files: for rdo-tox-molecule https://review.rdoproject.org/r/#/c/23212/14:09
*** akahat has quit IRC14:14
rlandymarios|rover: left comment14:14
marios|roverrlandy: ack thanks14:14
*** derekh has quit IRC14:15
*** derekh has joined #oooq14:15
zbrmarios|rover: the correct molecule limit is at https://review.rdoproject.org/r/#/c/23158/ -- but depends on fixing ansibe-lint14:15
marios|roverrlandy: container-push already there adding others14:15
*** dsneddon has joined #oooq14:15
rlandyack ok14:16
marios|roverrlandy: not sure about build-images.sh i can only see it referenced in one place don't think we use it14:17
rlandymarios|rover: wasn't sure about that one14:18
rlandymaybe just the script then14:18
marios|roverrlandy: ack thanks for checking14:18
*** dsneddon has quit IRC14:20
chandankumarpanda|drappt: software factory zuul app is installed now on containers/libpod repo14:34
chandankumarpanda|drappt: we are good to go ahead with further patches14:34
chandankumartime to go now, see ya tomorrow14:35
*** chandankumar is now known as raukadah14:35
*** amoralej|afk is now known as amoralej14:43
*** dtrainor has quit IRC14:46
*** surpatil has quit IRC14:49
*** surpatil has joined #oooq14:52
*** dsneddon has joined #oooq14:54
*** dsneddon has quit IRC14:58
weshayrlandy, ready14:59
*** epoojad1 has joined #oooq14:59
*** panda|drappt is now known as panda15:09
pandageez. two 30 seconds visit turned into a mess of traffic, parking and waiting.15:09
weshayrlandy, https://review.opendev.org/#/c/688534/15:15
weshaypanda, sounds like you are in the US15:17
*** epoojad1 has quit IRC15:24
*** surpatil has quit IRC15:28
*** dsneddon has joined #oooq15:33
*** dsneddon has quit IRC15:37
*** epoojad1 has joined #oooq15:47
raukadaharxcruz: weshay: rlandy marios|rover I would like to present successful os_tempest fs020 job http://logs.rdoproject.org/34/23034/5/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/2516d0c/logs/stestr_results.html15:47
*** epoojad1 has quit IRC15:47
*** epoojad1 has joined #oooq15:48
marios|roverrlandy: nice ... waiting for that one still https://review.rdoproject.org/r/#/c/23159/15:48
marios|rovererr rlandy sorry was for raukadah ^15:48
rlandyoh gee15:48
rlandyanother r* nick15:48
*** marios|rover is now known as rmarios|rover15:48
raukadahhehe15:48
*** rmarios|rover is now known as marios|rover15:48
rlandyraukadah: you need to pick another name - sorry15:49
rlandyr* is overpopulated15:49
raukadahrlandy: :-)15:49
* rlandy change my nick to the_original_r15:49
*** rfolco|ruck has quit IRC15:49
rlandytype two letters people before tab complete15:50
raukadahmarios|rover: I have runned from check not from periodic15:50
raukadaharxcruz: weshay fs021 porting to os_tempest, we need to make some changes on os_tempest side15:51
marios|roverraukadah: ack15:51
raukadahwinter is arriving in india15:53
*** surpatil has joined #oooq15:53
raukadahas OpenStack Train just arrived15:53
*** epoojad1 has quit IRC15:56
raukadahmarios|rover: wrt to this http://logs.rdoproject.org/71/20171/62/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/b9094fd/logs/tempest.html.gz 300 tests missing15:57
* raukadah will figure out tomorrow15:58
*** jpena is now known as jpena|off16:00
*** tesseract has quit IRC16:08
*** dsneddon has joined #oooq16:09
*** dsneddon has quit IRC16:14
*** rfolco has joined #oooq16:15
*** marios|rover has quit IRC16:16
*** rfolco has quit IRC16:18
*** rfolco has joined #oooq16:18
raukadahpanda: weshay https://review.opendev.org/#/c/688148/ one review much needed16:19
rfolcodear google wifi, please collaborate. I just need to work. Thanks.16:19
*** rfolco is now known as rfolco|ruck16:20
chemrfolco|ruck: hey, I search for something similar but one standalone job failed on "There was an issue creating /root/DLRN as requested: [Errno 13] Permission denied: '/root/DLRN'"16:23
chemrfolco|ruck: search in tripleo-bug (the log is there https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_acd/688846/1/gate/tripleo-ci-centos-7-standalone/acdc890/job-output.txt )16:24
rfolco|ruckchem, looking16:24
chemrfolco|ruck: just FYI, I'm going to the recheck as this should be working just fine (maybe a tmp error or somethnig)16:25
rfolco|ruckchem, there is a good chance of a regression there16:25
chemrfolco|ruck: argh16:25
rfolco|ruckchem, you can do a recheck to save time meanwhile, I'll try to dig more into the problem and see what I find16:26
pandaraukadah: in 688148, why did you use a block ?16:26
*** jbadiapa has quit IRC16:27
chemrfolco|ruck: ack, cool and thanks.16:27
rfolco|ruckchem, np I keep you posted16:27
raukadahpanda: ah nice catch16:28
raukadahrlandy: https://review.opendev.org/#/c/688148 needs some love16:29
rlandyraukadah: k - will update16:30
*** jfrancoa has quit IRC16:34
*** jbadiapa has joined #oooq16:35
rlandyraukadah: panda: rfolco|ruck: https://review.opendev.org/#/c/688148 updated16:35
raukadahrlandy: thanks!16:36
*** udesale has quit IRC16:40
*** aakarsh has joined #oooq16:41
*** chem is now known as chem|eod16:43
*** dsneddon has joined #oooq16:44
rfolco|ruckrlandy, git blame shows you were the last one to change this... do you know why this is being run as root somehow ?16:44
rfolco|ruckhttps://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_acd/688846/1/gate/tripleo-ci-centos-7-standalone/acdc890/job-output.txt16:44
rfolco|ruckrlandy, There was an issue creating /root/DLRN as requested: [Errno 13] Permission denied: '/root/DLRN'16:44
rlandyrfolco|ruck: what file are you looking at?16:48
rlandywhere I made the change16:48
rfolco|ruckrlandy, https://opendev.org/openstack/tripleo-quickstart-extras/commit/2cbbc898b2772d839c197352781a05e89e255b0916:49
*** dsneddon has quit IRC16:49
rfolco|ruckrlandy, not blaming you at all, just asking if know what could be happening. I don't think it should be running as root there16:51
rlandy Ronelle Landy 2 years ago16:51
rlandyengage long term memory16:51
*** dtantsur is now known as dtantsur|afk16:51
rfolco|ruckrlandy, and this happens in gate just now16:51
rlandy+build_repo_dir: "{{ ansible_user_dir }}"16:52
rfolco|rucklol16:52
rlandyrfolco|ruck: can you check what {{ ansible_user_dir }} is defined as?16:52
rlandyif the user is root16:52
rlandythen maybe it would go to root16:52
rlandycompare with previous passing job16:53
rlandyto see where it went16:53
rfolco|ruckrlandy, primary says16:53
rfolco|ruckansible_user_dir: /home/zuul16:53
rfolco|ruckhttps://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_acd/688846/1/gate/tripleo-ci-centos-7-standalone/acdc890/zuul-info/host-info.primary.yaml16:53
*** derekh has quit IRC17:01
rlandyhttp://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone17:02
rlandyrfolco|ruck: idk - you have mixed results here17:02
rlandy2019-10-16 11:35:09.529318 | primary | TASK [build-test-packages : Ensure DLRN dir is present] ************************17:03
rlandy2019-10-16 11:35:09.556894 | primary | Wednesday 16 October 2019  11:35:09 +0000 (0:00:01.280)       0:04:33.486 *****17:03
rlandy2019-10-16 11:35:10.574757 | primary | changed: [undercloud]17:03
rlandy2019-10-16 11:35:10.631253 | primary |17:03
rlandy^^ fine17:03
raukadahweshay: what is the issue with fs039 reproducer?17:04
raukadahis it on libvirt or rdo-cloud?17:04
weshayraukadah, rdo-cloud .. need to run it w/ the new repro17:04
weshaythe old one won't work in this case I think17:04
raukadahweshay: you have the box handy17:04
raukadahweshay: fire up, we can check tomorrow17:04
raukadahplease someone fix waiting on logger issue17:05
rfolco|ruckrlandy, thx will compare those17:05
weshayraukadah, oh.. where I'm running it?17:05
weshayUGH.. I know17:05
weshayraukadah, I think we need to pull new containers again17:06
raukadahweshay: I updated the containers using docker-compose17:06
raukadahbut no idea it does not work17:06
* raukadah not sure doing something wrong there17:06
raukadahmight be arxcruz knows!17:06
raukadahrlandy: https://review.opendev.org/#/c/688860/ please +w it17:07
* raukadah bugs rlandy a lot for +w sorry17:08
rlandyraukadah: not a problem - that is what I am here for17:08
raukadahrlandy: :-)17:09
raukadahweshay: will I just replace the launcher-playbook and we are done for fs039?17:11
weshayraukadah, we need to debug tempest basic-ops on 39 master/train17:12
raukadahweshay: but we need a reproducer na?17:13
raukadahweshay: I am in the box 38.145.34.6 with tmux advent17:13
weshayah cool man17:14
weshaywill check in a few.. yet another call17:14
raukadahrlandy: can we include mosh for new reproducer?17:18
raukadahit is easier for slow connection17:18
rlandyraukadah: on a bunch of other things now - so not working on any reproducer stuff atm17:18
rlandywe can make a list of enhancements17:19
*** dsneddon has joined #oooq17:20
raukadahrlandy: sure no hurry!17:22
raukadahzbr: Hello17:24
zbrraukadah: hi17:24
raukadahzbr: please have a look at this user story https://tree.taiga.io/project/tripleo-ci-board/us/1321?milestone=242589 if need any help on creating task, let us know17:24
zbryeah, my plan was to work in that today but today everything went wrong: both ansible-lint and molecule getting broken....17:25
*** dsneddon has quit IRC17:25
raukadahzbr: time to share the knowledge, make other team members molecule experts17:25
raukadahzbr: no problem take your time :-)17:26
zbrended up writing one line in ~4h : https://review.rdoproject.org/r/#/c/23215/ -- and creating almost a dozen of tickets: virtualenv/setuptools/ansible-lint/tox/...17:30
zbrmainly setup.py is EVIL17:31
raukadahzbr: need any help on setup.py stuff17:32
raukadahFailed to load file list: 4 Expected application/json or text/plain; received Content-Type: text/html;charset=utf-8 why review.rdoproject.org17:33
raukadahpanda: from the conversation I am seeing on podman channel can be easily solve by zuul17:34
raukadahwe just need to sell them17:34
raukadahf31 issue or release stuff17:35
zbrraukadah: need help review https://review.rdoproject.org/r/#/c/23215/ and its related one.17:36
raukadahzbr: sure will look17:36
zbrraukadah: it is critical, blocking any change to rdo-jobs17:37
raukadahweshay: http://38.145.34.6:9000/t/tripleo-ci-reproducer/stream/2ab8178e552d4137ac13029f92f3e968?logfile=console.log17:37
raukadahweshay: please have a look when free once deployment fs039 finishes17:37
weshayraukadah++17:37
weshayI'm on it now17:37
weshayraukadah, and to get on the undercloud?17:39
weshayI'm on 34.617:39
weshayah.. see it.. reprozuul17:39
raukadahweshay: 38.145.34.133 might be undercloud17:39
weshayk k17:39
raukadahweshay: you can access from 38.14517:40
weshayraukadah, ya.. as long as I'm in ~/tripleo-ci-reproducer I can get to the undercloud17:40
weshay:)17:40
weshaythank you thank you17:41
raukadahah nice, there is no waiting on logger message17:47
raukadahzbr: sorry time to hide, catch rlandy for next merge17:48
* raukadah runs17:49
*** dsneddon has joined #oooq17:53
*** holser has quit IRC17:55
*** dsneddon has quit IRC17:59
*** dsneddon has joined #oooq18:03
*** TrevorV has joined #oooq18:05
raukadahweshay: nkinder.keycloak role is installing is it something new got added?18:08
weshayraukadah, yes18:08
weshayraukadah, you can see in the long dep chain in that patch18:09
raukadahyup, quite a long series18:10
rlandyrfolco|ruck: weshay: vexxhost stack create issue shows up on rdocloud ...18:15
weshayoh lord18:15
weshayrfolco|ruck, all the time?18:15
rlandyhttps://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ovb-master18:15
*** surpatil has quit IRC18:16
weshayrlandy,  /bin/sh: line 1: /home/zuul/workspace/ovb/.venv/bin/openstackstack: No such file or directory18:17
weshay?18:17
rlandyhttp://logs.rdoproject.org/60/685460/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ovb-master/f1b562d/job-output.txt18:18
rlandyweshay: ^^18:18
weshayhttp://logs.rdoproject.org/60/685460/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ovb-master/f1b562d/logs/failed_ovb_stack.log18:18
weshayrfolco|ruck, need a bug on this ^..18:18
rfolco|ruckweshay, ok18:19
weshayrlandy, ya.. but that doesn't show the error18:19
rlandyweshay: there is already a bug ...18:19
rlandyhttps://bugs.launchpad.net/tripleo/+bug/184522518:19
openstackLaunchpad bug 1845225 in tripleo "OVB master branch job is broken, can't create bmc" [High,Triaged]18:19
weshayrlandy, the error looks to be in failed_ovb_stack.log no?18:19
rlandycorrect18:20
rlandyhttp://logs.rdoproject.org/60/685460/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ovb-master/f1b562d/logs/failed_ovb_stack.log18:20
weshayya.. that looks like openstack failed to pip install to me18:20
weshaythe client18:21
rfolco|ruckweshay, openstackclient bug is affecting many jobs18:21
rfolco|ruckweshay, any job that installs the client in venv and runs the unit tests18:21
weshayrfolco|ruck, do we have that pinned somewhere?18:22
rfolco|ruckI've been looking into several gate failures that may look different but point to the same bug18:22
raukadahrlandy: weshay https://github.com/openstack/tripleo-ci/blob/master/zuul.d/base.yaml#L3618:22
raukadahovb is pinned at stable/1.018:23
raukadahis it running from maste?18:23
*** kopecmartin is now known as kopecmartin|off18:24
rfolco|ruckweshay, dunno, have to look18:24
raukadahrlandy: do we collect ovb version in the logs?18:25
weshayraukadah, some of these jobs check openstack-virtual-bm itself18:25
weshay2019-10-15 15:26:46.438905 | primary | ZUUL_CHANGES=openstack/openstack-virtual-baremetal:master:refs/changes/60/685460/318:25
weshayhttp://logs.rdoproject.org/60/685460/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ovb-master/f1b562d/job-output.txt18:25
weshayrlandy, both cases18:26
weshay2019-10-16 13:10:34.648718 | primary | ZUUL_CHANGES=openstack/openstack-virtual-baremetal:master:refs/changes/60/685460/3^openstack/openstack-virtual-baremetal:master:refs/changes/61/685461/518:26
weshayrfolco|ruck, ugh.. I need to merge this18:26
weshayhttps://review.rdoproject.org/r/#/c/23144/18:26
rfolco|ruckweshay, w+ https://review.opendev.org/#/c/687388/, should be fixed soon18:27
weshayk18:27
rfolco|ruckweshay, need to refresh on tox situation18:28
raukadahrlandy: weshay https://review.opendev.org/#/c/688945/ one more +w18:28
rfolco|ruckchecking18:28
weshayrlandy, that list is all check jobs from openstack virtual bm18:29
raukadahweshay: rlandy here master is used from zuul inventory18:30
raukadahfor ovb18:30
weshayraukadah, rlandy not much changing https://review.opendev.org/#/q/project:openstack/openstack-virtual-baremetal+status:merged18:31
rfolco|ruckweshay, rechecked your change, the workaround for tox-linters, https://review.rdoproject.org/r/#/c/23144/ merged18:33
rlandyhmm...18:34
rlandyraukadah: the ovs stuff is in?18:35
*** dsneddon has quit IRC18:36
raukadahrlandy: it is related to ovs to ovn migration18:37
rlandythat is done enough to enable the tests?18:37
raukadahrlandy: https://bugs.launchpad.net/tripleo/+bug/183216618:37
openstackLaunchpad bug 1832166 in tripleo "[FUll Tempest job][master][periodic]3 Tempest tests related to DNS failing " [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)18:37
raukadahrlandy: yes18:37
rlandyok18:38
zbrpanda: sshnaidm|pto do you happen to read this? I need to make a decision on https://github.com/pycontribs/pytest-molecule/issues/4518:40
rlandyweshay: according to the bug, it's the master branch only18:40
*** dtrainor has joined #oooq18:40
rlandymaybe if we don;t define the branch as master it picks something working18:40
zbrothers are welcomed to comment/vote there, in the end we are all using it, better be user friendly than not.18:40
weshayrlandy, query you sent is for a job that is only checking the ovb repo18:42
raukadahif job is runned against ovb, then branch-override is ignored18:43
raukadahzuul checks out the current patch18:43
rlandyweshay: ack - but it shows that the failure is not vexxhost specific18:43
rlandyit's branch18:43
weshayrlandy, raukadah we use stable/1.018:45
weshayhttp://codesearch.openstack.org/?q=ovb_repo_version&i=nope&files=&repos=18:45
zbrrfolco|ruck: rlandy -- https://review.rdoproject.org/r/#/c/23158/ fixes molecule job.18:46
raukadahweshay: yes18:47
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/search?q=ovb_repo_version&unscoped_q=ovb_repo_version18:47
rlandysame there18:47
rlandywhen does get overridden with master?18:48
raukadahweshay: https://review.opendev.org/#/c/685461/ might have issue18:48
raukadahwhich we are not seeing18:48
raukadahmight be18:49
weshayraukadah, rlandy let's chat18:50
raukadahor is it something related ipv6 networking issue18:50
weshayhttps://meet.google.com/eka-okfx-xyp18:50
rlandyno - ipv418:50
weshayraukadah, rlandy why are you guys chasing check jobs on the ovb project?18:51
weshaywhy is nhicher using master?18:51
*** amoralej is now known as amoralej|off18:51
*** dsneddon has joined #oooq19:07
rlandyOct 16 17:57:52 overcloud-controller-2 systemd[1]: Failed to start DHCP interface ovs-system.19:08
rlandyOct 16 17:57:52 overcloud-controller-2 systemd[1]: Unit dhcp-interface@ovs-system.service entered failed state.19:08
rlandyOct 16 17:57:52 overcloud-controller-2 systemd[1]: dhcp-interface@ovs-system.service failed.19:08
*** dsneddon has quit IRC19:12
rfolco|ruckweshay, https://review.rdoproject.org/r/#/c/23215/ should have fixed tox-linters.... zbr around ?19:21
zbrrfolco|ruck: kinda of, i was about to leave19:21
rfolco|ruckzbr, tox-linters still failing after https://review.rdoproject.org/r/#/c/23215/ merged19:22
rfolco|ruckzbr, thats ok, I'll take a look19:22
weshayzbr, go away.. we'll get it19:22
rfolco|rucksorry yeah too late for you19:23
rfolco|ruck /kick sorry'n19:24
rfolco|ruckweshay, it does not make sense, testing locally in a new venv19:29
rfolco|ruckweshay, bug says its for py3...19:30
*** dsneddon has joined #oooq19:30
*** tosky has quit IRC19:30
rfolco|ruckweshay, virtualenv=16.7.619:37
rfolco|ruckhttps://review.rdoproject.org/r/#/c/23215/4/tox.ini19:37
zbrrfolco|ruck: send me link to job still failing.19:38
rfolco|ruckzbr, go away19:38
rfolco|ruck:)19:38
rfolco|ruckzbr, almost fixing it, let me try it19:39
zbrfyi, i made a pytest-molecule release few minutes ago https://pypi.org/project/pytest-molecule/#history -- you may want to double check molecule jobs in case something goes wrong.19:41
rfolco|ruckzbr, thx19:42
rlandyweshay: ever seen 2019-10-16 18:10:42 | Trying to ping 172.17.0.88 for local network 172.17.0.0/24.19:44
rlandy2019-10-16 18:10:42 | Ping to 172.17.0.88 failed. ... (full log ... http://logs.rdoproject.org/66/16566/13/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-vexxhost/25ea796/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) where ... 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.119:44
weshayrlandy, can he hold the nodes?19:47
rlandyweshay: that job already failed but I can rekick19:47
rlandythere is a master job running now19:47
rlandyin overcloud deploy19:48
rlandyI want to see if that job reports the same issue contacting the one network19:48
rlandyif it doesn;t we don't have an env issue19:48
rlandyalthough ...19:48
*** holser has joined #oooq19:50
rfolco|ruckweshay, combined the tox-linters fix in your change, hope it is ok19:58
rfolco|ruckhttps://review.rdoproject.org/r/#/c/23144/19:58
weshayrfolco|ruck, did that break rdo-tox-molecule FAILURE in 10m 51s (non-voting)19:59
weshay?19:59
rfolco|ruckweshay, no, see ps2, it was failing before that20:00
weshayrfolco|ruck, https://review.rdoproject.org/r/2322520:06
weshayrfolco|ruck, fyi https://review.rdoproject.org/r/2322620:09
rfolco|ruckweshay, scen10 is failing now right20:09
weshayrfolco|ruck, not sure how.. but it wasn't running in periodic at all20:10
weshayrfolco|ruck,  https://review.rdoproject.org/r/2322520:10
rfolco|ruckweshay, success rate for the week 85%... fine20:11
weshayit always should run20:11
weshaydoesn't have to be criteria20:11
rfolco|ruckweshay, meh need to fix linters in config too20:11
weshayrfolco|ruck, it's a gating job20:11
rfolco|ruckk20:11
weshayrlandy, you wf... https://review.opendev.org/#/c/688945/20:21
weshaythe test removed from the skip list failed afaict20:21
weshayrlandy, http://logs.rdoproject.org/45/688945/1/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020/cd32c66/logs/tempest.html.gz20:21
rlandyweshay: I didn't wf it yet20:22
weshayrlandy, you did20:22
weshayRonelle Landy20:22
weshay12:39 PM20:22
weshay20:22
weshayPatch Set 1: Workflow+120:22
rlandyPatch Set 1: Workflow+120:23
rlandyoh I did ok20:23
weshay:)20:23
rlandy<rlandy> that is done enough to enable the tests?20:23
rlandy<raukadah> rlandy: yes20:23
weshayraukadah--20:24
rlandysorry - believed that w/o checking20:24
weshayaye20:24
rlandymy fault20:24
weshaynp20:24
rlandyI am core - I should check - sorry20:24
weshayI've launched jewish guilt20:24
rlandythat is always in flight20:25
weshayI am truly sorry20:25
rlandyno launch required20:25
weshaylolz20:25
weshayrlandy,  what we need is a captcha on on workflow20:25
weshaylolz20:25
weshayrlandy, you should probably remove the +2 as well :)20:26
rlandydone20:26
rlandythanks for catching that20:26
weshayrlandy, np.. /me just watching the gate from time to time20:27
rlandyon other matters ...20:27
weshayrfolco|ruck, comments on https://review.rdoproject.org/r/#/c/23150/  nice.. minor comments20:27
rlandyhttps://review.rdoproject.org/r/#/c/23227/ doesn't kick anything20:27
rlandyso unless I messed up testproject review, nhicher change is needed to kick something20:29
rlandyon branch jobs20:29
weshayrlandy, hrm.. so it should I think vexx is hooked into rdo sf20:29
weshayI had a patch do that to me yesterday20:30
rlandyweshay: nhicher jobs does kick ...20:30
rlandyhttp://logs.rdoproject.org/66/16566/13/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-vexxhost/25ea796/20:30
weshayrlandy, try to resubmit w/ a diff commit hash20:30
weshayjust for giggles20:30
rlandysure20:30
weshaymay be unrelated to the job20:30
weshay3hr queue in rdo atm20:30
rlandyjust to check I am not totally drunk today20:30
weshayfor check20:30
rlandyit wasn;t even the four cups of wine holiday20:31
weshaylolz20:38
rlandyhttps://review.rdoproject.org/r/#/c/23228/ - didn't help20:40
rlandythat job never ran before20:40
rlandynhicher is right there20:40
*** holser has quit IRC20:54
recheck[gh-pytest-molecule] 1.2.3 → https://github.com/pycontribs/pytest-molecule/releases/tag/1.2.320:55
*** holser has joined #oooq21:08
weshayrfolco|ruck, marios master is promoting :)21:11
weshayI'm about to kill these tox tests21:11
weshayrfolco|ruck, why is this running on a pipeline change?21:12
rlandyweshay: got the master and stein vexxhost nodes held now21:15
rlandymaster timed out ...21:15
rlandy2019-10-16 20:03:08 | ok: [overcloud-controller-1] => {21:15
rlandy2019-10-16 20:03:08 |     "ansible_facts": {21:15
rlandy2019-10-16 21:04:25 | Overcloud configuration failed.21:15
rlandy2019-10-16 21:04:25 |         "python_cmd": "python2"21:15
rlandy^^ took an hour21:15
rlandyand then the deploy times out21:16
*** TrevorV has quit IRC21:24
weshayrlandy, I'll look at it tomorrow.. thanks for checking it out21:25
rlandyweshay: k - we can discuss more tomorrow - am debugging stein and master21:26
rlandyat least they get to deployment21:26
*** holser has quit IRC21:45
*** Goneri has quit IRC21:48
*** holser has joined #oooq21:48
*** aakarsh has quit IRC21:52
*** brault has quit IRC22:12
*** brault has joined #oooq22:12
*** holser has quit IRC23:01
*** rfolco has joined #oooq23:25
*** rfolco|ruck has quit IRC23:25
*** rfolco has quit IRC23:34
*** rfolco has joined #oooq23:34
*** rlandy has quit IRC23:48
*** rfolco has quit IRC23:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!