Monday, 2019-08-26

*** dsneddon has joined #oooq00:00
*** dsneddon has quit IRC00:05
*** d0ugal has quit IRC00:05
*** d0ugal has joined #oooq00:21
*** dsneddon has joined #oooq00:38
*** dsneddon has quit IRC00:43
*** dsneddon has joined #oooq01:11
*** dsneddon has quit IRC01:16
*** dsneddon has joined #oooq01:50
*** dsneddon has quit IRC01:55
*** dsneddon has joined #oooq02:27
*** dsneddon has quit IRC02:32
*** dsneddon has joined #oooq03:06
*** dsneddon has quit IRC03:11
*** dsneddon has joined #oooq03:44
*** hamzy_ has joined #oooq03:48
*** aakarsh|2 has quit IRC03:49
*** dsneddon has quit IRC03:49
*** hamzy has quit IRC03:51
*** ykarel has joined #oooq03:59
*** udesale has joined #oooq04:05
*** surpatil has joined #oooq04:18
*** ykarel has quit IRC04:19
*** dsneddon has joined #oooq04:23
*** ykarel has joined #oooq04:24
*** dsneddon has quit IRC04:27
*** raukadah is now known as chkumar|rover04:37
*** dsneddon has joined #oooq05:02
*** bhagyashris has joined #oooq05:11
*** dsneddon has quit IRC05:35
*** jaosorior has joined #oooq05:51
*** surpatil has quit IRC06:02
*** dsneddon has joined #oooq06:06
*** dsneddon has quit IRC06:11
*** ykarel is now known as ykarel|afk06:21
*** skramaja has joined #oooq06:23
*** ykarel|afk is now known as ykarel06:32
*** dsneddon has joined #oooq06:37
*** dsneddon has quit IRC06:41
*** bhagyashris has quit IRC06:47
*** ykarel is now known as ykarel|afk07:01
*** jtomasek has joined #oooq07:07
*** bogdando has joined #oooq07:08
*** dsneddon has joined #oooq07:10
*** jpena|off is now known as jpena07:11
*** dsneddon has quit IRC07:15
*** sshnaidm|afk is now known as sshnaidm07:16
sshnaidmchkumar|rover, commented on https://review.opendev.org/#/c/678472/07:29
*** jfrancoa has joined #oooq07:29
sshnaidmchkumar|rover, well, it's not enough.. I'll start from reverting this07:34
chkumar|roversshnaidm: https://review.opendev.org/#/c/678472/1/roles/run-test/templates/oooq_common_functions.sh.j2@86 it should be venvpath/share /usr/local/share/tripleo-quickstart/roles na ?07:36
chkumar|roversshnaidm: ok07:38
*** ykarel|afk is now known as ykarel07:43
*** apetrich has joined #oooq07:43
*** dsneddon has joined #oooq07:50
*** dsneddon has quit IRC07:58
sshnaidmchkumar|rover, https://review.rdoproject.org/r/#/c/21881/07:59
chkumar|roversshnaidm: will I also +w it?08:02
sshnaidmchkumar|rover, feel free :)08:02
*** jfrancoa has quit IRC08:03
*** derekh has joined #oooq08:09
*** jfrancoa has joined #oooq08:18
*** ykarel is now known as ykarel|lunch08:26
*** dsneddon has joined #oooq08:27
*** dsneddon has quit IRC08:32
chkumar|roversshnaidm: fs039 tempest runs using os_tempest https://logs.rdoproject.org/24/639324/43/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/69d3757/job-output.txt.gz#_2019-08-26_08_31_14_619952 but it got failed08:35
chkumar|roverno logs so we cannot do anything08:36
sshnaidmipalib.errors.NotFound: controller-0.ooo.test: host not found08:40
sshnaidmseems like ipa dns didn't work08:40
chkumar|roversshnaidm: where ctrlplane network is used?08:41
chkumar|roveris it related to undercloud08:42
sshnaidmchkumar|rover, it's related to IPA setup. On supplemental node we have IPA server which serves as DNS for all nodes - undercloud, overcloud.08:43
sshnaidmchkumar|rover, maybe worth to see how they run tempest downstream, maybe need some additional settings08:44
*** dtantsur|afk is now known as dtantsur08:46
*** dsneddon has joined #oooq09:01
*** dsneddon has quit IRC09:06
*** ykarel|lunch is now known as ykarel09:07
chkumar|roversshnaidm: sure09:21
chkumar|roversshnaidm: putting a bug and then take it from there09:21
chkumar|roversshnaidm: https://9b8de9f26d5cfa9d157d-87a4d4201f92d032de2e8328e3e57fa8.ssl.cf2.rackcdn.com/678476/2/check/tripleo-ci-centos-7-scenario004-standalone/0f3b02d/logs/09:34
sshnaidmchkumar|rover, ?09:35
chkumar|roversshnaidm: https://review.opendev.org/#/c/678476/09:35
chkumar|roversshnaidm: let's merge the revert09:35
sshnaidmchkumar|rover, aah, ok09:35
*** chem has joined #oooq09:37
*** dsneddon has joined #oooq09:40
*** dsneddon has quit IRC09:44
*** apetrich has quit IRC09:49
*** saneax has joined #oooq09:59
chkumar|roverarxcruz: https://review.opendev.org/#/c/678418/ will fix the undefined tempest_install_method undefined10:04
arxcruzchkumar|rover: okay, for now i'm adding all the variables on fs001, then i'll move to baremetal-full-overcloud-validate.yml10:05
arxcruzbecause there are a lot of vars, every run i got a different error10:05
arxcruzand i don't want to keep adding depends on, that ill increase the time of checking10:05
*** dsneddon has joined #oooq10:19
*** ksambor is now known as ksambor|lunch10:23
*** dsneddon has quit IRC10:24
*** jaosorior has quit IRC10:26
*** sanjayu_ has joined #oooq10:43
*** saneax has quit IRC10:43
*** surpatil has joined #oooq10:55
*** ksambor|lunch is now known as ksambor10:56
*** dsneddon has joined #oooq10:59
sshnaidmarxcruz, chkumar|rover, do you know why is that happening? No package pytho3n-barbican-tests-tempest available. http://logs.rdoproject.org/45/21945/3/check/periodic-tripleo-ci-rhel-8-scenario002-standalone-master/1c89d62/job-output.txt.gz11:01
chkumar|roversshnaidm: chekcing11:02
chkumar|roversshnaidm: fxing it11:02
arxcruzsshnaidm: chkumar|rover pytho3n11:04
arxcruzsshnaidm: chkumar|rover https://review.opendev.org/#/c/678520/11:04
*** dsneddon has quit IRC11:04
*** udesale has quit IRC11:04
sshnaidmarxcruz, cool :)11:04
*** tesseract has joined #oooq11:12
*** ccamacho has joined #oooq11:21
*** jpena is now known as jpena|lunch11:25
*** dsneddon has joined #oooq11:30
*** dsneddon has quit IRC11:35
*** apetrich has joined #oooq11:41
*** jaosorior has joined #oooq11:44
*** rlandy has joined #oooq11:59
*** rlandy is now known as rlandy|ruck11:59
*** derekh has quit IRC12:00
rlandy|ruckchkumar|rover: hi12:00
rlandy|ruckchkumar|rover: just two more days :)12:01
rlandy|ruckchkumar|rover: ok if I move https://trello.com/c/ZvV5ul81/1029-cixlp1836046tripleociproa-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-on-queens to done?12:01
weshay_MODykarel chkumar|rover you guys looking at scen003 packstack in master?12:05
weshay_MODError while evaluating a Resource Statement, Could not find declared class ::ceilometer::agent::central12:05
*** dsneddon has joined #oooq12:06
ykarelweshay_MOD, https://review.opendev.org/#/c/678169/12:06
weshay_MODykarel thank you.. please open a lp in case it doesn't promote in time12:07
*** weshay_MOD is now known as weshay12:07
ykarelweshay, ack, will open against packstack12:08
*** dsneddon has quit IRC12:10
rlandy|ruckchkumar|rover: sshnaidm: does the revert close out https://bugs.launchpad.net/tripleo/+bug/1841405 or is there still more work required here?12:11
openstackLaunchpad bug 1841405 in tripleo "role 'dump_vars' not found leading to logs not getting collect in post" [Critical,Confirmed]12:11
chkumar|roverrlandy|ruck: revert worked12:12
rlandy|ruckchkumar|rover: k - thank - closing that bug12:12
arxcruzchkumar|rover: woot, tempest run, it fails, but it runs12:13
chkumar|roverarxcruz: error log?12:13
chkumar|roverarxcruz: please try with config_drive fix12:14
chkumar|roverarxcruz: https://review.opendev.org/#/c/639324/45/config/general_config/featureset039.yml@15012:15
rlandy|ruckchkumar|rover: from ^^ ...  ok if I move https://trello.com/c/ZvV5ul81/1029-cixlp1836046tripleociproa-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-on-queens to done?12:15
rlandy|ruckthere was a failure in the last run but it didn't get to tempest12:15
chkumar|roverrlandy|ruck: yes, move it to done12:16
arxcruzchkumar|rover: i believe it's related to the ip address for tempest_cidr12:16
chkumar|roverarxcruz: might be or might not be as we have no logs12:17
arxcruzchkumar|rover: http://logs.rdoproject.org/00/673400/13/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/d534e97/job-output.txt.gz12:17
rlandy|ruckweshay: still want to keep https://trello.com/c/rPZGwJmU/1081-cixlp1839532tripleociproa-tripleo-gate-jobs-are-failing-to-pull-containers-when-running-on-ovh-provider-with-unauthorized-error on the escalation board?12:19
weshayrlandy|ruck let's explain it today and close it today12:20
pandarfolco: ping me whenever12:21
rfolcopanda, ping12:21
chkumar|roverrlandy|ruck: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario001-standalone does not looks good12:21
rlandy|ruckchkumar|rover: better than it was :) - adding on etherpad to investigate12:22
rfolcopanda, do you have a task for me ?12:24
pandarfolco: wanna sync ?12:24
rfolcopanda, sure12:25
*** surpatil has quit IRC12:25
rfolcopanda, my bj or yours?12:26
chkumar|roverarxcruz: please try fs01 with compute_feature_enabled.config_drive': 'True' and let see if it works or not, then we need to dig what is wrong12:28
*** jpena|lunch is now known as jpena12:30
*** Vorrtex has joined #oooq12:31
pandarfolco: I'm in yours12:32
rfolcok k12:33
*** dsneddon has joined #oooq12:44
weshayrlandy|ruck chkumar|rover let's look at https://trello.com/c/dq5ntFjS/1084-cixlp1840763tripleociproa-queensfs037-upgrade-jobs-are-failiing-while-doing-overcloud-minor-update-with-ssh-error12:45
rlandy|ruckweshay: k - your bj?12:45
weshayrlandy|ruck looks like ssh is failing.. is that the consistent issue?12:46
* rlandy|ruck will confirm with latest failure - sec12:46
chkumar|roverweshay: that ssh issue is consistent failure12:47
weshaychkumar|rover rlandy|ruck ok.. any chance we can setup a reproducer and ask jose luis or other upgrade folks to take a look?12:48
rlandy|ruck2019-08-26 06:15:10 |     "msg": "SSH Error: data could not be sent to remote host \"192.168.24.3\". Make sure this host can be reached over ssh",12:48
rlandy|ruckweshay: yep - same one12:48
chkumar|roverweshay: that we call do12:48
*** dsneddon has quit IRC12:48
chkumar|rover*can12:50
rlandy|ruckweshay: ack - we can try a reproducer - just want to check if we have similar failures on other upgrade12:50
weshayk.. thanks12:50
rlandy|ruckweshay: chkumar|rover: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein12:51
rlandy|ruck^^ stein is all green12:51
weshayrockin.. that is still the important one :)12:52
rlandy|ruckas a comparison12:52
weshayah.. thought you meant the status.. ignore me12:52
rlandy|ruckweshay: no the same upgrade in stein12:52
rfolcoscrum time13:00
* chkumar|rover headed home, will connect from there13:00
*** Goneri has joined #oooq13:02
jfrancoaweshay: rlandy|ruck hey, I'm having a look at the very same issue. Just for my understanding, is this something that has been happening for long time?13:10
jfrancoaweshay: rlandy|ruck or did it start recently?13:10
rlandy|ruckjfrancoa: it's been going a while13:10
rlandy|ruckjfrancoa: here is the history ...13:11
rlandy|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-queens13:11
rlandy|ruck^^ so pretty much as long as we are keeping history13:12
jfrancoarlandy|ruck: oh yes..it's been already a while. Ok, I think I'll give a try to the reproducer to debug the issue, but I'm considering it could be the missing overcloud-ssh-user in "overcloud update prepare"13:12
rlandy|ruckjfrancoa: as a comparison, the stein and rocky jobs are green ... https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-rocky13:13
jfrancoarlandy|ruck: if you check any job in rocky or stein, we pass the user zuul during overcloud deploy and during prepare too13:13
rlandy|ruckjfrancoa: this job runs in periodic ... so no check13:13
rlandy|ruckjfrancoa: but yes, the rocky and stein equivalent jobs pass13:14
rlandy|ruckper links above13:14
rlandy|ruckjfrancoa: I was just starting to set up a reproducer ...13:14
jfrancoarlandy|ruck: could you give me access once it finishes, please? or are you running it locally?13:14
rlandy|ruckjfrancoa: but if you have a fix you suspect might work, we can spin on that13:15
rlandy|ruckjfrancoa: ack - it will be on my rdocloud tenant - pls pm send me your key13:15
jfrancoarlandy|ruck: if not, I'll try one in my rdo cloud account. So far it's a hint...that's why I would like to have an environment to confirm it13:15
rlandy|ruckjfrancoa: ack - will ping you with env13:15
jfrancoarlandy|ruck: http://pastebin.test.redhat.com/79167113:16
rlandy|ruckthanks13:16
jfrancoarlandy|ruck: thanks. I'll investigate meanwhile why do we pass the zuul user in rocky and stein, but nothing for queens..13:16
rlandy|ruckso that might really be it13:18
*** dsneddon has joined #oooq13:18
weshaychkumar|rover you are coming to the pre-planning mtg right?13:21
arxcruzchkumar|rover: that option is already in the fs00113:22
*** dsneddon has quit IRC13:23
*** skramaja has quit IRC13:25
*** ykarel is now known as ykarel|away13:26
chkumar|roverweshay: yes13:30
chkumar|roverarxcruz: the syntax is different13:31
arxcruzchkumar|rover: what you mean ?13:31
chkumar|rovertempest_tempest_conf_overrides:13:31
chkumar|rover  compute_feature_enabled.config_drive: 'True'13:32
chkumar|roverarxcruz: ^^13:32
arxcruzchkumar|rover: oh, yeah, because it's os_tempest, shit...13:38
chkumar|roverack13:45
*** dsneddon has joined #oooq13:56
rlandy|rucksshnaidm: hi - reproducer keeps failing on ansible-role-tripleo-ci-reproducer : Wait for zuul tenant error copied in http://pastebin.test.redhat.com/791691. any suggestions?14:00
*** dsneddon has quit IRC14:00
*** apetrich has quit IRC14:00
rlandy|ruckI can see the :9000 auul14:01
rlandy|ruckzuul14:01
rlandy|ruckno jobs is launched14:01
sshnaidmrlandy|ruck, need to see logs, could be anything.. take a look at scheduler logs, it's most critical service14:02
*** apetrich has joined #oooq14:03
chkumar|roverrlandy|ruck: time to remove pike job https://review.opendev.org/#/c/678154/ from where we will start?14:03
rlandy|ruckchkumar|rover: not sure I understand question 'from where we will start'?14:04
rlandy|ruckie: which jobs to remove first?14:04
chkumar|roverrlandy|ruck: yes which jobs to remove first?14:05
chkumar|roverfirst remove from promotion criteria14:05
rlandy|ruckchkumar|rover: just fighting with reproducer now - will look in a bit14:05
chkumar|roverrlandy|ruck: ack14:05
rlandy|ruckbasically job definitions go last14:05
rlandy|ruckchkumar|rover: everywhere they are called can be removed14:05
rlandy|ruckpipeline etc.14:06
chkumar|roverrlandy|ruck: ok, I will take a look and propose the patches14:06
rlandy|ruckchkumar|rover: go to review.rdoproject.org14:06
rlandy|ruckremove pipeline14:06
rlandy|ruckfirst14:06
rlandy|ruckthen other check inclusions14:07
ykarel|awayrlandy|ruck, chkumar|rover fyi ocata-eol tag job failed14:11
ykarel|awayi see tags are created, but stable/ocata not removed14:11
ykarel|awaywill followup with openstack/releases14:11
ykarel|awayhttps://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_9c/9c30af482980c34ed865db11bbcd56bf3fd4cf1f/release-post/tag-releases/ba9f773/job-output.txt14:12
*** aakarsh|2 has joined #oooq14:12
chkumar|roverykarel|away: checking14:13
rlandy|rucksshnaidm: k - http://pastebin.test.redhat.com/791697 - scheduler not starting properly14:13
rlandy|ruckdid we qualify against python 3.7?14:14
chkumar|roverykarel|away: openstack/os-cloud-config.git this one failed14:15
rlandy|ruckpanda: web search shows you hit this error with reproducer ... http://pastebin.test.redhat.com/791697 - remember how you got out of it?14:16
ykarel|awaychkumar|rover, hmm i see tag not exist for it, neither ocata-em nor ocata-eol14:16
ykarel|awayhttps://github.com/openstack/os-cloud-config/releases14:17
chkumar|roverrlandy|ruck: yes14:18
pandarlandy|ruck: does the web search show you if I ever did ? Seems you need to add some host to the /etc/hosts, but as I said, I had to give up on testing the reproducer because it was not working for me, maybe that's one of the errors that I could not get out of.14:20
jfrancoarlandy|ruck:  I believe this could fix the job https://review.opendev.org/#/c/678572/ , let's try it out once the reproducer is ready14:20
rlandy|ruckjfrancoa: I'm having some reproducer troubles - I will set up a testproject job to run your fix with fs037 queens14:21
chkumar|roverrlandy|ruck: I am taking care of above upgrade test14:21
chkumar|rovertesting in testproject14:21
rlandy|ruckchkumar|rover: k - pls add jfrancoa to the review14:22
jfrancoachkumar|rover: rlandy|ruck awesome, thanks. let's see if at least we can progress from that step (it might appear some different issue though)14:23
rlandy|ruckok - will watch progress14:23
chkumar|roverrlandy|ruck: jfrancoa https://review.rdoproject.org/r/#/c/21946/14:25
rlandy|ruckchkumar|rover: thanks14:26
rlandy|ruckchkumar|rover: why there - rather than testproject?14:27
chkumar|roverrlandy|ruck: it also works there14:27
rlandy|ruckyeah - just interested14:27
chkumar|roverrlandy|ruck: I left using testproject, we can do the same thing with rdo-jobs so14:28
*** dsneddon has joined #oooq14:30
*** dsneddon has quit IRC14:35
*** jbadiapa has quit IRC14:36
rlandy|ruckgo something going npw14:38
rlandy|rucknow14:38
*** jbadiapa has joined #oooq14:43
rlandy|ruckjfrancoa: pls try ssh zuul@38.145.32.4614:45
rlandy|ruckthat is your reproducer env14:45
jfrancoarlandy|ruck: I'm in, did the job already fail? Is it supposed to be the undercloud that host?14:46
jfrancoarlandy|ruck: I can't locate the stackrc14:47
rlandy|ruckjfrancoa: you're on the undercloud14:47
rlandy|ruckit's just started running14:47
rlandy|rucksending you zuul info14:47
jfrancoarlandy|ruck: ahh ok. I'll wait then14:47
rlandy|ruckjfrancoa: will ping you when the test stops14:49
jfrancoarlandy|ruck: ack, thank you14:49
rlandy|ruckweshay: ping re: https://bugs.launchpad.net/tripleo/+bug/1833465 - hitting that error14:54
openstackLaunchpad bug 1833465 in tripleo "tripleo reproducer fails w/ "waiting on logger"" [Critical,Incomplete]14:54
rlandy|ruckattached patch was never merged14:54
rlandy|ruckwhat did you do to fix?14:54
rlandy|ruckupstream-cloudinit-centos-7 image14:56
rlandy|ruckUpdated At14:56
rlandy|ruck    2019-05-10T11:31:44Z14:56
rlandy|rucknvm - it carried on14:58
rlandy|ruckoh no- it's going again14:58
rlandy|ruckchkumar|rover: how's it going with the pike job removals? need any help there?15:00
chkumar|roverrlandy|ruck: working on reviews15:00
rlandy|ruckk15:00
chkumar|roverweshay: rfolco meeting time15:01
rfolcoyeah I am there15:01
rfolcoweshay, ping15:03
*** dsneddon has joined #oooq15:08
chkumar|roverrfolco: ping, we are waiting on bj15:11
*** sanjayu_ has quit IRC15:11
*** bogdando has quit IRC15:12
*** ykarel|away has quit IRC15:12
*** dsneddon has quit IRC15:13
sshnaidmrlandy|ruck, Name or service not known - need to check dns config15:23
rlandy|rucksshnaidm: got by that error15:24
rlandy|rucksshnaidm: now dealing with 'waiting on logger' as in https://bugs.launchpad.net/tripleo/+bug/18334615:24
openstackLaunchpad bug 183346 in ubuntu-dev-tools (Ubuntu) "requestsync files a bug with the wrong version in the title, not in the changelog" [Low,Fix released] - Assigned to Michael Bienia (geser)15:24
rlandy|ruckoops15:24
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/183346515:24
openstackLaunchpad bug 1833465 in tripleo "tripleo reproducer fails w/ "waiting on logger"" [Critical,Incomplete]15:24
rlandy|ruck^^ that one15:24
rlandy|rucksshnaidm: possible my tenant has old images?15:25
rlandy|ruckopenstack-infra-centos-7?15:25
sshnaidmrlandy|ruck, wait, it's not from scheduler?15:25
rlandy|rucksshnaidm: the initial error was - fixed that one15:25
rlandy|ruckon to another error completely15:26
sshnaidmrlandy|ruck, and what is the error now?15:26
rlandy|rucksshnaidm: "waiting on logger" as shown in https://bugs.launchpad.net/tripleo/+bug/183346515:26
openstackLaunchpad bug 1833465 in tripleo "tripleo reproducer fails w/ "waiting on logger"" [Critical,Incomplete]15:26
rlandy|ruckthere is a review attached15:26
rlandy|rucknever merged15:26
sshnaidmmm.. haven't seen this a long time15:27
sshnaidmrlandy|ruck, on which task did it start?15:27
rlandy|rucksshnaidm: http://10.10.125.142:9000/t/tripleo-ci-reproducer/stream/b29bf1a2b633452f8132e25a2e3c2b39?logfile=console.log15:27
rlandy|rucksshnaidm: 2019-08-26 15:22:43.524755 | TASK [persistent-firewall : List current ipv4 rules]15:28
rlandy|ruck^^ starts on that tasks15:28
rlandy|ruckhappens on a few on them15:28
rlandy|ruckuntil jobs dies15:28
sshnaidmrlandy|ruck, can I log in to this machine?15:28
rlandy|rucksshnaidm: the machine running zuul or the instance?15:29
sshnaidmrlandy|ruck, where containers are15:29
sshnaidmrlandy|ruck, it was suspect that it's related to bad performance of rdo cloud host15:29
sshnaidmand poor network connection15:30
rlandy|rucksshnaidm: it's running from  my minidell - instances on my rdocloud tenant15:30
rlandy|ruckI can add your key to my minidell15:30
*** hamzy_ is now known as hamzy15:30
sshnaidmrlandy|ruck, try to rerun the job, just curios if it's reproducible15:32
rlandy|rucksshnaidm: this is my second run15:32
rlandy|rucksame deal15:32
rlandy|rucknot sure how weshay got rid of his errors15:32
rlandy|ruckalso, the images should be upstream-cloudinit-centos-715:33
rlandy|ruckmaintained centrally15:33
sshnaidmrlandy|ruck, started job on my tenant, let's see if it has same error15:33
sshnaidmrlandy|ruck, yeah15:33
rlandy|rucksshnaidm: thanks15:33
rlandy|ruckI'm running mutinode15:33
weshayrfolco chkumar|rover https://docs.google.com/document/d/1lXxKY794_TCByXovoFwkf6JfKOkOirL6dyAcw1zYjrY/edit#15:44
*** dsneddon has joined #oooq15:45
*** dsneddon has quit IRC15:50
rlandy|rucksshnaidm: second job - died15:53
rlandy|ruckweshay: chat about fs021?16:02
*** jpena is now known as jpena|off16:02
*** jfrancoa has quit IRC16:03
rlandy|ruckwhy is tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost running in periodic???16:05
*** sabedevops has joined #oooq16:06
rlandy|ruckhttps://review.rdoproject.org/r/#/c/21881/'16:06
rlandy|ruckreally???16:06
chkumar|roverrlandy|ruck: I was added temprory with check pipeline16:07
chkumar|rover*it16:07
rlandy|rucknot sure I would agree16:07
rlandy|ruckmore failures16:07
chkumar|roverrlandy|ruck: it will help to test the stuff for vexhost16:08
chkumar|roversshnaidm: ^^16:08
*** dsneddon has joined #oooq16:08
rlandy|ruckchkumar|rover: every 4 hours?16:09
rlandy|ruckI have no problem with a periodic vexxhost job16:09
rlandy|ruckthat pipeline is for promotions16:09
rlandy|ruckbut it16:09
sshnaidmrlandy|ruck, why not?16:09
rlandy|ruck's there now16:09
sshnaidmrlandy|ruck, the pipeline is just time trigger based16:10
sshnaidmrlandy|ruck, it shouldn't affect promotions16:10
rlandy|rucksshnaidm: it is - but I don't get why we need to debug that failing job every four hours16:10
rlandy|ruckit wont16:10
sshnaidmI need to run it periodically as frequent as possible16:10
sshnaidmrlandy|ruck, why do you need to debug it?16:10
rlandy|ruckimho there are better places to run that job16:11
rlandy|ruckI am not going to remove it or anything16:11
rlandy|ruckjust my personal opinion16:11
sshnaidmrlandy|ruck, where for example?16:12
rlandy|rucksshnaidm; any other temp periodic pipeline16:12
rlandy|ruckor at least call the job periodic16:12
sshnaidmrlandy|ruck, we run promotions in all periodic pipelines16:12
sshnaidmrlandy|ruck, and to define same zuul pipeline for one job is an overhead16:13
rlandy|rucksshnaidm: question is - is your intention to check vexxhost job with the latest hash - or just to run a check job 4 times a day?16:14
sshnaidmrlandy|ruck, I didn't call it periodic just for that reason, not to confuse it with periodic promotion jobs16:14
sshnaidmrlandy|ruck, just to run a job as much as possible16:14
rlandy|ruckhonestly not our biggest problem right now16:14
rlandy|ruckjust surprised to see it show up16:14
rlandy|ruckand I assume we agree that the expection is not on ruck/rover to keep debugging this16:15
sshnaidmrlandy|ruck, yeah, no need atm, I'm working on this16:15
*** matbu has quit IRC16:17
*** sabedevops has quit IRC16:19
rlandy|rucksshnaidm: back to reproducer - do you see the logger issue?16:29
rlandy|ruckI might be better off going the libvirt option16:29
rlandy|ruckif it's an rdocloud network issue16:29
weshayrlandy|ruck I was wrong about 21..  7.7 rpms are not there yet.. sorry16:43
weshaychkumar|rover set me straight16:43
rlandy|ruckchkumar|rover++16:45
* chkumar|rover is working on bring a bot soon16:45
chkumar|rover*bringing16:45
rlandy|ruckugh reproducer16:46
*** tesseract has quit IRC16:57
sshnaidmrlandy|ruck, for me job worked in reproducer17:01
sshnaidmrlandy|ruck, do you use this image? | 6a6d23d7-65e7-43ea-9307-71b2e17d2ead | upstream-cloudinit-centos-7               | active |17:03
sshnaidmrlandy|ruck, maybe try "docker-compose down -v; docker-compose up -d"17:04
rlandy|rucksshnaidm: trying that17:18
rlandy|rucksame image17:18
chkumar|roversee ya tomorrow, Good night, Bye!17:21
*** chkumar|rover is now known as raukadah17:21
*** dtantsur is now known as dtantsur|afk17:25
raukadahrlandy|ruck: https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/logs_60/652060/6/check/tripleo-ci-centos-7-standalone-os-tempest/97289e7/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz17:31
raukadahmore breakage is coming17:31
raukadahrlandy|ruck: https://review.opendev.org/#/c/678622/ and we need change in tripleo-ansible spec file17:31
rlandy|ruckraukadah: over and above https://review.opendev.org/#/c/673366/17:33
rlandy|ruckI see discussion on #tripleo17:33
raukadahrlandy|ruck: :-)17:37
rlandy|ruck2019-08-26 12:06:13.362741 | primary | TASK [os_tempest : Execute tempest tests] **************************************17:41
rlandy|ruck2019-08-26 12:06:13.847162 | primary | Monday 26 August 2019  12:06:13 -0400 (0:00:00.750)       2:02:29.520 *********17:41
rlandy|ruck2019-08-26 16:27:29.380476 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]17:41
rlandy|ruck2019-08-26 16:27:29.380779 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]17:41
rlandy|ruck2019-08-26 16:27:31.116683 |17:41
raukadahrlandy|ruck: which job?17:41
rlandy|ruckraukadah: looking at master promotion ... http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-master/f93ae1d/logs/undercloud/home/zuul/17:41
rlandy|ruck4 hours on tempest17:42
rlandy|ruck4.517:42
rlandy|ruckexcessive, no?17:42
raukadahrlandy|ruck: too much excessive and expensive17:42
rlandy|ruckraukadah: too many tests running?17:43
rlandy|ruckraukadah: not commonly ... https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-rhel-8-standalone-master17:43
rlandy|ruckraukadah: ^^ rerunning to check17:44
rlandy|ruckfirst timeout we've seen17:44
rlandy|ruckpossibly a htch17:44
raukadahrlandy|ruck: ok, I am adding to my morning checklist17:44
rlandy|ruckraukadah: let's see if it reproducer first17:44
rlandy|ruckreproduces17:44
rlandy|ruckoh no17:45
rlandy|rucklog failures again17:45
rlandy|ruckweshay: when was the last time you ran a libvirt reproducer?18:38
rlandy|ruckweshay: did it pass?18:38
weshayrlandy|ruck w/ the new repro?18:38
weshayit's been quite some time18:38
rlandy|ruckweshay: wow - no easy getting the reproducer functional18:39
rlandy|ruckThe error was: libvirtError: Cannot get interface MTU on 'virbr0': No such device18:39
rlandy|ruckfailed: [localhost] (item={u'flavor': u'control', u'name': u'subnode-1'}) => {"changed": false, "item": {"flavor": "control", "name": "subnode-1"}, "msg": "Cannot get interface MTU on 'virbr0': No such device"}18:39
rlandy|ruck^^ seen that one?18:39
rlandy|rucktrying libvirt as using tenant gave me the log error18:40
weshayrlandy|ruck ur trying on an rdo box?18:40
rlandy|ruck./reproducer-zuul-based-quickstart.sh -l -e libvirt_volume_path=/home/temp/images18:40
rlandy|ruck^^ no on my minidell18:40
rlandy|ruckworked a while back18:40
weshayrlandy|ruck I have no insight.. other than the ci jobs running daily18:49
rlandy|ruckok18:50
*** brault has joined #oooq18:51
*** brault has quit IRC18:56
rlandy|ruckweshay: can you vote on https://review.opendev.org/#/c/678290/ pls19:50
weshaydone19:54
weshayrlandy|ruck have you seen:20:00
weshayTransaction check error:20:00
weshay2019-08-26 16:05:31 |   file /usr/share/ansible/roles/tripleo-hieradata/tasks/hieradata_vars.yaml conflicts between attempted installs of openstack-tripleo-common-11.1.1-0.20190826025903.29b7c8a.el7.noarch and tripleo-ansible-0.2.1-0.20190826144854.bf61a6f.el7.noarch20:00
weshayhttps://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/logs_28/678028/3/gate/tripleo-ci-centos-7-undercloud-containers/ed39d44/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz20:00
rlandy|ruckweshay: ack20:00
rlandy|ruckwas supposed to be fixed by kevin's patches20:00
rlandy|ruckdiscussion on #tripleo20:00
weshay+120:01
rlandy|ruckweshay: merged about 1 hr ago20:01
rlandy|ruckif you are still seeing it20:01
rlandy|ruckwe have a problem20:01
* weshay just catching up20:02
weshayimpossible20:02
rlandy|ruckgate jobs are clearer now20:02
rlandy|ruckwe had mass failures a while back20:02
rlandy|ruckcheck jobs may need to be rebased20:02
rlandy|ruck2019-08-26 19:37:28,050 29646 INFO     promoter Promoting the container images for dlrn hash 2485770cdc7f29d75b79dc9b6c7c1d099e926c86 on master to current-tripleo20:04
rlandy|ruckweshay: redhat8 is promoting now20:04
rlandy|ruckwas stalled for a bit20:04
weshaywooot!20:04
rlandy|ruckstein should also - if fs020 passes20:05
rlandy|ruckqueens promoted as well20:05
rlandy|rucklog server keeps clogging though20:05
rlandy|ruckwe are runing at 95%20:05
rlandy|ruckand each time the prune script runs, we go over the top20:06
rlandy|ruckother than getting reproducer to work, other stuff is ok20:06
rlandy|ruckreproducer is killing me20:28
weshayrlandy|ruck ya..  if I could squeeze out time, I'd be working on that20:29
weshaymy own pet project... maybe in november20:30
rlandy|ruckI am giving up on rdocloud20:31
rlandy|ruckneed to get libvirt working then20:31
rlandy|ruckI am the only one with this logger issue?20:31
*** aakarsh|2 has quit IRC20:40
*** Vorrtex has quit IRC20:45
*** Goneri has quit IRC20:57
rlandy|ruckweshay: can you tell me how virbr0 is configured on the box where you run libvirt?23:03

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!