Monday, 2019-06-03

*** dsneddon has joined #oooq00:03
*** dsneddon has quit IRC00:09
*** rfolco has quit IRC00:31
*** dsneddon has joined #oooq00:40
*** Goneri has joined #oooq00:46
*** dsneddon has quit IRC00:47
*** saneax has quit IRC00:52
*** dsneddon has joined #oooq01:18
*** Goneri has quit IRC01:24
*** Goneri has joined #oooq01:28
*** dsneddon has quit IRC01:31
*** Goneri has quit IRC01:34
*** dsneddon has joined #oooq02:06
*** dsneddon has quit IRC02:11
*** dsneddon has joined #oooq02:37
*** dsneddon has quit IRC02:42
*** dsneddon has joined #oooq02:58
*** dsneddon has quit IRC03:02
*** dsneddon has joined #oooq03:29
*** dsneddon has quit IRC03:34
*** tcw has joined #oooq03:36
*** ykarel|away has joined #oooq03:53
*** dsneddon has joined #oooq04:07
*** rascasoft has quit IRC04:14
*** rascasoft has joined #oooq04:16
*** dsneddon has quit IRC04:20
*** altlogbot_0 has quit IRC04:36
*** altlogbot_0 has joined #oooq04:36
*** raukadah is now known as chandankumar04:41
*** dsneddon has joined #oooq04:49
*** dsneddon has quit IRC04:53
*** sshnaidm is now known as sshnaidm|off05:03
*** quiquell has joined #oooq05:22
*** dsneddon has joined #oooq05:24
*** udesale has joined #oooq05:26
*** dsneddon has quit IRC05:29
*** jtomasek has joined #oooq05:31
*** saneax has joined #oooq05:33
*** sanjayu_ has joined #oooq05:34
*** sanjayu_ has joined #oooq05:34
*** marios has joined #oooq05:37
*** dtantsur|afk is now known as dtantsur05:44
*** ykarel|away is now known as ykarel05:44
*** dsneddon has joined #oooq05:45
*** saneax has quit IRC05:46
*** skramaja has joined #oooq05:57
*** ksambor has quit IRC05:59
*** jbadiapa has joined #oooq06:02
arxcruze7YPGgQhyY06:03
ykarelwhat password is this :)06:04
mariosarxcruz: delete !06:04
marios:D06:04
arxcruznah, not password06:05
arxcruzu can try :D06:05
mariosarxcruz: what's the name of your first teacher (just curious) also any favourite pets?06:05
arxcruzmarios: lol06:05
arxcruzmarios: for that i only use one answer for everything06:06
arxcruzwhat's the name of your first teacher: answerxyz06:06
arxcruzyour favorite pets: answerxyz06:06
marios:D06:06
arxcruzI was just testing a password generator for gnome, and this was in the clipboard :)06:06
arxcruzneed a replacement for 1password for linux :( the only thing i miss from mac06:07
*** ksambor has joined #oooq06:08
arxcruzneed to go to brazilian embassy to renew my daughter passport and start my visa process, brb06:16
*** ykarel is now known as ykarel|afk06:20
*** ccamacho has joined #oooq06:29
*** jfrancoa has joined #oooq06:37
*** holser_ has joined #oooq06:43
*** d0ugal has joined #oooq06:44
*** dsneddon has quit IRC06:49
*** fmount has quit IRC06:54
chandankumararxcruz: what about using pass and diceware06:58
*** tosky has joined #oooq06:58
*** fmount has joined #oooq06:59
*** kopecmartin|off is now known as kopecmartin06:59
chandankumarmarios: https://code.engineering.redhat.com/gerrit/#/c/167633/1107:14
chandankumarmarios: epel is not needed, we need to check with sf guys regarding rh-python3507:15
*** jfrancoa has quit IRC07:16
*** amoralej|off is now known as amoralej07:18
marioschandankumar: ack but that is blocked see latest comment from apevec07:19
*** dsneddon has joined #oooq07:24
chandankumarmarios: ack!07:28
*** jfrancoa has joined #oooq07:29
*** dsneddon has quit IRC07:32
*** aakarsh has quit IRC07:41
*** hamzy has quit IRC07:41
*** irclogbot_1 has quit IRC07:44
*** irclogbot_1 has joined #oooq07:44
*** jaosorior has joined #oooq07:44
arxcruzchandankumar: what is diceware?07:46
chandankumararxcruz: random password generator which you can remember07:46
arxcruzchandankumar: Right now I am using lesspass.com so, right now I have no idea what my password is07:46
arxcruzfor each site, I have a different password that is generated based on a master key in real time07:47
*** jjoyce has quit IRC07:47
*** dtantsur has quit IRC07:47
*** jjoyce has joined #oooq07:52
*** dtantsur has joined #oooq07:52
*** aakarsh has joined #oooq07:53
*** hamzy has joined #oooq07:53
*** jpena|off is now known as jpena07:55
*** ykarel|afk is now known as ykarel07:55
*** bogdando has joined #oooq08:00
*** dsneddon has joined #oooq08:01
zbri also use 1Password, quite good. I also have a free for open-source team-account if anyone is interested. I used LastPass in the past, good two.08:02
*** dsneddon has quit IRC08:06
mariosperiodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/07633b8/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2019-05-30_02_39_5208:08
mariossorry08:08
*** apetrich has joined #oooq08:10
*** yolanda has joined #oooq08:11
marioschandankumar: check comment 4 pls https://bugs.launchpad.net/tripleo/+bug/183122308:16
openstackLaunchpad bug 1831223 in tripleo "[rocky][fs020] test_resize_server_revert_with_volume_attached and test_show_update_rebuild_list_server failing after moving to 2 compute nodes with Virtual Interface creation failed" [Critical,Triaged]08:16
*** holser_ has quit IRC08:16
chandankumarmarios: on it08:18
chandankumarmarios: commented back, it is coming in 2 comp nodes jobs08:20
chandankumarin 1 compute one got skipped and other were passing08:21
marioschandankumar: ack /me afk for a bit08:21
mariosbiab08:21
*** marios has quit IRC08:21
*** ykarel is now known as ykarel|lunch08:37
*** dsneddon has joined #oooq08:41
dtantsurhi folks! is there a nice way to tear down a quickstart environment (only tear down, not rebuild)08:43
*** dsneddon has quit IRC08:46
*** chem has joined #oooq08:54
dtantsurarxcruz: ^^09:05
arxcruzthere's a --teardown option in quickstart.s h09:07
dtantsurarxcruz: will it skip creating the environment?09:09
arxcruzdtantsur: I believe there's an option for that as well09:09
dtantsurdoesn't seem to be. maybe use --tags with some non-existing tag..09:11
* dtantsur tries09:12
*** marios has joined #oooq09:17
dtantsurhmm, seems to have worked09:17
*** marios_|ruck has joined #oooq09:17
*** dsneddon has joined #oooq09:18
*** dtantsur has quit IRC09:20
*** holser_ has joined #oooq09:22
*** dsneddon has quit IRC09:22
*** dtantsur has joined #oooq09:26
*** ykarel|lunch is now known as ykarel09:34
zbrmarios: panda : please check https://review.opendev.org/#/c/661994/09:37
zbrchandankumar: ^09:38
chandankumarzbr: my query about rhel and rhel-8 files?09:39
zbrchandankumar: my irc client is so annoying as instead of your nick it gives me "ChanServ" as autocomplete :p09:39
marioschandankumar: please check comment #10 https://bugs.launchpad.net/tripleo/+bug/1830351 i see abishop commented there but yesterday has green?09:40
openstackLaunchpad bug 1830351 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens fails tempest run for tempest.api.volume.admin.test_volumes_backup test group" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)09:40
marioszbr: ack will do09:41
ykareldtantsur, u could try running teardown playbooks like https://github.com/openstack/tripleo-quickstart/blob/master/playbooks/teardown.yml with -p quickstart option09:42
dtantsurI guess I ended up doing something like that with "-T all -t none"09:43
ykareldtantsur, and everyting got cleaned up? like vms on virthost09:44
ykarelwhich run under stack user09:44
dtantsuryep, seems so09:44
marioschandankumar: also that please comment #16 @ https://bugs.launchpad.net/tripleo/+bug/1792862  (sorry they are all promotion blockers and seem to be tempest related ) cc arxcruz and kopecmartin can you check -  we merged the skip so removed the promotion blocker. but there is still ongoing work here? the bug is pretty old...09:49
openstackLaunchpad bug 1792862 in tripleo "[master] Telemetry Tempest integration tests failed giving Unable to complete operation on subnet b13c76ec-c851-48d0-91a9-245a2fdcad9b: One or more ports have an IP allocation from this subnet." [Critical,Triaged] - Assigned to chandan kumar (chkumar246)09:49
ykareldtantsur, ack09:50
ykarelseems somehow teardown role is included09:50
ykareland tags teardown-all, none making the cleanup09:51
arxcruzmarios: so it start again?09:52
marios_|ruckarxcruz: no it is skipped09:53
marios_|ruckarxcruz: i'm standing in for panda today and so am catching up on all the blockers.09:53
marios_|ruckarxcruz: i see the skip merged 6 days ago. wondering what else is left there. i removed the promotion-blocker tag, but is there more work09:54
*** dsneddon has joined #oooq09:55
arxcruzmarios_|ruck: ok, so, it's on skip, i'll bring an env and check the root cause, and update the bug ok ?09:55
marios_|ruckarxcruz: ack thanks sounds good cc chandankumar ^09:56
chandankumarmarios_|ruck: yes,09:56
*** dsneddon has quit IRC10:00
*** ykarel_ has joined #oooq10:05
*** ykarel has quit IRC10:08
marios_|ruckchandankumar: arxcruz kopecmartin please see comment #10 when you get a chance did something merge over weekend for green run at https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens green periodic run this morning10:19
marios_|ruckhttps://trello.com/c/dOpKEsnk/985-cixlp1830351tripleociproa-periodic-tripleo-ci-centos-7-ovb-1ctlr1comp-featureset020-queens-fails-tempest-run-for-tempestapivolum10:21
marios_|ruckchandankumar: arxcruz its there so will likely come up on call later  ^^10:21
*** ykarel_ is now known as ykarel10:22
chandankumarmarios_|ruck: we need to ingore check jobs, I was testing the queens by updating tempest tag to 19 there10:22
marios_|ruckchandankumar: ack but i see green *periodic* this morning10:22
chandankumarmarios_|ruck: I think we can wait till tomorrow, as it sometimes it passes or fails10:22
marios_|ruckchandankumar: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens https://review.rdoproject.org/zuul/build/8471a37d9127445b8eae4c3dc0e59a3610:23
marios_|ruckchandankumar: ah ok it isn't consistent?10:23
chandankumarmarios_|ruck: nope, not consistent10:23
marios_|ruckchandankumar: can you please tell launchpad and/or trello about that10:23
chandankumarmarios_|ruck: will update the bz soon10:23
marios_|ruckchandankumar: hold on i'll copy paste chate10:23
marios_|ruckchandankumar: ack :D10:23
ykarelmarios_|ruck, chandankumar i think i posted reason of inconsistency in fs020 queens, may be u guys missed that10:25
ykarelon Friday10:25
ykarelor may be i was wrong so been ignored10:25
marios_|ruckykarel: ack yeah i wasn't racking bugs then panda was ruck/rover so ... was it email/irc?10:25
marios_|ruckykarel: s/racking/tracking :D10:25
ykarelIRC10:25
chandankumarykarel: on tripleo or here?10:26
ykarel:)10:27
ykarelok finding10:27
marios_|ruckchandankumar: good question :D ykarel i/we can find but chan would help for logs search10:27
marios_|ruckykarel: i assume you were discussing with panda/10:27
ykareli think it was chandan, but let me find10:27
*** dsneddon has joined #oooq10:27
chandankumarykarel: I was discussing about other issue related to rocky failure10:28
marios_|ruckykarel: chandankumar k thanks gonna go look for some logs on a new f28 issue will check that in bit chandankumar maybe you can update trello i updated comment there i.e. we need to wait for few more runs10:28
ykarelykarel> chandankumar, i checked ur comment in fs020-queens, i think the issue is happening because of duplicated cinder service names(cinderv3) with different service type(volume,volumev3)10:28
ykarel<ykarel> and that's why we see random SUCCESS in periodic pipeline10:28
ykarel* amoralej is now known as amoralej|off10:28
ykarel<ykarel> abishop and u can relate more on it as it's related to tempestconf and cinder10:28
ykarelchandankumar, marios_|ruck ^^10:28
chandankumarykarel: yes, it is updated in bz comments alos10:28
chandankumarykarel: abishop has replied to it10:29
marios_|ruckykarel: ack is that tempest or cinder issue? or tht issue. i know we were creating a 'fake' endpoint for v1 (which is removed/deprecated) but it wpointed to v310:29
marios_|ruckykarel: maybe thats why there is duplicate?10:29
ykarelchandankumar, okk good, so it's related or not, let me check10:29
* ykarel reading bz10:29
marios_|ruckykarel: chandankumar ack thanks10:29
ykarelchandankumar, i don't see my message there in bz10:31
ykareli only see urs an alan's10:31
chandankumararxcruz: do we know why we need to check for both v2 and v3 https://opendev.org/openstack/python-tempestconf/src/branch/master/config_tempest/services/volume.py#L4510:31
chandankumari think v2 is also deprectated10:31
ykarelme seeing https://bugs.launchpad.net/tripleo/+bug/1830351, may be it's also tracked in different bz10:31
openstackLaunchpad bug 1830351 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens fails tempest run for tempest.api.volume.admin.test_volumes_backup test group" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)10:31
arxcruzchandankumar: nope10:32
*** dsneddon has quit IRC10:32
ykarelmarios_|ruck, so issue from my point of view is:- two services are created for v3 with name: cinderv3 and type: volume and volumev3, and reason of inconsistent failures, when tempestconf detects volumev3 it pass, and when it detects volume it fails10:38
ykarelbut it's good if tempest guys look at it10:39
ykarelas they can more quickly dig the code10:39
marios_|ruckykarel: thank you10:40
ykarellet me know if needed, i can dig again to tempetconf, but issue that i looked on Friday was ^^10:40
ykarelmarios_|ruck, can u review https://review.rdoproject.org/r/#/c/20989/10:52
ykarelmarios_|ruck, it's mainly to get ansible-2.8 in RDO Train, which breaked last time, so we reverted, and now re-revert after handling the last issue10:52
ykarelit should be good now, but as u are |ruck currently u should be aware10:53
ykarelin case if the bump causes new issue10:53
marios_|ruckykarel: ack are we ready to try again /me looks10:59
ykarelmarios_|ruck, yes11:00
marios_|ruckykarel: ok lets see11:00
marios_|ruckykarel: thanks11:00
ykareli added link in commit message to the fix we trying11:00
marios_|ruckykarel: yeah i recall that thanks11:00
ykareli also tested container build job with the fix11:00
ykarelso we should be good11:00
marios_|ruckykarel: anything in a playbook with package: ansible remove you tend to remember :D11:01
marios_|ruckykarel: ack11:01
ykarelmarios_|ruck, :) i also tried completely remove ansible usage, but for that we need to wait for upgrade of software factroy zuul11:02
ykarelmarios_|ruck, https://review.opendev.org/#/c/660247/ fyi11:03
marios_|ruckykarel: thanks11:03
*** dsneddon has joined #oooq11:08
*** tosky has quit IRC11:11
*** dsneddon has quit IRC11:13
*** jpena is now known as jpena|lunch11:31
weshay0/11:33
weshaychandankumar hey buddy... rfolco should be settting up a pre-sprint planning mtg11:37
weshayccccccdlvueffgehujglbbkkinneejflflvihiitduuj11:38
weshaydang11:38
marios_|ruckweshay: you have cats?11:38
weshayheh.. just on a make shift desk atm11:38
weshaypanda how is the ruck11:39
weshaymarios_|ruck why are you ruck?11:39
*** udesale has quit IRC11:40
*** udesale has joined #oooq11:40
marios_|ruckweshay: public holiday i believe11:40
weshayah k k11:41
weshaymarios_|ruck need anything?11:41
marios_|ruckweshay: so mostly things are OK - few ongoing promotion blockers but should be clearing. i see some new ones that i will file but no big problems afaics @ http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=now-7d&to=now11:41
weshayya.. bank holiday in ireland11:41
marios_|ruckweshay: panda isn't tracking in etherpad/etc so going on that ^^ updated a few trello cards in prod chain11:42
weshayya.. I saw we hit 90% pass in upstream.. that is unusually high11:42
marios_|ruckweshay: and been harrassing tempest folks chandankumar++ arxcruz++ for ongoing/known issues tempest related11:42
weshaymarios_|ruck k great.. thanks :)11:42
weshayzbr howdy sir11:42
marios_|ruckweshay: introspection issue ongoing at https://bugs.launchpad.net/tripleo/+bug/1831053 comment #211:42
openstackLaunchpad bug 1831053 in tripleo "BMC node in periodic OVB jobs fails to retrieve metadata information" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)11:42
marios_|ruckweshay: ack will let you know if things escalate and obv will hand off later before i go11:43
weshayk.. marios_|ruck if it's introspection it would be worth a baremetal run or vexhost11:43
weshayjust to eliminate any questions about rdo-cloud11:44
*** dsneddon has joined #oooq11:45
weshaymarios_|ruck rdo-cloud jobs are running at 86% pass also a little higher pass rate than normal :)11:45
*** saneax has joined #oooq11:46
*** holser_ is now known as holser|lunch11:47
chandankumarmarios_|ruck: /me is digging into the code11:48
marios_|ruckchandankumar: ack well fyi theres a couple new tempest related ones on master like http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/fef4dda/logs/tempest.html.gz11:49
marios_|ruckchandankumar: but waiting to see it more times before filing bug11:49
marios_|ruckchandankumar: also f28 https://github.com/marios/tripleo_ruck_job_tool.git11:49
marios_|ruckchandankumar: oops that tempest http://logs.openstack.org/61/662561/4/gate/tripleo-ci-fedora-28-standalone/b708bd0/logs/tempest.html.gz11:50
*** dsneddon has quit IRC11:50
ykarelmarios_|ruck, re. fs030 and other ovb jobs they are failing/timeout since 31st11:51
marios_|ruckykarel: ack looks like bug was filed on 30th11:52
chandankumarzbr: Hello11:52
chandankumarzbr: I tested the os_tempest patch here https://code.engineering.redhat.com/gerrit/#/c/171117/11:52
ykarelmarios_|ruck, /me not aware of bug, but marios_|ruck, https://review.rdoproject.org/r/#/c/20990/ should fix it and preventive action:- https://review.rdoproject.org/r/#/c/20991/11:52
chandankumarzbr: it is not working https://sf.hosted.upshift.rdu2.redhat.com/logs/17/171117/13/check/tripleo-ci-rhel-8-standalone-rhos-15/12b44bd/job-output.txt.gz#_2019-06-03_07_28_31_09870011:52
marios_|ruckykarel: this one https://bugs.launchpad.net/tripleo/+bug/183105311:53
openstackLaunchpad bug 1831053 in tripleo "BMC node in periodic OVB jobs fails to retrieve metadata information" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)11:53
marios_|ruckykarel: looking thanks11:53
ykarelmarios_|ruck, ^^ is different11:53
marios_|ruckykarel: thanks don't think we even have/had a bug for that one then?11:55
ykarelmarios_|ruck, /me don't think so bug is there11:56
*** tesseract has joined #oooq11:57
chandankumarzbr: please have a look on the patch11:57
*** rfolco has joined #oooq11:58
weshayrfolco morning...  bug triage? chandankumar you avail?11:59
chandankumarweshay: Yes11:59
rfolcoweshay, give me 3 min please, switching rooms11:59
marios_|ruckweshay: rfolco chandankumar do you need ruck there?11:59
weshaymarios_|ruck hrm..  if you have things under control otherwise it could help.  If you are busy tracing errors no worries12:01
marios_|ruckweshay: ac,]12:01
marios_|ruckweshay: ack can join12:01
*** rlandy has joined #oooq12:06
*** tosky has joined #oooq12:06
marios_|ruckchandankumar: rfolco weshay this https://bugs.launchpad.net/tripleo/+bugs?orderby=-datecreated&start=012:08
chandankumarmarios_|ruck: f28 bug https://bugs.launchpad.net/tripleo/+bug/183071812:09
openstackLaunchpad bug 1830718 in tripleo "[fedora28][check] tempest boot volume pattern and network basic ops tests are failing " [Critical,Triaged]12:09
rlandyweshay; hello12:15
weshayhttps://bugs.launchpad.net/tripleo/+bug/182973612:15
openstackLaunchpad bug 1829736 in tripleo "periodic-tripleo--master-containers-build is failing with "Failed to import docker-py - cannot import name __version__"" [Critical,In progress]12:15
weshayhttps://bugs.launchpad.net/tripleo/+bug/182946812:16
openstackLaunchpad bug 1829468 in tripleo "OVB jobs failing in overcloud-prep-images with timeout on introspection" [Critical,Incomplete]12:16
marios_|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build-push12:16
*** amoralej is now known as amoralej|lunch12:16
rlandyweshay: welcome back ... I need to chat with you re: perf testing some time today12:17
weshayhttps://bugs.launchpad.net/tripleo/+bug/182938212:17
openstackLaunchpad bug 1829382 in tripleo " ovb-3ctlr_1comp_1supp-featureset039 failing overcloud deploy "Failed to connect to NOVA_API database "" [High,Triaged] - Assigned to Martin Schuppert (mschuppert)12:17
*** dsneddon has joined #oooq12:21
*** rfolco has quit IRC12:21
weshaymarios_|ruck https://bugs.launchpad.net/tripleo/+bug/182786312:22
openstackLaunchpad bug 1827863 in tripleo "(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups" [High,Triaged] - Assigned to Marios Andreou (marios-b)12:22
weshayrlandy hey :)12:22
weshayfew min12:23
weshaychandankumar https://bugs.launchpad.net/tripleo/+bug/182712112:24
openstackLaunchpad bug 1827121 in tripleo "TestServerBasicOps failing on gate: SSHException: Error reading SSH protocol banner" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)12:24
rlandysshnaidm|off: you around by any chance ( despite the obvious |off)?12:24
*** rfolco has joined #oooq12:24
weshaymarios_|ruck https://bugs.launchpad.net/tripleo/+bug/182672912:25
openstackLaunchpad bug 1826729 in tripleo "Requested MTU is too big, maximum is 1500" [Undecided,New]12:25
weshaymarios_|ruck can be closed https://bugs.launchpad.net/tripleo/+bug/1826179 ?12:25
openstackLaunchpad bug 1826179 in tripleo "standalone-upgrade fails during deploy steps (generate config) step1 for "Evaluation Error:... Could not find class ::tripleo::profile::base::placement::api" " [High,Triaged] - Assigned to Marios Andreou (marios-b)12:25
*** holser|lunch is now known as holser_12:26
*** apetrich has quit IRC12:26
*** dsneddon has quit IRC12:27
*** apetrich has joined #oooq12:27
*** jpena|lunch is now known as jpena12:29
*** vinaykns has joined #oooq12:29
chandankumarmarios_|ruck: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/requirements.txt#L312:30
chandankumarmarios_|ruck: we need to update the ansible version here also12:31
rlandymarios_|ruck: hey - do you know anything about placement? and the recent placement changes?12:33
zbrweshay: hi! i am back.12:33
weshayzbr hey there :)  You are still my point guy for OSP 15, but atm it looks pretty calm..  Any work in progress on osp-15 that I need to be aware of?12:35
*** saneax has quit IRC12:35
zbrweshay: nope. the meeting were indeed calm, with no direct actions for us.12:36
weshayk cool :)12:36
ykarelmarios_|ruck, re. https://bugs.launchpad.net/tripleo/+bug/1831053 the actual issue is what panda mentioned in description12:36
openstackLaunchpad bug 1831053 in tripleo "BMC node in periodic OVB jobs fails to retrieve metadata information" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)12:36
zbri managed to setup the VPN on Friday in order to be able to run RHEL stuff from my homelab.12:37
ykarelwe have seen this issue from time to time and related to rdo-cloud12:37
ykarelmentioning here after i saw ur comment in CIX call12:37
zbrweshay: i am trying to help chandan with a change to ansible os-tempest role (related to multi-os), should finish in soon.12:38
*** rascasoft has quit IRC12:39
*** rascasoft has joined #oooq12:41
*** jschlueter has quit IRC12:43
*** jschlueter has joined #oooq12:43
weshayrlandy hey.. can chat for a few if needed12:47
rlandysure12:48
rlandyjoining your bj12:48
*** vinaykns has quit IRC12:48
zbrchandankumar: I do not understand the link you showed me about "not working", is not from my change: https://review.opendev.org/#/c/661994/12:48
*** vkapalav has joined #oooq12:49
chandankumarzbr: on this review https://code.engineering.redhat.com/gerrit/#/c/171117/ we are testing the changes downstream12:49
chandankumarcheck depends-on12:49
chandankumarit runs RHOS-15 on RHEl-812:50
zbrchandankumar: aha. ok I will add some debug for that string. somehow ansible output was useless in that case.12:51
zbrchandankumar: i was confused because i did not see any reply from zuul on that change.12:51
chandankumarzbr: rhel and rhel-8 file, can we merge it in a single one?12:51
chandankumarzbr: it is not voting back12:51
zbrchandankumar: not really, rhel is more of rhel-7.12:51
chandankumarthat is in the plan in another sprints12:51
zbrwe need to keep it compatible with old versions, which means that we need py2 packages for rhel/centos-712:52
zbrthat is why I added the new ones on -8 file.12:52
zbrbut is overriding those from the unversioned file12:53
*** vkapalav is now known as vinaykns12:57
*** dsneddon has joined #oooq12:59
rfolcoscrum time13:02
rfolco marios, sshnaidm, weshay, panda, rlandy, arxcruz, rfolco, chkumar, zbr, kopecmartin13:02
*** dsneddon has quit IRC13:04
*** Vorrtex has joined #oooq13:09
*** aakarsh has quit IRC13:11
*** jaosorior has quit IRC13:15
*** amoralej|lunch is now known as amoralej13:26
chandankumarmarios_|ruck: rlandy https://tree.taiga.io/project/tripleo-ci-board/issue/1096 gating tempest downstream patches13:30
*** dsneddon has joined #oooq13:31
*** aakarsh has joined #oooq13:35
*** dsneddon has quit IRC13:36
*** panda has quit IRC13:39
*** panda has joined #oooq13:42
*** blueDuckRandom has joined #oooq13:59
*** ykarel_ has joined #oooq14:00
arxcruzmarios_|ruck: so, i do have an env up and I can reproduce the issue with telemetry14:00
arxcruzi don't see what changed to make it start to fail14:01
*** ykarel has quit IRC14:03
*** skramaja has quit IRC14:07
*** dsneddon has joined #oooq14:07
arxcruzmarios_|ruck: we can ask someone from telemetry to check14:09
marios_|ruckarxcruz: ack thanks vry much can you hold the env for a while and maybe note on the bug so panda can pickup tomorrow14:11
*** dsneddon has quit IRC14:12
arxcruzmarios_|ruck: sure14:16
rlandymarios_|ruck: weshay: these are the two outstanding reviews for rhos-15 ... https://review.opendev.org/#/c/659794/ (marios_|ruck already voted) and https://code.engineering.redhat.com/gerrit/#/c/171119/ (where I have removed the depends-on to merge)14:22
*** quiquell has quit IRC14:22
*** Goneri has joined #oooq14:24
arxcruzzbr: so https://review.opendev.org/#/c/661979/ or https://review.opendev.org/#/c/661994 ?14:24
marios_|ruckrlandy: ack will check in bit14:26
*** ykarel_ is now known as ykarel14:26
chandankumarzbr: please have a look at version compair https://review.opendev.org/#/c/661979/11/vars/redhat.yml@2514:37
rfolcorlandy, https://code.engineering.redhat.com/gerrit/#/c/172246/ would you merge this one please?14:37
chandankumarI am certainly doing something wrong there14:37
rlandyrfolco: I can - this is a testable repo - is it possible to post test results?14:38
*** dsneddon has joined #oooq14:39
rfolcorlandy, I can run a dnm with depends on if you prefer... I tested it without chdir, and then realized the modules does not support it14:39
rfolcothe archive module I mean14:39
rlandyrfolco: k - I'll merge this one - in future, where testable results are easier to review14:40
rfolcorlandy, ok, thanks14:40
*** dsneddon has quit IRC14:43
zbrchandankumar: the code from downstream does not seem to respect the depends-on14:47
zbrlook at https://sf.hosted.upshift.rdu2.redhat.com/logs/17/171117/13/check/tripleo-ci-rhel-8-standalone-rhos-15/0af8ad6/job-output.txt.gz#_2019-06-03_09_36_59_211945 -- there is no debug task before it.14:48
chandankumarzbr: did we changed depends-on?14:48
chandankumarzbr: I think you updated the https://review.opendev.org/#/c/661979/ this patch, /me has not changed the depends-on14:49
zbrchandankumar: no. please look at and tell me https://code.engineering.redhat.com/gerrit/#/c/171117/14:49
zbrafter i added the debug statement, I did a recheck on the downstream change.14:49
chandankumarzbr: I think you have updated my patch14:49
zbroops...14:50
zbrchandankumar: btw, very sad that ansible gives such a cryptic error on "Install distro packages" task.14:51
chandankumarzbr: yes, better we need to improve that error msg on ansible side which might come in new release14:52
zbrchandankumar: file a bug, i will support it.14:56
chandankumarzbr: sure, will do it14:56
chandankumarweshay: rlandy https://review.rdoproject.org/r/#/c/20975/ to move 1 comp fs020 jobs14:59
*** ccamacho has quit IRC15:01
*** ykarel_ has joined #oooq15:01
rlandychandankumar: just checking those jobs are not referenced anywhere15:03
*** ccamacho has joined #oooq15:04
*** ykarel has quit IRC15:04
rlandychandankumar: k - w+ed15:06
zbrmarios_|ruck: failure to download packages from rax mirrors, is known? http://logs.openstack.org/65/649965/32/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/6f217e7/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz#_2019-06-03_10_13_3115:07
*** ykarel_ is now known as ykarel15:07
zbri wonder why I do not see any retries around these, afaik yum is supposed to retry 10 times by default.15:08
zbrand clearly Error 14 (transfer closed) is a recoverable one.15:08
*** tosky has quit IRC15:08
*** irclogbot_1 has quit IRC15:10
*** irclogbot_0 has joined #oooq15:10
*** altlogbot_0 has quit IRC15:11
*** altlogbot_0 has joined #oooq15:11
*** dsneddon has joined #oooq15:12
*** ykarel is now known as ykarel|away15:12
chandankumarweshay: rfolco https://sf.hosted.upshift.rdu2.redhat.com/etherpad/p/tripleoci-rhel815:13
mariosrlandy: o/ i have an ai to ping you about that https://bugs.launchpad.net/tripleo/+bug/1831053 introspection issue we're trying to track down - raised in prod chain escalation call today and asked for ironic to check it out, but, do we have logs from a bm job we can check or can you perhaps kick a bm job to see if we can hit it?15:15
openstackLaunchpad bug 1831053 in tripleo "BMC node in periodic OVB jobs fails to retrieve metadata information" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)15:15
* rlandy looks15:16
*** dsneddon has quit IRC15:17
mariosrlandy: voted at https://code.engineering.redhat.com/gerrit/#/c/171119 already +2 in https://review.opendev.org/#/c/65979415:20
rlandymarios; thanks15:23
rlandysorry - git distracted with downstream - looking at ovb issue now15:23
rlandygot15:23
rlandymarios: as far as long from baremetal, yes15:24
rlandybut baremetal does not use bmc15:24
rlandymarios; you can look at bm logs here ...15:24
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envB-3ctlr_1comp-featureset001-stein/5e1a98c/15:25
marios_|ruckrlandy: ack thanks15:25
marios_|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/15:26
*** dsneddon has joined #oooq15:26
rlandymarios_|ruck: looking at the bug now15:26
rlandymarios_|ruck: oh - you want master logs15:27
rlandygetting15:27
marios_|ruckrlandy: ack /me almost out but am thinking where best to note that link either on the bug or maybe better on the escalation card https://trello.com/c/hQ41vura/989-cixlp1831053tripleociproa-bmc-node-in-periodic-ovb-jobs-fails-to-retrieve-metadata-information15:27
rlandyha ... https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/0066e43/job-output.txt.gz15:28
rlandyalso timed out15:28
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/0066e43/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz15:28
rlandynothing in the log though15:28
marios_|ruckrlandy: ack ok thanks for pointer anyway will check closer tomorrow15:29
rlandymarios_|ruck: hmmm ... we also have a timeout in master bm15:29
rlandyon introspection15:30
rlandyyou may have a real issue here15:30
marios_|ruckrlandy: i can see all the jobs there https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/15:30
marios_|rucki think thats the best link to share15:30
rlandyI can get you/ironic devs on the hardware15:30
marios_|ruckrlandy: ack cool well we escalated today to them dfg:hardprov so lets see  i'll add pointer to logs for now15:30
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/0066e43/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz15:30
rlandylet me see if I can find an error15:31
marios_|ruckrlandy: well usuasll we can find the errors in bmc log but we don't have it there15:31
rlandy2019-06-02 16:07:53.686193 | primary | TASK [overcloud-prep-images : Prepare the overcloud images for deploy] *********15:32
rlandy2019-06-02 16:07:53.848610 | primary | Sunday 02 June 2019  16:07:53 +0000 (0:00:06.785)       0:00:51.912 ***********15:32
rlandy2019-06-02 18:32:09.629911 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]15:32
rlandy^^ definitely a hang there15:32
rlandylook at the time15:32
marios_|ruckrlandy: ack thats the right symptom for the bug anyway15:32
* marios_|ruck shutdown sequence weshay need something before i go15:34
marios_|ruck?15:34
*** tcw has quit IRC15:35
*** tcw has joined #oooq15:35
*** sanjayu_ has quit IRC15:36
*** tcw has quit IRC15:36
*** tcw has joined #oooq15:37
*** tcw has quit IRC15:37
*** tcw has joined #oooq15:38
*** tcw has quit IRC15:38
*** tcw has joined #oooq15:38
*** tcw has quit IRC15:39
chandankumarrfolco: weshay I am logging out see ya tomorrow15:43
*** chandankumar is now known as raukadah15:43
* marios_|ruck too weshay rfolco ttyl going in aminute15:43
*** marios_|ruck has quit IRC15:44
*** marios has quit IRC15:44
*** zbr has quit IRC15:47
*** bogdando has quit IRC15:49
*** zbr has joined #oooq15:55
*** ykarel|away has quit IRC15:56
*** dtantsur is now known as dtantsur|afk16:15
*** tesseract has quit IRC16:27
zbr@oooq: i think you may want to try ssh plugin on vscode: https://code.visualstudio.com/docs/remote/ssh --- i am surprised how nice it works, mainly only the editor is local.16:34
*** panda has quit IRC16:35
*** panda has joined #oooq16:37
sshnaidm|offrlandy, hi16:51
rlandysshnaidm|off: hey - just carrying on with the ovb stuff16:51
rlandyno worries - been in touch with kforde-16:51
rlandywe can chat tomorrow16:51
*** holser_ has quit IRC16:52
sshnaidm|offrlandy, ok, did he find something?16:52
rlandysshnaidm|off: yeah - he said the hotfix for ovb was not implemented completely16:53
rlandyI am running again now16:53
sshnaidm|offrlandy, heh..16:53
rlandybut still seems very slow16:53
rlandysshnaidm|off: left notes on the card16:53
sshnaidm|offrlandy, thanks, will read16:54
weshayrlandy sshnaidm|off could it be the branch of ovb we're using?16:59
weshayor pin?16:59
sshnaidm|offweshay, it's not ovb fault, it's upshift16:59
rlandyweshay: I think it's the same one working in rdocloud16:59
rlandythe communication is extremely slow16:59
*** jpena is now known as jpena|off17:04
*** jfrancoa has quit IRC17:18
*** rlandy has quit IRC17:22
*** amoralej is now known as amoralej|off17:22
*** rlandy has joined #oooq17:22
weshayrlandy right.. but rdo-cloud I think is patched for ovb17:22
rlandyweshay: yep so is upshift17:23
weshayoh.. $new info for me17:23
weshayk17:23
weshayrlandy apparently I think the qe folks are running ovb on it17:23
weshayrlandy Alex may have an example job17:23
rlandyweshay: ack - we are comparing with the upgrade deployments as well17:24
weshayk17:24
weshaythanks17:24
rlandymy vpn cut out17:25
rlandyreconnecting17:25
rlandythe nodes just shut down now17:25
*** Vorrtex has quit IRC17:38
*** udesale has quit IRC17:41
rfolcoweshay, I'm trying to collect more data to understand the root cause of connect reset by peer issue. Do you have any ideas on how to unblock me ?17:58
rfolcohttps://code.engineering.redhat.com/gerrit/#/c/171832/12/playbooks/tripleo-buildcontainers/Dockerfile.j217:59
raukadahrfolco: is it possible to build this dockerfile locally?18:06
raukadahon the same env18:06
rfolcoraukadah, I did, cannot reproduce the same issue though18:14
*** altlogbot_0 has quit IRC18:43
*** altlogbot_3 has joined #oooq18:43
*** kopecmartin is now known as kopecmartin|off18:47
rlandysshnaidm|off: this is where it leaves off ... afaict dhcp never completes ... https://sf.hosted.upshift.rdu2.redhat.com/logs/68/171268/16/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-internal/835f89e/logs/undercloud/var/log/containers/ironic-inspector/dnsmasq.log.txt.gz18:51
rlandyweshay:is this log intentionally unreadable: overcloud_prep_images_introspection_debug.pcap.log.txt.gz?18:51
weshayrlandy it's binary18:52
weshaythat has to be read w/ wireshark18:53
weshaydownload and import18:53
*** dsneddon has quit IRC19:07
*** altlogbot_3 has quit IRC19:21
*** altlogbot_2 has joined #oooq19:25
*** dsneddon has joined #oooq19:31
*** dsneddon has quit IRC19:37
rlandyweshay: pls vote on https://review.opendev.org/#/c/659794/ to go forward with this change19:46
*** hamzy_ has joined #oooq20:06
*** dsneddon has joined #oooq20:12
*** hamzy has quit IRC20:15
weshayrlandy voted and wf20:16
*** jtomasek has quit IRC20:16
rlandyweshay: thanks20:17
*** Vorrtex has joined #oooq20:31
*** altlogbot_2 has quit IRC20:45
*** Goneri has quit IRC20:49
*** jbadiapa has quit IRC20:52
*** rlandy is now known as rlandy|brb21:04
*** rlandy|brb is now known as rlandy21:18
*** sshnaidm|off has quit IRC21:34
*** sshnaidm has joined #oooq21:40
*** sshnaidm is now known as sshnaidm|off21:41
*** Goneri has joined #oooq21:47
*** aakarsh has quit IRC22:24
*** aakarsh has joined #oooq22:49
*** zbr has quit IRC23:00
*** sanjayu_ has joined #oooq23:02
*** Vorrtex has quit IRC23:12
*** dsneddon has quit IRC23:22
*** dsneddon has joined #oooq23:26

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!