Tuesday, 2018-04-24

hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario007-multinode-oooq-container | check logs @ https://review.openstack.org/472607 a00:20
*** rlandy has quit IRC00:52
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario007-multinode-oooq-container | check logs @ https://review.openstack.org/472607 a02:20
myoungpanda|bbl, quiquell|off: (reminder) please update https://etherpad.openstack.org/p/tripleo-ci-squad-meeting @29 w/ CI status prior to the #tripleo meeting tomorrow @ 1400 UTC03:20
*** ykarel|away has joined #oooq04:05
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master | check logs @ https://review.openstack.org/472607 and fix them ASAP.04:20
*** ratailor has joined #oooq04:20
*** ratailor has quit IRC04:21
*** ratailor has joined #oooq04:35
*** ratailor has quit IRC04:35
*** ykarel|away is now known as ykarel04:40
*** skramaja has joined #oooq04:56
*** yolanda_ has joined #oooq05:27
*** yolanda has quit IRC05:28
*** ccamacho has quit IRC05:37
*** marios has joined #oooq05:39
*** ratailor has joined #oooq05:40
*** quiquell|off is now known as quiquell|rover05:40
*** links has joined #oooq06:16
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.06:20
*** jtomasek has joined #oooq06:25
*** anande has joined #oooq06:28
*** kopecmartin has joined #oooq06:33
ykarelchandankumar, arxcruz https://review.openstack.org/#/c/562316 is breaking ocata, please check06:41
ykarelhttps://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-ocata/226/consoleText06:41
ykarelquiquell|rover, ^^06:42
quiquell|roverykarel: A lot of stuff is going on, give a few minutes06:42
ykarelnp06:42
ykareltake ur time06:43
*** tesseract has joined #oooq06:49
arxcruzquiquell|rover: don't worry, i'm on it06:50
quiquell|roverykarel: New master blocker of the day :-/ https://bugs.launchpad.net/tripleo/+bug/176649306:53
openstackLaunchpad bug 1766493 in tripleo "Building kolla openstack-base No package python2-positional available" [Critical,New] - Assigned to Quique Llorente (quiquell)06:53
ykarelquiquell|rover, i think this should be tracked under the bug proposed yesterday06:53
quiquell|roverykarel: Also I am rover now, so I will look into it too (or at least learn howto).06:53
ykarelproposed/reported06:54
quiquell|roverykarel: the neutron-base or the mirroring ?06:54
ykarelmirroring one06:54
ykarelgperftools06:54
quiquell|roverThey have just push gperftools06:54
quiquell|roverNothing to do with this one06:54
quiquell|roverThey will have to add also python2-positional ?06:55
ykarelyes but similar to gperftools, positional is also missed06:55
ykarelyes06:55
quiquell|roverOhhh, I will relate to the other06:55
*** ccamacho has joined #oooq06:55
ykareli pinged amoralej|off to check this when he is available06:55
quiquell|roverIs good to have both bugs open06:55
quiquell|roverOk, will go back to rover duties06:55
ykarelokk06:55
quiquell|roverThanks ykarel06:55
*** jfrancoa has joined #oooq06:56
*** bogdando has joined #oooq06:56
*** jfrancoa has quit IRC07:00
*** jfrancoa has joined #oooq07:00
arxcruzchandankumar: do we have stestr in pike ?07:03
arxcruzykarel: i think it's also breaking newton07:03
ykarelarxcruz, yes it should, but i haven't checked logs07:04
ykarelin pike i saw stestr there07:04
ykarelhttps://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-pike/07:04
*** jfrancoa has quit IRC07:04
arxcruzpike is ok07:05
*** jfrancoa has joined #oooq07:05
*** jaosorior has joined #oooq07:08
*** jfrancoa1 has joined #oooq07:08
*** quiquell|rover is now known as quique|rover|afk07:09
arxcruzykarel: https://review.openstack.org/#/c/563843/07:09
*** jfrancoa has quit IRC07:10
ykarelarxcruz, you forgot to remove it from the list in previous task07:10
arxcruzgod dammit!07:10
*** amoralej|off is now known as amoralej07:11
arxcruzdone07:11
ykarelarxcruz, so os-testr not required in newton/ocata?07:12
arxcruzykarel: nope, it's a new package introduced in whatever version to replace testr07:12
arxcruzinitially was wrapping testr, but now it's completly independent07:12
arxcruzchandankumar can tell more about it i might be wrong in the details07:12
ykarelokk07:12
ykarelack07:13
arxcruzsshnaidm|off: https://review.openstack.org/#/c/562992/ if you don't mind07:13
*** sshnaidm|off is now known as sshnaidm07:14
chandankumararxcruz: yup, I have added it in pike07:17
chandankumararxcruz: where os-testr >= 1.0.0 is used stestr is there07:18
*** agopi is now known as agopi|zzz07:24
*** holser__ has joined #oooq07:40
*** quique|rover|afk is now known as quiquell|rover07:49
*** yolanda_ has quit IRC07:49
*** hamzy has quit IRC07:51
*** hamzy has joined #oooq07:51
amoralejykarel, quiquell|rover https://review.openstack.org/#/c/563850/07:54
amoralejykarel, quiquell|rover i'm running locally build-containers with patch for kolla07:57
ykarelamoralej, ack07:58
*** yolanda_ has joined #oooq08:02
*** yolanda_ is now known as yolanda08:02
*** ykarel is now known as ykarel|lunch08:03
quiquell|roveramoralej: ack08:08
quiquell|roverAdding it to the bug08:08
amoralejquiquell|rover, i did it08:08
amoralejwhen is next periodic run?08:08
*** tosky has joined #oooq08:13
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.08:20
*** lucas-afk is now known as lucasagomes08:31
*** moguimar has joined #oooq08:32
*** ykarel|lunch is now known as ykarel08:40
*** gkadam has joined #oooq08:41
ykarelamoralej, next periodic is in next 3 hours08:41
ykarel12:10 UTC08:41
*** yolanda has quit IRC08:48
*** moguimar has quit IRC08:57
*** yolanda has joined #oooq09:01
*** dtantsur|afk is now known as dtantsur09:09
*** anande has quit IRC09:12
*** jaosorior has quit IRC09:18
panda|bblquiquell|rover: morning, do we have a bug for this ? https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/d451ae3/undercloud/home/jenkins/tempest/tempest.log.txt.gz#_2018-04-23_00_50_14_95709:25
*** panda|bbl is now known as panda|ruck09:25
quiquell|roverGood morning09:25
quiquell|roverhttps://bugs.launchpad.net/tripleo/+bug/176597509:27
openstackLaunchpad bug 1765975 in tripleo "ssh timeout in tempest basic ops scen007 gate jobs" [Critical,Fix released] - Assigned to yatin (yatinkarel)09:27
quiquell|roverCheck if it covers it09:27
quiquell|roverA have update the etherpad09:27
quiquell|roverhttps://review.rdoproject.org/etherpad/p/ruckrover-sprint1209:27
quiquell|roverIt's all yours now09:27
panda|ruckyay09:28
panda|ruck!gatestatus09:30
openstackpanda|ruck: Error: "gatestatus" is not a valid command.09:30
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.09:30
panda|ruckmmh, zuul queue is at 15 hours, but gates seem fine09:32
quiquell|roverpanda|ruck: The mirror problem from yesterday has prevent people from merging09:37
quiquell|roverso the queues are piled up with stuff to do09:37
panda|ruckah, ok.09:38
*** jaosorior has joined #oooq09:44
panda|ruckquiquell|rover: after cleaning dowstream tenat did you launch test jobs for rhos-12 ?09:48
quiquell|roverWe are still unable to clean up09:48
quiquell|roverCan not delete the stack, I am opening another ticket09:49
quiquell|roverI can remove by hand all the stacks stuff09:49
quiquell|roverbtw do you know what kind of ticket do we have to open here ?09:49
*** moguimar has joined #oooq09:49
quiquell|roverhttps://redhat.service-now.com/pnt?id=pnt_index09:50
quiquell|roverpanda|ruck: Do we have any meeting today ?09:50
panda|ruckquiquell|rover: yes09:51
panda|ruckquiquell|rover: TripleO at 14 UTC, TOCI community right after (optional)09:52
amoralejquiquell|rover, panda|ruck wrt https://bugs.launchpad.net/tripleo/+bug/1766493 i think we can wait for https://review.openstack.org/#/c/563850/ and https://review.rdoproject.org/r/#/c/13516/09:52
openstackLaunchpad bug 1766493 in tripleo "Building kolla openstack-base No package python2-positional available" [Critical,In progress] - Assigned to Quique Llorente (quiquell)09:52
panda|ruckquiquell|rover: TripleO is on IRC09:52
amoraleji'd prefer not to revert rocky bootstrap deps09:52
panda|ruckquiquell|rover: even if if stack are not deleted, resources may be free09:54
panda|ruckamoralej: consider gate queue is at 15 hours now, and you still need reviews. So it may take one day to merge those09:56
quiquell|roverpanda|ruck: Let's open the ticket, maybe one resource is ok but another is not09:56
amoralejpanda|ruck, not, it's not tripleo gate09:56
amoralejit should go fast i think09:56
panda|ruckoh kolla09:56
amoralejyes, kolla and rdoinfo09:57
*** chem has joined #oooq10:02
panda|ruckquiquell|rover: well, nevermind, downstream tenant is currently stuck10:04
panda|ruckah, no , only the images "unable to get"10:04
panda|ruckquiquell|rover: cleaning up the networks10:05
quiquell|roverpanda|ruck: I have also open the ticket, in case we found more problems10:06
quiquell|roverLet me know when the network cleanup is finished I will launch the job10:06
*** quiquell|rover has quit IRC10:14
*** quiquell has joined #oooq10:15
*** zoli is now known as zoli|lunch10:15
quiquellpanda|ruck: https://review.openstack.org/#/c/561911/ is just missing the +1w ?10:17
panda|ruckquiquell: no, it's missing also the two +210:17
quiquellYes, I mean if some timestamp is still missing10:18
quiquellOr just the approval10:18
panda|ruckquiquell: I think we have all now10:20
quiquellOk will try to move it10:20
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.10:20
*** quiquell is now known as quiquell|rover10:20
panda|ruckquiquell|rover: Done network cleanup, I had to manually remove interfaces from routers10:23
panda|ruckquiquell|rover: also deleted all the stacks10:26
panda|ruckquiquell|rover: mmhh, nevermind the last phrase10:26
quiquell|roverstacks are screw10:26
quiquell|roverCool will start job10:27
panda|ruckquiquell|rover: ok, was able to delete stack10:27
quiquell|roversure ?10:27
quiquell|roverI see a lot of DELETE_FAILED10:28
quiquell|roverIt's the rhos-dev-ci tenant ?10:28
quiquell|roverBut resource will be ok10:29
panda|ruckquiquell|rover: all stacks deleted10:29
quiquell|roverDidn't them give you problems with volumes ?10:31
quiquell|roverI will close the case10:31
panda|ruckquiquell|rover: I see not volumes10:32
panda|ruckquiquell|rover: no*10:32
quiquell|roverWeird ok10:32
quiquell|roverbetter10:32
panda|ruckquiquell|rover: maybe there was some cleanup already for the volumes. I also had to remove manually the networks and suspend the stack before deleting them10:32
quiquell|roverJob started10:32
panda|ruckok, let's see if we can restore finally the job10:33
quiquell|roverMaybe there were something in progress10:33
quiquell|roverpanda|ruck: Have just star it, without using gerrit10:33
quiquell|roverIt's that ok ?10:33
panda|ruckquiquell|rover: from jenkins directly ?10:35
quiquell|rover"Build Now"10:36
panda|ruckquiquell|rover: ok, I'm not sure you need to change something in the main builder to make it work without a change to check10:36
quiquell|roverpanda|ruck: Let's see if it works10:41
*** jaosorior has quit IRC10:43
*** moguimar has quit IRC10:45
*** moguimar has joined #oooq10:47
ykarelpanda|ruck, hi10:51
panda|ruckykarel: hey10:51
ykarelpanda|ruck, can you check why phase 1 is not promoting for: 5466f249bd36900a1dac573cdc83e7a11493aea2_0c8f7f9510:51
ykarelpanda|ruck, all required jobs passed yesterday only: https://trunk.rdoproject.org/api-centos-queens/api/civotes_detail.html?commit_hash=5466f249bd36900a1dac573cdc83e7a11493aea2&distro_hash=0c8f7f95a9dace8504905c03862036fe270f154510:52
*** moguimar has quit IRC10:53
*** adarazs is now known as adarazs_lunch11:01
panda|ruckykarel: it's removed from the promotion criteria11:01
panda|ruckykarel: it detects that that hash is older than the one currently promoted11:01
quiquell|roverpanda|ruck: How can we test this ? https://review.rdoproject.org/r/#/c/13520/111:04
quiquell|roverIt's a revert on the Force TCG now that we have the logs11:04
ykarelpanda|ruck, but the hash is from 22nd and current promoted is from 21st, no?11:05
panda|ruckquiquell|rover: there's no easy way to test those, we don't have a stagin area for our scripts11:06
quiquell|roverWe have to break CI and the unbreak it to have it ?11:07
quiquell|roverpanda|ruck, ykarel: do you know if the logs are still necessary for https://bugzilla.redhat.com/show_bug.cgi?id=1565179 ?11:08
openstackbugzilla.redhat.com bug 1565179 in qemu-kvm-rhev "qemu-kvm fails when run with -machine accel=kvm:tcg nested KVM" [Unspecified,New] - Assigned to pbonzini11:08
ykarelquiquell|rover, yes logs are required from Compute node11:08
panda|ruckykarel: I sede 5644 has timestamp 1524473301 which is Mon Apr 23 08:48:21 UTC 201811:09
quiquell|roverok11:09
ykarelquiquell|rover, looks like dmanchand forgot to add logs11:09
ykarelpanda|ruck, looks like some issue then, it's not promoting from yesterday itself11:10
ykarelso something is wrong with timestamp11:10
panda|ruckykarel: currently promoted has Mon Apr 23 10:37:42 UTC 201811:11
panda|ruckykarel: f69480243f6475757f72af1fb1eecac9a1327c4d_035e37fd11:12
quiquell|roverykarel: But I see they have already the .__repo_setup.sh.log so we are good11:13
panda|ruckykarel: you're looking into it, or I can start to poke around ?11:15
ykarelpanda|ruck, f694 is from 2111:15
panda|ruckykarel: this is what the timestamp says11:15
panda|ruckis tehre any way to reset the timestamp ?11:16
panda|ruckmaybe a manual promotion11:16
ykarelpanda|ruck, no idea11:17
ykarelquiquell|rover, those 2 were different issues11:20
quiquell|roverykarel: Ups, ok11:25
*** ykarel is now known as ykarel|mtg11:34
*** zoli|lunch is now known as zoli11:35
*** lucasagomes is now known as lucas-hungry11:41
*** moguimar has joined #oooq11:46
*** jaosorior has joined #oooq11:48
*** adarazs_lunch is now known as adarazs11:57
*** quiquell|rover is now known as quiquell|rover|l11:57
*** quiquell|rover|l is now known as quique|rover|lch11:57
*** jaosorior has quit IRC12:15
*** jaosorior has joined #oooq12:16
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.12:20
*** amoralej is now known as amoralej|lunch12:22
*** quique|rover|lch is now known as quiquell|rover12:25
*** lucas-hungry is now known as lucasagomes12:29
*** jaganathan has quit IRC12:34
*** rlandy has joined #oooq12:35
*** trown|outtypewww is now known as trown12:36
*** panda|ruck is now known as panda|ruck|lunch12:37
rlandyquiquell|rover: are you all set with the stack deletion on ci-rhos now?12:38
*** Goneri has joined #oooq12:39
quiquell|roverrlandy: Yep12:40
quiquell|roverrlandy: I have try to run the tqe rhos-12 without gerrit but with no luck12:41
quiquell|roverDo I need to review a change to run it ?12:41
rlandyquiquell|rover: what do you mean 'without gerrit'?12:42
rlandyjust kick a run?12:42
rlandysure you can - through jenkins12:43
*** ykarel|mtg is now known as ykarel12:45
quiquell|rover"Build now" fails at gerrit12:45
quiquell|roverat git fetchin I mean12:45
quiquell|roverFailed to fetch from ssh://rdo-ci-downstream@review.openstack.org:29418/openstack/tripleo-quickstart-extras.git12:46
rlandyyou need to set up another POC job that does not use gerit12:46
* rlandy looks if I have one12:46
quiquell|roverI can do a new one copy this and removing gerrit12:48
rlandyquiquell|rover: pretty much ... you can recreate or hack up and use: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/poc-rlandy-master-ovb-multiple-nics/configure12:49
quiquell|roverrlandy: Cool thanks will try12:50
rlandyI don't need that jobs anymore12:50
myoungquiquell|rover, panda|ruck|lunch, thx for updating status12:53
rlandytrown; are your overcloud deployments succeeding? mine are failing with "" "Error: Execution of '/bin/yum -d 0 -e 0 -y install ntp' returned 1: Error downloading packages:", when I use the supersede change over the append change12:53
trownrlandy: not sure my env is a good litmus at this point... I have just been deploying over snapshots trying to figure out this fs037 thing12:54
trownrlandy: and that deploy fails still12:55
rlandytrown: ok - np - I'll go back to append and see if it complete12:55
rlandyI need to run through a few more test scenarios with reproducer anyways12:55
trownk, I was going to spend the morning on fs037 because I think I am close to figuring it out, but then I will update my snapshot patch to add in the reproducer script integration12:57
myoungtrown: I signed up to help QE https://trello.com/c/3aSQKPt6/710-add-snapshotting-capabilites-to-the-workflow, do you have a min this morning to quickly sync?12:57
trownmyoung: might be easier to QE after the reproducer script integration part is up in the review12:57
quiquell|roverrlandy: Maybe we can add a checking script for rhos-dev-ci that runs on jenkins and disabled it if it's full12:57
myoungtrown: ack, I'll work on something else for now12:58
quiquell|roverDo the cleanup and then re-enabled it12:58
rlandyquiquell|rover: I'd rather run a cron that cleans up old jobs12:58
rlandyreally, mostly they are cleaned up12:58
rlandyif you experiment there, manual clean up is required12:59
*** atoth has joined #oooq13:20
rlandytrown: btw ... https://review.openstack.org/#/c/563763/ getting rid of cloud-init did reduce the reboot time.13:40
*** amoralej|lunch is now known as amoralej13:40
*** atoth has quit IRC13:43
*** panda|ruck|lunch is now known as panda|ruck13:43
*** skramaja has quit IRC13:50
*** ratailor has quit IRC13:50
ykarelpanda|ruck, any update for 5466f249bd36900a1dac573cdc83e7a11493aea2_0c8f7f95, manual promotion or skipping it?13:52
panda|ruckykarel: manual promotion may be the only option, changing timestamps requires changes to the database, not sure what happened with the promotio order13:54
ykarelpanda|ruck, manual promotion would be ok then, please do13:55
*** jaosorior has quit IRC13:55
*** jaosorior has joined #oooq13:55
*** atoth has joined #oooq13:58
*** sdoran has quit IRC13:58
*** sdoran has joined #oooq13:58
rlandypanda|ruck: where are we running fs003?14:02
rlandyI am testing the reproducer14:02
rlandyand can do a run with a singlenode job14:02
panda|ruckrlandy: upstream multinode14:03
panda|ruckrlandy: well, singlenode14:03
rlandypanda|ruck: so iiuc, the requirement would be to keep the nodes_file the same14:03
rlandyand just switch the fs?14:03
panda|ruckrlandy: yes, if we don't want to parametrize the number of nodes, we'll keep the overhead14:04
* rlandy looks at job14:04
rlandylet's analyze this14:04
*** atoth has quit IRC14:04
*** jaosorior has quit IRC14:07
rlandypanda|ruck: ok - so I don't see it (fs003) explicitly listed - I see scenario003 and  tripleo-ci-centos-7-3nodes-multinodeâ—† tripleo-ci-centos-7-nonha-multinode-oooq14:08
rlandywhich one am I reproducing here14:09
panda|ruckrlandy: where are you looking ?14:09
rlandyhttp://cistatus.tripleo.org/14:09
rlandyI see undercloud jobs14:09
rlandyâ—† tripleo-ci-centos-7-undercloud-containers14:09
rlandyâ—† tripleo-ci-centos-7-undercloud-oooq14:09
rlandyfs02714:10
rlandywhat am I missing here?14:10
panda|ruckundercloud-oooq should be fs00314:10
rlandyoh I see14:11
rlandyhttp://logs.openstack.org/06/563806/1/gate/tripleo-ci-centos-7-undercloud-oooq/c2fedd3/job-output.txt.gz14:11
rlandyno so explicit from the name14:11
rlandygot it now14:11
quiquell|roverykarel: Can we close this ? https://bugs.launchpad.net/tripleo/+bug/176619514:11
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Fix committed] - Assigned to Quique Llorente (quiquell)14:11
*** moguimar has quit IRC14:14
myoungo/ CI community meeting starts soon, immediately following #tripleo meeting (in progress) ==> https://bluejeans.com/7050859455, https://etherpad.openstack.org/p/tripleo-ci-squad-meeting (agenda)14:15
*** atoth has joined #oooq14:16
*** atoth has quit IRC14:16
*** atoth has joined #oooq14:17
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.14:20
ykarelquiquell|rover, should wait for the next periodic run14:23
quiquell|roverOk14:23
ykarelalso i noticed fs20 in pike is failing randomly with similar failures related to nova-compute, so it would be good to track and fix it, is there already a bug for it14:24
ykarelquiquell|rover, panda|ruck ^^14:24
panda|ruckarxcruz: ping14:26
arxcruzpanda|ruck: pong14:27
panda|ruckarxcruz: let's talk about featureset02114:27
arxcruzpanda|ruck: sure14:27
*** zoli is now known as zoli|brb14:28
sshnaidmpanda|ruck, I don't think 021 is relevant anymore, but we need some jobs to gate python-tempestconf and tempest14:28
panda|ruckarxcruz: what does it do , and can we remove it ?14:28
sshnaidmarxcruz, chandankumar ^^14:28
ykarelahh so same failure is in fs002 pike so it's a real issue14:28
arxcruzpanda|ruck: here on irc, or bj or what ?14:28
chandankumarsshnaidm: we will be replacing with containerized uc job there, but we can keep till we get the job up14:29
panda|ruckarxcruz: here on IRC, still in trpleo meeting, but we can extend in the community meeting14:29
arxcruzpanda|ruck: up to you14:30
panda|ruckchandankumar: isn't there any working job that is currently working ?14:30
chandankumarpanda|ruck: apart from f21, there is no other job against tempest14:31
chandankumaron tempestconf there is a tripleo-ci-centos-7-containers-multinode job there14:32
panda|ruckchandankumar: not other OVB job is running tempest consistently ?14:32
panda|ruckchandankumar: so we can use that one instead ?14:33
chandankumarpanda|ruck: yup, we can use that14:33
chandankumartripleo-ci-centos-7-containers-multinode as third party due to time limition against tempest14:33
sshnaidmpanda|ruck, why not to convert 021 to be containers-based?14:34
chandankumarsshnaidm: panda|ruck this one is coming https://review.openstack.org/#/c/563879/ f060 is coming for the same14:34
sshnaidmchandankumar, neither tempest nor tempestconf have branches, right?14:35
chandankumarsshnaidm: correct14:35
sshnaidmchandankumar, does it make sense to run there branchful jobs like fs001 with ocata, pike, etc..?14:36
*** quiquell|rover is now known as quiquell|off14:37
panda|rucksshnaidm: chandankumar: can we talk in the communitiy meeteing ?14:37
sshnaidmpanda|ruck, is it now?14:37
panda|rucksshnaidm: yes, it's rgith after tripleo meeting , whenever it ends14:38
sshnaidmpanda|ruck, I think it's ended14:38
panda|rucksshnaidm: yes it is14:38
chandankumarsshnaidm: but it is not going to gates the tempest changes na? As it is picking the stuff from tempest rpm, for each release we have pinned stuff to a known version in rdo.info14:38
chandankumarfor tempest and tempestconf14:39
panda|ruckmyoung: https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build/838/14:39
sshnaidmchandankumar, yeah, only for checking if nothing is broken in previous releases..14:39
sshnaidmchandankumar, I'm not sure something could be broken, just a thought14:40
sshnaidmchandankumar, because we have branchful jobs in tripleo-ci and oooq for example, just to check that we are ok with previous branches14:41
chandankumarsshnaidm: it would be useful when we bump a tag for tempest for a particular release14:41
chandankumarsshnaidm: let's try it, how it goes, if things will not work, we can revert it14:41
*** links has quit IRC14:42
chandankumarpanda|ruck: meeting link14:42
adarazspanda|ruck: are you going to be able to join myoung and me for the hubbot setup walkthrough?14:42
arxcruzhttp://cistatus.tripleo.org/promotion/14:43
arxcruzpanda|ruck: myoung ^14:43
panda|rucksshnaidm: chandankumar https://bluejeans.com/705085945514:43
myoungarxcruz: ya i know we have sova, but again my point was it's not *focused* on a particular promotion attempt (for a given hash, at a given phase).  e.g. , if you're a developer looking for "what's blocking promotion" - you have to know what you're looking at, and cross-ref with the textual promotion logs14:47
myoung^^ just mean there's an opportunity for better / groovier dashboards14:47
*** ajo has quit IRC14:50
*** ajo has joined #oooq14:51
*** yolanda_ has joined #oooq14:55
adarazspanda|ruck: another question: is there a way to deploy only a specific server with infra-setup besides deleting the others from ci-config/ci-scripts/infra-setup/tenant_vars/tripleo-infra/servers.yml14:55
*** moguimar has joined #oooq14:57
*** yolanda has quit IRC14:58
panda|ruckadarazs: no .. I thought it needed to be idempotent, install everything, delete none14:59
panda|ruckadarazs: omg, that's you in the meeting room with wes15:00
*** jaosorior has joined #oooq15:05
*** tesseract has quit IRC15:06
*** tesseract has joined #oooq15:06
panda|ruckmyoung: etherpadman15:09
panda|ruckbehold! the etherpadman!15:09
panda|ruckyou get a etherpad, you get a etherpad!15:09
myounghttps://etherpad.openstack.org/p/tripleo-ci-hubbot-configuration15:09
myoung:)15:10
* myoung is just trying to get his TCness on15:10
rlandypanda|ruck: supersede is not working for my testing - I am putting in a new patch to run again with append15:10
panda|ruck...15:10
panda|ruckwhat's in the resolv.conf after the dhcp finishes ?15:11
rlandyjust 8.8.8.815:11
rlandyneeds the original entry15:11
rlandywhich refers to the virthost15:11
rlandyI am testing fs00315:12
panda|ruckwhy does it need the original entry ?15:12
panda|ruckrlandy: in a meeting now, can we chat a bit more later on this ? there's still something I don't get sorry15:13
*** zoli|brb is now known as zoli15:19
*** ccamacho1 has joined #oooq15:21
*** ccamacho has quit IRC15:21
ykarelpanda|ruck, queens phase1 promotion started for a new hash15:23
ykarel2018-04-24 15:19:35,555 6812 INFO     promoter Promoting the container images for dlrn hash fee47ee9af3d38834c33bec31694f4be6b0c4396 on queens to current-tripleo-rdo15:23
ykarelso we can skip the previous one15:23
panda|ruckykarel: yeah, sorry, I had a stream of meeting, and wanted to check first that nothing was running15:23
ykarelpanda|ruck, np15:24
ykarelpanda|ruck, i saw there are failures in pike jobs from last few days, please check those15:25
ykarelonce you are free with other stuffs15:25
* ykarel leaving15:25
panda|ruckykarel: thanks, I'll look15:26
ykarelahh promotion jobs not started for master and queens at 12:10 UTC15:27
ykarelamoralej, is this because of that fs037 job?15:27
amoralejykarel, yes15:28
amoralejykarel, let me check i replied too fast15:28
amoralejykarel, yes, at that time previous run hadn't finished yet because of the fs037 job15:29
ykarelOkk, so we are clear for next run15:29
amoralejykarel, anyway we need https://review.openstack.org/#/c/563850/ for master15:30
amoralejykarel, did we get promotion in queens in last run?15:30
*** hubbot` has joined #oooq15:30
ykarelamoralej, need to check15:30
amoralejyeah, we did15:30
amoralejhttp://dashboards.rdoproject.org/rdo-dev15:30
adarazs%gatestatus15:32
hubbot`adarazs: Missing value for config sshCommand, changeID, changeURL. Use 'config plugins.GateStatus.<config_name> <value>' to setup this plugin. Check 'config help plugins.GateStatus.<config_name>' for detailed help and 'config list plugins.GateStatus' for all available variables.15:32
ykarelamoralej, for fee47ee9af3d38834c33bec31694f4be6b0c4396, promoter is currently running15:35
*** moguimar has quit IRC15:35
ykarelthis hash is from 23rd15:35
ykareli rechecked the phase1 scenario004 after poi patch merged15:36
ykarelamoralej, so we need kolla patch and for that have to clear current FTBFS: https://review.rdoproject.org/r/#/q/topic:rdo-FTBFS+status:open15:38
amoralejykarel, yes15:40
ykarelit would be good if we are consistent before next run15:40
ykarelamoralej, can you clear that15:41
adarazs%gatestatus15:41
hubbot`FAILING CHECK JOBS on stable/queens: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/560452, master: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/56044515:41
amoraleji'm checking the nova one15:41
ykarelokk15:41
*** ykarel is now known as ykarel|away15:42
*** hubbot` has quit IRC15:44
*** hubbot` has joined #oooq15:44
*** hubbot` has quit IRC15:46
*** hubbot` has joined #oooq15:47
*** kopecmartin has quit IRC15:48
panda|ruckrlandy: you have time for a small chat ?15:49
*** jfrancoa1 has quit IRC15:49
rlandypanda|ruck: sec - juts logging results from fs003 run15:49
rlandyjust15:49
panda|ruckadarazs: Ill ping you tomorrow morning with some questions on the review15:52
myoungrasca: you still around/avail?15:52
*** bogdando has quit IRC15:53
rascamyoung, for short, but yes!15:53
myoungsame..come back, weshay and I are here15:53
myounghttps://bluejeans.com/705085945515:53
*** atoth has quit IRC15:53
*** ykarel|away has quit IRC15:54
*** ccamacho1 has quit IRC15:55
*** atoth has joined #oooq15:55
*** moguimar has joined #oooq15:55
*** jbadiapa has quit IRC15:55
rlandypanda|ruck: k - I have a passing test with using append15:58
rlandyto confirm., I am going to try supersede with the same fs and hash15:59
*** zoli is now known as zoli|gone16:03
*** zoli|gone is now known as zoli16:03
panda|ruckrlandy: you want to wait for those results before chatting ?16:05
*** lucasagomes is now known as lucas-afk16:06
rlandypanda|ruck: yes - this will be a direct comparison16:09
panda|ruck%gatecheck16:10
hubbot`panda|ruck: Error: "gatecheck" is not a valid command.16:10
*** gkadam has quit IRC16:18
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.16:21
*** marios has quit IRC16:31
*** trown is now known as trown|lunch16:45
*** ccamacho has joined #oooq16:49
*** moguimar has quit IRC17:01
*** atoth has quit IRC17:02
*** panda|ruck is now known as panda|ruck|off17:04
rlandyhmmm ... scratch that17:09
rlandyseems ok17:09
*** jtomasek has quit IRC17:10
*** udesale has joined #oooq17:15
*** atoth has joined #oooq17:17
*** dtantsur is now known as dtantsur|afk17:17
*** jaosorior has quit IRC17:24
rlandytrown|lunch: we're good on singlenode fs00317:38
*** holser__ has quit IRC17:40
*** agopi|zzz has quit IRC17:41
*** agopi|zzz has joined #oooq17:41
*** trown|lunch is now known as trown17:55
trownrlandy: nice. I got fs037 working this morning17:55
trownit is iptables... I am making a patch for nodepool setup role17:56
*** udesale has quit IRC17:57
*** atoth has quit IRC18:08
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.18:21
*** amoralej is now known as amoralej|off18:24
rlandytrown: good to hear18:26
rlandytrown: wrt testing cards, should we move those to 'In progress'?18:28
rlandyfs010 has been tested a bunch18:28
rlandyyou're on fs03718:29
rlandyand I'm on fs00318:29
trownrlandy: ya, I think maybe we should just leave them and hope somebody besides us tests them18:29
* trown is not really sure18:29
rlandyI'm fine with that until Thursday18:29
rlandyif nobody claims them by then18:29
rlandywe can move18:29
trownI am only on fs037 because I was using it to test my snapshotting, and in the process figured out the issue that plagued me all last sprint18:30
trownI guess one of the other services (not present in keystone-only) properly sets up iptables as does proper nodepool18:31
*** sshnaidm has quit IRC18:32
trownrlandy: myoung would you prefer I make a new etherpad for snapshot? or just add to the bottom of what we have going for the rest?18:33
rlandytrown: whatever works for you18:34
rlandyI am ready to test when you have the instructions available18:34
rlandyI am just updating the last of my reviews18:34
trownok, I can put up some quick instructions, updated WIP patch just now18:34
trownI will make a new etherpad, but link from the old one and the card18:35
rlandytrown: cool - will take a look while my last reproducer check is running18:42
trownrlandy: ok put up https://etherpad.openstack.org/p/tqe-libvirt-snapshot please dont hesitate to reach out if something doesnt work18:56
rlandytrown: k - looking - also I updated the doc for the libvirt option - https://review.openstack.org/#/c/563319/ - let me know if you think something is missing18:57
rlandytrown: if the snapshot option should be available in the reproducer, we will have to doc/add that18:58
trownrlandy: ya, I plan to add it on top of your patch19:01
trownrlandy: ya your patch looks good, just waiting for CI to produce one I can try out19:02
rlandysure19:02
rlandytrown: myoung: panda|ruck|off: weshay: just fyi - will be out tomorrow morning - have to attend a funeral of a dear friend19:03
trownrlandy: :( sorry to hear that19:04
trownrlandy: I have been through that, though thankfully not recently, maybe you should take the whole day?19:04
rlandytrown: thanks - idk - will see how it goes. the poor girl had a double lung transplant that did not succeed - so I am sure her family will have lots of support upfront that day. I'll probably go spend more time there later.19:06
trownhttps://review.openstack.org/564050 is the patch to make fs037 work btw, testing it now on a fresh environment19:06
*** agopi|zzz has quit IRC19:07
trownrlandy: cool, we will be ok here in any case :), do what you need to do19:07
*** yolanda__ has joined #oooq19:15
*** sshnaidm has joined #oooq19:16
*** yolanda_ has quit IRC19:18
myoungtrown, rlandy: I'm back now, giving the etherpads a look.  re: etherpads I'm fine with whatever makes sense.19:23
trownmyoung: k, opted to keep it seperate since it is optional, and really could be performed anywhere in the workflow19:24
myoungrlandy: sorry to hear about your news, please prioritize self :(19:24
*** tesseract has quit IRC19:26
*** agopi|zzz has joined #oooq19:31
rlandymyoung: thanks19:44
*** rasca has quit IRC20:04
*** weshay has quit IRC20:05
*** weshay has joined #oooq20:07
*** rasca has joined #oooq20:07
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.20:21
*** agopi|zzz has quit IRC20:22
*** agopi|zzz has joined #oooq20:32
rlandyugh - not sure why just sometimes vms are unreachable20:35
*** hubbot` has quit IRC20:36
*** dmellado has quit IRC20:38
*** dmellado has joined #oooq20:43
rlandytrown: +100 on having libvirt_volume_path be a variable20:48
rlandyI have been making that correct to my scripts all along20:48
trownrlandy: ya it is just copied from other role, but I put it that way in the etherpad for you :)20:49
rlandymuch appreciated :)20:49
trownmy nodepool-setup patch for fs037 works \o/20:50
rlandyneed to update the original work to allow that - will put up a patch20:50
rlandywoohoo20:50
rlandywe're three for three on the fs!!!20:50
rlandysnapshot under test now20:50
*** sshnaidm is now known as sshnaidm|off20:52
rlandytrown: is it a requirement for libvirt workflow to use .quickstart as the workspace?20:52
rlandyin keeping with the reproducer, I have ignored that20:52
rlandyusing the workspace the reproducer creates - which wfm20:52
trownno it should work either way20:53
trowni actually need to figure out what in the reproducer script is broken with that option... I always have to manually edit it20:54
*** hubbot` has joined #oooq20:54
rlandyhmmm ... I have no snapshots listed20:56
rlandyoh I do20:56
trownya they wont show up in virsh20:56
rlandyyeah - I see them in dir20:57
rlandy-rw-r--r--. 1 qemu qemu     6094848 Apr 24 16:55 subnode-0.qcow220:57
rlandy-rw-------. 1 qemu qemu 10757865472 Apr 24 16:51 subnode-0.qcow2.bak20:57
rlandy-rw-r--r--. 1 qemu qemu     5963776 Apr 24 16:56 subnode-1.qcow220:57
rlandy-rw-------. 1 qemu qemu 10773463040 Apr 24 16:51 subnode-1.qcow2.bak20:57
rlandyta da!!20:57
rlandyhello snapshots20:57
trownand if you do "qemu-img info subnode-0.qcow2" it should show that the .bak is a backing file for it20:57
rlandybacking file: /home/temp/images/subnode-0.qcow2.bak20:58
rlandyindeed it does20:58
rlandyok - that part works20:58
rlandylet's retore20:58
rlandyrestore20:58
* myoung is behind you two...will be onto second etherpad (snapshots) in a bit20:59
rlandymyoung: I have my reproducer already edited so it's quicker for me to get to the start point20:59
* myoung nods21:00
rlandyok - let's deploy and then restore21:00
trownhave a good night folks21:02
*** trown is now known as trown|outtypewww21:02
rlandymyoung: cleaning up tenant - will be back on this in a bit21:08
myoungtrown|outtypewww: cya21:08
*** Goneri has quit IRC21:22
*** tcw has quit IRC21:23
*** tcw has joined #oooq21:24
*** jbadiapa has joined #oooq21:38
*** agopi|zzz has quit IRC21:52
myoungrlandy, trown|outtypewww: iterating and guessing I'm hitting what yall hit last week.  tracking issues in the etherpad.21:55
*** agopi|zzz has joined #oooq21:57
rlandymyoung: worked for me22:17
rlandywhat issues are you hitting?22:17
rlandywith the snapshots?22:17
rlandyyou may need some of the patches I put out22:18
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.22:21
*** dtrainor has quit IRC22:41
myoungrlandy: just getting the POC stuff working22:48
myoungrlandy: mostly things like packages being present, stack user created, etc.23:05
*** strattao has quit IRC23:16
*** rlandy has quit IRC23:25
*** tosky has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!