Monday, 2018-05-07

hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429100:47
*** atoth has quit IRC01:38
*** links has joined #oooq02:17
*** ratailor has joined #oooq02:39
*** EvilienM is now known as EmilienM02:45
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429102:47
*** jaosorior has joined #oooq04:05
*** udesale has joined #oooq04:11
*** sshnaidm has joined #oooq04:17
*** pgadiya has joined #oooq04:25
*** pgadiya has quit IRC04:28
*** ratailor_ has joined #oooq04:28
*** saneax-_-|AFK is now known as saneax04:29
*** ratailor has quit IRC04:31
*** ykarel has joined #oooq04:35
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429104:47
*** sshnaidm has quit IRC05:14
*** jtomasek has joined #oooq05:24
*** ratailor__ has joined #oooq05:37
*** ratailor_ has quit IRC05:39
*** marios has joined #oooq05:46
*** quiquell|off is now known as quiquell05:54
*** jbadiapa has joined #oooq05:56
*** jtomasek has quit IRC05:56
*** ratailor_ has joined #oooq06:29
*** ratailor__ has quit IRC06:32
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429106:47
*** kopecmartin has joined #oooq06:50
*** quiquell is now known as quiquell|afk06:53
*** jfrancoa has joined #oooq06:53
*** bogdando has joined #oooq06:54
*** jfrancoa has quit IRC06:58
*** jfrancoa1 has joined #oooq06:58
*** jaganathan has joined #oooq07:03
*** jfrancoa has joined #oooq07:09
*** florianf has joined #oooq07:11
*** jtomasek has joined #oooq07:12
*** holser__ has joined #oooq07:12
*** ykarel is now known as ykarel|lunch07:18
*** jtomasek has quit IRC07:20
*** saneax is now known as saneax-_-|AFK07:24
*** jtomasek has joined #oooq07:26
*** agopi has quit IRC07:51
*** udesale_ has joined #oooq07:55
*** tosky has joined #oooq07:57
*** udesale has quit IRC07:58
*** skramaja has joined #oooq08:03
*** ykarel|lunch is now known as ykarel08:10
*** skramaja has quit IRC08:11
*** skramaja_ has joined #oooq08:11
*** skramaja_ is now known as skramaja08:23
*** quiquell|afk is now known as quiquell08:38
chandankumararxcruz: kopecmartin tosky weshay https://review.openstack.org/#/q/topic:tempest_log+(status:open+OR+status:merged)08:42
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429108:47
*** brault has joined #oooq09:08
*** saneax-_-|AFK is now known as saneax09:09
*** dtantsur|afk is now known as dtantsur09:11
*** jaosorior has quit IRC09:17
*** jaosorior has joined #oooq09:19
*** sshnaidm has joined #oooq09:35
*** sshnaidm is now known as sshnaidm|rover09:35
*** skramaja has quit IRC09:43
*** skramaja_ has joined #oooq09:43
*** dtantsur is now known as dtantsur|brb09:48
*** matbu has joined #oooq10:17
*** ratailor__ has joined #oooq10:29
*** ratailor_ has quit IRC10:30
*** yolanda_ is now known as yolanda10:35
*** skramaja has joined #oooq10:35
*** skramaja_ has quit IRC10:36
*** amoralej is now known as amoralej|brb10:45
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429110:47
*** skramaja has quit IRC10:52
*** skramaja_ has joined #oooq10:52
*** pgadiya has joined #oooq11:00
*** pgadiya has quit IRC11:00
*** dtantsur|brb is now known as dtantsur11:03
*** skramaja has joined #oooq11:08
*** skramaja_ has quit IRC11:09
quiquellHi,11:10
quiquellI am cleaning up a little before CI Sprint 1311:11
quiquellLooks like some featureset related to upgrades are no longer used11:11
quiquellhttps://review.openstack.org/#/c/566546/11:11
quiquellIs this ok ?11:11
*** matbu has quit IRC11:11
quiquellfs012, fs013, fs014, sf01511:11
sshnaidm|roverquiquell, I don't think it11:13
sshnaidm|rover's worth11:13
sshnaidm|roverquiquell, to remove featuresets I mean11:13
sshnaidm|roverquiquell, we didn't remove anything yet, need to check if it doesn't create problems11:15
sshnaidm|roverquiquell, and if you remove, need to remove too from: https://github.com/openstack/tripleo-quickstart/blob/e8938e090f3130f8470d92f7caceaa2e79413612/doc/source/feature-configuration.rst11:16
chandankumararxcruz: kopecmartin I will be back in another 2 hours I have to do some courier stuff11:20
kopecmartinchandankumar, ack11:22
*** atoth has joined #oooq11:29
*** skramaja has quit IRC11:33
*** skramaja has joined #oooq11:34
*** tcw has joined #oooq11:49
*** jaosorior has quit IRC11:49
*** udesale__ has joined #oooq11:55
*** tcw has quit IRC11:57
*** udesale_ has quit IRC11:57
sshnaidm|roverykarel, hi11:59
ykarelsshnaidm|rover, hi11:59
sshnaidm|roverykarel, re https://bugs.launchpad.net/tripleo/+bug/1767049 - should gabbi 1.42 be in queens repo too11:59
openstackLaunchpad bug 1767049 in tripleo "Error during test discovery : 'must specify exactly one of host or intercept'" [High,In progress] - Assigned to Quique Llorente (quiquell)11:59
sshnaidm|rover?11:59
ykarelsshnaidm|rover, we build it for master only, not sure if it really needed for queens, current heat-tempest-plugin for queens neend gabbi 1.3312:01
ykarelsshnaidm|rover, if we move the pin for head-tempest-plugin for queens then we would need gabbi 1.4212:01
ykarelchandankumar, would that be done ^^12:02
sshnaidm|roverykarel, ok, so we can revert pins for master for now?12:02
ykarelsshnaidm|rover, master pins are reverted already12:02
ykareland they are working12:02
ykarelok bug not updated yet, i updated the trello card but forgot to update the bug12:03
sshnaidm|roverykarel, ack12:03
weshaymorning12:04
weshayrasca, howdy sir12:04
*** trown|outtypewww is now known as trown12:12
sshnaidm|rovermyoung|ruck, hi, do you have the ruck etherpad?12:22
quiquellsshnaidm|rover: Removing featuresets reduce the complexity of doing refactors12:30
quiquellIt helps with the sprint 1312:30
*** jfrancoa1 has joined #oooq12:30
*** tcw has joined #oooq12:30
*** jfrancoa has quit IRC12:30
*** rlandy has joined #oooq12:35
*** amoralej|brb is now known as amoralej12:35
*** ratailor__ has quit IRC12:36
*** skramaja_ has joined #oooq12:40
*** skramaja has quit IRC12:40
myoung|rucksshnaidm|rover: o/12:42
myoung|rucksshnaidm|rover: good morning.  etherpad is https://review.rdoproject.org/etherpad/p/ruckrover-sprint13, I'm in CIX meeting now, will be updating it this morning.  it's a little messy atm12:42
quiquellmyoung|ruck: Good morning, do old ruck/rover need to send you and e-mail with the stuff we did ?12:43
myoung|rucksshnaidm|rover: has misc details from the failures we were seeing friday from rdo cloud12:43
myoung|rucksshnaidm|rover: planning to look at rdo2 as a focus over next week12:43
myoung|ruckquiquell, panda|off: that would be helpful as time allows.  I would like to have the sprint 12 summary drafted by end of (NA) working day12:43
myoung|ruckquiquell: I can also parse your etherpad however if you are nose down on sprint tasks12:44
quiquellmyoung|ruck: Let me humanize the etherpad on an e-mail with panda as CC12:44
myoung|ruckquiquell: \o/12:46
*** skramaja has joined #oooq12:47
*** skramaja_ has quit IRC12:47
panda|offmyoung|ruck: quiquell I have 10 minutes before the next meeting12:47
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/56044512:47
myoung|ruckpanda|off, quiquell: I know time is precious, emails's fine12:48
quiquellpanda|off: Going to put a summary at the top of https://review.rdoproject.org/etherpad/p/ruckrover-sprint1212:48
quiquellThen you can add whatever is missing12:49
quiquelland myoung|ruck Can read it and send the summary12:49
myoung|ruckrasca: I actually have a 9-9:30, can chat after that12:50
myoung|ruckpanda|off: I'm in CIX now --> 912:50
*** jaosorior has joined #oooq12:54
rascamyoung|ruck, no worries, ping me when you're avail12:56
rlandyquiquell: ping re: https://wiki.openstack.org/wiki/Tripleo-upgrades-fs-variables12:56
rlandyundercloud_upgrade n -> n + 312:57
quiquellrlandy: Thanks, changed.12:58
quiquellrlandy: There is a lot of redudancy mixed_upgrade and overcloud_upgrade are the same12:58
rlandyquiquell: lol - I was just about to point that out wrt mixed upgrade12:59
quiquellThe mean the same but are not the same12:59
quiquellTented to refactor them and replaced mixed_upgrades for overcloud_upgrade12:59
rlandywe need to differentiate those12:59
rlandyquiquell: there is a lot of old stuff out there13:01
rlandywe will have to decide how much we are supporting - obviously, we can't break anything that is running13:01
*** tcw1 has joined #oooq13:02
rlandyquiquell: lastly, is there anything like containerize_undercloud_upgrade ?13:02
rlandyoh meeting13:03
*** jtomasek_ has joined #oooq13:04
*** links has quit IRC13:05
*** tcw has quit IRC13:11
*** jtomasek has quit IRC13:11
*** quiquell has quit IRC13:11
*** Goneri has joined #oooq13:16
*** quiquell has joined #oooq13:18
quiquellrlandy: Have verify some stuff, the doc is wrong, really a WIP.13:23
*** skramaja has quit IRC13:26
sshnaidm|roverrlandy, trown if we are talking about reviews, what's about namservers patch? https://review.openstack.org/#/c/565839/  ignoring it is the same as -2 actually :)13:43
rlandysshnaidm|rover: we didn't ignore it - we just haven't come to an agreement13:46
chandankumarykarel: if we pin to a known commit having gabbi 1.42 then we requires new gabbi13:46
sshnaidm|roverrlandy, ok, so what are points we need to discuss?13:46
ykarelchandankumar, any plan to move the pin for heat-tempest-plugin?13:46
trownsshnaidm|rover: not giving +2 (+0) is clearly not the same as giving -2 :P13:46
rlandysshnaidm|rover: I think trown, myoung|ruck  and I agree with this comment ""this requires an ability to set the option in the reproducer script, if there is not a default that works for everyone.13:47
sshnaidm|rovertrown, right, but for that patch it's the same, as it was problem in your and rlandy environment :P13:47
trownsshnaidm|rover: I have not been convinced to give it +2, but I wont keep you from convincing someone else13:48
sshnaidm|roverrlandy, trown do you still have this environment that required a special dns?13:48
chandankumarykarel: https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L658813:48
sshnaidm|roverrlandy, trown can I/we try and run there with that patch?13:48
rlandysshnaidm|rover: trown; I suggested adding the option to the reproducer earlier but it was ditched13:48
chandankumarykarel: https://github.com/openstack/heat-tempest-plugin/commit/07a6bd4a5b8dc2e94d2b7f4fb74b472b3a4f562c13:49
chandankumarykarel: it is already pinned13:49
chandankumarwhich does not need gabbi-1.4213:49
rlandysshnaidm|rover: yes - we still have that env13:49
ykarelyes it's pinned, i want to know is there any plan to move this pin forward?13:49
ykareljust next one needs 1.4213:49
ykarelnext commit13:49
sshnaidm|roverrlandy, I think it's not normal use case to have not-working dns, and that's why I'm against to add additional options for workaround13:50
chandankumarykarel: if gabbi 1.42 us available then we can move to latest commit13:50
sshnaidm|roverrlandy, can I get access and run it there with my patch?13:50
ykarelsshnaidm|rover, ^^13:50
sshnaidm|roverrlandy, I'm sure it was just temporary problem13:50
chandankumarykarel: no we cannot update the pinn https://github.com/redhat-openstack/rdoinfo/blob/master/deps.yml#L63013:50
rlandythe question is not whether the problem was temporaray or not13:51
chandankumarfor queens gabbi is still 1.33.13:51
sshnaidm|roverykarel, chandankumar queens deps doesn't have 1.42 so far13:51
rlandyit happened often enough for three of us to hit it13:51
chandankumarsshnaidm|rover: yup that is what i am saying13:51
sshnaidm|roverrlandy, if you have the same environment it's one case, not three13:52
rlandysshnaidm|rover: trown: myoung|ruck: I have one other suggestion ...13:52
chandankumarsshnaidm|rover: sending patch for prmoting gabbu13:52
rlandyhttps://review.openstack.org/#/c/566155/2/roles/create-reproducer-script/templates/README-reproducer-quickstart.html.j213:52
rlandysshnaidm|rover: myoung|ruck: trown: ^^13:52
rlandyif we remove the default 8.8.8.813:53
rlandydoc here where/how to define it13:53
rlandyI have already left a note about it13:53
rlandywe could tell people where to add that option13:54
sshnaidm|roverrlandy, yeah, totally agree13:54
rlandyas in: edit the reproducer to add a -e line13:54
rlandymyoung|ruck: trown: thoughts on that idea?13:54
myoung|ruckrlandy, trown: i have thoughts :) https://bugs.launchpad.net/tripleo/+bug/1769532 RFE (libvirt-reproducer): add mechanism allowing user to keep/pass params to script13:55
openstackLaunchpad bug 1769532 in tripleo "RFE (libvirt-reproducer): add mechanism allowing user to keep/pass params to script" [Medium,Triaged]13:55
rlandy< here we go with design discussion on 20% time >13:55
trownrlandy: sure13:56
* myoung|ruck trying to find a balance between "policy wonk" and "anarchist"13:56
* rlandy reading RFEs now13:56
* trown defaults to anarchist13:56
* rlandy runs for cover13:57
rlandymyoung|ruck: https://bugs.launchpad.net/tripleo/+bug/1769532 - this is an excellent idea13:57
openstackLaunchpad bug 1769532 in tripleo "RFE (libvirt-reproducer): add mechanism allowing user to keep/pass params to script" [Medium,Triaged]13:57
rlandytake out -d13:58
rlandyand add a extras parameters file13:58
rlandysshnaidm|rover: ^^ I think this is the correct way to go about it - is a general extras file14:00
ykarelThanks chandankumar14:01
rlandywe take out the -d option14:01
ykarelchandankumar++14:01
hubbotykarel: chandankumar's karma is now 314:01
rlandyand replace it with a general settings file14:01
rlandythen we have a clear way to offer people to add that14:02
rlandyI think if we add that ability to the reproducer and doc it, we will get approval14:02
* rlandy is confused if this qualifies as 20% work14:03
rlandythere is a some design/doc required14:03
sshnaidm|roverrlandy, yeah, another option to have it as "-e ... -e ..." as it's in ansible14:03
rlandysshnaidm|rover: yep and we change the doc to refer to that14:04
rlandythe reproducer as a whole could benefit from that14:04
myoung|ruckrlandy, sshnaidm|rover, trown: (personally) I like the idea of a yml I can keep that has my storage/dns settings, and is reusable across multiple invocations of the reproducer script14:05
myoung|ruckI absolutly have a non-standard storage setup, and care where Vm's land14:05
rlandymyoung|ruck: I agree - too many -e's is confusing14:05
myoung|ruck(err. care where the volume pool path points to)14:05
sshnaidm|roverrlandy, how about this? https://review.openstack.org/#/c/546932/14:05
sshnaidm|rovermyoung|ruck, rlandy in https://review.openstack.org/#/c/546932/ it's possible both yaml file and -e args14:06
rlandysshnaidm|rover: I would like to see that option on all playbook calls14:07
sshnaidm|roverrlandy, yep, just need to add it to all playbooks14:07
rlandyie; an options file you add to the reproducer that can contain settings for any playbook14:07
sshnaidm|roverrlandy, I think jfrancoa1 isn't against we pick up his patch14:07
rlandyotherwise it's a maintenance burden14:07
rlandysshnaidm|rover: ok - but we need clear doc on this14:07
sshnaidm|roverrlandy, this patch already contains option for file14:08
*** ykarel is now known as ykarel|away14:08
rlandysshnaidm|rover: have you tested that out?14:08
jfrancoa1rlandy: I did, and quiquell too. We used it mostly to pass the flavor (as sometimes rdo cloud doesn't have resources to use m1.large)14:10
rlandyjfrancoa1: we like your idea - but would like to expand on it14:11
jfrancoa1rlandy: sshnaidm|rover:  but it might be interesting to test it a little further (it's a week or so I don't use it)14:11
rlandygeneralize to any playbook14:11
jfrancoa1rlandy: feel free to use the patch or submit something base on it, I think it's an useful thing14:11
rlandykeep the -e in the playbook call14:11
myoung|rucksshnaidm|rover: thanks for including link to LP :)14:15
sshnaidm|rovermyoung|ruck, sure14:15
myoung|ruckrasca: apologies for delay, are you still available?14:26
rascamyoung|ruck, yes, for 20mins or so, let's do it14:26
myoung|rucksame, i have a hard stop in 2014:27
myoung|ruckbluejeans.com/matyoung14:27
myoung|ruckrasca: (or https://bluejeans.com/7050859455) if you are using the app14:27
rascamyoung|ruck, comin14:28
myoung|ruckweshay: do you have a hot sec? ^^14:29
*** strattao has quit IRC14:29
myoung|ruck(not required but we're paraphrasing your intent...we think we have it but if you're bored :))14:32
*** yolanda has quit IRC14:38
sshnaidm|rovermyoung|ruck, so, back to etherpards/trellos/lp - how do you track you ruckness? :)14:41
*** jfrancoa1 has quit IRC14:43
*** holser__ has quit IRC14:44
*** yolanda has joined #oooq14:45
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/56044514:47
*** quiquell is now known as quiquell|off14:50
myoung|rucksshnaidm|rover: TBH thus far there's been very little actual rucking going on, but https://review.rdoproject.org/etherpad/p/ruckrover-sprint13 is where am tracking for this sprint14:50
myoung|rucksshnaidm|rover: just got off BJ with rasca, getting queens promotion unstuck.  will update etherpad...but after tempest squad standup (in 9 min)14:51
myoung|rucksshnaidm|rover: queens rdo2 bare metal UC (https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/queens.ini#L43)14:52
*** udesale__ has quit IRC14:52
*** ykarel|away has quit IRC14:53
rascasshnaidm|rover, by the enf of today the new slave will be ready so we'll be good also on that side14:54
sshnaidm|roverrasca, thanks!14:54
myoung|ruckchandankumar: arxcruz, weshay, kopecmartin, i will be ~2 mins late to scrum14:57
*** florianf has quit IRC14:58
arxcruzmyoung|ruck: ack14:58
weshayI see that at 330utc14:59
weshayin 1/2 hour14:59
*** florianf has joined #oooq15:01
*** brault has quit IRC15:02
* myoung|ruck had wires crossed and blushes15:02
myoung|rucksshnaidm|rover: I'm tracking higher level ruck/rover for sprint 13 on the etherpad @24, so that at end of sprint we don't need to do a summary after fact.15:09
myoung|rucksshnaidm|rover: could you please give https://review.rdoproject.org/r/#/c/13604 (fix for https://bugs.launchpad.net/tripleo/+bug/1768090 promoter script is not comparing timestamps correctly when folding hashe)  a review?  It's blocking a leftover CIX issue from sprint 12 (https://trello.com/c/H7ZaoTJ4/573-cixlp1768090tripleociproa-promoter-script-is-not-confronting-timestamps-correctly-when-folding-hashes)15:14
openstackLaunchpad bug 1768090 in tripleo "promoter script is not comparing timestamps correctly when folding hashes" [High,Fix committed] - Assigned to Gabriele Cerami (gcerami)15:14
*** brault has joined #oooq15:16
sshnaidm|rovermyoung|ruck, will look15:16
myoung|ruckTHX15:17
myoung|ruckthx rather15:17
myoung|ruck(didn't mean to yell)15:18
weshaysshnaidm|rover, we have a little tech debt to take out of these undercloud vars15:20
*** brault has quit IRC15:20
weshaysshnaidm|rover, e.g. undercloud_undercloud_hostname15:20
weshayvs. undercloud_hostname15:20
sshnaidm|roverweshay, why do we want to take undercloud out..?15:21
weshayhttps://etherpad.openstack.org/p/weshay_notes15:21
weshaysshnaidm|rover, not sure what you mean15:21
weshayafaict.. most of what I see in https://review.openstack.org/#/c/566501/1/config/environments/containerized_undercloud.yml15:21
weshaycan be refactored back into default vars if we spend a little time with it15:22
sshnaidm|roverweshay, not sure I understand15:22
weshaysshnaidm|rover, dang15:23
weshayok.. let me explain on blue15:23
weshayhttps://bluejeans.com/u/whayutin/15:23
weshaysshnaidm|rover, https://review.openstack.org/#/c/566501/1/config/environments/containerized_undercloud.yml15:26
weshayhttps://etherpad.openstack.org/p/weshay_notes15:26
myoung|ruckdoes openstack have a link shortner (e.g. FOSS version of bit.ly)15:31
rlandypanda|off: ping re: https://trello.com/c/NMX5beYn/737-ensure-update-check-jobs-are-also-gating-voting15:32
weshaymyoung|ruck, no but I often see folks use https://hootsuite.com/pages/owly for obvious reasons15:36
myoung|ruckweshay: awesome thx15:41
myoung|ruckweshay++15:41
hubbotmyoung|ruck: weshay's karma is now 215:41
chandankumararxcruz++15:47
hubbotchandankumar: arxcruz's karma is now 115:47
panda|offrlandy: I'm off today, can you put definition and DOD with weshay  ?15:50
*** bogdando has quit IRC15:51
rlandypanda|off: ack - I just picked up that card15:51
weshay0/15:52
panda|offrlandy: if it hasnt' clear description and DOD is not really mature to be started.15:52
*** udesale has joined #oooq15:56
*** udesale has quit IRC15:58
chandankumarmyoung|ruck: weshay DFG OSP-13 GSS knowledge transfer presentations - please plan regarding this email on rhos-announce we also need to plan16:03
*** kopecmartin has quit IRC16:05
rascarlandy, do you have the guide for configuring the jenkins slave handy somewhere?16:07
rascaI remember a link we shared16:07
rascamyoung|ruck, I think I need to remove old SSH host key from rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com for pidone slave, can you do that?16:11
rascamyoung|ruck, error is http://pastebin.test.redhat.com/58705816:12
myoung|ruckrasca: afaik if you add http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/keys/beaker/rdoci-jenkins/id_rsa_rdoci-jenkins.pub to your new slave should work without having to add a new key to jenkins.  nose down on something atm...can look in a few if that doesn't work16:14
myoung|ruckrasca: we do have playbooks that do a pile of stuff not generally needed.16:14
rascamyoung|ruck, yeah but if you look at the pastebin since the IP is different it is supposing a spoof16:14
myoung|ruckother than a java sdk (headless) there's not much we need there on the slave16:14
* myoung|ruck looks16:15
myoung|ruckrasca: BJ, can do it quick now16:15
myoung|ruckif avail16:15
rascamyoung|ruck, sure16:15
rascamyoung|ruck, I'm in your channel16:16
myoung|ruckrasca: /me is INCOMING with adds (goblins or gremlins, not sure...perhaps an orc commander too...have crowd control ready!)16:17
weshaypanda|off, let's chat about https://bugs.launchpad.net/tripleo/+bug/1768997 tomorrow16:18
openstackLaunchpad bug 1768997 in tripleo "containerized-undercloud-upgrades: dlrn tasks don't run" [Critical,Triaged]16:18
weshay%gatestatus16:27
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/56044516:27
*** marios has quit IRC16:30
myoung|ruckrasca: whoop whoop!  https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/computer/pidone-private is back oinline16:33
*** dtantsur is now known as dtantsur|afk16:43
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/56044516:47
*** dsneddon has quit IRC16:51
*** trown is now known as trown|lunch16:52
*** dsneddon has joined #oooq16:53
rlandyrasca: hmm - that was a guide myoung|ruck wrot17:17
rlandywrote17:17
myoung|ruckrlandy: we got it sorted.  selinux was stomping on /home/rhos-ci/.ssh/authorized_keys and needed a restorecon17:18
myoung|ruckrlandy: the pidone slave for BMU jobs is now running off a virtualized slave VS BM17:18
weshaymyoung|ruck, made a small change (refactor) https://docs.google.com/document/d/1mxoxOuxhublB0UfANkGcbEHlY4sblyLswvBmij8Fqm0/edit?ts=5af086f417:19
weshayack to send17:19
weshaymyoung|ruck, reminder to print/format it as text17:19
myoung|ruckweshay: ack17:19
rlandyok17:25
*** jaosorior has quit IRC17:26
*** ccamacho has quit IRC17:27
*** jaosorior has joined #oooq17:28
weshaymyoung|ruck, send it :)17:30
weshayrlandy, trown|lunch for this sprint.. I have -1 just to get reviews etc. first17:31
weshayhttps://review.openstack.org/#/q/topic:gate_update+(status:open+OR+status:merged)17:31
rlandyweshay: I picked up https://trello.com/c/NMX5beYn/737-ensure-update-check-jobs-are-also-gating-voting17:32
weshayrlandy, ah thanks17:32
rlandywe should talk about what jobs need to go where17:32
rlandyI actually was going to ping you about that17:32
rlandywe need a DoD17:33
weshayk.. /me goes to the card17:33
rlandyweshay,:since you did the work already - you can remove my name17:33
rlandyunless there is more to do17:34
weshayrlandy, I'm not 100% I have it right yet17:34
rlandyweshay: well, that depends on DoD17:34
myoung|ruckweshay: fire in the hole!17:36
rlandyweshay: I added you to the card17:36
rlandyand out me in QE role17:37
weshayk17:38
weshayrlandy, how does the DOD look?17:38
weshayrelated to this card, but out of scope for it.. is I want to start thinking and checking what else we can run as a gate upstream17:38
rlandyusing the same dlrn_hash and containers??17:38
weshayhowever I suspect it's best to wait until we have the release fixed17:39
rlandywhat is meant by same dlrn hash17:39
*** tcw1 has quit IRC17:39
weshayrlandy, atm.. that job runs an "update" w/ the same dlrn hash and the same containers17:39
weshayit just redeploys the same containers that were originally deployed17:39
rlandyweshay: for my education ... the job starts with the base has before the change is applied17:40
rlandyand then updates the containers to include the change?17:40
rlandyafter the update?17:40
rlandyie: where does the change get incorporated?17:41
weshaybecause it's single release.. in the normal spot17:41
rlandyweshay: so - base release is installed, change is added, update is done17:42
weshayhttp://logs.openstack.org/53/562353/10/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/a20cc72/logs/delorean_logs/fc/b5/fcb539afb834be781f3a4ab88ecc5762c53b7b43_dev/17:42
weshayrlandy, the update.. is essentially a no-op for content... but it does walk through the workflow17:43
weshaymeaning.. you have the same rpms, same containers after deploy and update17:43
rlandyweshay: ok - that is what I am trying to workout17:43
rlandywhen we do a container update anyways17:44
weshayhttp://logs.openstack.org/53/562353/10/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/a20cc72/logs/undercloud/home/zuul/overcloud_update_prepare.log.txt.gz17:44
weshayhttp://logs.openstack.org/53/562353/10/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/a20cc72/logs/undercloud/home/zuul/overcloud_update_converge.log.txt.gz17:44
weshayhttp://logs.openstack.org/53/562353/10/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/a20cc72/logs/undercloud/home/zuul/overcloud_update_run-Controller.sh.txt.gz17:44
weshayrlandy, at least you should17:44
weshaynot sure if the patch to improve the logging ever merged :(17:44
weshaywhere we would have captured the rpm/containers versions after deploy, and then after upgrade/update to compare17:45
rlandyovercloud_update_prepare.log17:45
weshaywhaaa whaaa waaaaaaaaaaaaaaaaaaaaaaa17:45
weshaywe also still do not have time stamps17:45
weshaywhaaa whaaa waaaaaaaaaaaaaaaaaaaaaaa17:45
rlandyso mostly, the container value before update and after update should be the same17:48
rlandyhash value17:48
rlandybecause we update containers on the undercloud before the original deploy17:48
myoung|ruckI can update that logging patch, it just needs a rebase and a minor tweak17:49
*** brault has joined #oooq17:51
*** tcw has joined #oooq17:56
* myoung|ruck transmogrifies into a taxi and will return in 1hr18:01
rlandyweshay: one more on https://review.openstack.org/#/c/563526/18:02
rlandywrt DOD also18:02
rlandy Goal: in two diff commits. Use the update job to gate: - tripleo-quickstart - tripleo-quickstart-extras - tripleo-upgrade18:02
*** tcw has quit IRC18:02
rlandy Use the update job to check non-voting - tripleo-$other repos18:03
rlandythe commit message are all the same as the dependent patch18:04
*** jfrancoa has joined #oooq18:06
rlandyweshay: https://review.openstack.org/#/c/565523/1/zuul.d/multinode-jobs.yaml (line 136). How is a job its own parent?18:27
rlandyis that possible>18:27
weshayrlandy, wellllll... it was a template in base but I think Emilien updated that18:30
weshayhttps://github.com/openstack-infra/tripleo-ci/commit/c065512020b38a60193ad5123d3f89d2020ff7da#diff-ac5a223332f824ad473fce19f950eb5218:32
rlandyweshay: ok - now I  understand where that came from18:33
*** trown|lunch is now known as trown18:33
rlandy parent: tripleo-ci-dsvm-multinode18:33
weshayya. that is right18:34
weshayrlandy, to be honest... I don't think we need this part anymore.. https://review.openstack.org/#/c/565523/1/zuul.d/multinode-jobs.yaml18:34
weshayline 13518:34
weshayhttps://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/multinode-jobs.yaml#L30518:35
weshayis marked voting now18:35
weshaywhere previously it's wasn't18:35
weshayrlandy, however we're missing something18:36
weshaybecause that job is still not gating18:36
rlandyweshay: the whole patch set is a bit confusing (also I am not super familiar with it - so I am learning on the fly here). since you -1'ed, I guess it can wait until you confirm what is for review18:36
*** atoth has quit IRC18:36
weshayrlandy, ya.. my goal would be to have this wrapped up by thrs18:37
weshayfriday the latest18:37
rlandyweshay: ok - will leave it for the moment18:37
*** amoralej is now known as amoralej|off18:42
* trown is working on using https://review.openstack.org/#/c/566565/1 to actually make updates job do an update18:46
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/56044518:47
rlandyhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L11618:51
rlandykind of steals the update thunder18:51
weshayrlandy, ?18:52
weshaytrown, ya.. I saw that patch18:52
weshaytrown, was wondering what the next step looked like18:52
*** florianf has quit IRC18:53
trownweshay: that array should be passed/created by the python script18:53
trownweshay: then it is dynamic per job, and we can change releases one job at a time18:53
weshayya.. ok.. I get it now18:55
weshaythe formatting threw me off a bit18:55
weshaytrown, one job at a time or playbook at a time18:55
weshayI think any job that calls multinode-overcloud-update.yml would see a  change right18:56
weshaysorry should use upgrade as an example18:57
weshayany job that called ["multinode-overcloud-upgrade.yml"], would have some additional logic then either made it n+1 nor n+318:58
weshayam I understanding that correctly?18:58
weshayn+1 or n+3 is determined by the python script and featureset config18:59
weshayjust highlighting the diff between changing the release "one job at time" vs.. jobs that use a particular playbook18:59
*** brault has quit IRC19:01
*** florianf has joined #oooq19:03
*** holser__ has joined #oooq19:04
*** florianf has quit IRC19:10
weshayrlandy, ok.. this makes more sense now https://review.openstack.org/#/c/565523/19:11
* rlandy looks19:11
rlandyweshay: that change seems sane - but it doesn't match the commit message19:15
rlandyweshay: no need to fix now - I'll juts add that comment to the review19:17
trownweshay: no one job at a time... each job has a different array of per playbook releases19:20
weshaytrown, OH.. so the array of playbooks has to match.. huh19:21
weshaynot sure if I picked that up from the review19:21
weshayinteresting19:21
*** Goneri has quit IRC19:23
weshayrlandy, ok.. think I have them all updated now19:37
rlandyweshay: k - will look in a bit - just testing another review atm19:37
weshaytrown, it would be interesting to come up w/ a simple version of said tool and test w/ https://review.openstack.org/#/c/566067/19:40
weshayand https://bugs.launchpad.net/tripleo/+bug/176899719:40
openstackLaunchpad bug 1768997 in tripleo "containerized-undercloud-upgrades: dlrn tasks don't run" [Critical,Triaged]19:40
weshaymyoung|ruck, just a reminder that you probably want to promote rasca's job today vs. tomorrow19:46
myoung|ruckweshay: aye i'm doing that now19:47
myoung|rucksshnaidm|rover, rlandy: https://review.openstack.org/#/c/544696 has been updated per your comments19:48
rlandymyoung|ruck: +1'ed it - waiting on CI for +219:50
rlandywant to see what the logs look like19:51
myoung|ruckrlandy: ack, i didn't run an upgrade to test (doing ruck things too) - also want to see CI +2 :)19:51
rlandymyoung|ruck: actually - left a question on that review - not urgent19:57
myoung|ruckrlandy: ack, answered20:08
myoung|ruck%gatestatus20:09
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429120:09
*** saneax is now known as saneax-_-|AFK20:15
*** jfrancoa has quit IRC20:25
*** brault has joined #oooq20:30
*** brault has quit IRC20:35
rlandyweshay: ok - that looks more understandable now - reviewing20:39
rlandythanks20:39
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429120:47
rlandymyoung|ruck: you familiar with this error: http://logs.openstack.org/23/565523/3/check/tripleo-ci-centos-7-3nodes-multinode/30e0fc4/logs/devstack-gate-setup-workspace-new.txt?20:49
rlandyweshay: ?20:49
rlandyunrelated failure on your jobs20:49
weshayouch..  nope have not seen it20:50
*** jjoyce has joined #oooq20:50
rlandy2018-05-07 20:33:08.232 | + /home/zuul/workspace/devstack-gate/functions.sh:setup_workspace:L458:   rsync -a '/home/zuul/src/*/openstack/tripleo-incubator/' tripleo-incubator20:50
rlandy2018-05-07 20:33:08.236 | rsync: change_dir "/home/zuul/src/*/openstack/tripleo-incubator" failed: No such file or directory (2)20:50
rlandy2018-05-07 20:33:08.342 | rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]20:50
rlandy10.209.1.48 | FAILED | rc=23 >>20:50
* rlandy remembers patch20:50
rlandyincubator was removed20:51
rlandylooking20:51
weshayrlandy, they are talking about in openstack-infra20:51
myoung|ruckweshay: fyi that queens hash has been reported...waiting for promoter to loop around and catch it20:51
*** saneaxL has joined #oooq20:51
*** saneaxL is now known as saneax20:52
* myoung|ruck mutters something about unformatted logs on the promoter and vows to make a dashboard for this20:52
rlandyhttps://review.openstack.org/#/c/565847/20:52
*** saneax-_-|AFK has quit IRC20:53
*** fuzzball81 has quit IRC20:53
rlandyweshay: ^^20:53
*** bandini has quit IRC20:56
*** bandini has joined #oooq20:58
*** trown is now known as trown|outtypewww21:05
myoung|ruckweshay: have a quick sec to chat re: queens?21:07
*** Goneri has joined #oooq21:07
weshaymyoung|ruck, what's up?21:07
rlandyweshay: changes look sane but let's see what Co does with them when it actually runs21:08
rlandy+121:08
rlandyfor the moment21:08
weshayheh21:08
weshaythanks21:08
myoung|ruckweshay: fun with BM.  for the current queens rdo2 promotion, we run 4 metal jobs.  rasca's, and the 3 other jobs (env B, D, E).  rasca's passed (but promoter was screwed up).  I've logged that to dlrnapi21:09
rlandytripleo-multinode-ci-only-minimal  - where was that actually called - only see it referenced in one place21:09
myoung|ruckof the remaining 3, D instafailed (will look into it later), B passed (but is ignored), and E (tripleo-quickstart-queens-rdo_trunk-baremetal-hp_dl360_enve-single_nic_vlans) which the promoter wants, failed on OC deploy, after being green for a while21:10
rlandymyoung|ruck: D is out of the picture21:10
* myoung|ruck nods21:10
* myoung|ruck nods again and will disable job21:10
weshaymyoung|ruck, why is D out of the picture?21:10
weshaymyoung|ruck, no need to disable it21:10
myoung|ruckB passed (but is ignored by promoter), E failed (and is not ignored).  do we want to change the ini, or rekick E (or both)?21:10
weshayright?21:10
weshayhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/all%20top%20level%20multijobs/job/rdo-promote-queens-rdo_trunk/73/21:11
weshaymyoung|ruck, it only needs to pass once on a hash man21:11
weshayfor E21:11
myoung|ruckthat's a diff hash21:11
myoung|ruck2018-05-07 20:55:36,938 8692 INFO     promoter Skipping promotion of current-tripleo-rdo to current-tripleo-rdo-internal, missing successful jobs: ['tripleo-quickstart-queens-rdo_trunk-baremetal-hp_dl360_enve-single_nic_vlans']21:12
myoung|ruckhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rdo-promote-queens-rdo_trunk/75/21:12
myoung|ruck^^ https://trunk.rdoproject.org/centos7-queens/f8/90/f8906417f6905331948fa1b79dd4bd3f4644c85c_85b157a921:13
weshaymyoung|ruck, k.. taking care of it21:13
myoung|ruck^^ we can promote that one if criteria is for B --> queens.  not sure why E failed, it's been otherwise green.  going to rekick to see if fail repros but for tomorrow meeting would like to have a queens promo21:14
myoung|ruckrdo221:14
myoung|ruck(passed on this hash) https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-queens-rdo_trunk-baremetal-dell_fc430_envB-single_nic_vlans/3821:14
weshaymyoung|ruck, fyi https://review.rdoproject.org/r/#/c/13659/121:15
myoung|ruckweshay: ack thx.  FYI https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/dlrnapi-manual-job-report/4/console is how I promoted rasca's job.  this way no passwords are revealed for dlrnapi and we can do this again a little easier21:16
myoung|ruckrlandy: what did you mean re: "D is out of picture" ?  I see it's ssh unreachable in latest job(s)...has it been taken out of rotation or loaned out?21:17
rlandymyoung|ruck: there was an issue with one of the overcloud boxes21:17
rlandywe took it out of rotation21:17
rlandymaybe the env is ok now21:17
rlandyI have not been on the ruck/rove roation recently to check21:18
myoung|ruckrlandy: it's not :)  00:03:39.293 fatal: [rdoci-hp-01.v100.rdoci.lab.eng.rdu.redhat.com]: UNREACHABLE!21:18
rlandyreboot21:18
myoung|ruckrlandy: oh aye...just didn't know if there was new news.  still figuring out how long it's been down...queens has been attempting for a while21:19
rlandya long time21:19
*** jtomasek_ has quit IRC21:20
rlandyweshay: where was tripleo-multinode-ci-only-minimal actually referenced?21:21
rlandyI see the definition - which is correctly removed21:22
weshayrlandy, I killed it21:23
weshayit was referenced in some of the other project's zuul config21:23
rlandyI see - may it rest in peace21:23
weshayaye21:23
weshayrlandy, got verbal approval to add it containers-minimal21:23
weshayor irc approval?21:23
rlandyweshay: that is fine - I was just looking for why you had removed a definition but I could not find where it was actually used21:24
weshayk21:24
rlandyif nowhere now, that is fine21:24
*** holser__ has quit IRC21:29
rlandyweshay: will gate actually run on this job? https://review.openstack.org/#/c/565523/21:33
*** dsneddon has quit IRC21:41
*** dsneddon has joined #oooq21:41
*** Goneri has quit IRC21:44
*** Goneri has joined #oooq21:44
myoung|ruckarxcruz, chandankumar: looking at a recent check fail, do you know what's going on here?  http://logs.openstack.org/50/566050/9/check/tripleo-ci-centos-7-3nodes-multinode/346a704/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2018-05-07_16_43_03 (check job from https://review.openstack.org/#/c/566050) - is non-voting but was curious if this was expected21:44
* rlandy now has the fun job of going to a wedding of someone I hardly know21:49
rlandybbl21:49
myoung|ruckbbl21:52
*** Goneri has quit IRC21:57
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429122:47
*** Goneri has joined #oooq22:51
-openstackstatus- NOTICE: Any devstack job failure due to rsync errors related to tripleo-incubator can safely be rechecked now22:55
*** Goneri has quit IRC23:11
*** tosky has quit IRC23:30
*** Goneri has joined #oooq23:58
myoung|ruckweshay: you still here?23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!