Wednesday, 2018-07-18

*** tosky has quit IRC00:12
*** brault has joined #oooq00:48
*** brault has quit IRC00:52
hubbot`FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message)01:02
*** rlandy has joined #oooq01:35
rlandyweshay: still around? any idea why this combination does not trigger the fs053 test? https://review.openstack.org/#/c/581488/ https://review.rdoproject.org/zuul3/status.html01:37
rlandywhat did I miss?01:37
weshayrlandy, /me looks01:37
rlandyhttps://review.rdoproject.org/r/#/c/14772/01:37
rlandyshould define the job triggers01:37
rlandyshould trigger on changes to tq/tqe/tripleo-ci as well as browbeat01:38
weshayrlandy, oh I think we learned that the zuul config in rdo sf sucks in that it does not trigger unitil merged01:40
weshayrlandy, however I wonder if that is just for triggers on repos outside of sf01:41
rlandyhmmm01:41
rlandywhat's a catch 2201:41
weshayrlandy, so I wonder if we set it up to say.. trigger on ci-config01:42
weshaywhich does exist there..01:42
weshaytest it.. w/ that01:42
rlandyI'll have to test is with a reproducer01:42
weshayrlandy, /me looks at this01:42
rlandyweshay: the alternative is the to merge it triggering just one file in browbeat so we can see what it does01:43
weshayrlandy, aye.. do you have core?01:43
rlandyweshay: yes - do you?01:44
weshayrlandy, no01:44
rlandyhmmm01:44
weshayrlandy, we can fix that.. so I'll just resubmit under a diff commit id01:45
weshayrlandy, sec01:45
weshayrlandy, I think we should try to trigger w/ ci-config though too01:45
weshayfwiw01:45
rlandyI don;t know what 'trigger w/ ci-config' is01:45
rlandythe ci-config doesn't trigger that test01:46
rlandyok - let me put in another commit01:46
rlandyI don't want to merge this tonight w/o anyone's approval01:48
weshayrlandy, I think...  https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/projects.yaml#L483301:51
rlandyoh - I see01:52
weshayrlandy, and then we add a test patch to ci-config w/ a depends on ur zuul change01:52
weshayrlandy, not sure01:52
weshaybut I think it's worth a shot01:52
rlandyweshay: actually I think putting in a temp patch to trigger on only one file is a good idea01:53
rlandybut I 'll ask for merge tomorrow01:53
weshayok.. that is fine too01:53
rlandyso I am not the one pushing through my own code01:53
rlandythen we can merge the job definition and one file trigger and work from there01:54
rlandythanks for your help01:54
weshayk01:55
rlandyhttps://review.rdoproject.org/r/#/c/14880/01:59
*** rlandy has quit IRC02:02
*** jaganathan has joined #oooq02:37
*** gkadam has joined #oooq02:55
*** gkadam has quit IRC02:59
hubbot`FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message)03:02
*** skramaja has joined #oooq03:17
*** skramaja_ has joined #oooq03:21
*** skramaja has quit IRC03:21
*** udesale has joined #oooq03:24
*** ykarel has joined #oooq04:24
*** trown has quit IRC04:24
*** trown|brb has joined #oooq04:34
*** holser_ has joined #oooq04:54
*** udesale has quit IRC04:55
*** udesale has joined #oooq04:55
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message)05:02
*** lucasagomes_ has joined #oooq05:12
*** links has joined #oooq05:15
*** hamzy has quit IRC05:16
*** lucasagomes has quit IRC05:16
*** Tengu has quit IRC05:16
*** chkumar|ruck has quit IRC05:28
*** chandankumar has joined #oooq05:29
*** chandankumar is now known as chkumar|ruck05:29
*** hamzy has joined #oooq05:35
*** Tengu has joined #oooq05:35
*** ratailor has joined #oooq05:35
*** quiquell|off is now known as quiquell05:39
quiquellsshnaidm|rover, chkumar|ruck: new rrcockpit url http://38.145.34.131/d/7q6lisOik/cockpit?orgId=105:42
quiquellTell me if you are missing somet hing05:43
*** sanjayu_ has joined #oooq06:05
chkumar|ruckquiquell: I wants me to login06:06
ykarelchkumar|ruck, join #rhos-ops, all ovb jobs affected by rdo cloud issue06:06
*** jfrancoa has joined #oooq06:07
ykarelchkumar|ruck, i have asked csastri to look06:07
chkumar|ruckykarel: thanks :-)06:12
chkumar|ruck%gatestatus06:15
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009 (1 more message)06:15
chkumar|ruckquiquell: which credentials to use?06:17
*** holser_ has quit IRC06:24
quiquellchkumar|ruck: Forgot to activate anonymous access, wait06:30
*** kopecmartin has joined #oooq06:34
quiquellchkumar|ruck: need another spin to activate it, give me a few, ok ?06:46
chkumar|ruckquiquell: yup totally ok06:47
*** bogdando has joined #oooq06:49
*** florianf has joined #oooq06:50
*** holser_ has joined #oooq06:56
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message)07:02
*** amoralej|off has joined #oooq07:06
*** amoralej|off is now known as amoralej07:06
*** gkadam has joined #oooq07:08
*** ccamacho has joined #oooq07:08
*** brault has joined #oooq07:17
*** tesseract has joined #oooq07:18
*** brault has quit IRC07:19
*** brault has joined #oooq07:19
*** ykarel is now known as ykarel|lunch07:19
quiquellchkumar|ruck: Ok now anonymous works07:32
*** zoli|gone is now known as zoli07:44
*** jtomasek has joined #oooq07:50
*** jtomasek has quit IRC07:50
*** jtomasek has joined #oooq07:50
*** skramaja_ is now known as skramaja08:05
*** tosky has joined #oooq08:06
*** sanjayu_ is now known as saneax08:14
*** ykarel|lunch is now known as ykarel08:15
*** lucasagomes_ is now known as lucasagomes08:15
*** holser__ has joined #oooq08:20
*** holser_ has quit IRC08:21
*** amoralej_ has joined #oooq08:22
*** amoralej has quit IRC08:22
chkumar|ruckquiquell: thanks :-)08:27
quiquellmarios, sshnaidm|rover: fs as strings needed rebase https://review.openstack.org/#/c/58302208:36
mariosquiquell: ack08:38
jfrancoaanybody else has lost connectivity with the internal irc channels? quiquell, marios ?08:58
chkumar|ruckRDO cloud outage has just started08:59
mariosjfrancoa: checking08:59
mariosjfrancoa: no i'm still on internal08:59
mariosjfrancoa: maybe try re/connect vpn09:00
jfrancoamarios: yes, I'll try it. thanks09:00
quiquellWill check09:00
quiquellI am good there now09:00
jfrancoaquiquell: marios: restarting helped, as usual :-D thanks to both09:01
hubbot`FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message)09:02
*** amoralej has joined #oooq09:08
*** amoralej has quit IRC09:09
*** amoralej has joined #oooq09:09
*** amoralej_ has quit IRC09:10
*** dtantsur|afk is now known as dtantsur09:20
*** zoli is now known as zoli|lunch09:38
*** brault has quit IRC09:44
*** brault has joined #oooq09:45
*** Tengu has quit IRC09:48
sshnaidm|roverchkumar|ruck, seems like we hit it again: http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e548c5a/job-output.txt.gz#_2018-07-18_07_44_08_93687809:48
sshnaidm|roverhttp://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/6232ced/job-output.txt.gz#_2018-07-18_07_49_30_45969209:48
sshnaidm|roverchkumar|ruck, is the bug still opened?09:48
bogdandosshnaidm|rover: hi, https://review.openstack.org/#/c/465047/ looks good to go09:48
bogdandoat least for -2 removing :)09:49
sshnaidm|roverbogdando, ack, will remove09:49
sshnaidm|roverbogdando, btw, in which job this code runs?09:49
bogdandosshnaidm|rover: tested in https://review.openstack.org/#/c/583515/09:50
sshnaidm|roverbogdando, and job name?09:50
bogdandosshnaidm|rover: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/zuul.d/multinode-jobs.yaml#n39509:51
bogdandojfrancoa: ^^ am I right>09:51
sshnaidm|roverbogdando, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades ?09:52
bogdandoyes, should be it09:52
jfrancoabogdando: right. I triggered the job a while ago, so the results should start appearing soon09:56
bogdandothanks09:56
*** jaganathan has quit IRC10:07
chkumar|rucksshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/178187110:08
openstackLaunchpad bug 1781871 in tripleo "RDO cloud jobs failing with SSH Error: data could not be sent to remote host \"38.145.32.100\". Make sure this host can be reached over ssh" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)10:08
chkumar|rucksshnaidm|rover: but it is coming on openstack servers na not on rdo cloud10:09
sshnaidm|roverchkumar|ruck, of course10:10
sshnaidm|roverchkumar|ruck, it happens everywhere10:10
*** ykarel is now known as ykarel|afk10:13
*** atoth has quit IRC10:14
*** hubbot` has quit IRC10:23
quiquellsshnaidm|rover: What is the extra_node_keypair used for ?10:26
sshnaidm|roverquiquell, will be used for a specific ovb job "all tls"10:28
sshnaidm|roverquiquell, anyway, it's not a place to remove it, every commit should have a scope10:29
*** hubbot has joined #oooq10:29
quiquellsshnaidm|rover: I have just added, but then I need to put a extra_key.pub somewhere to test this10:29
quiquellsshnaidm|rover: Don't understand why it's needed in infra pieces like promoter or rrcockpit10:30
sshnaidm|roverquiquell, not sure I follow..10:32
sshnaidm|roverquiquell, I think we need to sync about this patch a little10:35
quiquellsshnaidm|rover: Yep, when ever you have some time we can bj on it10:35
sshnaidm|roverquiquell, cool, will ping you10:36
quiquellsshnaidm|rover: ack10:36
*** dmellado has quit IRC10:36
*** zoli|lunch is now known as zoli10:40
*** dmellado has joined #oooq10:42
quiquellBasic question, I see conditionals at master.yaml or featureset051.yaml files10:46
quiquellthey are treated like jinja templates ?10:46
sshnaidm|roverquiquell, jinja templates are only in variable definition11:01
sshnaidm|roverquiquell, the files themselves are yaml format11:01
quiquellsshnaidm|rove: Found that we can have conditionals for the contents of variables11:02
sshnaidm|roverquiquell, but ansible evaluates jinja lang inside vars11:02
quiquellsshnaidm|rover: IThat's going to be soo helpful11:02
sshnaidm|roverquiquell, yeah, there is a limited flexibility11:02
quiquellsshnaidm|rover: Can you take a look at this sprint review ?11:03
quiquellsshnaidm|rover: https://review.openstack.org/#/c/582885/, is not very big11:03
*** atoth has joined #oooq11:03
quiquellsshnaidm|rover: Have check the conditional manually11:03
chkumar|rucksshnaidm|rover: I need some help on this bug https://bugs.launchpad.net/tripleo/+bug/178231711:04
openstackLaunchpad bug 1782317 in tripleo "[master] scenario008 multinode job failing at undercloud giving Invalid local_interface specified. br-ex is not available." [Critical,Triaged]11:04
quiquellrfolco: Check this https://review.openstack.org/#/c/582885/20/playbooks/tripleo-ci/vars/common.yaml11:04
chkumar|ruckwhere can i look for local_interface11:04
quiquellrfolco: That's what I mean by moving conditionals to the .yaml include vars11:05
quiquellrfolco: Like releases files, ej. master.yaml11:05
sshnaidm|roverquiquell, if tripleo_root and workspace are the same, why to use both?11:08
quiquellsshnaidm|rover: playbooks are expecting tripleo_root variable, but the scripts where working with WORKSPACE11:09
quiquellsshnaidm|rover: Just want to keep the naming11:09
quiquellsshnaidm|rover: We can also create diferent directories for both11:09
sshnaidm|roverchkumar|ruck, br-ex is not available  - I suspect it's leftovers from last CI sprint11:10
sshnaidm|roverquiquell, tripleo_root was for folder which includes all repos, if we save them in workspace, then no need to keep it11:10
chkumar|rucksshnaidm|rover: scenario 008 is broken from last 2 weeks11:11
quiquellsshnaidm|rover: a -e tripleo_root=workspace will be needed for the underlying playbooks11:11
chkumar|ruckso today I filed a bug, whom can i bug to get it fixed?11:11
sshnaidm|roverchkumar|ruck, yeah, it's what I'm talking about11:11
sshnaidm|roverchkumar|ruck, I'll handle it..11:12
chkumar|rucksshnaidm|rover: Thanks :-)11:12
sshnaidm|roverquiquell, I see.. then I don't understand why we copy repos to workspace11:13
quiquellsshnaidm|rover: All the underlying playbooks expectr to be at one root directory11:13
sshnaidm|roverquiquell, and how exactly to define where to use tripleo_root and where workspace, if they are the same11:14
quiquellsshnaidm|rover: wait... I have find in idea11:14
quiquellsshnaidm|rover: Playbooks are only using tripleo_root ansible var11:15
quiquellsshnaidm|rover: And from them access to the different repos11:15
quiquellsshnaidm|rover: workspace is for the script, but we can make the scripts use tripleo_root too11:16
quiquellsshnaidm|rover: We cannot set tripleo_root to /home/zuul/src/git.openstack.org/openstack/11:16
quiquellbecause we have /home/zuul/src/git.openstack.org/openstack-infra/tripleo-ci/11:16
quiquelltripleo-ci is at openstack-infra not openstack11:16
sshnaidm|roverquiquell, it's easily solved with symlink11:17
quiquellsshnaidm|rover: That's alreay don't at run-v3.yaml11:17
quiquellwith hardlinks11:17
quiquellprevious sprint did that11:18
quiquellso, the bash scripts use workspace and the playbooks use tripleo_root11:18
quiquellI can remove workspace and make the script use tripleo_root11:18
quiquellWhat do you think ?11:18
sshnaidm|roverquiquell, so we will continue to copy repos to workspace, I don't think it's good idea. Workspace should be place for our scripts, logs, images, other artifacts. tripleo_root/anything should be root folder for repositories11:20
sshnaidm|roverquiquell, actually tripleo_root should be /home/zuul/src/git.openstack.org/openstack11:20
quiquellsshnaidm|rover: Cannot be there11:20
quiquelltripleo-ci is a openstack-infra11:20
quiquellThis is the main issue11:20
sshnaidm|roverquiquell, ln -s /home/zuul/src/git.openstack.org/openstack-infra/tripleo-ci /home/zuul/src/git.openstack.org/openstack/tripleo-ci11:20
quiquellWe don't have permissions there11:21
sshnaidm|roverquiquell, what do you mean?11:21
quiquellWe can write only at zuul.project.src_dir11:21
quiquellsshnaidm|rover: But let me try11:21
sshnaidm|roverquiquell, I think I had a patch that does it, and everything worked11:22
quiquellsshnaidm|rover: Let me check11:22
quiquellsshnaidm|rover: Would be nice to reduce links11:22
quiquellsshnaidm|rover: Still we use tripleo_root for generated stuff11:22
sshnaidm|roverquiquell, I will start process of moving tripleo-ci to openstack namespace also11:22
quiquellI don't know if /home/zuul/src/git.openstack.org/openstack/ is a good place for it11:22
sshnaidm|roverso maybe it will be temporary hack with symlink11:23
sshnaidm|roverquiquell, generated stuff?11:23
sshnaidm|roverquiquell, do you mean jinja templates?11:23
quiquellwait wait11:23
*** ykarel|afk is now known as ykarel11:24
quiquellpufff we have at least a playbook that uses the env var WORKSPACE :-/11:26
quiquellbut non try to write a tripleo_root11:26
quiquellsshnaidm|rover: So is a good thing to have both and point tripleo_root to openstack11:27
quiquellsshnaidm|rover: We can move tripleo-ci to openstack namespace ?11:27
quiquellsshnaidm|rover: It would take years ?11:27
sshnaidm|roverquiquell, hopefully not11:27
chkumar|ruckweshay: sshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1782267 this bug does not have logs, is it related to the https://bugs.launchpad.net/tripleo/+bug/1782211 comments?11:34
openstackLaunchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged]11:34
openstackLaunchpad bug 1782088 in tripleo "duplicate for #1782211 [master][promotion]undercloud install failed by giving ImportError: cannot import name utils at Starting OpenStack Neutron Destroy Patch Ports" [Critical,Triaged]11:34
quiquellsshnaidm|rover: Added the changes to the review, let's see how it works11:34
sshnaidm|roverchkumar|ruck, you can see that last is dup of first11:35
sshnaidm|roverchkumar|ruck, it's not from job, it's from local run11:35
quiquellmarios, sshnaidm|rover: This one is ready for merge https://review.openstack.org/#/c/583022/11:35
quiquellmarios, sshnaidm|rover: featureset numbers as strings11:35
*** ratailor has quit IRC11:38
*** jaganathan has joined #oooq11:44
quiquellsshnaidm|rover: Linking of just tripleo-ci at openstack namespace is working11:49
sshnaidm|roverquiquell, left additional comments in https://review.openstack.org/#/c/582885/11:51
quiquellsshnaidm|rover: Thanks, very helpful, good for an ansible newbie like myself11:58
*** trown|brb is now known as trown12:00
*** skramaja_ has joined #oooq12:05
weshaychkumar|ruck, ya.. no worries12:06
weshaychkumar|ruck, I see Alan's comments12:06
weshaychkumar|ruck, are you interested in giving status on the program call?12:07
chkumar|ruckweshay: sure I can update that12:08
*** skramaja has quit IRC12:08
weshaychkumar|ruck, k.. join https://bluejeans.com/7253947007/12:10
weshaychkumar|ruck, search for your name, "Chandan" in https://docs.google.com/document/d/13StMviF7pnXve42F2vigeZoopoofEtqOyZj8CxVpxko/edit#12:10
weshaychkumar|ruck, review the status I added for July 18th12:11
weshaythat is all you need to say :)12:11
weshaychkumar|ruck, make sense?12:11
chkumar|ruckweshay: yup correct12:12
*** amoralej is now known as amoralej|lunch12:18
weshaychkumar|ruck, almost up :)12:25
weshaychkumar|ruck, well done12:26
chkumar|ruckweshay: thanks boss :-)12:26
* chkumar|ruck was fearing while speaking12:26
*** tcw has quit IRC12:27
*** tcw has joined #oooq12:27
*** tcw1 has joined #oooq12:28
*** tcw has quit IRC12:28
rfolcoquiquell, are you going to rebase https://review.openstack.org/#/c/582885 later on top of my patch ?12:28
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo- (1 more message)12:29
ssbarnea1is there a trick i can use to be able to bundle several CRs on gerrit? like 5-10 of them, unrelated.12:30
quiquellssbarnea1: topic maybe12:34
quiquellrfolco: This is the one that can be in the top https://review.openstack.org/#/c/582466/12:35
*** rlandy has joined #oooq12:35
quiquellrfolco: But they are not releated12:35
quiquellrfolco: This one https://review.openstack.org/#/c/582885/ "common.yaml" can be at t he top of yours12:35
quiquellrfolco: so you can use "workspace" and "tripleo_root" and "periodic" variable12:35
weshayssbarnea1, ping12:36
quiquellrfolco: After your patch with featureset numbers as string I think we have to merge the common.yaml12:36
rfolcoquiquell, so things are not like we discussed yesterday12:37
quiquellrfolco: So we can move to cmmon.yaml stuff rom toci_gate_test and toci_quickstart12:37
weshayssbarnea1, have a sec to join early?12:37
weshayquiquell, since today is actually wednesday.. I'm ignoring you :)12:37
rascasshnaidm|rover, hi, so this https://review.openstack.org/#/c/582932/ passed with fs 10, can we proceed for the merge?12:37
weshaysshnaidm|rover, chkumar|ruck 21 gate failures?12:38
quiquellweshay: You are nog ignoring me, as you are saying that you are ignoring me12:38
weshayis that accurate12:38
weshayquiquell, SOB!!!!12:38
quiquellrfolco: As I remember it was to move in top of your patches this https://review.openstack.org/#/c/582466/12:38
quiquellrfolco: But also we talk about moving this other as the base https://review.openstack.org/#/c/582885/12:39
quiquellrfolco: I can be wrong totally12:39
chkumar|ruckweshay: 21 gates failures all the ovb jobs failed in the morning, due to little hiccup in http on controller node12:39
chkumar|ruckweshay: https://etherpad.openstack.org/p/8jz2JhpZWc12:39
weshayssbarnea1, https://bluejeans.com/411356779812:39
rfolcoquiquell, that sounds reasonable. I'll make the changes to reflect this.12:40
chkumar|ruckweshay: or you are referring to something else?12:40
quiquellrfolco: I am not going to add more stuff at common.yaml12:40
quiquellrfolco: So we can start to think about merge it,12:40
weshaychkumar|ruck, http://38.145.34.131/d/7q6lisOik/cockpit?orgId=112:40
* weshay looking at the gate jobs12:40
quiquellso it's safe to reparent now12:40
rlandysshnaidm|rover: panda|off: rfolco: I need to start testing the browbeat job in in CI. I have tried 'Depends-On' but the fs035 jobs does not trigger. Can we merge https://review.rdoproject.org/r/#/c/14808/ and https://review.rdoproject.org/r/#/c/14880/ so that fs053 job will trigger on just one file change so we can test that way?12:40
weshaychkumar|ruck, ovb jobs can not be gate12:40
weshayrlandy, good morning.. thanks for your help yesterday12:41
chkumar|ruckweshay: others are related to thisobne https://bugs.launchpad.net/tripleo/+bug/178187112:41
openstackLaunchpad bug 1781871 in tripleo "RDO cloud jobs failing with SSH Error: data could not be sent to remote host \"38.145.32.100\". Make sure this host can be reached over ssh" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)12:41
rlandywe need to watch timing and access12:41
weshayrlandy, the ovb introspection error is fixed https://bugs.launchpad.net/tripleo/+bug/178226812:41
openstackLaunchpad bug 1782088 in tripleo "duplicate for #1782268 [master][promotion]undercloud install failed by giving ImportError: cannot import name utils at Starting OpenStack Neutron Destroy Patch Ports" [Critical,Fix released] - Assigned to yatin (yatinkarel)12:41
*** quiquell is now known as quiquell|lunch12:41
chkumar|ruckweshay: ssh issue happening in openstack  jobs12:42
*** agopi|off has quit IRC12:42
rlandyweshay: lol - what would we do w/o ykarel???12:42
bogdandoweshay, sshnaidm|rover: https://review.openstack.org/#/c/465047/13 worked12:42
weshaychkumar|ruck, what issue?12:42
bogdandoin https://review.openstack.org/#/c/583515/, it passed the needed upgrade job12:42
sshnaidm|roverweshay, it's infra problem with ssh12:42
bogdandotripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades12:42
chkumar|ruckweshay: http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e548c5a/job-output.txt.gz#_2018-07-18_07_44_08_93687812:42
weshaychkumar|ruck, sshnaidm|rover thanks12:43
weshaychkumar|ruck, sshnaidm|rover they haven't posted anything to their wiki or twitter12:44
chkumar|ruckswitched primary address for openstackci pypi account from review@o.o to infra-root@o.o so that it doesn't get mixed in with gerrit backscatter (we can switch to a dedicated alias later if needed)12:44
chkumar|rucklastmessage not seen12:44
weshaychkumar|ruck, sshnaidm|rover in these cases.. imho it would be informative to describe the issue and error and add a tracking bug to tripleo w/ the alert tag so that folks can get the status w/o having to bug you guys :)12:45
rlandyalso ..12:45
bogdandosshnaidm|rover: commented12:45
bogdandowe can't do by script name12:45
rlandyweshay: sshnaidm|rover: marios: https://review.openstack.org/#/c/581484/ - I think this can merge w/o causing issues12:45
rfolcorlandy, sorry if I miss something.... why do you need browbeat in all featuresets if you only run browbeat on fs053 ?12:46
rlandyrfolco: I wasn't sure I do12:47
rlandybut if you look all the jobs have upgrades included12:47
rlandyeven though not all jobs run them12:47
chkumar|ruckweshay: sshnaidm|rover https://review.rdoproject.org/r/#/c/14870/ is applied in the rdo afs mirror machine after pypi caching change by apevec, if anything odd is seen, feel free to report12:47
rlandyalso there was error  a few runs ago12:47
rlandysaying the playbook was not available ... see12:48
*** quiquell|lunch is now known as quiquell12:48
rlandyhttps://logs.rdoproject.org/88/581488/8/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/27bdb3f/job-output.txt.gz#_2018-07-17_01_12_49_45244412:48
rlandy Could not find or access '/home/zuul/workspace/.quickstart/playbooks/browbeat-minimal.yml'12:48
rlandyeven though that fs did not use the playbook12:48
rfolcorlandy, oh I see all those jobs have been translated by script... there is much duplication.12:48
rlandysshnaidm|rover: ^^ am I right about that?12:49
rlandyrfolco: it's a complicated now12:49
quiquellrfolco: Added some comments to your patches12:49
rlandyand really I am guessing my way through this transformed code12:49
*** ykarel is now known as ykarel|away12:49
rlandybut we need to start testing now12:50
rfolcorlandy, this .quickstart dir works now ?12:50
rfolcoit seems odd to me12:50
rlandyrfolco: we have passing tests now12:50
rlandyme too12:50
rlandyfeel free to explain this to me :)12:50
rfolcorlandy, still trying to understand where this playbook is defined12:51
rfolco:)12:51
chkumar|ruckarxcruz: https://review.rdoproject.org/r/#/c/14886/12:51
chkumar|ruckajo: fs020 is already in experimental, will i replace fs020 tofs021 in experimental?12:52
weshayssbarnea1, https://ci.centos.org/view/rdo/view/tripleo-gate/12:52
arxcruzweshay: ^12:52
chkumar|ruckajo: sorry12:53
ajochkumar|ruck: not sure if that was to me, what's fs021? :)12:53
ajohehe12:53
chkumar|ruckarxcruz: fs020 is already in experimental, will i replace fs020 tofs021 in experimental?12:53
ajoack12:53
chkumar|ruckajo: featureset021 == fs02112:53
rlandyrfolco: this playbook - which one? there are a bunch in the  mix12:54
bogdandosshnaidm|rover: updated by your comments12:55
arxcruzchkumar|ruck: yes, it's the same, except for the skip list, so i think fs021 should also be in experimental, as non voting12:55
arxcruzthe idea of this is to see how much we are covering on tempest and see what we can remove from skip list during time12:55
chkumar|ruckarxcruz: ack12:56
mariosrlandy: ack12:56
rfolcorlandy, so browbeat is not a playbook you call from the job definition. Featreset053 runs toci_gate which calls browbeat under .quickstart dir (only if browbeat-minimal.yml is modified --> ansible/oooq/browbeat-minimal.yml12:59
rlandyunder the workspace12:59
rlandybut yes12:59
rfolcorlandy, I did not look at toci_gate... but... if toci_gate clones browbeat, you don't need it in required-projects13:00
rlandyrfolco: then why the error?13:00
*** hubbot has quit IRC13:00
rlandyI'd happily take it out13:00
rfolcorlandy, the error you pointed me last time was that you were trying to use browbeat from /home/zuul/git.openstack.org/openstack/browbeat .... use = copy to workspace13:01
rfolcorlandy, required-projects makes zuul to clone the repo into  /home/zuul/src/git....13:01
rfolcobut if this job is already done by zuul-cloner or quickstart, you don't need it13:02
rfolcorlandy, cannot find browbeat in tq tqe or tripleo-ci... where is it?13:03
rlandyrfolco: it's in its own repo13:04
*** hubbot has joined #oooq13:04
rlandyhttps://github.com/openstack/browbeat13:04
rfolcorlandy, but how toci_gate fs053 runs it ?13:04
rfolcothe run playbook in https://review.rdoproject.org/r/#/c/14808/7/playbooks/legacy/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/run.yaml runs toci_gate13:04
rlandyhttps://review.openstack.org/#/c/581488/11/playbooks/baremetal-quickstart-extras.yml13:04
*** skramaja_ has quit IRC13:05
rfolcoaaaah13:05
rfolco:)13:05
rlandysshnaidm|rover merged it13:05
rlandyI am happy to change it in another patch13:05
sshnaidm|roverrlandy, it's fine to fix it, it doesn't run anywhere for now..13:06
rlandysshnaidm|rover: thanks - yeah, we just need a test space13:06
sshnaidm|roverrlandy, sorry if I merged it before13:06
rlandyno - thank you13:06
rlandywe are missing some settings13:07
rlandywaiting for agopi/ rook to be available to add them somewhere13:08
rlandythen we can run13:08
weshayssbarnea1, http://cistatus.tripleo.org/gates/13:08
*** agopi|off has joined #oooq13:10
*** agopi|off is now known as agopi13:10
rlandyagpoi: hi13:12
sshnaidm|roverarxcruz, chkumar|ruck what is credentials to tempestvm? is it in our infra-setup group?13:12
rlandyagpoi: can we talk about the settings you took out and where to put them now?13:12
rookrlandy: what settings are m,issing13:13
arxcruzsshnaidm|rover: tempestmail ? not yet, i still need to create the playbook for that13:13
arxcruzsshnaidm|rover: gimme your creds, i'll add it13:13
agopirlandy, hi13:13
chkumar|rucksshnaidm|rover: i donot have13:14
*** ykarel|away has quit IRC13:14
sshnaidm|roverarxcruz, https://github.com/sshnaidm.keys13:14
rlandyagopi: rook: https://review.openstack.org/#/c/581484/5..6/config/general_config/featureset053.yml13:14
agopirook, i can create a config file hardcoding most of the variables13:14
agopibut some can't be put in config file13:15
rlandyagopi: rook: we can keep the enable/disable settings13:15
agopilike the benchmark var itself13:15
rlandyand the config file13:15
rlandythe cloud_name can move to rdocloud settings13:15
rlandyhosts should move13:16
rlandynothing env specific should be in this file13:16
*** vinaykns has joined #oooq13:16
rlandymake sense?13:16
agopiokay got it, those hosts are basically random urls because config file needs hosts in there.13:17
agopiack rlandy13:17
chkumar|ruckweshay: sshnaidm|rover I need some help on adding fs021 in 24 periodic queue https://review.rdoproject.org/r/#/c/14887/13:18
chkumar|ruckplease have a look13:18
rlandyagopi: rook; ok - let's move those to rdocloud config13:18
rlandyweshay: ^^ agree?13:18
rookok, so the browbeat specific config will be rdocloud?13:18
* chkumar|ruck moves to home13:18
rlandyrook: well cloud_name: rdo_cloud13:18
rookwhy not just out the entire browbeat_rdo cloud in the browbeat dir13:19
rookwhy segment it?13:19
rlandythe hosts will be different per env13:19
agopirlandy, i can hardcode elastic_enabled, elastic_host, grafana_enabled and grafana_host so that we dont have to worry about it now13:19
rookit will still be rdo_cloud though, right?13:19
agopithose hosts are just for the sake of being there13:19
rlandyagopi: the enabled stuff can stay13:20
sshnaidm|roverbogdando, I didn't understand your argument about script though, not critical just curious13:20
arxcruzchkumar|ruck: https://review.openstack.org/#/c/573220/ \o/13:20
rfolcorlandy, I think you can test without browbeat as required-project... If I am not missing anything, it is installed as a requirement here https://review.openstack.org/#/c/581484/6/quickstart-extras-requirements.txt13:20
sshnaidm|roverchkumar|ruck, firstly let's decide which branches you want to run it on13:21
rlandyagopi: rook: my objection is the rdo_cloud thing mostly13:21
sshnaidm|roverchkumar|ruck, 24 hr is mostly for pike and ocata13:22
rookrlandy: can you show me where?13:22
agopii put cloud_name: rdo_cloud rook13:22
rlandyone sec - let's step back13:22
rookthat is literally just a string... it has no meaning to anything really.13:22
rookin the grand scheme of things13:22
rlandylet's talk about where these settings **should** be13:22
rookit could be "bobs carwash"13:23
rfolcorlandy, if you are not using it from /home/zuul/src/git.openstack.org/openstack/browbeat, then its a signal you don't need it... I'll comment your patch if you don't mind13:23
*** quiquell is now known as quiquell|launch13:23
*** quiquell|launch is now known as quiquell|lunch13:23
sshnaidm|roverchkumar|ruck, also you need a periodic job, not regular, please see how other periodic jobs are defined13:23
rlandyrfoloc; one moment - will gat back to you13:23
rookif we were using elasticsearch it becomes important.13:23
*** vinaykns has left #oooq13:23
rookelastic + collectd/grafana13:23
rlandyrook: agopi: the goal is you should be able to use this featureset as well13:24
rlandyso,13:24
rooksince we are not, it has no significance13:24
sshnaidm|roverchkumar|ruck, what is purpose of running it in periodic?13:24
rlandybefore we move anything ...13:24
rlandythis is a browbeat fs13:24
rlandyso we can have browbeat specific settings in there13:25
rlandyjust not env settings13:25
rlandyif, in CI where this fs is used13:25
rlandythe hosts are just fake, that is fine13:25
rlandyI am just trying to keep this uniform and env-agnostic13:26
rlandyrook; agopi: pls see https://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/config/testenv/ovb-rdocloud.yml13:26
rlandyas an example13:26
mariosweshay: posted this btw https://review.openstack.org/#/c/583547/ for that bug you pointed me at yesterday https://bugs.launchpad.net/openstack-infra/+bug/178125513:26
openstackLaunchpad bug 1781255 in tripleo ""Error: centos-release-ceph-luminous conflicts with centos-release-ceph-jewel-1.0-1.el7.centos.noarch" [Undecided,New]13:26
mariosweshay: even though we are gonna stop using it we should probably land that fix for now anyway /first13:27
rlandywe would keep your cloud_name in here13:27
weshaymarios++13:28
hubbotweshay: marios's karma is now 313:28
rlandymaybe the rest can stay13:28
mariosweshay: ha :) thanks was an easy fix and i was already staring at that code13:29
rlandyfor sure the enable can13:29
rlandyalright13:29
rlandyI'll try a test run13:29
agopiack rlandy13:29
rlandyagopi: is the cloud name important?13:30
rlandycould I switch it to rdocloud?13:30
rlandyno dash?13:30
agopibecause we arent collecting data its not imp13:30
rlandyok13:30
rlandylet me try and we will discuss from there13:30
agopiokay13:31
rookso we need to template out that from the browbeat config13:31
agopiyes rook as i mentioned above i can hardcode them13:31
agopihttps://github.com/openstack/browbeat/blob/master/ansible/oooq/roles/template-configs/templates/browbeat-minimal-ci.yaml.j2 create a template specific to this job13:32
rooki was under the impression rlandy is suggesting to store cloud_name here : ovb-rdocloud.yml -- so, i don't get the sense hardcoding it is what we want.13:34
rlandyrook: yep - no hardcoding13:34
rlandyjust env settings where they belong13:34
rlandyI am moving the cloud_name and the hosts - that is all13:34
agopioh gotcha.13:35
rookso in the grand scheme of things, I think what rlandy is asking is everything under workloads: we keep... things under browbeat: ealsticsearch: grafana: they keep13:35
rookwhich gets me to another point about our browbeat config13:35
rooki want to break it up13:35
rookbrowbeat-infra.yaml13:36
*** myoung|off is now known as myoung13:36
rookwhich will have that shit, and browbeat-workloads.yaml which will be the workload definition13:36
rookagopi: ^13:36
*** tcw1 is now known as tcw13:36
rlandyI am just trying to keep your testing uniform with upstream, that is all13:36
rookyeah13:36
rookunderstood rlandy... I think I am tracking13:36
*** amoralej|lunch is now known as amoralej13:37
agopimakes sense rook, rlandy. i now get what you're saying.13:38
rlandypatches coming up13:38
*** Tengu has joined #oooq13:44
*** tcw has quit IRC13:46
*** tcw has joined #oooq13:47
*** quiquell|lunch is now known as quiquell13:54
rlandysshnaidm|rover: https://review.rdoproject.org/r/#/c/14880/ what is missing here for merge?13:58
rlandy Needs Verified13:58
rlandyVerified13:58
rlandy+1 Zuul CI13:58
rlandyoh nvm13:58
rlandyno - wait - depends-on is merged13:59
myoungo/  happy weds!14:00
rlandystill?14:00
sshnaidm|roverrlandy, seems zuul queue was reloaded14:01
rlandysshnaidm|rover: I rechecked14:01
sshnaidm|roverrlandy, no need, gate jobs started14:01
arxcruzsshnaidm|rover: weshay rlandy trown could you guys please, please, pretty please review https://review.openstack.org/#/c/573220/ ? only this is missing to close a card :D14:01
arxcruz<314:01
rlandycool14:03
sshnaidm|roverarxcruz, where does this code execute?14:03
arxcruzsshnaidm|rover: scenario00414:03
rlandyyay!!!14:05
rlandywe have test kick14:05
rlandylegacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master14:05
sshnaidm|roverarxcruz, did you test it on different from master branch?14:05
agopirlandy++14:06
hubbotagopi: rlandy's karma is now 1414:06
rlandyrfolco: as soon as I see one clear run, I'll remove browbeat from required projects14:06
rlandythen we can see how that works14:06
arxcruzsshnaidm|rover: other branches are not affectred, only master and rocky14:07
rfolcorlandy, sounds like a plan14:07
rlandyagopi: rook: weshay: we have a queued job in https://review.rdoproject.org/zuul/status.html14:07
rlandylet's see what shakes out14:07
arxcruztempestconf on other branches also doesn't have the patch, so we are good14:07
weshayrlandy, which review?14:08
rlandyopenstack/browbeat14:08
rlandy583581,214:08
sshnaidm|roverarxcruz, yeah, i just wasn't sure about such jinja construct14:08
weshayrlandy+++14:09
*** quiquell is now known as quiquell|off14:12
*** dtrainor has quit IRC14:20
weshayrlandy, sshnaidm|rover fyi.. so the introspection bug on libvirt deployments is most likely caused by a baseos / libvirt change https://bugzilla.redhat.com/show_bug.cgi?id=157646414:25
openstackbugzilla.redhat.com bug 1576464 in libvirt "Hash operation not allowed during iteration" [High,Verified] - Assigned to mprivozn14:25
weshayfyi14:25
* weshay getting into it14:25
*** udesale_ has joined #oooq14:26
*** udesale has quit IRC14:27
rlandywell14:28
rlandyinteresting find14:29
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo- (1 more message)14:29
sshnaidm|roverweshay, so waiting for libvirt-4.3.0-114:32
myoungpanda|off: If you are available for a quick sync after training today, please ping.  I need 3-5 min.14:32
weshaysshnaidm|rover, why would this work in ci.centos?14:35
chkumar|rucksshnaidm|rover: we wanted a 24 periodic job for all branches running fs21 so that we can clear the skip list at regular interval and fix the job14:35
weshaymy brain hurts14:35
chkumar|rucksshnaidm|rover: it should not be part of promotion14:35
weshaysshnaidm|rover, https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-375/undercloud/var/log/extra/rpm-list.txt.gz14:40
weshayI don't get it14:40
sshnaidm|roverweshay, chkumar|ruck one by one, please :)14:41
weshayhttps://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-375/172.19.2.142/var/log/extra/rpm-list.txt.gz14:41
weshaylolz14:41
weshaysshnaidm|rover, I'm just fyi14:41
weshay'ing you14:41
weshaychkumar|ruck, it can be in the 24hr queue next to the ocata/pike jobs14:42
weshaychkumar|ruck, we just wont add it to https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/dlrnapi_promoter/config14:42
weshaychkumar|ruck, make sense?14:42
chkumar|ruckweshay: yup make sense, lots of new things to grasp in last 2 weels14:43
sshnaidm|roverchkumar|ruck, part of promotion or not - decides criteria files like this https://github.com/rdo-infra/ci-config/blob/614e886f17e146c9327b77d57009cd82394649f7/ci-scripts/dlrnapi_promoter/config/queens.ini#L114:43
sshnaidm|roverchkumar|ruck, we can run it with all other periodic jobs, no problem14:44
chkumar|rucksure, I will create the zuulv3 job for fs21 for stable branches14:44
sshnaidm|roverchkumar|ruck, but: 1) it should be periodic 2) better to have it for all branches and run it according to branch run policy14:44
*** dtrainor has joined #oooq14:45
sshnaidm|roverchkumar|ruck, like 24 hr for pike and ocata, openstack-periodic for master queens14:45
sshnaidm|roverchkumar|ruck, just look at repo how other periodic jobs are defined14:45
chkumar|rucksshnaidm|rover: do we need to keep for ocata, it is already EOL?14:46
sshnaidm|roverchkumar|ruck, is it?14:46
chkumar|rucki mean downstream RHOS-11 EOLed already14:46
chkumar|ruckwe have kept it for fast forward upgrades14:46
sshnaidm|roverchkumar|ruck, well, when it will be EOLed we'll delete just everything, including 012 job14:49
sshnaidm|roverchkumar|ruck, it won't add us additional work14:50
sshnaidm|rovers/012/021/14:50
sshnaidm|roverchkumar|ruck, but up to you14:50
chkumar|rucksshnaidm|rover: sure,14:50
rascachkumar|ruck, can you have a look at this https://review.openstack.org/#/c/573255/ and maybe drive the +2 +workflow so we can merge?14:56
*** kopecmartin has quit IRC14:56
weshaysshnaidm|rover, +1 correct on the direction w/ that job14:58
weshaychkumar|ruck, sshnaidm|rover I'm not terribly concerned about ocata, it's not EOL upstream14:58
weshaychkumar|ruck, sshnaidm|rover let's focus on master/rocky, queens, pike14:58
weshaychkumar|ruck, especially queens and pike the skip list is terrible14:59
weshaysshnaidm|rover, hrm.. I don't see a fix in centos15:03
weshayhttps://errata.devel.redhat.com/advisory/34155/builds15:03
weshayhttps://cbs.centos.org/koji/packageinfo?packageID=13015:03
sshnaidm|roverweshay, last update is form 2018-05-0915:05
weshayya15:05
sshnaidm|roverweshay, doesn't seems like people hurry to build it..15:05
weshaysshnaidm|rover, we could rebuild from src to test15:05
weshaysshnaidm|rover, btw.. I still can't get master rdo2 fs20 to get passed the undercloud15:05
sshnaidm|roverweshay, maybe something with slave..?15:07
weshaynaw15:07
sshnaidm|roverweshay, if it fetches the right code and uses old, seems like it has the old somewhere15:08
sshnaidm|roverweshay, cache, temp dirs..15:09
weshaysshnaidm|rover, I rm -Rf /home/rhos-ci/jenkins/workspace15:09
weshaysshnaidm|rover, could be a temp dir I suppose15:09
weshaysshnaidm|rover, maybe the image?15:09
weshaysshnaidm|rover, maybe /var/cache/tripleo-quickstart?15:09
arxcruzholly shit! weshay i FOUND THE PROBLEM WITH SCENARIO00215:12
weshayarxcruz, oh my15:13
weshay?15:13
weshayarxcruz, so in the endpoint url?15:13
arxcruzweshay: no15:13
arxcruzweshay: for some reason, tempest.conf is returning the region as RegionOne, so it's getting different endpoint15:14
arxcruzthe correct is regionOne15:14
sshnaidm|roverweshay, I'd terminate the slave totally..15:14
arxcruzI ALWAYS told this regionOne was a terrible idea15:14
weshaysshnaidm|rover, ?15:15
weshaysshnaidm|rover, who are you Arnold15:15
weshay?15:15
weshayarxcruz, oh man15:16
chkumar|ruckarxcruz: camelcasing was always a problem another example was SwiftOperator15:16
weshayarxcruz, it's such a nice moment when you find that shit15:16
weshayand so frustrating all at the same time15:16
sshnaidm|roverweshay, annihilate it15:17
weshaysshnaidm|rover, I basically did man15:17
weshaysshnaidm|rover, the next thing to try would be to clean the virthosts15:17
arxcruzchkumar|ruck: weshay tosky so, we have a bug in tempestconf15:17
arxcruzhttp://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest_container.sh.txt.gz15:17
weshaysshnaidm|rover, we could pin that job to a particular host15:17
arxcruzit's passing the tempest-deployer-input.conf that has the regionOne set, but on tempest.conf is not there http://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest/etc/tempest.conf.txt.gz15:18
chkumar|ruckarxcruz: https://github.com/openstack/tempest/blob/master/tempest/config.py#L13215:19
chkumar|ruckmay be it is taking the default one15:19
toskyarxcruz: so it's a regression? It used to work, and also it didn't show in other jobs15:20
toskywe merged patches until few hours ago15:20
arxcruztosky: checking other jobs to see if that is also the case in other jobs15:20
chkumar|ruckoh it is capital R15:20
arxcruztosky: so, it's only on scenario002, let's see why15:21
weshaymarios, https://review.openstack.org/#/c/583547/315:22
toskyarxcruz: what is the difference? Is it containerized and the others are not?15:23
toskyarxcruz: maybe the file is not mounted in the container correctly (see http://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-07-18_13_01_36 ? )15:23
arxcruztosky: checking right now15:23
rascaweshay, chkumar|ruck, so the error I get in rdocloud deploying master seems totally reproducible -> "Unable to find any of pip2, pip to use.  pip needs to be installed."15:23
rascaweshay, chkumar|ruck, is some sort of known issue?15:23
mariosweshay: checking15:23
toskyarxcruz: that's it - -v is for volumes, not files15:25
chkumar|ruckrasca: nope not seen15:25
toskyarxcruz: the file is not passed to the container15:25
arxcruztosky: do you know the option for file ?15:25
rascarlandy, maybe you can give me a hint here (look above please)15:25
chkumar|rucktosky: arxcruz but whitelist or skipfile are getting passed there15:25
toskyarxcruz: I don't think you can pass files15:26
toskychkumar|ruck: sure?15:26
rlandyrasca: using the reproducer or quickstart.sh directly?15:27
chkumar|rucktosky: http://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-07-18_13_01_3615:27
toskychkumar|ruck: how do you know that they are passed and working?15:27
weshaymarios, think you can just assume to nuke ceph from orbit15:27
rascarlandy, quickstart at the moment15:27
rlandywe merged a patch yesterday that weshay wrote to expose pip errors15:27
chkumar|rucktosky: check test results15:27
toskychkumar|ruck: no - that's the command line which says that you are passing them, it does not mean that it works15:27
rlandylooking for it15:27
mariosweshay: ack locking on15:27
rascarlandy, consider about the reproducer that I'm running this locally on my rdocloud tenant, so how can I obtain a reproducer?15:28
rlandyrasca: you can take one from another job and edit the changes to incorporate15:29
rascarlandy, I did it once but I think that by this time I'll need to change it again15:29
rascarlandy, can you give me a link as a reference?15:29
toskychkumar|ruck, arxcruz: but anyway, regardless of whether the files are passed or not, how is it working if the region is incorrect?15:30
rlandyrasca: wget this file: https://logs.rdoproject.org/84/581484/6/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0118a75/logs/reproducer-quickstart.sh15:30
rlandy^^ fs00115:30
rlandyyou can change that15:30
rlandy${TOCI_JOBTYPE:="ovb-3ctlr_1comp-featureset001"}15:30
arxcruztosky: chkumar|ruck give me a few minutes, i'm debugging15:31
rlandy: ${ZUUL_CHANGES:="openstack/tripleo-quickstart:master:refs/changes/84/581484/6"}15:31
rlandy^^ edit that15:31
rlandywill give you an ovb job15:31
*** jaganathan has quit IRC15:31
rascarlandy, trying it at once thanks15:34
mariosweshay: https://review.openstack.org/#/c/583547/4/scripts/tripleo.sh15:37
mariosgf15:39
sshnaidm|roverweshay, look much more green now15:40
weshaysshnaidm|rover, I can't forget but we need to build jobs w/ centos fasttrack15:45
weshayto help sniff out issues w/ minor releases15:45
weshayI'll get that going15:45
sshnaidm|roverweshay, what is fasttrack15:46
sshnaidm|rover?15:46
weshaysshnaidm|rover, they start to drop in centos 7.6 updates15:46
weshayso we can flag issues before another minor update of cenots15:47
weshaycentos15:47
* weshay has a card on it some where15:47
rascarlandy, just one last clarification, TOCI_JOBTYPE is not clear to me15:47
sshnaidm|roverweshay, is it about testing new centos release or just issues?15:47
rascarlandy, the featureset part, yes, it is15:47
rascarlandy, but the previous, not15:47
weshaysshnaidm|rover, fasttrack is only about the $next release15:47
arxcruzchkumar|ruck: how can i run tempest and get into the docker container?15:47
weshaysshnaidm|rover, so imho.. master jobs only..15:48
rascarlandy, take : ${TOCI_JOBTYPE:="ovb-3ctlr_1comp-featureset041"} how am I supposed to change "ovb-3ctlr_1comp"?15:48
weshayin a weekly or something pipeline15:48
sshnaidm|roverweshay, but these changes are introduced in rhel firstly, isn't it?15:48
weshaysshnaidm|rover, of course15:48
weshaysshnaidm|rover, we don't have any rdo on rhel jobs atm15:48
sshnaidm|roverweshay, and before that they are introduced into fedora afaik15:48
weshaynot until we get them going in the internal sf15:48
rlandyrasca: yep - needs to change15:48
weshaysshnaidm|rover, and the sky is blue :)15:49
rlandybut we have done a lot of refactoring15:49
rlandyso you may hit other problems15:49
sshnaidm|roverweshay, so maybe worth to catch them in fedora, sooner is better15:49
weshaysshnaidm|rover, tripleo doesn't support fedora15:49
sshnaidm|roverweshay, aren't we gonna test it on python3?15:50
sshnaidm|roverweshay, on fedora raw15:50
sshnaidm|roverweshay, iirc15:51
rascarlandy, yes but my problem is that I don't know what to change!15:51
chkumar|ruckarxcruz: http://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest-setup.sh15:51
arxcruzchkumar|ruck: but it will execute tempest as well15:51
sshnaidm|roverweshay, well, anyway, if we have these repos anywhere, worth to test them of course, need a new release file15:52
arxcruzi want to stay inside the docker container to check some settings15:52
arxcruzchkumar|ruck: i remember you did something15:52
chkumar|ruckarxcruz: in this script ttp://paste.openstack.org/show/726211/15:52
chkumar|ruckhttp://paste.openstack.org/show/726211/15:52
chkumar|rucks/-i/-it and replace last script part15:53
arxcruzchkumar|ruck: k15:53
*** agopi has left #oooq15:53
*** agopi has joined #oooq15:53
*** links has quit IRC15:55
*** jfrancoa has quit IRC15:56
chkumar|ruckarxcruz: does it worked?15:58
*** gkadam is now known as gkadam-afk15:59
arxcruzchkumar|ruck: no, it's stucked16:01
*** bogdando has quit IRC16:04
chkumar|ruckarxcruz: can I get into that system?16:06
*** zoli is now known as zoli|PTO16:06
*** zoli|PTO is now known as zoli16:06
arxcruzchkumar|ruck: your keys ?16:06
*** saneax has quit IRC16:07
chkumar|ruckarxcruz: http://paste.openstack.org/show/726213/16:07
rlandyrasca: 41 is your fs?16:09
rascarlandy, yes it is16:09
rlandy{TOCI_JOBTYPE:="ovb-3ctlr_1comp-featureset041"} is correct then16:10
rlandyif you're running on ovb16:11
rlandyyou are correct16:11
*** sshnaidm|rover is now known as sshnaidm|bbl16:12
*** udesale_ has quit IRC16:13
*** udesale has joined #oooq16:13
rascarlandy, yeah I get to the same conclusion :) I'm completing the installation on the undercloud now16:16
rlandyrasca: apologies I misunderstood your original question16:17
rlandyhappy to hear you are moving along16:17
*** udesale has quit IRC16:18
*** gkadam-afk is now known as gkadam16:23
hubbotFAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (1 more message)16:29
*** florianf has quit IRC16:31
rfolcodoes anybody know why containers-multinode is timing out ?16:35
rfolcochkumar|ruck, ^16:36
chkumar|ruckrfolco: https://bugs.launchpad.net/tripleo/+bug/178188816:37
openstackLaunchpad bug 1781888 in tripleo "[stable/queens]tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades job timing out on stable/queens noop jobs " [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)16:37
chkumar|ruckyou mean this one?16:37
chkumar|ruckrfolco: https://bugs.launchpad.net/tripleo/+bug/178210216:38
openstackLaunchpad bug 1782102 in tripleo "[master] Multinode periodic RDO promotion jobs are timing out on FS035, 001, 002" [Critical,Triaged]16:38
rfolcochkumar|ruck, http://logs.openstack.org/22/583022/2/gate/tripleo-ci-centos-7-containers-multinode/20bde14/16:38
chkumar|ruckrfolco: it is a known issue16:39
chkumar|ruckrfolco: one min16:39
chkumar|ruckrfolco: https://bugs.launchpad.net/tripleo/+bug/178187116:40
openstackLaunchpad bug 1781871 in tripleo "RDO cloud jobs failing with SSH Error: data could not be sent to remote host \"38.145.32.100\". Make sure this host can be reached over ssh" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)16:40
chkumar|ruckrfolco: paul is working on that16:40
chkumar|ruckit tracks for both rdo and openstack16:40
rfolcochkumar|ruck, thanks16:46
*** myoung is now known as myoung|lunch16:50
* chkumar|ruck out now16:55
*** atoth has quit IRC16:58
*** atoth has joined #oooq16:58
*** trown is now known as trown|lunch17:05
weshaymarios, doesn't the bootstrap role have to be in tripleo-ci?17:15
weshayhttps://review.openstack.org/#/c/583195/2/playbooks/tripleo-ci/run-v3.yaml17:15
weshaydoes the pre playbook have access to the tqe roles at that point?17:16
*** dtantsur is now known as dtantsur|afk17:26
*** tesseract has quit IRC17:35
*** gkadam has quit IRC17:38
*** amoralej is now known as amoralej|off17:39
*** atoth has quit IRC17:39
*** atoth has joined #oooq17:40
weshayrlandy, so you just have the job up.. browbeat is not integrated yet right?17:45
rlandyweshay: I missed changing https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L13217:45
rlandyI changed baremetal-full-deploy.yml17:46
rlandyI need to add another patch to trigger that playbook17:46
rlandyand some way to not run it if var is not set17:46
rascarlandy, weshay, so the job is in the deploy overcloud phase, which is really good. I'd say that we can consider the patches good.17:46
rlandyrasca: good news17:47
rascarlandy, about baremetal-full-deploy.yml remember that inside one of the pending reviews there's a modification on that file17:47
rlandyweshay: kind of confusing with this playbook override17:47
rlandy " $QUICKSTART_SH_JOBS " =~ " $TOCI_JOBTYPE "17:47
rlandyneed to check that condition17:47
rascarlandy, but in any case I would like to have a clearer idea about how the reproducer thing works17:47
rlandyrasca: we can go through that tomorrow17:48
rascarlandy, yes, please it would be great.17:48
*** jtomasek has quit IRC17:50
*** dsneddon has quit IRC17:51
rlandyweshay: adding another patch and will rerun17:52
rfolcorlandy,  " $QUICKSTART_SH_JOBS " =~ " $TOCI_JOBTYPE --> I replace this with 2 simple ifs... one for ovb, another for multinode... https://review.openstack.org/#/c/582385/6/playbooks/tripleo-ci/templates/toci_gate_test.sh.j217:56
rfolcoas part of toci_jobtype replacement17:57
rlandyrfolco: sounds good - working with what is there atm17:57
rlandyanyone know if it is possible to block include playbooks?18:04
weshayrlandy, weird.. now power on is working.. but introspection still times.. just fyi'ing / complaining not looking for help18:07
rlandyweshay: while we are complaining ... the old way of including playbooks is very wasteful18:08
weshay?18:08
rlandyall things considered, would be way easier to add an extra playbook to the run on fs05318:08
rlandyat some point18:08
*** myoung|lunch is now known as myoung18:09
rlandyI need to conditionally include a playbook in toci_gate_test18:09
weshayrlandy,  you mean specify the extra playbook in the fs config?18:09
rlandythe playbook specification is set out in toci_gate_test18:10
rlandyso ... for the moment ...18:10
rlandyI am looking if it is possible to block include playbooks18:10
weshayrlandy, you mean adding another section like.. https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L17318:11
weshayrlandy, browbeat)18:11
weshay?18:11
rlandyno - I was going to always include it18:12
myoungI've finally completed excavating email/trello/things from the weeks away.  With remaining time this sprint per guidance would like to jump in and help QA cards.  Are there specific cards that need more QA attention more than others at this point?18:12
rlandyand block: when: the playbook itself18:12
weshayrlandy, and block?18:13
rlandyblock:18:13
rlandywhen:18:13
rlandy - include18:13
rlandybut I don;t think that is possible with playbooks18:14
rlandyjust tasks18:14
weshayrlandy, for JOB_TYPE_PART in $(sed 's/-/ /g' <<< "${TOCI_JOBTYPE:-}") ; do18:14
weshayso currently.. as things stand the old way18:14
rlandyyaeh - but I don't want a whole section18:14
weshayrlandy, if browbeat is in the job_type string..18:14
weshayyou can case on it18:14
rlandyjust to checf or fs05318:14
weshayrlandy, hrm..18:14
rlandyand then add another playbook18:14
rlandywhich I can do18:14
rlandyI think18:14
weshayrlandy, or you could add some logic in the ovb section18:15
weshayto check the fs file18:16
weshayas well18:16
weshayand set the playbooks there18:16
weshaythis is the problem with software18:16
weshaythere are multiple ways of doing things18:16
weshay always18:16
weshayrlandy, so forward looking18:16
rlandythings will be much better18:16
weshayrlandy, what happens when we need n+1 browbeat jobs18:16
rlandywith the new configuration18:17
rlandyweshay: if we want more browbeat jobs, then we have to do this:18:17
rlandyalways include the browbeat playbook for ovb18:17
rlandyand switch case within the playbook18:18
rlandyas I was trying to do at the start18:18
rlandyto go back to my original question then ...18:19
rlandyhow does one use blocks to include playbooks?18:19
rlandyidk if that is possible18:19
*** dsneddon has joined #oooq18:28
weshayrlandy, bah18:28
rlandyhttps://paste.fedoraproject.org/paste/kNPAcCI5MVozPqdnX9lMuw18:28
rlandyweshay: ^^ is that legitimate ansible18:28
rlandyfor a playbook??18:28
* rlandy hopes against the odds18:28
rlandyhack, hack, hack18:29
hubbotFAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (1 more message)18:29
rlandyI could just include when: on every line18:30
weshayrlandy, bah18:30
weshaywhy18:30
weshaywhy18:30
weshaywhy18:30
rlandywhy do I want to do that?18:30
* weshay is debating between18:30
weshayrlandy, let's blue18:31
weshaysorry18:31
rlandyk - joining18:31
*** atoth has quit IRC18:39
weshayhttps://review.openstack.org/58370018:40
weshayrlandy, ^18:40
Tenguweshay: heya! did you see my comment on your review for the validations in CI?18:56
weshayaye19:12
weshayhttps://review.openstack.org/#/c/583275/19:13
weshayI swear19:20
weshay2018-07-18 16:33:50 |                 "fatal: [localhost]: FAILED! => {\"changed\": true, \"cmd\": [\"ntpdate\", \"-u\", \"pool.ntp.org\"], \"delta\": \"0:00:08.711899\", \"end\": \"2018-07-18 16:33:47.729579\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2018-07-18 16:33:39.017680\", \"stderr\": \"18 Jul 16:33:47 ntpdate[6200]: no server suitable for synchronization found\", \"stderr_lines\": [\"18 Jul 16:33:47 ntpdate[6200]:19:20
weshayno server suitable for synchronization found\"], \"stdout\": \"\", \"stdout_lines\": []}",19:20
weshayI'm going to burn jenkins down19:20
myoungweshay: got a link to that job?  will add to the IT ticket I opened last night19:39
weshayhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/19:40
weshayvs.19:41
weshayhttp://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/environments/oooq-internal.yml19:41
weshaymyoung, I want to try this w/o the ci-script though19:41
*** atoth has joined #oooq19:43
* myoung looks19:43
weshayhttps://code.engineering.redhat.com/gerrit/#/c/144346/19:45
rlandyagpoi: hi - how hard is it to get a change merged in browbeat?19:50
agopirlandy,19:51
agopieasy enough19:51
agopirook,19:51
agopiis it this one rlandy ?19:51
agopihttps://review.openstack.org/#/c/583581/19:51
rlandyI need to edit it - sec19:51
agopiokay, please ping rook when ready to be merged.19:52
rookagopi: one sec.19:52
agopiokay rook19:52
rlandyagopi: rook: https://review.openstack.org/583581 Add conditional to browbeat minimal test19:53
rlandyI may need to undo that at some point but it would help if that was in atm19:54
rookagopi: load that page :)19:54
agopiaww19:54
agopirook++19:54
hubbotagopi: rook's karma is now 119:54
agopi<319:54
rookagopi++19:54
hubbotrook: agopi's karma is now 119:54
*** atoth has quit IRC19:59
*** atoth has joined #oooq19:59
weshaymyoung, you can close the ticket20:02
weshayit's something in our tooling20:02
weshayhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/tripleo-environments/config/environments/oooq-internal.yml/*view*/20:02
myoungweshay: i've been backtracking thru the past few days of deltas, why don't they match?  was about to hop on that node20:02
rlandyrook: agopi: we are including the minimal playbook in the list of default CI playbooks, hence this review will protect other featuresets from kicking browbeat: https://review.openstack.org/#/c/583581/20:03
myoungsagi's changes are not reflected, but it looks from the sha that we're pulling down tripleo-env at the right revision to include it20:03
weshaymyoung, vs.. https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/20:03
weshaymyoung, is there any place where we pin tripleo-environments?20:03
rlandyty20:03
agopioh okay understood rlandy, and i understand this makes easier to kick browbeat off on other tests too right?20:03
myoungnot that i'm aware of...or at least that i recall.  looking...20:04
myoungor do we have a problem with workspace clean actually cleaning?20:04
* myoung looks20:04
rlandyagopi: ack20:05
rlandymore generic20:05
agopisounds good rlandy20:06
myoungweshay: /ws/tripleo-environments/config/environments/oooq-internal.yml is the one from git, and if jenkins is to be trusted it's wonky that ws/config/environments/oooq-internal.yml (the one copied by unpacking of the egg) is stale20:06
myoungcan i hope on that node?  what's the timestamp delta?  seems like the old oooq-internal in the destination location isn't being overridden20:07
myounghop*20:07
myoungweshay: ahh you just wiped it ;)20:09
weshayI did previously as well20:09
weshaybut this time disconnected the slave20:09
myoungya i saw, i'm on it too20:10
* myoung won't change anything...just poking it20:10
agopirlandy, not sure how it happened but if you checkc https://review.openstack.org/#/c/583717/1/ansible/oooq/browbeat-minimal.yml it only shows your comment as the diff20:10
weshaymyoung, https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/20:10
weshaystill a problem20:10
agopiit shouldn't as your change hasn't been merged yet20:10
agopihttps://github.com/openstack/browbeat/blob/master/ansible/oooq/browbeat-minimal.yml20:10
myoungwtf where is that coming from20:10
agopior am i missing something20:11
myoungweshay: /me is digging thru https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/113/console20:11
rlandyagopi, that is correct20:11
rlandybuilt off the change I asked to merge20:11
agopiokay gotcha20:12
*** sshnaidm|bbl is now known as sshnaidm|rover20:18
rfolcosshnaidm|rover, any ideas why gate containers-multinode fails? https://review.openstack.org/#/c/583022/20:21
rfolcotiming out :(20:21
sshnaidm|roverrfolco, yes, known infra problem, recheck20:22
rfolcosshnaidm|rover, recheck won't run gate again.20:22
sshnaidm|roverrfolco, what do you mean?20:22
rfolcosshnaidm|rover, nm, will recheck20:23
rfolcothx20:24
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo- (1 more message)20:30
myoungweshay: i think it's coming from the pip cache somehow...debugging it20:30
myoungweshay: i think would make sense to obliterate /home/rhos-ci/.cache/ and see what happens20:31
weshayya.. just did that on https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/computer/rdo-manager-slave_rdo-ci-fx2-01-s2/log20:31
weshayhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/20:33
weshaymyoung, worked20:33
weshaymyoung, so maybe the ticket to support is.. what is the proper way to disable jenkins pip cache per slave20:34
myounghrm...wonder if the --no-cache-dir added https://github.com/openstack/tripleo-quickstart/commit/df55d3908d053b5b3a33d0dfa22ed7cbe65ea1b6 a couple weeks ago would make this work better....just run with OPT_CLEAN20:35
weshaythat's a straight up jenkins bug20:39
weshaymyoung, oh in the full-deploy.sh script20:39
myounghrm...i have a theory20:39
weshayadding --clean20:40
weshayya20:40
myoungi wonder if the pip running in the context of quickstart.sh isn't honoring the change we put in to handle the concurrency pip bug, so it's looking in the OLD pip cache location (the default one) ~/.cache/pip, instead of the location overridden by the jenkins global env variable that makes it ~/.cache/{executor_number}/pip (in this case ~/.cache/0/pip20:41
weshaymyoung, in fact.. --bootstrap should be replaced w/ --clean20:41
myoungthat would make it find an old version of the file20:41
weshay--bootstrap just installs the rpms20:41
weshayhttps://review.openstack.org/58373620:44
myounghttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/115/injectedEnvVars/ has XDG_CACHE_HOME=/home/rhos-ci/.cache/$EXECUTOR_NUMBER (correct), but on the slave i was on if somehow quickstart.sh's pip install wasn't seeing/using that override, it would abs()-ly find the older verion of oooq-internal.yml20:44
myoungweshay: that patch makes sense, however in this case (internal schtuff) I think the egg is being unpacked here: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/ci-scripts/prep-internal-rhel.sh?id=39976b947b01b1fb92e14a91a347aea50711f12c#n3920:45
*** gkadam has joined #oooq20:45
weshaymyoung, ya.. internal def sets it up first20:46
myoungkind of want an option to have quickstart.sh bletch out all the env vars (with a --verbose flag) it knows about - to validate the XDG_CACHE_HOME theory20:47
myoungi guess we could nuke the old cache location and see if it populates it on the offline slave.  care if I give it a whack?20:47
myoung(it shoulnd't...it's set env wide)20:47
weshaymyoung, sure20:48
*** agopi is now known as agopi|brb20:53
*** agopi|brb has quit IRC20:57
*** trown|lunch is now known as trown|outtypewww20:59
*** gkadam has quit IRC21:04
myoungweshay: ruled out that theory.  it's not ignoring XDG_CACHE_HOME, is using the right pip cache, but it does appear to be pulling it from the cache and picking up the older version.  nuking the cache or --clean should be the fix.21:28
* myoung has to dash to fetch a wee one21:29
weshayhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/21:29
weshayshould work21:29
* myoung looks quickly21:29
* myoung nods21:30
myoungweshay: the --force-reinstall just forces a uninstall then reinstall (from cache).  we need the --no-cache-dir 4 sure.21:31
myoungweshay: heh pip -vvv is spammy but made that clear21:31
myoungso "quickstart.sh --clean $theRest FTW"21:32
*** myoung is now known as myoung|afk21:32
ssbarnea1pip cache isolation workaround works only in some cases, downstream i had several failures due to tox.ini not passing the custom cache to venvs. i did check the ticket but i got the impression that they already fix it by now in pip.22:10
ssbarnea1but also cleaning the cache every week was another way to lower the risks.22:11
panda|offrfolco: merged22:19
*** chem has quit IRC22:26
hubbotFAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci- (1 more message)22:30
*** holser__ has quit IRC22:39
*** brault has quit IRC22:46
*** rlandy has quit IRC22:57
*** honza has quit IRC22:59
*** myoung|afk has quit IRC23:03
*** faceman has quit IRC23:03
*** myoung has joined #oooq23:03
*** faceman has joined #oooq23:05
*** tosky has quit IRC23:20

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!