Thursday, 2018-07-12

*** chem has quit IRC00:47
hubbot`FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @  (1 more message)01:00
*** agopi has joined #oooq02:11
hubbot`FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @  (1 more message)03:01
*** skramaja has joined #oooq03:02
*** brault has joined #oooq03:09
*** brault has quit IRC03:14
*** sanjayu has joined #oooq03:50
*** udesale has joined #oooq03:55
*** ykarel has joined #oooq04:44
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @  (1 more message)05:01
*** links has joined #oooq05:03
*** ratailor has joined #oooq05:20
*** sanjayu has quit IRC05:33
*** quiquell|off is now known as quiquell05:42
*** bogdando has joined #oooq05:55
*** d0ugal_ has joined #oooq05:55
*** chandankumar is now known as chkumar|ruck05:56
*** d0ugal has quit IRC05:56
chkumar|ruck%gatestatus05:59
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @  (1 more message)05:59
*** quiquell has quit IRC06:01
*** quiquell has joined #oooq06:01
*** jtomasek has joined #oooq06:18
*** sanjayu_ has joined #oooq06:36
*** jfrancoa has joined #oooq06:40
*** matbu has quit IRC06:41
*** matbu has joined #oooq06:41
*** chem has joined #oooq06:42
*** skramaja has quit IRC06:57
*** sanjayu_ is now known as saneax07:00
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @  (1 more message)07:01
*** amoralej|off is now known as amoralej07:02
*** gkadam has joined #oooq07:14
*** kopecmartin has joined #oooq07:23
*** holser_ has joined #oooq07:29
ssbarneasimple fix https://review.openstack.org/#/c/571180/08:00
*** d0ugal_ has quit IRC08:03
*** d0ugal has joined #oooq08:03
*** d0ugal has quit IRC08:03
*** d0ugal has joined #oooq08:03
*** tosky has joined #oooq08:03
*** ccamacho has joined #oooq08:17
*** links has quit IRC08:24
*** florianf has joined #oooq08:26
*** links has joined #oooq08:26
*** sshnaidm|off has quit IRC08:27
*** saneax has quit IRC08:27
ssbarneaa new tox bug that is likely to affect lots of us https://github.com/tox-dev/tox/issues/899#issuecomment-40442784508:29
*** saneax has joined #oooq08:32
quiquellssbarnea: it will appear at CI, don't know what tox version we are running on08:35
ssbarneaalready found it on ara builds yesterday, expect more to appear on CI.08:35
ssbarneait was introduced by very new tox release,  i hope they will take it down as it woudl minimize the "damage" :)08:36
*** tesseract has joined #oooq08:37
quiquellssbarnea: We will have to pin all the tox requirements.txt :-)08:37
quiquellfun08:37
ssbarneaeven worse I got a very weird behaviour on my machine: it attempted to test using py34 when the tox.ini was mentioning only pe35.08:37
arxcruzmarios: hey, can you +2+w https://review.openstack.org/#/c/580384/ ?08:38
ssbarneais even worse, tox is not in requirement.txt for obvious reasons.08:38
*** jaganathan has quit IRC08:38
*** jaganathan has joined #oooq08:39
quiquellssbarnea: Humm you are right08:40
mariosarxcruz: ack08:40
arxcruzmarios: thanks08:40
arxcruzchkumar|ruck: https://review.rdoproject.org/r/#/c/14554/ is it still needed, from amoralej comments08:41
chkumar|ruckarxcruz: nope08:42
chkumar|ruckarxcruz: I need to abandon all those patches08:42
chkumar|ruckarxcruz: I need to take a different route all together08:42
arxcruzchkumar|ruck: https://review.rdoproject.org/r/#/c/14553/ as well08:42
ssbarneaglat to hear the "-e VAR" works on docker, makes cleaner to passenv.08:43
chkumar|ruckarxcruz: all abandoned on rdo side08:44
arxcruzchkumar|ruck: thanks :)08:44
chkumar|ruckquiquell: seen this error http://paste.openstack.org/show/725664/ while creating env using reproducer script08:47
chkumar|ruck?08:47
*** sshnaidm has joined #oooq08:48
*** sshnaidm is now known as sshnaidm|rover08:48
quiquellchkumar|ruck: Could be a problem with your tenant08:48
quiquellchkumar|ruck: Or that RDO is out of capacity08:49
chkumar|ruckquiquell: https://github.com/openstack-infra/tripleo-ci/blob/master/toci_quickstart.sh#L96 failing at this part08:50
arxcruzmarios: done08:50
quiquellchkumar|ruck: Humm reproducer is broken, ENVIRONMENT is not set08:51
quiquellchkumar|ruck: find the TOCI_JOBTYPE env var at the top of the reproducer08:51
quiquellpaste it here08:51
chkumar|ruckquiquell: ${TOCI_JOBTYPE:="multinode-1ctlr-featureset033"}08:51
chkumar|ruckquiquell: http://logs.openstack.org/43/579843/13/experimental/tripleo-ci-centos-7-scenario009-multinode-oooq/31c1bec/logs/reproducer-quickstart.sh08:51
*** chem has quit IRC08:52
mariosthanks arxcruz revoted08:52
arxcruzsshnaidm|rover: can you take a look on https://review.openstack.org/#/c/580384/ when you have time ?08:53
sshnaidm|roverarxcruz, sure08:53
quiquellchkumar|ruck: How did you executed the reproducer ?08:54
quiquellchkumar|ruck: Did you sources env_vars_to_src.sh ?08:54
chkumar|ruckquiquell: so I found the problem08:55
chkumar|ruckquiquell: so i source env_vars_to_src.sh first08:55
chkumar|ruckthen installed tmux08:55
sshnaidm|roverarxcruz, don't you need to specify value when set "-e" for docker?08:55
chkumar|ruckthen runned the script that's why env is not exported08:55
arxcruzsshnaidm|rover: according panda|off no08:56
quiquellchkumar|ruck: Yep, you have to source after tmux08:56
quiquellchkumar|ruck: Let's see if we can put tmux on reproducer08:56
chkumar|ruckquiquell: I will propose a patch for tmux switch08:56
sshnaidm|roverarxcruz, and why?08:56
arxcruzsshnaidm|rover: https://review.openstack.org/#/c/580384/4/roles/validate-tempest/templates/run-tempest.sh.j2@4908:57
quiquellchkumar|ruck: btw, this is known ? http://logs.openstack.org/23/578023/5/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/3ce1249/logs/undercloud/home/zuul/tempest/tempest.html.gz09:00
hubbot`FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @  (1 more message)09:01
sshnaidm|roverarxcruz, oh, so you set pythonwarnings before?09:01
*** skramaja has joined #oooq09:01
*** saneax has quit IRC09:01
chkumar|ruckquiquell: yup09:01
arxcruzsshnaidm|rover: yes09:01
chkumar|ruckquiquell: it comes sometimes and sometimes not09:01
chkumar|ruckquiquell: https://bugs.launchpad.net/tripleo/+bug/177332509:02
openstackLaunchpad bug 1773325 in tripleo "tempest.api.object_storage.test_object_services is failing on scenario002" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)09:02
arxcruzsshnaidm|rover: since this script has some includes it's hard on the review to understand that :)09:02
sshnaidm|roverarxcruz, in which line do you set it?09:03
*** saneax has joined #oooq09:03
arxcruzsshnaidm|rover: when we source overcloudrc file, it's on the overcloudrc09:04
*** links has quit IRC09:10
*** links has joined #oooq09:12
*** panda|off is now known as panda09:20
pandaarxcruz: more according to https://docs.docker.com/engine/reference/run/#env-environment-variables than to me :)09:22
arxcruzsshnaidm|rover: ^09:22
*** kopecmartin is now known as mkopec|lunch09:32
mariosarxcruz: any idea about this tempest error we get on ci-centos-7-scenario002-multinode-oooq-container at https://review.openstack.org/#/c/578023/5 (see my latest comment from just now there) please09:52
quiquellpanda: Looks like cleanup dependencies works https://review.openstack.org/#/c/581668/09:56
quiquellDon't know if I have to test it with a patch at TQE to exercise all CI09:57
pandaquiquell: I think it's pretty straightforward patch09:59
quiquellpanda: ack09:59
sshnaidm|roverquiquell, hi10:10
sshnaidm|roverquiquell, I saw you talked with rdo folks about moving grafana, did you reach some conclusions..?10:11
quiquellsshnaidm|rover: Let's wait for software-factory.org to upgrade10:13
sshnaidm|roverok10:13
quiquellsshnaidm|rover: They don't support changes in grafana for commits yet10:13
quiquellsshnaidm|rover: sf at RDO is old10:13
sshnaidm|roverquiquell, and then they'll support grafyaml?10:13
quiquellsshnaidm|rover: Yep, but I am changing grafyaml, it misses a lot of features we need10:14
quiquellsshnaidm|rover: Also trying to port to it, we can do this while wait for sf upgrade10:14
sshnaidm|roverquiquell, where is its sources?10:14
sshnaidm|roverquiquell, yeah, I know they don't have influxdb support..10:15
quiquellsshnaidm|rover: openstack/grafyaml10:15
quiquellsshnaidm|rover: I was going to start with table panels10:15
sshnaidm|rovergreat10:15
quiquellsshnaidm|rover: very easy to add new stuff, also pabelenger is quick merged10:16
quiquellsshnaidm|rover: It' doesn't need 2 +2s10:16
quiquellsshnaidm|rover: Example or adding adhoc variables https://review.openstack.org/#/c/581368/10:16
sshnaidm|roverquiquell, nice10:18
sshnaidm|roverquiquell, do you have a list of features to implement?10:19
quiquellsshnaidm|rover: nop, just porting little by little and implementing the missing one10:19
quiquellsshnaidm|rover: Let me push what I have so far10:20
quiquellsshnaidm|rover: https://review.rdoproject.org/r/1480410:22
arxcruzmarios: concurrency :(10:23
*** mkopec|lunch is now known as kopecmartin10:27
quiquellsshnaidm|rover: Have concerns with grafyaml, it's going to be always a subset of the latest grafana10:27
*** kopecmartin has quit IRC10:28
quiquellsshnaidm|rover: And also I don't know how easy is to changes big dashboards with commits10:28
*** kopecmartin has joined #oooq10:28
*** florianf has quit IRC10:30
*** florianf has joined #oooq10:32
sshnaidm|roverquiquell, well, I don't know too.. maybe need something for testing10:34
sshnaidm|roverquiquell, like testing dashboard or kind of..10:34
*** udesale has quit IRC10:35
quiquellsshnaidm|rover: meanwhile we can store into yaml what we have now, to simplify editing10:37
quiquellsshnaidm|rover: It's good enough and it's not a subset10:37
quiquellsshnaidm|rover: testing dashboard... we can use some kind of GUI testing10:38
sshnaidm|roverquiquell, but we can't store in yaml everything we have now, right?10:39
quiquellsshnaidm|rover: Yep, and change grafana-export to read yaml or json10:40
quiquellsshnaidm|rover: Depending on the extension or a command line arg10:40
quiquellsshnaidm|rover: Also there are some stuff that we can clean tooo10:40
quiquellsshnaidm|rover: It's a poor man grafyaml but more flexible10:40
sshnaidm|roverquiquell, yeah10:41
sshnaidm|roverquiquell, so we need another instance to use grafyaml in it10:41
sshnaidm|roverand test patches10:42
quiquellsshnaidm|rover: Kind of, or we don't use grafyaml all together10:42
quiquellsshnaidm|rover: We are going to update their schema forever10:42
sshnaidm|roveryeah...10:43
quiquellsshnaidm|rover: Maybe it's better to have a custom grafana with all the plugins that we want10:44
quiquellsshnaidm|rover: And the version that we want, it doesn't requires to much of maintenance10:44
chkumar|rucksshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1780224 we are hitting again in master noop job scenario 110:54
openstackLaunchpad bug 1780224 in tripleo "ERROR configuring keystone_init_tasks in tripleo-ci-centos-7-scenario001-multinode-oooq-container job" [High,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)10:54
hubbot`FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @  (1 more message)11:01
quiquellsshnaidm|rover: What's the best way to store secrets at openstack ?11:15
sshnaidm|roverquiquell, in openstack..?11:15
*** atoth has joined #oooq11:17
quiquellsshnaidm|rover: Found how we do at tripleo-infra11:17
*** amoralej is now known as amoralej|lunch11:18
ssbarneaquick one https://review.openstack.org/#/c/581788/ please, already passed.11:20
sshnaidm|roverquiquell, you mean ansible vault?11:22
quiquellsshnaidm|rover: Humm was replicating what we do with promoter at ci-scripts/infra-setup/tenant_vars11:23
quiquellwill look into ansible vault11:23
sshnaidm|roverquiquell, we don't have vault yet in tripleo-infra11:23
sshnaidm|roverquiquell, all secrets are just deployed manually11:24
quiquellsshnaidm|rover: Then I will go to same as promoter11:24
quiquellok11:24
sshnaidm|roverquiquell, yeah, that's fine, the ansible-vault is the plan for future11:24
sshnaidm|roverssbarnea, it's not tested in jobs11:27
sshnaidm|roverssbarnea, otherwise it would fail on syntax error11:27
sshnaidm|roverssbarnea, this kind of patches we need to test locally unfortunately11:27
sshnaidm|roverssbarnea, I don't know how even ansible lint missed this, need to check it..11:28
pandamarios: do you know what's happening in http://logs.openstack.org/76/581376/5/check/tripleo-ci-centos-7-3nodes-multinode/b141c5e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-11_22_34_43 ?11:28
pandamarios: what's that file that is missing ?11:29
ssbarneaquiquell: exactly, I did had a good impression about ansilbe-linter, not to mention that I had the impression that the gate was testing the code,...or at least most of it.11:29
ssbarneathis kind of error should have being catched by either linter or the gate, but this time human did the best work :D11:29
mariospanda: no looks like a new one ... seems to be about the upgradeinit but not sure what is giving this error ? mistral?ansible? looking more11:30
pandamarios: thanks, 3 nodes is the only job missing to migrate at this point, standalone works11:31
sshnaidm|roverssbarnea, I'm sshnaidm|rover  :) I know it's confusing..11:32
sshnaidm|roverssbarnea, linter doesn't check this playbook: http://logs.openstack.org/88/581788/1/check/openstack-tox-linters/0b8f715/job-output.txt.gz#_2018-07-12_08_13_04_64693111:33
ssbarneano worry, before fixing the typo i will try to test if I can make the linter catch it11:33
mariospanda: trying to find where that came from still, but it cant be upgrade related anyway since there is no upgrade here in this job (though the error is indeed about the upgrade init file...)11:40
pandamarios: that's what I imagined.11:40
bogdandofolks, shall I dup these https://review.openstack.org/#/c/465047/5/roles/collect-logs/defaults/main.yml for tripleo-ci collect logs?11:43
bogdandoor is it enough to only change either place of two?11:43
bogdandoPTAL btw11:43
bogdandothe patch is wanted by upgrades team to diverge tht for undercloud vs overcloud and mixed upgrades11:44
sshnaidm|roverbogdando, is it used in CI? /home/*/undercloud_custom_tht_script.log11:45
bogdandosshnaidm|rover: it becomes in the patch11:46
sshnaidm|roverbogdando, it's very controversial patch11:47
sshnaidm|roverbogdando, firstly, please don't change overcloud_templates_path11:47
*** quiquell is now known as quiquell|lunch11:47
bogdandosshnaidm|rover: I believe it has to11:48
bogdandootherwise we have a mess un the tht script11:48
bogdandoin11:48
sshnaidm|roverbogdando, it's default and used so in all featuresets11:48
*** skramaja_ has joined #oooq11:49
sshnaidm|roverbogdando, that's fine if we need to override it in some of tasks11:49
bogdandolook folks, there is a real mess for mixed updates11:49
*** skramaja has quit IRC11:49
bogdandoyou can only install tht from rpm to /usr/share11:49
bogdandothen, you have to diverge for mixed undercloud vs overcloud upgrades, which technically may be using 4 versions11:49
*** skramaja_ is now known as skramaja11:50
bogdandopre-update UC, pre-update OC, post-update UC, post-update OOC11:50
sshnaidm|roverpanda, I believe there is a lot of design questions, can you look: https://review.openstack.org/#/c/46504711:50
bogdandoso we really should not mess in /usr/share IMO, but it's up to DFG upgrades how they would want thatpatch11:50
bogdandoso yeah, I can improve that default changing not  globally prolly11:51
sshnaidm|roverbogdando, who from upgrades is working on that?11:51
bogdandojfrancoa: ^^11:51
sshnaidm|roverjfrancoa, bogdando I think we need a proper problem definition before..11:51
bogdandoright, IIUC jfrancoa wanted exactly what I described - a smooth way to diverge t-h-t used for mixed upgrades11:52
pandaOMG this is huge.11:52
bogdandogiven that now we have not only overcloud deployed from it, but undercloud as well, and each has its upgrade path11:52
bogdandoso there is 2x2 matrix or so11:53
bogdandoI let upgrade folks to comment on that11:53
bogdandopanda: just cuz of huge shell scripts moved from role to another one ;)11:54
bogdandoto be shared for OC and Uc11:54
bogdandomarios: ^^11:54
*** agopi has quit IRC11:55
pandabogdando: so the problem is "I want to be able to inject two sets of custom THT in a way that I can test upgrades too" ?11:55
bogdandopanda: yes, as I can see the request11:55
*** udesale has joined #oooq11:57
*** saneax has quit IRC11:58
bogdandosshnaidm|rover: the problem definition https://bugs.launchpad.net/tripleo/+bug/178122711:58
openstackLaunchpad bug 1781227 in tripleo "Overcloud upgrade CI job doesn't load templates from /usr/share/openstack-tripleo-heat-templates" [High,In progress] - Assigned to Bogdan Dobrelya (bogdando)11:58
bogdandobasically, that's why I wanted to ditch that path11:59
bogdandoand rely on working dir11:59
mariosbogdando: which patch? reading back but agree on not messing with /usr/share for tht12:00
bogdandomarios: see https://bugs.launchpad.net/tripleo/+bug/178122712:00
openstackLaunchpad bug 1781227 in tripleo "Overcloud upgrade CI job doesn't load templates from /usr/share/openstack-tripleo-heat-templates" [High,In progress] - Assigned to Bogdan Dobrelya (bogdando)12:00
bogdandohttps://review.openstack.org/#/c/465047/12:00
mariospanda: re that multinode job. fwiw the error is coming from this https://github.com/openstack/tripleo-common/blob/master/tripleo_common/templates/deployments.yaml#L3 however i also see the file it complains about, is there at http://logs.openstack.org/76/581376/5/check/tripleo-ci-centos-7-3nodes-multinode/b141c5e/logs/undercloud/var/lib/mistral/d9b8772a-af39-427f-b9dd-6f53baa26ccc/ControllerApi/centos-7-ovh-bhs1-0000668818/UpgradeInitDeployment.g12:01
mariosbogdando: ack checking12:03
mariosbogdando: ack i'll have a look at the review its a bigun ;) but probably tomorrow12:04
bogdandomarios: I think that patch may also help for https://trello.com/c/vzjMRcHt/28-upgrades-ci-with-the-standalone-installer12:08
bogdandoif just thinking of undercloud_* as standalone_*12:08
bogdandoack, waiting for feedback tomorrow then :)12:09
mariosbogdando: cool thanks for pointer to the card12:09
*** trown|outtypewww is now known as trown12:10
*** ratailor has quit IRC12:11
*** quiquell|lunch is now known as quiquell12:12
pandamarios: the missing file ends with a 612:15
pandamarios: centos-7-ovh-bhs1-0000668818 vs centos-7-ovh-bhs1-000066881612:16
mariospanda: ah weird12:16
pandaprobably the file for the 3rd node is missing12:16
mariospanda: well spotted ;)12:17
*** amoralej|lunch is now known as amoralej12:20
*** saneax has joined #oooq12:20
*** atoth has quit IRC12:22
*** ykarel is now known as ykarel|afk12:25
*** ykarel|afk has quit IRC12:30
*** apetrich has quit IRC12:32
jfrancoabogdando: sshnaidm|rover: I just came from lunch12:33
* jfrancoa having a look at https://review.openstack.org/#/c/46504712:34
bogdandoyes, please go for it, I've added enough comments to have it looking not so big change in fact12:34
bogdandothe only change is basically for overcloud_templates_path defaults, but that's totally discussible12:35
bogdandoother changes should not be breaking for any release (hopefully) or config matrix :)12:35
*** skramaja has quit IRC12:37
*** rlandy has joined #oooq12:38
*** ykarel|afk has joined #oooq12:38
weshaysshnaidm|rover, chkumar|ruck upload is still failing I take it?12:44
sshnaidm|roverweshay, yeah, last time it failed, fixed by tristan: https://review.rdoproject.org/r/#/c/14799/12:45
sshnaidm|roverweshay, looking now12:45
sshnaidm|roverweshay, I have images, so I'm ready to push them for master..12:45
*** agopi has joined #oooq12:46
weshaysshnaidm|rover, perfect! thank you12:46
weshaychkumar|ruck, I was looking for the trello card I created re: tempest w/o a skip list12:46
weshaycouldn't find it :(12:46
weshaytrello--12:47
rlandypanda: hi - I need to add a sprint card for the work I am doing with the perf team12:47
sshnaidm|roverweshay, let's wait 30 min and wee what happens with new upload job, if it fails - I'll push12:47
weshaysshnaidm|rover, aye12:47
chkumar|ruckweshay: https://trello.com/c/sIF5LG9K/857-add-24hr-periodic-tempest-job-w-o-skip-list12:48
rlandysshnaidm|rover: hi ... wrt zuulv3 migration in rdocloud, have you come across this,12:48
rlandy  The project "git.openstack.org/openstack/browbeat" was not found.  All12:48
rlandy  projects referenced within a Zuul configuration must first be added to12:48
rlandy  the main configuration file by the Zuul administrator.12:48
weshaypanda, anything we need to cover before the scrum?12:48
weshaychkumar|ruck, thanks12:49
rlandynot on that repo exactly but with referenced projects12:49
pandarlandy: ok12:49
sshnaidm|roverrlandy, hmm... which job is it?12:49
weshayssbarnea, doing well?12:49
rlandysshnaidm|rover: I am adding a new project in our testing ... https://review.rdoproject.org/r/#/c/14772/12:50
rlandyclearly I missed a step12:50
rlandyI guess I don;t know what the main configuration file is12:51
ssbarneaweshay: yep well, regot acces to RHU labs and also learnign about how to bring oooq changes to meargeable state.12:51
*** atoth has joined #oooq12:52
ssbarneaalso discovered an interesting tox change this morning.12:52
rlandyykarel|afk: ^^ do you know where I find this main configuration file?12:53
rlandypanda: do I just add the card?12:53
ykarel|afkrlandy, yes, let me grab a link12:54
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/projects.yaml?12:54
*** ykarel|afk is now known as ykarel12:54
ykarelrlandy, you need to submit a patch like:- https://softwarefactory-project.io/r/#/c/12800/12:55
ykarelin softwarefactory config12:55
pandarlandy: yes, we can discuss it in the meeteing12:56
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @  (1 more message)13:01
pandarlandy: meeeting ?13:01
pandassbarnea: meeting ?13:01
pandarfolco: meeting ?13:01
ssbarneaoops, sorry, calendar didn't popup.13:02
*** udesale has quit IRC13:15
rlandyykarel: stupid question - but where is that repo?13:19
chkumar|ruckweshay: sshnaidm|rover what to od with scenario002 job having swift tempest tests failing with 412 error code while creating swift container tests, there is nothing in swift container logs also?13:20
ykarelrlandy, git clone https://softwarefactory-project.io/r/config13:20
sshnaidm|roverchkumar|ruck, does it fail always?13:20
rlandyykarel: ty :)13:20
chkumar|rucksshnaidm|rover: it fails one time, then do recheck it passes, for most of the patches, http://zuul.openstack.org/builds.html?job_name=tripleo-ci-centos-7-scenario002-multinode-oooq-container13:21
chkumar|ruckthe job is more than 8k sec, it has failed with same reason13:22
chkumar|ruckit is started poping up from yesterday13:22
chkumar|rucksshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1773325 , I am not sure what to do in this case13:23
openstackLaunchpad bug 1773325 in tripleo "tempest.api.object_storage.test_object_services is failing on scenario002" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)13:23
*** anande has joined #oooq13:25
chkumar|rucksshnaidm|rover: ykarel post failure with periodic job https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/60b1009/job-output.txt.gz#_2018-07-12_13_07_44_40292913:26
ykarelchkumar|ruck, https://review.rdoproject.org/r/#/c/1480713:32
chkumar|ruckykarel: I think we need to reschedule the periodic job then after merging this?13:33
ykarelchkumar|ruck, if all jobs pass sshnaidm|rover will promote manually13:34
ykarelsshnaidm|rover, from where u will get the image? will u build that?13:36
sshnaidm|roverykarel, yeah, building locally13:36
ykarelsshnaidm|rover, okk13:36
* chkumar|ruck headed home13:37
sshnaidm|roverchkumar|ruck, I'm off tomorrow, I think weshay will cover me13:40
weshayaye13:40
rlandyykarel: do I need to be added to that project? rlandy@softwarefactory-project.io: Permission denied13:41
weshaysshnaidm|rover, https://images.rdoproject.org/master/rdo_trunk/e0c5c24e3bca25349894b8055589d88a16f4b894_0d0d3bed/13:41
weshaysshnaidm|rover, is that manual or from the job?13:42
ykarelrlandy, your pub keys to gerrit https://softwarefactory-project.io/r/#/settings/ssh-keys13:42
sshnaidm|roverweshay, being uploaded now13:43
sshnaidm|roverweshay, from my laptop13:43
weshayk13:43
weshaysshnaidm|rover, please hand off anything I need to fix the upload issue before you go :)13:44
sshnaidm|roverweshay, which issue?13:44
weshayimage upload13:44
sshnaidm|roverweshay, ??13:44
weshayassuming it did not work again.. or did the latest patch not land in time13:44
sshnaidm|roverweshay, I've fixed it13:45
weshaysshnaidm|rover, ok.. so it's fixed, but have not yet see a job upload images successfully, should happen $next run13:45
weshayis that correct?13:45
sshnaidm|roverweshay, yeah, sure, we'll see13:45
weshayk. .thanks13:46
sshnaidm|roverweshay, oh, sorry, my fault, read you message incorrectly13:46
rlandyykarel: https://softwarefactory-project.io/r/#/c/13019/13:47
sshnaidm|roverweshay, stopped for now promoter service..13:47
weshaysshnaidm|rover, ?13:47
weshayI'm not following..13:47
ykarelrlandy, need to ask tristanC or paul to get it merged13:48
rlandyykarel: ok - will do - thanks13:48
sshnaidm|roverweshay, I stopped promotion service while images are being uploaded13:48
weshayok13:48
weshaysshnaidm|rover, do we have images for queens and master?13:48
sshnaidm|roverweshay, for master, queens is in the progress13:48
weshayrock13:49
*** links has quit IRC13:50
ykarelrlandy, also you need to add job definition to rdo-jobs and project config to config, like https://review.rdoproject.org/r/#/c/14516/ and https://review.rdoproject.org/r/#/c/14508/13:51
ykarelconfig/zuul and config/jobs directories were for zuul v2, which are not used now13:53
ykarelrefering https://review.rdoproject.org/r/#/c/1477213:53
radezrlandy: the env template you gave the me other day, do you have one that deploys ironic in the overcloud?13:53
*** anande has quit IRC13:58
rlandyradez: I don't but that's a good idea13:59
rlandyweshay: ^^ is this something we want to add a featureset for?13:59
radezrlandy: ok, I'm actually working on that so i'll try and pass it back to you when I get it working13:59
weshayrlandy, for deploying ironic in the overcloud?14:04
*** apetrich has joined #oooq14:05
weshayrlandy, I guess that should show up in the ci/environment scenarios14:05
rlandyweshay: yeah - I don;t think we have/test that14:08
radezI'll get back to you what I come up with incase you want to add a job for it14:09
rfolcopanda, would you please w+ https://review.openstack.org/#/c/581814/14:14
pandarfolco: you're late14:15
rfolcomeh14:15
pandarfolco: needed to chat ?14:15
*** ccamacho has quit IRC14:16
arxcruzsshnaidm|rover: panda rlandy https://review.openstack.org/#/c/570884/14:16
*** quiquell is now known as quiquell|off14:16
rfolcopanda, we can chat tomorrow, I'll play a bit more with the code. My question was about 'periodic' jobs... and dryrun cases....14:16
arxcruzwhen you guys have time14:16
arxcruzweshay: ^14:16
pandarfolco: ok14:17
weshayarxcruz, chkumar|ruck https://review.openstack.org/#/c/574731/14:20
*** ccamacho has joined #oooq14:21
weshaysshnaidm|rover, on Sunday / Monday can you please work w/ bogdando on https://review.openstack.org/#/c/465047/14:23
bogdandoon Sunday? :O14:24
weshaysshnaidm|rover, help him bring the design to the team14:24
sshnaidm|roverweshay, if Ci will be ok14:24
bogdando:D14:24
weshaysshnaidm|rover, right.. as time permits of course14:24
weshaybogdando, this is a fairly large patch14:24
bogdandodo you guys work on Sundays?.. og gush14:24
sshnaidm|roverweshay, I think it could be just review session like we did sometimes14:24
* weshay reads through the bugs14:24
sshnaidm|roverbogdando, yeah,  who breaks CI remains to work in Sunday14:25
weshaybogdando, we may need your assistance in review14:25
bogdandookkkay14:25
bogdando:)14:27
*** ykarel is now known as ykarel|away14:27
bogdandodo you chain the duties for the most guiltiest?14:27
bogdandolike in army :D14:28
bogdandobut one can't take more than 3 in a raw IIRC14:28
*** quiquell|off is now known as quiquell14:32
weshaysshnaidm|rover, what was the name of the tool dci was coming up w/ ?14:32
quiquellpanda, rfolco: Maybe we can try to merge https://review.openstack.org/#/c/581331/14:33
quiquellpanda, rfolco: Does it needs a extensive CI run ?14:33
pandaquiquell: change this https://review.openstack.org/#/c/579653/ to depend on your patch14:34
pandaquiquell: so we can test scenarios too14:34
pandaat least14:34
pandaquiquell: you can replace the existing depends on, that change is merged14:35
quiquellpanda: Will do, tomorrow we decide14:35
quiquellcreated one with tqe too https://review.openstack.org/#/c/582198/14:37
sshnaidm|roverbogdando, see you in Sunday, man14:37
*** quiquell is now known as quiquell|off14:37
jristhey guys14:38
jristhow do we re-enable tempest on oooq undercloud install?14:38
sshnaidm|roverarxcruz, ^^14:39
jristasking for a friend... :)14:39
arxcruzjrist: run_tempest: true14:40
jristpardon my ignorance, but where? :)14:40
jristunrelated: getting lots of these14:40
jrist[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|failed` instead use `result is failed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.14:40
jristarxcruz: pardon my ignorance, but where? :)14:43
arxcruzjrist: in the featureset, or parse this as env in ansible14:44
*** dtrainor has joined #oooq14:47
bogdandosshnaidm|rover: ;)14:48
bogdandoupdated the Scarry Patch, to have less work on sunday14:48
bogdandono more global changes!14:48
sshnaidm|roverbogdando, ok, so we can work only day in Sun, and rest in night!14:50
bogdandoyay14:50
jristarxcruz: sorry you've lost me again. what do you mean by "in the featureset"14:51
sshnaidm|roverjrist, which job is it?14:54
chkumar|ruckjrist: for https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset003.yml#L715:00
chkumar|ruckfor undercloud oooq15:00
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @  (1 more message)15:01
weshayarxcruz, ping15:01
weshaymtg15:01
arxcruzweshay: i'm on your bj15:01
chkumar|ruckarxcruz: bj link15:01
*** ratailor has joined #oooq15:02
kopecmartinarxcruz, chkumar|ruck https://bluejeans.com/7050859455/15:02
*** saneax has quit IRC15:04
chkumar|ruckpanda: sshnaidm|rover https://review.openstack.org/#/c/572347/15:08
*** bogdando has quit IRC15:11
sshnaidm|roverchkumar|ruck, commented15:13
jristchkumar|ruck, ssbarnea just trying to do oooq from my laptop to a machine with tempest installed and configured15:14
rlandysshnaidm|rover: https://review.rdoproject.org/r/#/c/14808/ - did I do that right?15:19
rlandyoh I need playbooks15:20
sshnaidm|roverrlandy, yep, need to create playbooks/legacy/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/run.yaml and post15:21
rlandysshnaidm|rover: yep - just noticed that - adding15:21
*** ccamacho has quit IRC15:26
sshnaidm|roverrlandy, is timeout ok there? it's 3 hours, maybe need more?15:32
rlandysshnaidm|rover: idk - I don;t want this to run forever15:32
*** ykarel|away has quit IRC15:32
rlandywe want minimal testing15:33
rlandysshnaidm|rover: nothing longer than upgrades - what is that?15:33
sshnaidm|roverrlandy, I dunno.. anyway, we can always increase15:34
rlandysshnaidm|rover: also  not sure what to do about https://review.rdoproject.org/r/#/c/14772/4/jobs/tripleo-upstream.yml - do we just remove that now?15:34
rlandythat I have added the legacy job?15:35
rlandyor we keep both?15:35
sshnaidm|roverrlandy, it's not used anymore15:35
rlandyok - will remove that15:35
rlandysshnaidm|rover: so this right? https://review.rdoproject.org/r/#/c/14772/4/zuul.d/tripleo.yaml and I get rid of https://review.rdoproject.org/r/#/c/14772/4/zuul/upstream.yaml?15:36
rlandyor I define  tripleo-browbeat-check-branchless to contain legacy?15:37
sshnaidm|roverrlandy, you configure jobs in one place15:38
sshnaidm|roverrlandy, in rdo-jobs15:38
chkumar|ruckweshay: https://bugs.launchpad.net/tripleo/+bug/177332515:38
openstackLaunchpad bug 1773325 in tripleo "tempest.api.object_storage.test_object_services is failing on scenario002" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)15:38
sshnaidm|roverrlandy, so you can leave there project template only15:38
rlandysshnaidm|rover: ok  - what about triggering that test with tq/tqe?15:39
sshnaidm|roverrlandy, sorry?15:40
rlandysshnaidm|rover: https://review.rdoproject.org/r/#/c/14772/4/zuul/upstream.yaml15:40
rlandyI get rid of the template here and add legacy jobs explicitly to https://review.rdoproject.org/r/#/c/14772/4/zuul.d/tripleo.yaml?15:41
sshnaidm|roverrlandy, sorry, I don't follow15:41
sshnaidm|roverrlandy, you configured jobs in rdo-jobs repo patch15:42
sshnaidm|roverrlandy, so now need to configure project template to trigger them, and you do it in right place15:42
*** jfrancoa has quit IRC15:42
sshnaidm|roverrlandy, but jobs definitions shouldn't be there, because you set them in rdo-jobs15:42
rlandysshnaidm|rover: I am talking about the template ...15:43
rlandy tripleo-browbeat-check-branchless looks wrong15:43
sshnaidm|roverrlandy, template of what?15:44
rlandysshnaidm|rover: I want to run the 053 legacy test with changes in browbeat, tq, tqe, tripleo-ci15:44
rlandywhat do I need?15:45
*** gkadam has quit IRC15:47
sshnaidm|roverrlandy, you need to change this file only now: https://review.rdoproject.org/r/#/c/14772/4/zuul.d/tripleo.yaml15:49
sshnaidm|roverrlandy, maybe this too, not sure https://review.rdoproject.org/r/#/c/14772/4/sanity-check.py15:49
sshnaidm|roverbut won't hurt15:49
sshnaidm|roverrlandy, lines 829-839 look correct15:50
sshnaidm|roverbut tripleo-browbeat-check-branchless template doesn't exist15:51
weshaychkumar|ruck, http://tripleo.org/contributor/check_gates.html#troubleshooting-a-failed-job15:52
weshayhttps://github.com/openstack-infra/tripleo-ci/blob/master/scripts/getthelogs15:52
sshnaidm|roverrlandy, if you want to create it, you need to define "project-templates:" like in https://review.rdoproject.org/r/#/c/14772/4/zuul/upstream.yaml15:52
rlandysshnaidm|rover: how about this ... https://review.rdoproject.org/r/#/c/14772/15:55
rlandyseem sane?15:55
sshnaidm|roverrlandy, yep, exactly, commented16:02
sshnaidm|roverrlandy, didn't you want it on tripleo-ci too?16:03
*** hamzy_ has quit IRC16:04
*** hamzy_ has joined #oooq16:04
rlandysshnaidm|rover: yep - will add to tripleo-ci16:06
*** vinaykns has joined #oooq16:06
*** sanjayu has joined #oooq16:06
rlandysshnaidm|rover:  https://review.rdoproject.org/r/#/c/14772/ and https://review.rdoproject.org/r/#/c/14808/ (one more time, pls)16:08
*** sanjayu has quit IRC16:11
*** panda is now known as panda|off16:12
*** ykarel|away has joined #oooq16:12
sshnaidm|roverrlandy, last comment on https://review.rdoproject.org/r/#/c/1477216:12
rlandysshnaidm|rover: thanks  - fixed16:15
vinayknsHello channel...I am having issue with deploying 000q on a VM.16:17
vinayknshttp://pastebin.test.redhat.com/61702016:17
vinayknsAt least what i can see is...it is unable to modify the disk image.16:18
*** ratailor has quit IRC16:23
sshnaidm|roverrlandy, thanks, +w16:27
sshnaidm|roverrlandy, you can try trigger them in 5 min16:27
sshnaidm|roverweshay, promoter SUCCESS promoting tripleo-ci-testing as current-tripleo16:30
sshnaidm|roverweshay, for master, queens is in progress16:30
weshaysshnaidm|rover++16:31
hubbot`weshay: sshnaidm|rover's karma is now 116:31
weshaysshnaidm|rover++16:31
hubbot`weshay: sshnaidm|rover's karma is now 216:31
weshaysshnaidm|rover++16:31
hubbot`weshay: sshnaidm|rover's karma is now 316:31
weshaythank you16:32
*** ykarel|away has quit IRC16:32
chkumar|rucksshnaidm|rover++ A great Rover and Friend :-)16:42
hubbot`chkumar|ruck: sshnaidm|rover's karma is now 416:42
sshnaidm|roverchkumar|ruck, :)16:42
chkumar|ruckdoes hubbot` stores comments also?16:43
sshnaidm|roverchkumar|ruck, I don't think so16:46
sshnaidm|roverrlandy, can you take a look please at https://review.openstack.org/#/c/576816/ ? I think it's better way to install additional roles than adding them to extras file16:47
sshnaidm|roverrlandy, like browbeat, etc16:48
sshnaidm|roverrlandy, in the scope of this sprint we can even configure it as different var for job16:48
rlandysshnaidm|rover: weshay and I discussed that  yesterday16:48
rlandysshnaidm|rover: I think that review has a lot of merit for something we want to add occasionally16:49
rlandyweshay mentioned that we are going to need to start adding tempest related repos in requirements as well16:49
rlandywe can certainly discuss the best way to conquer that16:49
sshnaidm|roverrlandy, yeah16:50
rlandybut it is my understanding that if we just add it in quickstart, we leave out some use cases where people call ansible-playbook directly16:50
rlandyso I am +1 on the idea16:50
rlandyjust not sure it fits in this case16:50
rlandyopen to discussion on it16:50
sshnaidm|roverrlandy, in CI we still use quickstart.sh for bootstrap16:51
sshnaidm|roverrlandy, and in any other way we need to use something to bootstrap - installing venv, packages and roles16:51
sshnaidm|roverrlandy, ansible-playbook is already next step16:51
sshnaidm|roverrlandy, you can't just call ansible-playbook without setup environment with oooq16:52
*** trown is now known as trown|lunch16:53
*** kopecmartin has quit IRC16:54
rlandysshnaidm|rover: so I would consider adding another separate requirements file but not only through quickstart16:54
rlandysshnaidm|rover: I think this should be put to the team - esp considering the upcoming tempest work. Monday's meeting?16:56
sshnaidm|roverrlandy, yeah16:58
*** agopi is now known as agopi|lunch16:59
rlandysshnaidm|rover: I +1'ed your patch in the mean time as I think it's a good idea in general16:59
hubbot`FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @  (1 more message)17:01
rfolcosshnaidm|rover, I thought we've agreed that the original script should not be modified by the j2 templates... rdo-cloud jobs won't follow the run-v3 workflow and use the original one ? https://review.openstack.org/#/c/581331/12/playbooks/tripleo-ci/run-v3.yaml17:03
sshnaidm|roverrfolco, well, that was an idea17:05
rfolcosshnaidm|rover, so let me put this way... do rdo-cloud jobs follow a different workflow (not run-v3) ? coz if they use the same run-v3, it will use the modified templates.17:09
*** dtrainor has quit IRC17:10
*** dtrainor has joined #oooq17:14
sshnaidm|roverrfolco, rdo cloud jobs are defined mostly here: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/zuul-legacy-jobs.yaml  and here https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/jobs.yaml17:15
sshnaidm|roverrfolco, as you see it needs to be configured separately.. until we know how to configure them in upstream repos17:16
*** tesseract has quit IRC17:16
rfolcosshnaidm|rover, ok so they do not follow run-v3, thats what I wanted to understand. Thanks17:17
sshnaidm|roverrfolco, yep17:17
sshnaidm|roverrfolco, but they use toci_* scripts in old way, that's what I warned about17:17
rfolcoyep, we don't touch original toci_ scripts. Safe.17:18
weshayrlandy, nice https://review.rdoproject.org/r/#/c/14772/17:19
weshayrlandy, bah.. but doesn't trigger the real job until merge17:20
weshayrdoproject--17:20
rlandyweshay: we still need the playbook17:26
weshayk17:26
rlandywaiting for agopi|lunch to be back from lunch17:26
rlandyonce we have that - we can try run something and see how long it takes17:27
rlandyweshay: hmmm ... probably should not have merged that - don't really want to add failing jobs :(17:30
weshay?17:31
weshayrlandy, which?17:31
weshaythe doc17:31
*** sshnaidm|rover is now known as sshnaidm|afk17:31
weshayrlandy, which one?17:32
rlandyno https://review.rdoproject.org/r/#/c/14772/17:32
rlandyoh - maybe it didn't17:32
weshaymeh17:32
rlandyignore me17:32
weshayit's fine17:32
rlandyupdating the tq review so we are ready to go there17:32
*** sshnaidm|afk has quit IRC17:36
weshaytrown|lunch, so scen009 is non-containerized only?17:36
*** holser_ has quit IRC17:36
rlandyagopi|lunch: ping when you are back ... we need to chat about https://review.openstack.org/#/c/581488/1/playbooks/baremetal-quickstart-extras.yml and the playbook to add17:38
rlandythat is the missing piece17:38
agopi|lunchhey rlandy17:41
agopi|lunchsorry was out for lunch17:41
*** agopi|lunch is now known as agopi17:41
rlandyagopi: np17:51
amoralejhi, i'm confused about how are containers updated in oooq jobs17:51
amoralejwhen is container-check used?17:51
rlandyagopi: looks good ... https://gist.github.com/aakarshg/73b4a1a7a87b0867d816ce94bb83f76e17:51
rlandyhow long will it run?17:51
rlandyagopi: we will need to enable this only when enable_minimal_browbeat: true17:52
rlandyper https://review.openstack.org/#/c/581484/3/config/general_config/featureset053.yml17:52
rlandyagopi: the current job time limit is 3 hrs17:55
rlandywe can extend if need be17:55
agopihello rlandy im guessing it shouldnt take more than 30-50 mins just for the playbook i sent.17:55
agopiand also17:55
weshayamoralej, it's been moved into the undercloud install17:56
weshayamoralej, outside of ci17:56
weshayamoralej, it's also moving from overcloud-prep-containers --> tripleo-common17:56
weshayamoralej, the logic is still the same17:56
weshayof when it is used17:56
rlandyagopi: ok - let's try it and see how it goes17:56
agopirlandy, how will we go about passing variables?17:57
amoralejweshay, so in master is done by https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L34-L37 ?17:57
agopiare we assuming same setup as previous jobs ?17:57
rlandywe'll have to do an include when17:57
weshayamoralej, ya17:57
amoralejweshay, so repo-name is always gating-repo https://github.com/openstack/tripleo-quickstart-extras/blob/1340ff513fc9e3f89cb0d1ec0ccde223e0b43ce3/roles/undercloud-deploy/templates/containers-prepare-parameter.yaml.j2#L1717:58
amoralejright?17:58
amoralejit used to be configurable before using gating_repo_name17:58
* rlandy thinking about where we should put the when: condition17:58
rlandyagopi: I'll add then when17:59
agopirlandy,18:00
rlandyupdating review18:00
agopihttps://github.com/openstack/browbeat/blob/master/browbeat-config.yaml18:00
agopibrowbeat needs a config file like this18:00
agopia minimal config file is here18:00
agopihttps://github.com/openstack/browbeat/blob/master/ansible/oooq/roles/template-configs/templates/browbeat-basic.yaml.j218:00
agopias you see we need browbeat_cloud_name, elastic_host, grafana_host18:01
rlandyagopi: https://github.com/openstack/tripleo-quickstart-extras/blob/master/config/environments/rdocloud.yml18:01
rlandyso we would need to edit that18:02
rlandythe values are stored above: ^^18:02
agopiokay rlandy18:02
*** trown|lunch is now known as trown18:03
rlandyagopi: you'll need a grafana dashboard on our graphana, right?18:03
trownweshay: it is deploying openshift, so ya18:03
rlandycloud_name: rdocloud18:03
trownweshay: it doesnt deploy openstack at all (other than undercloud)18:03
rlandytrown: hey - how's the new role?18:04
trownrlandy: so far so good :P... a ton to learn18:04
agopiwe already have hosts for ELK including grafana18:04
rlandyhttps://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/config/testenv/ovb-rdocloud.yml18:04
agopihttp://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/jenkins/jobs/tripleo-quickstart/tripleo-quickstart-builders.yml#L20318:05
rlandyagopi: ^^ sorry that has the grapaha values18:05
rlandyinfluxdb18:05
* rlandy updates review18:05
agopisorry i dont follow rlandy18:05
agopihow are you passing the password? well need to do same for grafana_apikey18:06
rlandyagopi: hmm not sure - will need to ask/check with sshnaidm18:07
rlandypanda|off: rfolco: ^^ do you remember18:07
rfolcosorry don't recall handling keys for grafana18:09
weshaytrown, ah k18:10
rlandyagopi: https://review.openstack.org/#/c/581488/ updated18:14
rlandyagopi: if you have the config file, we can add it18:15
amoralejmay i get some eyes on https://review.openstack.org/582327 ?18:15
amoraleji'm not sure if it's the right way18:15
trownweshay: I am working on the other path... deploying openshift inside of an openstack tenant... but just getting an env up to work with is challenging ... that is pretty far off from a CI perspective though18:16
trownweshay: undercloud deployed openshift (scenario009) seems pretty much done18:16
weshayk18:17
*** amoralej is now known as amoralej|off18:17
*** hamzy_ has quit IRC18:18
*** hamzy has joined #oooq18:20
*** vinaykns has quit IRC18:22
weshaysagi, chkumar|ruck, rlandy fyi.. I have disabled osp jobs https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-12-ci-rhos-ovb-minimal-pacemaker-public-bond/18:23
rlandyweshay: ok18:23
weshayamoralej|off, https://review.openstack.org/#/q/topic:bp/container-prepare-workflow+(status:open+OR+status:merged)18:25
rlandyrfolco: I think we had doc on it ... checking ruck/rover18:26
rfolcorlandy, I remember I did not get access to grafana server18:27
rlandywas it rdo infra or ours?18:27
rfolcoI believe it was rdo infra18:28
weshaytrown, is that something I can poke at w/ the reproducer ?18:28
weshayscen00918:28
trownweshay: ya, you may need https://review.openstack.org/#/c/582164/1 if 1.1.1.1 does not work for DNS for you... though that may be very specific to my home ISP18:29
weshayk18:29
trownweshay: but the above patch was all I needed to get it working via libvirt18:29
trownyou can even run oc commands against the openshift and make apps and what not18:30
trownwas a bit surprised it actually works given that part is not at all tested yet18:30
*** ykarel|away has joined #oooq18:32
hubbot`FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @  (1 more message)19:01
*** ykarel|away has quit IRC19:17
rlandymarios: still around? ping re: reparent failure19:17
*** vinaykns has joined #oooq19:28
*** jaganathan has quit IRC19:38
rlandymarios: panda|off: I think the error is unrelated to the change - rerunning19:56
*** apevec has joined #oooq20:35
weshay0/20:35
apevecweshay, here?20:35
weshayaye mate20:35
weshay<apevec> weshay, are network interface stats collected somewhere in oooq logs?20:36
apevecso, .*:  are network interface stats collected somewhere in oooq logs?20:36
weshay<weshay> apevec, can you ask in #oooq20:36
weshay<apevec> weshay, I need some guesstimate how much network traffic CI jobs are generating20:36
apevecoh thanks for CPaaS :)20:36
weshayapevec, what ever we have there would be $node/var/log/extra20:36
weshayapevec, if you give me some sample commands we can get them into the logs for you20:36
* weshay gets example20:37
weshayapevec, http://logs.openstack.org/26/581026/3/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/7242759/logs/undercloud/var/log/extra/network.txt.gz20:38
apevecbasic stats would be just e.g. ifconfig, that would show RX/TX packets20:38
weshayhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/tasks/collect.yml#L4720:38
weshayya.. it's there20:38
apevecah cool, /me looks20:39
apevecso assuming nodepool node is freshly created, we can assume all packets were generated by the CI job20:39
apevecok, cool, so can do some mining, thanks!20:40
weshaynp20:40
*** holser_ has joined #oooq20:40
apevecweshay, br-ex would be traffic to/from Internet?20:51
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci- (1 more message)21:01
weshayapevec, to the internet .. no21:02
weshayeth021:02
*** florianf has quit IRC21:03
weshay192.168.24.x is the public connection between the undercoud and overcloud21:04
*** trown is now known as trown|outtypewww21:04
*** holser_ has quit IRC21:18
*** rlandy is now known as rlandy|afk21:55
*** sshnaidm|afk has joined #oooq21:57
*** apevec has quit IRC21:57
*** florianf has joined #oooq22:17
*** florianf has quit IRC22:43
hubbot`FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci- (1 more message)23:01
*** tosky has quit IRC23:01
*** rlandy|afk is now known as rlandy23:18
rlandyoriginal message: could not locate file in lookup: ControllerApi/centos-7-inap-mtl01-0000693370/UpgradeInitDeployment"23:23
rlandy^^ consistent23:23
rlandyreal issue with 3 node23:23
*** vinaykns has quit IRC23:27
*** vinaykns has joined #oooq23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!