Thursday, 2019-04-18

rlandy|ruckweshay|rover: fyi ...00:17
rlandy|ruck    ttps://bugs.launchpad.net/tripleo/+bug/1825265 periodic-tripleo-ci-centos-7-singlenode-featureset027-stein is failing tempest -  NoOptionError: No option 'region' in section: 'identity'00:17
rlandy|ruck    Did not mark this as a promotion blocker yet.00:17
rlandy|ruck    Will see of it fails again.00:17
*** hrybacki_ has joined #oooq00:55
*** sshnaidm has quit IRC01:02
*** hrybacki has quit IRC01:02
*** hrybacki_ is now known as hrybacki01:02
*** irclogbot_2 has quit IRC01:05
*** irclogbot_0 has joined #oooq01:05
*** sshnaidm has joined #oooq01:08
*** rlandy|ruck has quit IRC01:38
*** ykarel has joined #oooq02:02
*** rascasoft has quit IRC02:17
*** ykarel has quit IRC02:18
*** ykarel has joined #oooq02:19
*** rascasoft has joined #oooq02:20
*** saneax has joined #oooq02:27
*** ykarel has quit IRC02:30
*** ykarel has joined #oooq03:48
*** ykarel_ has joined #oooq03:50
*** holser_ has joined #oooq03:50
*** ykarel has quit IRC03:52
*** holser_ has quit IRC03:54
*** holser_ has joined #oooq03:57
*** holser_ has quit IRC04:09
*** saneax has quit IRC04:31
*** ratailor has joined #oooq05:01
*** gkadam has joined #oooq05:01
*** gkadam_ has joined #oooq05:07
*** gkadam has quit IRC05:09
*** marios has joined #oooq05:40
*** kopecmartin|off is now known as kopecmartin05:47
*** rascasoft has quit IRC06:13
*** rascasoft has joined #oooq06:15
*** saneax has joined #oooq06:42
*** saneax has quit IRC06:58
*** gkadam_ has quit IRC07:01
*** marios has quit IRC07:20
sshnaidmchandankumar, seems like tempest failed there: https://review.openstack.org/#/c/639324/07:25
chandankumarsshnaidm: yes, related with overcloud osname07:26
chandankumarlooking and fixing it07:27
*** marios has joined #oooq07:34
*** tosky has joined #oooq07:42
*** ykarel_ is now known as ykarel07:44
*** ykarel is now known as ykarel|lunch07:45
chandankumarsshnaidm: I think there is no support of os_cloud for overcloud07:45
chandankumarsshnaidm: from tht there is only https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/post_deploy/standalone_post.yaml and https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/post_deploy/undercloud_post.yaml07:46
chandankumarI think I need to add overcloud post.yaml07:46
zbrcall for reviews on molecule change: https://review.openstack.org/#/c/651772/ -- i am more than happy to do a hands on experience session on BJ, just tell me.07:47
chandankumarhttps://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/utils/overcloudrc.py#n70 -> single reference to overcloud os cloud07:48
mariosbrb07:52
*** marios has quit IRC07:52
*** marios has joined #oooq07:54
*** bogdando has joined #oooq07:58
*** dtantsur|afk is now known as dtantsur08:25
zbr@oooq:  i need some support for adding a feature into bindep related to py3: https://review.openstack.org/#/c/639951/08:27
*** ykarel|lunch is now known as ykarel08:37
*** holser_ has joined #oooq08:57
*** ratailor_ has joined #oooq09:27
*** ratailor has quit IRC09:29
*** dsneddon has quit IRC10:02
ykarelpanda, i proposed https://review.openstack.org/#/c/653408/ to allow kolla rpm override10:04
pandaykarel: you rock10:23
pandaykarel: I paper10:23
pandaykarel: I win.10:23
ykarel:D10:28
*** ratailor__ has joined #oooq10:41
*** ratailor_ has quit IRC10:44
*** dsneddon has joined #oooq10:57
*** dsneddon has quit IRC11:05
*** panda is now known as panda|lunch11:26
*** vinaykns has joined #oooq11:59
*** dsneddon has joined #oooq11:59
mariosneed to merge that please if you have time needs reviews thanks https://review.openstack.org/#/c/651789/12:02
*** dsneddon has quit IRC12:08
panda|lunchmarios: python2-docer depends on docker ?12:28
chandankumarweshay|rover: Hello12:30
chandankumarweshay|rover: 1:1 time12:30
weshay|roverya12:30
marios panda|lunch: https://bugs.launchpad.net/tripleo/+bug/182430112:32
openstackLaunchpad bug 1824301 in tripleo "standalone-upgrade job fails in post-upgrade tasks for " Failed to import docker or docker-py - No module named docker"" [High,In progress] - Assigned to Marios Andreou (marios-b)12:32
zbrpanda|lunch: the correct python docker module is called just docker and it does not require docker. the other one is old/obsolete.12:33
mariospanda|lunch: more like, the docker_container module needs python-docker12:33
*** rlandy|ruck has joined #oooq12:34
panda|lunchmarios: zbr so the task that checks the existence of /usr/bin/docker is actually completely unneeded12:36
mariospanda|lunch: not quite... this replaces that so we can also solve the bug. if python2-docker is installed, we also then check the docker service state12:38
*** ratailor__ has quit IRC12:38
zbrpanda|lunch: having docker installed could mean two things: docker-client and docker-server12:39
zbrthey are installed by the same package12:39
zbri need to test a bit, but likely that docker cli is not needed for using python docker module, but I am not sure.12:40
zbrclearly it does not need the server running, this is for sure.12:41
rlandy|ruckweshay|rover: hello!12:41
zbrthat is what I use locally, DOCKER_HOST=ssh://root@n0 --- where n0 is my docker machine.12:41
marioszbr: panda|lunch fwiw, that makes the standalone-upgrade job green, it is red without12:42
mariospanda|lunch: zbr please add any concerns on the review so i can address them thanks12:42
weshay|roverrlandy|ruck hey :)12:43
rlandy|ruckweshay|rover: after the meeting can we touch base on what's left to sort out for promotions?12:44
mariosweshay|rover: rlandy|ruck fyi https://review.openstack.org/653701 for phase 1 jobs via ykarel earlier see comment fro ykarel here https://review.openstack.org/#/c/652078/12:47
weshay|roverrlandy|ruck sure12:48
rlandy|ruckmarios: https://review.openstack.org/#/c/653634/12:48
rlandy|ruckmarios: but if your patch gets in before mine, ok12:49
mariosrlandy|ruck: ack OK sorry obviously didn't see yours ... they do slightly different things (I removed the jinja %} but no problem i can abandon mine)12:50
rlandy|ruckmarios: pls don't12:50
mariosrlandy|ruck: k leave it for now12:50
rlandy|ruckmarios: leave it12:50
rlandy|ruckhas had trouble getting past check12:50
rlandy|ruckif your passes, let's accept it12:51
mariosk12:51
chandankumarweshay|rover: https://review.openstack.org/#/q/topic:rhel-element+(status:open+OR+status:merged)12:55
rlandy|ruckchandankumar: hi - could you look at https://bugs.launchpad.net/tripleo/+bug/1825265 and let me know if that is a familiar error to  you? I didn't mark it as a promotion-blocker yet as it only happened once.12:55
openstackLaunchpad bug 1825265 in tripleo "periodic-tripleo-ci-centos-7-singlenode-featureset027-stein is failing tempest - NoOptionError: No option 'region' in section: 'identity'" [Critical,Triaged]12:55
chandankumarrlandy|ruck: checking12:55
*** dsneddon has joined #oooq12:58
chandankumarrlandy|ruck: https://github.com/openstack/python-tempestconf/commit/50920bd50bbdde34ba0da716363011bdd69f3c4813:01
chandankumarrlandy|ruck: it caused it13:01
weshay|roverapetrich ping13:01
chandankumarrlandy|ruck: recently we have added heat tempest plugin tempest.conf support13:02
chandankumarin tempest.comf13:02
chandankumarkopecmartin: around?13:02
chandankumarkopecmartin: https://logs.rdoproject.org/openstack-periodic-latest-released/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset027-stein/a247eb3/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-04-17_23_05_46 heat tempest conf broke undercloud job13:03
chandankumarhttps://github.com/openstack/python-tempestconf/commit/50920bd50bbdde34ba0da716363011bdd69f3c48#diff-c56aef0c59df627fa3b5a05f6e2612b7R4213:03
rlandy|ruckchandankumar: k, thanks for investigating. let me know if there is any change also needed on our side13:03
*** dsneddon has quit IRC13:05
rlandy|ruckykarel: hi - did you hit any more failed stack issues due to dup ips so far?13:06
apetrichweshay|rover, sorry didn't hear the ding.13:06
ykarelrlandy|ruck, i have not pushed any patch today, so can't say13:07
*** aakarsh has quit IRC13:15
kopecmartinchandankumar, but the reason of the failure is missing region name under identity section, not heat support itself13:17
*** Goneri has joined #oooq13:17
chandankumarkopecmartin: if we pass oslcloud that time it will automatically set13:18
rlandy|rucksshnaidm: hi - can we discuss the sova changes for promotions?13:28
sshnaidmrlandy|ruck, yeah, after mtg13:28
rlandy|rucksshnaidm: ack13:28
weshay|roverpanda|lunch sshnaidm rfolco give me 5min13:30
weshay|roversorry for delay13:30
rfolcoweshay|rover, ack13:30
panda|lunchcoast is not clear13:30
panda|lunchrepeat coast is NOT clear13:30
panda|lunchdisengage13:30
weshay|roverrfolco open up your bluejeans.. and I'll move there13:31
weshay|roverin 4min13:31
rfolcoweshay|rover, ok13:32
*** dsneddon has joined #oooq13:32
rfolcopanda|lunch, sshnaidm https://bluejeans.com/587845809713:33
rfolcoweshay|rover, ^13:33
weshay|roverrfolco joining13:37
sshnaidmrlandy|ruck, do you know what is public net that we need to use in https://rhos-d.infra.prod.upshift.rdu2.redhat.com/dashboard/project/networks/ ?13:37
*** dsneddon has quit IRC13:38
rlandy|rucksshnaidm: hmm .. I had that in a settings file for the old setup - iirc, there were multiple and we could pick any one  - may be different now.13:39
rlandy|rucklet me check13:39
*** aakarsh has joined #oooq13:40
rlandy|ruckweshay|rover; editing https://trello.com/c/nBx3IF93/954-cixlp1825220tripleociproa-f28-master-standalone-job-fails-container-image-prepare-no-module-named-pkgresources - do we have a RCA? upstream change13:42
rlandy|rucksshnaidm: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/environments/rhos-ocp-upshift.yml#n3713:43
rlandy|ruck^^ is what I used to test the initial env13:43
rlandy|ruckit may be diff now13:43
sshnaidmrlandy|ruck, I saw provider_net_lab before, but it's not there now13:47
rlandy|rucksshnaidm: k - then I'd refer to the admins13:48
sshnaidmrlandy|ruck, provider_net_lab is visible for rhos-dev-ci-jenkins13:50
sshnaidmrlandy|ruck, but not for our internal tenant13:50
sshnaidmrlandy|ruck, are those different clouds? rhos-ocp and rhos-d13:51
rlandy|rucksshnaidm: yes - ocp was th original test cloud13:55
rlandy|ruckit's changed since then13:55
*** dsneddon has joined #oooq14:00
*** rlandy|ruck is now known as rlandy|ruck|mtg14:02
*** dsneddon has quit IRC14:08
*** fmount has joined #oooq14:15
*** jaosorior has quit IRC14:15
*** Vorrtex has joined #oooq14:31
*** dsneddon has joined #oooq14:32
*** mjturek has joined #oooq14:33
*** yolanda_ has quit IRC14:40
*** dsneddon has quit IRC14:41
chandankumarweshay|rover: sshnaidm https://github.com/openstack/ansible-role-collect-logs/blob/master/setup.cfg#L2314:58
*** ratailor has joined #oooq15:01
*** dsneddon has joined #oooq15:01
weshay|roverchandankumar thanks have a good night15:02
*** rlandy|ruck|mtg is now known as rlandy|ruck15:18
mariosrlandy|ruck: are you around tomorrow or is holiday for you?15:18
rlandy|ruckweshay|rover: marios has a fix for the broken ci.centos jobs that is further along then mine - let's approve this one ... https://review.openstack.org/#/c/653701/15:19
rlandy|ruckmarios: I'm off tomorrow15:19
mariosrlandy|ruck: ok... re https://code.engineering.redhat.com/gerrit/168285 i am happy to abandon it15:19
mariosrlandy|ruck: is just a job definition didn't take long... do you want to update https://tree.taiga.io/project/tripleo-ci-board/task/102415:19
rlandy|ruckmarios: I am trying to figure out why my card did not show ... https://tree.taiga.io/project/tripleo-ci-board/task/1024?kanban-status=144727415:20
mariosrlandy|ruck: weird... i see it in activities15:21
mariosrlandy|ruck: created custom attribute(Reviews)15:21
marioshttps://code.engineering.redhat.com/gerrit/168210 Add rocky standalone job to run on internal sf15:21
mariosrlandy|ruck: but it wasn't on the card when i looked.15:22
rlandy|ruckmarios: sorry for the duplicate work - I am pretty sure it was on the board when I added it15:23
rlandy|ruckI specifically didn;t want us to duplicate15:23
mariosrlandy|ruck: no problem at all15:24
rlandy|ruckmarios; k - taiga mystery ... you pick the review you want to take forward and let me know - all the same to me15:25
rlandy|rucklet's just go with one so there is no more confusion15:26
mariosrlandy|ruck: ack will do ... likely going with yours.. :) I'll have a look and update it tomorrow15:27
rlandy|ruckmarios: cool - thanks15:27
*** sshnaidm has quit IRC15:27
mariosbiab15:29
*** marios has quit IRC15:29
*** sshnaidm has joined #oooq15:38
rlandy|ruckykarel: pls vote on https://review.openstack.org/#/c/653701/ to fix ci.centos15:38
rlandy|ruckI had another patch but this one is passed zuul already15:39
ykarelrlandy|ruck, ack looking15:50
rlandy|ruckthanks15:51
ykarelrlandy|ruck, +115:55
rlandy|rucksshnaidm:just touching base about sova and the promotion pipelines - what is the issue with leaving https://github.com/sshnaidm/sova/blob/promotion/tripleoci/config.py#L65 ( to retain history) and adding the two new pipelines?15:55
sshnaidmrlandy|ruck, looking..15:55
rlandy|ruckykarel: thanks - w+ing it to clear ci.centos15:55
ykarelrlandy|ruck, ack15:55
weshay|roverrlandy|ruck need to chat ?15:57
ykarelrlandy|ruck, you have a minute?15:57
ykarelrlandy|ruck, can you check: openstack port list|egrep -e  "172.18.0" -e '172.17.0'15:58
ykarelrlandy|ruck, i would like to see mac address pattern in those ports15:58
rlandy|ruckykarel: looking16:01
sshnaidmrlandy|ruck, ok, added it: https://github.com/sshnaidm/sova/commit/3d1e9b434edeeb1642f3c96209e2433c92015b3716:03
sshnaidmrlandy|ruck, let's see how it works..16:03
sshnaidmrlandy|ruck, so, who can I ask about public network in rhos-d cloud?16:03
rlandy|ruckykarel: http://pastebin.test.redhat.com/75685516:04
rlandy|rucksshnaidm: thanks16:04
rlandy|rucklet me check the tenant - sec16:04
ykarelrlandy|ruck, so ^^ same what i doubted16:05
*** ratailor has quit IRC16:05
ykarelrlandy|ruck, so the examples i have are BE:E5:4F:B9:21:B0, 26:91:D7:84:DB:41 and 9E:CD:59:F2:42:2416:06
ykarelthese do not match with the port list output16:06
ykarelso something strange is happening16:07
rlandy|rucksshnaidm: start with kforde16:07
rlandy|ruckif he doesn't know, someone on apevec's team16:08
rlandy|ruckykarel: ok - from any particular job or just random mismatch?16:09
rlandy|rucksshnaidm: kforde gave us the shared user16:09
panda|lunchweshay|rover: 2019-04-18 11:59:20 | ERROR:kolla.common.utils.base:repository docker.io/rhel not found: does not exist or no pull access.16:10
panda|lunchweshay|rover: where can I find the base image for rhel containers ?16:10
ykarelrlandy|ruck, out of the three mac address, 2 are from logs i shared yesterday and 1 is from the bug description16:10
panda|lunchweshay|rover: nevermind I'll as reldel16:10
weshay|roverpanda|lunch not in docker.io16:11
rlandy|ruckfine16:11
weshay|rover:)16:11
*** bogdando has quit IRC16:11
*** panda|lunch is now known as panda16:11
rlandy|ruckweshay|rover: just wanted to touch base on what we had left to work on - no rush though16:12
*** Vorrtex has quit IRC16:12
rlandy|ruckI guess a couple question16:13
rlandy|ruckalso16:13
*** ykarel is now known as ykarel|away16:15
weshay|roverrlandy|ruck k.. /me grabs a cup of coffee and will ping you.. no coffee yet today.. drrrraaag16:17
*** marios has joined #oooq16:19
*** ykarel|away has quit IRC16:22
*** marios has quit IRC16:30
*** kopecmartin is now known as kopecmartin|off16:42
*** dtantsur is now known as dtantsur|afk16:50
*** ykarel has joined #oooq16:51
*** tosky has quit IRC16:56
*** dsneddon has quit IRC17:12
*** dsneddon has joined #oooq17:13
weshay|roversshnaidm you still around? https://review.rdoproject.org/r/#/c/20289/17:15
sshnaidmweshay|rover, seems like "last jobs" are broken acc. to your screenshot17:20
weshay|roversshnaidm /me looks.. I think I have this working..  is it too late to blue?17:21
weshay|roverI can show you live17:21
sshnaidmweshay|rover, also I don't think it's worth to have all this bugs in upper part17:21
weshay|roversshnaidm well would you repsect decent on that? I find it very helpful17:21
sshnaidmweshay|rover, if you see list of last jobs, from upstream and rdo, that's fine17:21
weshay|roversshnaidm want to spend two min w/ me?17:22
weshay|roverto get this how we both like it?17:22
sshnaidmweshay|rover, I need to scroll every time to jobs below, bugs are not changing so much and don't reflect CI status17:22
sshnaidmweshay|rover, yeah17:22
weshay|roverya.. so I think I agree w/ you on that17:22
weshay|roverlet's figure it out.17:22
* weshay|rover opens blue17:22
weshay|roverhttps://bluejeans.com/u/whayutin/17:23
*** holser_ has quit IRC17:54
*** dsneddon has quit IRC18:09
sshnaidmrlandy|ruck, do you know which flavors should we use for ovb in rhos-d? ci.* or m1.* ?18:12
weshay|roverrlandy|ruck master rdo p1 is promoting18:14
*** dsneddon has joined #oooq18:16
rlandy|ruckweshay|rover: yay!18:17
rlandy|rucksshnaidm: we only had access to m1 flavors before - the ci ones are new18:20
sshnaidmrlandy|ruck, ok, will try ci.* flavors..18:20
*** sshnaidm is now known as sshnaidm|off18:32
rlandy|ruckweshay|rover: https://review.rdoproject.org/r/20340 Add mini cleanup scripts for failed stacks and servers in error18:37
weshay|roverrlandy|ruck k.. and do we need an additional change to setup cron for them?18:38
weshay|roveror you want to take it one step at a time?18:38
rlandy|ruckweshay|rover: yep - once these are reviewed and approved18:38
rlandy|ruckbaby steps18:38
weshay|roverrlandy|ruck I ran both, there are no failed stacks or servers atm.. but not too concerned about it :)18:43
weshay|rovermerged18:43
*** ykarel is now known as ykarel|away18:44
rlandy|ruckweshay|rover: I am adding another review to modify the cron18:44
rlandy|ruckweshay|rover:  https://review.rdoproject.org/r/20342 Modify cron to add mini cleanup scripts18:54
rlandy|ruck^^ testing that cron18:54
rlandy|ruckweshay|rover: you just reset my whole world - cockpit reorg19:03
weshay|roverrlandy|ruck ha19:07
weshay|roverrlandy|ruck interested in your opinion :)19:07
rlandy|ruckweshay|rover: the jury is still out19:12
weshay|roverheh k19:12
rlandy|ruckweshay|rover: wow - happiness is ... http://rhos-release.virt.bos.redhat.com:3030/rhosp19:43
weshay|roverrlandy|ruck take a pic :)19:44
weshay|rovermeh.. I'll do it19:44
weshay|roverrlandy|ruck sent :)19:48
rlandy|ruckcool19:48
*** ykarel|away has quit IRC19:54
rlandy|ruckweshay|rover: just fyi ... bring bm back in triggering ...  https://code.engineering.redhat.com/gerrit/168340 Re-enable baremetal pipeline.20:30
weshay|roverooh20:30
* weshay|rover looks20:30
rlandy|ruckit's just a config job .. results are here ... https://code.engineering.redhat.com/gerrit/#/c/167766/20:31
rlandy|ruckweshay|rover: ^^20:31
rlandy|ruckstein might need a rework20:31
weshay|roverk20:31
rlandy|ruckweshay|rover: ^^ but I want to keep the hw spinning to check the networks20:34
rlandy|ruckand triggering when we enable pidone20:34
weshay|roverk20:36
weshay|roversounds good to me20:36
*** aakarsh has quit IRC20:43
rlandy|ruckweshay|rover; eek fs001 failing all over the place ...20:53
weshay|roverrlandy|ruck someone is putting in some big changes20:53
weshay|roverI don't think it's us20:53
weshay|roverI looked at a couple of the reviews20:54
rlandy|ruckweshay|rover: yeah - trying to get a pattern20:54
* weshay|rover looks again20:54
rlandy|ruckI don't see a consistent failure20:54
weshay|roverhrm.. need to look at https://review.openstack.org/#/c/651772/20:54
weshay|roverrlandy|ruck this I expect failures https://review.openstack.org/#/c/630644/20:55
rlandy|ruckchanges to tripleo-heat-templates20:55
rlandy|ruck2019-04-18 18:20:58.124519 | primary |20:56
rlandy|ruck2019-04-18 18:20:58.124790 | primary | TASK [overcloud-prep-images : Prepare the overcloud images for deploy] *********20:56
weshay|roverhttp://logs.rdoproject.org/59/652959/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/6d78668/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz20:56
rlandy|ruckintrospection20:56
weshay|roveris a prob, but a known prob20:56
* weshay|rover checks stacks20:56
* rlandy|ruck will stop panicking20:56
weshay|roverrlandy|ruck do we have the cron change up?20:57
rlandy|ruckweshay|rover: ack getting20:57
rlandy|ruckhttps://review.rdoproject.org/r/#/c/20342/20:58
rlandy|ruckweshay|rover: as long as line 144 doesn't explode, we can merge20:59
weshay|roverrlandy|ruck see my comment21:00
rlandy|ruckweshay|rover: ack - change coming up21:00
*** Goneri has quit IRC21:03
rlandy|ruckweshay|rover: https://review.rdoproject.org/r/#/c/20342/ updated21:03
weshay|rover+221:05
*** mjturek has quit IRC21:08
*** aakarsh has joined #oooq21:11
weshay|roverrlandy|ruck shall we merge?21:14
rlandy|ruckweshay|rover: sure21:15
rlandy|ruckhmm ... bm failures in connecting to host21:16
*** aakarsh has quit IRC21:16
*** fultonj has quit IRC21:23
rlandy|ruckfixed21:25
*** rlandy|ruck is now known as rlandy21:57
*** aakarsh has joined #oooq22:07
*** holser_ has joined #oooq22:46
*** holser_ has quit IRC23:23
*** holser_ has joined #oooq23:29

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!