Monday, 2019-04-22

« Sunday, 2019-04-21 Index Tuesday, 2019-04-23 »
*** rascasoft has quit IRC00:13
*** dsneddon has joined #oooq00:15
*** dsneddon has quit IRC00:19
hubbot1All check jobs are working fine on master, stable/pike, master, stable/rocky, stable/rocky, stable/ocata.00:42
*** Goneri has joined #oooq01:14
*** Goneri has quit IRC01:22
*** Goneri has joined #oooq01:39
*** ykarel|away has joined #oooq01:56
*** ykarel|away is now known as ykarel01:56
*** Goneri has quit IRC02:05
*** Goneri has joined #oooq02:31
hubbot1All check jobs are working fine on master, stable/pike, master, stable/rocky, stable/rocky, stable/ocata.02:42
*** gkadam has quit IRC03:00
*** gkadam has joined #oooq03:06
*** udesale has joined #oooq03:56
*** dsneddon has joined #oooq04:16
*** dsneddon has quit IRC04:20
hubbot1All check jobs are working fine on master, stable/pike, master, stable/rocky, stable/rocky, stable/ocata.04:42
*** Goneri has quit IRC04:42
*** dsneddon has joined #oooq04:51
*** dsneddon has quit IRC04:56
*** ratailor has joined #oooq05:07
*** marios has joined #oooq05:20
*** marios has quit IRC05:20
*** marios has joined #oooq05:21
*** matbu has quit IRC05:44
*** skramaja has joined #oooq05:49
*** dsneddon has joined #oooq05:53
*** ykarel is now known as ykarel|afk05:57
*** ykarel|afk has quit IRC06:01
*** jbadiapa has joined #oooq06:04
*** quiquell has joined #oooq06:09
*** quiquell is now known as quiquell|rover06:09
*** ykarel|afk has joined #oooq06:16
*** ykarel_ has joined #oooq06:19
*** ykarel_ is now known as ykarel06:19
*** ykarel|afk has quit IRC06:21
*** jfrancoa has joined #oooq06:40
hubbot1All check jobs are working fine on master, stable/pike, master, stable/rocky, stable/rocky, stable/ocata.06:42
*** quiquell|rover is now known as quique|rover|brb06:46
quique|rover|brbhumm we have to shutdown hubbot06:46
quique|rover|brbAt least this part of it06:46
*** saneax has joined #oooq06:58
*** ccamacho has joined #oooq06:58
*** sshnaidm|off is now known as sshnaidm07:00
*** quique|rover|brb is now known as quiquell|rover07:02
*** sanjayu_ has joined #oooq07:05
*** saneax has quit IRC07:07
*** amoralej|off is now known as amoralej07:15
*** yolanda_ has joined #oooq07:18
*** ykarel is now known as ykarel|lunch07:23
*** yolanda_ has quit IRC07:52
mariosquiquell|rover: do you know if sf-config is a trusted project (I think it is i can see it has secrets defined under zuul.d08:17
quiquell|rovermarios: yep08:19
quiquell|roversecrets = trusted08:19
*** ykarel|lunch is now known as ykarel08:19
mariosquiquell|rover: ack thanks thought so... was trying to work out if there was some other config the admin has to set from https://zuul-ci.org/docs/zuul/glossary.html#term-config-project to designate a given project as 'config'08:20
quiquell|rovermarios: nope just the tenant conifg there so specify wich ones are trusted and wich one aren't08:22
ykarelquiquell|rover, can you check if some port has mac address:- B6:A4:00:4F:53:8F08:23
ykarelError, some other host (B6:A4:00:4F:53:8F) already uses address 172.17.0.7308:23
ykarelseeing ^^ in one ovb job08:23
* quiquell|rover checking08:23
quiquell|roveropenstack --os-cloud nodepool port list  |grep -i B6:A4:00:4F:53:8F08:26
quiquell|rovernoi results here08:26
ykarelquiquell|rover, ack, so seems it's out of neutron dhcp08:27
ykarelquiquell|rover, can you also check if there are ports without mac address: fa:16:3e08:28
ykarelgrep -v fa:16:3e08:28
quiquell|roversure08:29
quiquell|rovernope08:29
ykarelokk,08:31
sshnaidmquiquell|rover, marios https://code.engineering.redhat.com/gerrit/#/c/16840808:37
mariossshnaidm: ack gimme few will check08:40
hubbot1All check jobs are working fine on master, stable/pike, master, stable/rocky, stable/rocky, stable/ocata.08:42
quiquell|roversshnaidm: I am taking care of reproducer08:42
quiquell|roverregarding opendev08:42
sshnaidmquiquell|rover, ack08:42
quiquell|roveropendev migration before PTG yey !!!08:43
quiquell|roveris RDO already migrated at zuul tenant ?08:44
quiquell|roverykarel: ^ ¿08:44
ykarelquiquell|rover, means?08:47
ykarelyou mean zuul config migrated from software factory?08:47
quiquell|roverykarel: I mean stuff like zuul-jobs now are at zuul/zuul-jobs08:48
quiquell|roverykarel: or some project moving from openstack-infra to openstack08:48
ykarelquiquell|rover, yes today morning opendev migration is done08:48
quiquell|roverykarel: I mean at zuul tenant08:48
quiquell|roverykarel: ack thanks08:48
ykarelquiquell|rover, see recent merges at https://review.rdoproject.org/r/#/q/owner:tdecacqu%2540redhat.com+status:merged08:49
ykarelif something missed please fix :)08:49
quiquell|roveryep08:49
quiquell|roverx/browbeat08:52
quiquell|roverx is external ?08:52
ykarelx i think is which is not under openstack/governance08:52
quiquell|roverack08:52
mariossshnaidm: too late :) next time08:55
sshnaidmmarios, it blocked me :)08:55
mariossshnaidm: quiquell|rover can you check this please https://code.engineering.redhat.com/gerrit/#/c/168410/ info in https://tree.taiga.io/project/tripleo-ci-board/task/988 item 3 on reviews. hoping it wil fix the issue in https://code.engineering.redhat.com/gerrit/#/c/167540/08:56
marioshttps://sf.hosted.upshift.rdu2.redhat.com/logs/40/167540/16/check/tripleo-ci-rhel-8-standalone-rhos-15/a53536a/job-output.txt.gz#_2019-04-19_11_47_26_21312108:56
mariosack sshnaidm no problem was mostly to tell you i didn't forget i just switch to check it and saw it merged08:56
quiquell|rovermarios: let's cross fingers libselinux is a pain in the a..08:58
*** sanjayu__ has joined #oooq08:58
quiquell|rovermarios: I am going to merge it08:58
quiquell|rovermarios: is config project08:58
mariosquiquell|rover: ack thanks08:58
mariosquiquell|rover: exactly thats my problem re the depends-on08:59
*** sanjayu_ has quit IRC08:59
quiquell|rovermarios: you also need to wait for a image rebuild I think08:59
mariosquiquell|rover: k was planning on switch now thanks so can try later08:59
quiquell|rovermarios: ack make sure that you are checking new image08:59
quiquell|rovermarios: when you come back to the dutie08:59
*** sanjayu__ has quit IRC09:00
*** sanjayu_ has joined #oooq09:00
mariosquiquell|rover: ack09:00
mariosquiquell|rover: you seen that one/known issue? standalone-upgrade fails at 2019-04-22 08:40:22 | 2019-04-22 08:40:22.950 102903 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-]         "CalledProcessError: Command '['systemctl', 'stop', u'tripleo_keystone.service']' returned non-zero exit status 1",[00m09:04
marioshttp://logs.openstack.org/89/651789/6/check/tripleo-ci-centos-7-standalone-upgrade/c6bfa88/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz09:04
mariosfood biab09:05
*** holser_ has joined #oooq09:09
*** holser_ has quit IRC09:11
quiquell|rovermarios: is standalone upgrade suppose to work ?09:14
chandankumarquiquell|rover: \o/09:16
chandankumarquiquell|rover: with standalone upgrade os_tempest job, we are seeing a weired issue while build packages09:16
chandankumarquiquell|rover: https://logs.rdoproject.org/25/20325/1/check/periodic-tripleo-ci-centos-7-standalone-upgrade-master/d8d56ea/job-output.txt.gz#_2019-04-18_10_53_27_06272209:16
chandankumarThere was an issue creating /root/DLRN as requested: [Errno 13] Permission denied: '/root/DLRN'09:17
quiquell|roverthat's new :-/09:17
chandankumarearlier we thought it is related to tempest playbook gather facts but it is not09:18
chandankumarsomehow, ansible user path is getting changed to /root09:18
quiquell|roverchandankumar:09:19
quiquell|roverThis is related to running build-test-packages with become: true09:19
quiquell|roversince build_repo_dir = ansible_user_dir09:19
quiquell|roverhumm09:20
chandankumarquiquell|rover: ok looking into that09:23
chandankumarquiquell|rover: if we see here https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/multinode-standalone-upgrade.yml#L4809:33
chandankumarbecome: true is not used09:33
quiquell|roveryep... I am looking where09:34
quiquell|roversince ansible_user_id is root09:34
quiquell|roverare we running the playbook as root ?09:34
chandankumarquiquell|rover: only tempest playbook is runned as root09:34
chandankumarquiquell|rover: https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/tempest.yml#L5309:36
chandankumarand after that standalone-upgrade playbook is called?09:37
quiquell|roverhumm build_repo_dir can be overwritten https://github.com/openstack/tripleo-quickstart-extras/blob/db85f97e1d4629a9677e222f51544dedc902f877/roles/build-test-packages/defaults/main.yml#L309:37
quiquell|roverchandankumar: I think so09:37
quiquell|roveror do we do upgrade and then tempest ?09:37
quiquell|roverbut thinking about it09:38
quiquell|roverif was running playbook as root it would have permissions09:38
quiquell|roveris like running with normal user but wrong build_repo_dir09:38
chandankumarquiquell|rover: so here were we are doing standalone deploy then running tempest and then calling standalone upgrade playbook09:39
quiquell|roverchandankumar: then we have to see two runs of build-test-packages09:40
quiquell|roverhumm noe09:40
quiquell|roverchandankumar: is like the first time standalone_upgrades run with os_tempest ?09:42
chandankumarquiquell|rover: yes09:42
chandankumarquiquell|rover: as per this https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/standalone-jobs.yaml;h=3fc7505f588dec3f5b94d1794756b57a05d355fd;hb=d8db8dee975ee4c219442c24c328f1f432b96fdc09:43
quiquell|roverthen it could be related to ansible_become: true09:43
quiquell|rovermaybe it get saved as a fact between playbooks09:44
chandankumarfirst quickstart will be called then standalone tag is called that time os_tempest role is called and then after that standalone upgrade is called.09:44
quiquell|roverbut os_tempest is called at differetn playbook ?09:44
chandankumaris it possible to destroy the facts once playbook fun finishes?09:44
chandankumarquiquell|rover: yes09:44
quiquell|roverbut we can have fact cached09:44
quiquell|rovercan you comment out os_tempest part and see what happends ?09:45
quiquell|roverjust the call to the role09:45
quiquell|roverthe task09:45
chandankumarhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/multinode-standalone.yml#L4909:45
chandankumarok09:45
chandankumarlet me do that09:45
chandankumarquiquell|rover: sshnaidm https://review.rdoproject.org/r/#/c/20350/ small patch for collect-logs rdo jobs09:49
sshnaidmchandankumar, +w09:50
quiquell|roverykarel: where did we have the rdo zuul tenannt connections yaml ?09:52
ykarelquiquell|rover, you mean https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul/rdo.yaml09:54
quiquell|roverykarel: nope the ini file with [connection] ... etc09:55
quiquell|roverHUmm resources maybe09:56
*** holser_ has joined #oooq09:58
mariosquiquell|rover: yeah well that patch is meant to fix it but that issue is a new one... i mean see it is green in check https://review.opendev.org/#/c/651789/09:59
*** holser__ has joined #oooq10:00
*** holser_ has quit IRC10:04
quiquell|rovermarios: I see it failing all the time http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-upgrade10:06
quiquell|rovermarios: maybe jfrancoa knows more10:06
mariosquiquell|rover: yes that patch is meant to fix it ...10:06
mariosquiquell|rover: 12:59 < marios> quiquell|rover: yeah well that patch is meant to fix it but that issue is a new one... i mean see it is green in check https://review.opendev.org/#/c/651789/10:06
quiquell|roverdamn ok10:06
mariosquiquell|rover: the job is only green in that patch cos that is the one that fixes it. at least it was green but now we have that new thing, which is why i was askign10:07
quiquell|rovergood at check fails at gate10:07
quiquell|roverlooks like new promotion has break it again10:07
quiquell|rovermaybe10:07
quiquell|roverbetween check and gate time window10:07
quiquell|roverahh damn this is check too10:07
mariosquiquell|rover: ack ok so you answered my question it isn't a known thing will debug a bit there thanks10:09
quiquell|rovermarios: yep looks like the upgrade is trying to stop some renamed sysmted services or the like10:09
quiquell|rovermaybe a kolla change ?10:10
mariosquiquell|rover: ack10:15
mariosquiquell|rover: how about that one is it known (same review, standalone failing on tempest http://logs.openstack.org/89/651789/6/check/tripleo-ci-centos-7-standalone/9e1b503/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-04-22_08_59_4110:16
mariosquiquell|rover: the standalone-upgrade isn't actually voting yet so we could merge that fix anyway though that is voting ^^10:16
chandankumarmarios: we have seen this tempest SSHException: Error reading SSH protocol banner issue few days, it comes and goes10:18
marioschandankumar: ack so recheck?10:18
chandankumarmarios: yes10:18
chandankumarmay be we can move to os_tempest better it is working fine on periodic jobs10:19
chandankumararxcruz|off|23: ^^ ?10:19
quiquell|rovermarios, chandankumar: That was related to old cirros image I think10:19
marioschandankumar: thanks10:19
mariosquiquell|rover: ack10:19
chandankumarmarios: wes was working on unifiing cirros image10:19
chandankumarlet me find the link10:19
chandankumarhttps://review.opendev.org/#/q/status:abandoned+topic:bug/1824243 related to cirros but got abandoned /me wait for weshay|rover to come online10:21
marioschandankumar: thanks10:21
*** quiquell|rover is now known as quiquell|ruck10:21
quiquell|ruckok one merged to opendev migration https://review.opendev.org/#/c/654300/10:23
ykarelquiquell|ruck, let's rerun periodic jobs10:27
quiquell|ruckykarel: still the other review is not merged10:27
quiquell|rucklet's wait for it10:28
ykarelquiquell|ruck, that's only for f2810:28
ykarelmainly we needed tq patch, that's merged10:28
ykarelcurrently zuul queue is nearly empty:- https://review.rdoproject.org/zuul/status, so good to run some jobs and have some early feedback10:29
quiquell|ruckykarel: we do with a test review or hack the time ?10:36
ykarelquiquell|ruck, we can temporary update periodic timing in config repo10:37
ykarelif going ^^ way let's reschedule  both master and stein10:38
chandankumarykarel: but fs027 master and stein is broken at tempest10:39
chandankumarykarel: we need this patch https://review.opendev.org/#/c/653742/ patch first na then reschedule the periodic10:39
ykarelchandankumar, i think we should run before to see if there is no new issue10:40
quiquell|ruckykarel: then let's just do some test review with jobs10:40
quiquell|ruckykarel: we don't even need a promotion just a re-run10:40
ykarelquiquell|ruck, all jobs?10:41
ykarelvia review10:41
quiquell|ruckykarel: yep10:41
ykarelquiquell|ruck, okk for me10:41
quiquell|ruckthe whole pipeline after container builds I suppose10:41
ykarelyes10:41
quiquell|ruckit's going to be easier to recheck in case of fix10:41
ykareliirc there is a review we can reuse it10:41
quiquell|rucknice10:41
hubbot1All check jobs are working fine on master, stable/pike, master, stable/rocky, stable/rocky, stable/ocata.10:42
quiquell|ruckHumm I am going to fix this hubbot thing10:42
ykarelquiquell|ruck, we can reuse https://review.rdoproject.org/r/#/c/20160/1/.zuul.yaml10:42
ykarelquiquell|ruck, /me updating the review10:43
quiquell|ruckykarel: thanks!10:44
chandankumarykarel: ack!10:45
ykarelquiquell|ruck, chandankumar started both stein/master at https://review.rdoproject.org/r/#/c/20160/ except fs027 and fs02110:57
chandankumarykarel: ack!10:58
quiquell|ruckykarel: noted10:58
quiquell|rucksshnaidm: I am going to update docker hub latest for reproducer11:03
quiquell|rucksshnaidm: there are issues with new config at opendev11:04
quiquell|rucksshnaidm: and old zuul version11:04
*** holser__ has quit IRC11:18
*** udesale has quit IRC11:34
jfrancoaquiquell|ruck: o/, sorry I missed your message...I was checking the log and it seems that paunch is trying to stop the containers implying that they're already managed by systemd (as they are in stein, systemd handles the podman containers). But this is an upgrade from docker to podman, so no systemd "tripleo_" units exist.. https://github.com/openstack/paunch/blob/master/paunch/utils/systemd.py#L12311:49
*** d0ugal has quit IRC11:49
jfrancoaquiquell|ruck: I'd create another lp bug for this, if it doesn't exist11:49
quiquell|ruckjfrancoa: ack thanks11:49
*** ykarel is now known as ykarel|afk11:51
*** d0ugal has joined #oooq11:51
mariosjfrancoa: quiquell|ruck added as issue '3' in https://tree.taiga.io/project/tripleo-ci-board/task/989 for now.. waiting to see it again in http://zuul.openstack.org/stream/7c90228da8a64be483c55149919f1cee?logfile=console.log and i'll file the bug11:52
jfrancoamarios: cool, thanks11:52
*** rlandy has joined #oooq12:06
*** rlandy is now known as rlandy|ruck12:06
*** amoralej is now known as amoralej|lunch12:09
quiquell|ruckmarios, ykarel|afk, rlandy|ruck: second patch for opendev migration merged12:10
rlandy|ruckquiquell|ruck: wow - yeah - I saw that12:10
quiquell|ruckrlandy|ruck: let's see if we are better now12:11
quiquell|ruckrlandy|ruck: https://review.rdoproject.org/r/#/c/20160/12:11
rlandy|ruckweshay|rover: we still don't have a dfg/rca/etc for https://trello.com/c/nBx3IF93/954-cixlp1825220tripleociproa-f28-master-standalone-job-fails-container-image-prepare-no-module-named-pkgresources12:11
quiquell|rucktest project to rerun periodics12:11
quiquell|ruckrlandy|ruck: it's not fixed ?12:11
*** quiquell|ruck is now known as quiquell|rover12:12
quiquell|roverunderstood it wrongly12:12
rlandy|ruckquiquell|rover: we were following a f28 failure ^^ and I don;t think we settled on a dfg there12:13
rlandy|ruckwhy that is happening12:13
rlandy|ruckwill have to check status of f2812:13
quiquell|roverWell everything is pretty broken12:13
quiquell|roverso is difficult to know12:13
quiquell|roverLet's see know that fixed are merged12:13
rlandy|ruckhttp://logs.rdoproject.org/openstack-periodic-master/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-master/62370f6/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz12:16
rlandy|ruckquiquell|rover; broken from opendev move?12:16
rlandy|ruckor other stuff12:16
weshay|roverrlandy|ruck aye I know :( I haven't been able to get a live one going12:18
weshay|rovermaybe quiquell|rover can get a f28 periodic repro going12:18
rlandy|ruckweshay|rover: k - so I'll move the card with an explanation12:19
quiquell|roverrlandy|ruck, weshay|rover: Let me take a look at f28 bug was ignoring it trying to fix reproducer for opendev12:19
quiquell|rovernow is working12:19
rlandy|ruckquiquell|rover: sure - I was just prep'ing for cix meeting12:20
quiquell|roverwhat a good comaback from PTO :-)12:21
quiquell|roverrlandy|ruck, weshay|rover: to fix reproducer + upgrade jobs https://review.opendev.org/65443612:21
rlandy|ruckquiquell|rover: I know :(12:22
quiquell|roverLet's feed the system12:22
*** quiquell|rover is now known as quique|rover|eat12:22
mariosquique|rover|eat: rlandy|ruck fyi just filed that https://bugs.launchpad.net/tripleo/+bug/182582612:23
openstackLaunchpad bug 1825826 in tripleo "standalone-upgrade failing during deploy steps for CalledProcessError: Command '['systemctl', 'stop', u'tripleo_keystone.service']' returned non-zero exit status" [High,Triaged] - Assigned to Marios Andreou (marios-b)12:23
*** rlandy|ruck is now known as rlandy|ruck|mtg12:36
*** rlandy|ruck|mtg is now known as rlandy|ruck12:42
weshay|roverrlandy|ruck|mtg quique|rover|eat shall we chat for a minute?12:42
hubbot1All check jobs are working fine on master, stable/pike, master, stable/rocky, stable/rocky, stable/ocata.12:42
weshay|roverhubbot1 YOU LIE12:42
hubbot1weshay|rover: Error: "YOU" is not a valid command.12:42
weshay|roverzbr ^12:42
rlandy|ruckweshay|rover: ack12:46
rlandy|ruckmarios: https://code.engineering.redhat.com/gerrit/#/c/168425/12:46
rlandy|ruckrhos-14 has a failure since new puddle12:46
mariosrlandy|ruck: ack12:47
rlandy|ruckweshay|rover: joined your blue12:48
*** gkadam has quit IRC12:49
weshay|roverpanda did you find the containers you were looking for?12:49
*** gkadam has joined #oooq12:52
*** quique|rover|eat is now known as quiquell|rover12:52
quiquell|roverweshay|rover: ready12:52
weshay|roverhttps://bluejeans.com/u/whayutin/12:56
quiquell|roverhttps://review.rdoproject.org/r/#/c/20160/13:01
mariosscrum now? new time?13:01
weshay|roverya13:01
*** ykarel|afk is now known as ykarel13:01
rf0lc0ping scrum13:02
*** rf0lc0 is now known as rfolco13:02
weshay|roversshnaidm scrum13:03
quiquell|roverykarel: this is related to the mac address you were searching before ? http://logs.rdoproject.org/60/20160/2/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/7cef42c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz13:03
ykarelquiquell|rover, ack13:03
ykarelquiquell|rover, seems different13:04
quiquell|roveris from the test review to re-run periodics13:04
ykarelsingle failure or multiple?13:05
*** skramaja has quit IRC13:06
*** rlandy|ruck is now known as rlandy|ruck|mtg13:08
*** ratailor has quit IRC13:16
sshnaidmrlandy|ruck|mtg, https://code.engineering.redhat.com/gerrit/16842913:16
rlandy|ruck|mtgsshnaidm: ^^ needs to be reviewed by admins13:19
rlandy|ruck|mtgadding our resources13:19
sshnaidmrlandy|ruck|mtg, I can't change connection via patch, asking in internal..13:20
rlandy|ruck|mtgsshnaidm: what do you mean via patch?13:21
rlandy|ruck|mtghttp://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/resources/tripleo-ci-internal.yaml#n8713:21
rlandy|ruck|mtgsshnaidm: ^^13:21
rlandy|ruck|mtgall that needs to change13:21
chandankumarweshay|rover: https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7-minimal13:23
*** amoralej|lunch is now known as amoralej13:24
*** aakarsh has joined #oooq13:25
sshnaidmrfolco, https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/project-templates.yaml#L42413:45
sshnaidmrfolco, I think you can replace pike with stein13:45
rfolcosshnaidm, we did not replace any pike jobs, are you sure we should replace? why not just add stein to branchless job template ?13:48
sshnaidmrfolco, I hope we'll do soon.. this will add +1 job to every patch, and not sure it worth it, but up to you13:48
*** mjturek has joined #oooq13:50
rfolcosshnaidm, ok, will propose it, thx for the reminder13:52
*** vinaykns has joined #oooq13:53
sshnaidmmarios, for me box reprovision worked only for 7.2, and then just "yum update -y"13:56
sshnaidmmarios, any later version freezes the box13:57
mariossshnaidm: ok thanks noted13:57
*** rlandy|ruck|mtg is now known as rlandy|ruck13:58
rfolcorlandy|ruck, my beaker box works, thanks14:06
rlandy|ruckrfolco: great - thanks14:06
rfolcowondering what is the quickest way to get a rhel7 box to test14:06
quiquell|roverrlandy|ruck: we can workflow the zuul reproducer fix for upgrades https://review.opendev.org/#/c/65443614:09
quiquell|roverrlandy|ruck: I have add a comment to the two kind of playbooks it generates14:10
quiquell|roverrlandy|ruck: Humm I see some failed to delete stacks14:15
quiquell|roverrlandy|ruck: Do we have any issue with nodepool ?14:15
mariosthanks for your time just now re beaker rlandy|ruck14:15
rlandy|ruckquiquell|rover: sec - just closing a review out - will check14:15
rlandy|ruckmarios: https://code.engineering.redhat.com/gerrit/#/c/168210/ - had to rebase that - it will fail due to opendev stuff. But they are non-voting so maybe it's ok to merge14:17
rlandy|ruckquiquell|rover: checking now14:17
rlandy|ruckquiquell|rover:ekk - we have a ton of failed stacks - checking why14:19
rlandy|ruck| stack_status_reason   | Resource CREATE failed: ResourceInError: resources.baremetal_env.resources.openstack_baremetal_servers.resources[1].resources.baremetal_server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"14:20
rlandy|ruckcleaning up14:20
rlandy|ruckquiquell|rover: we added two mini clean up scripts14:26
mariosrlandy|ruck: ack14:26
quiquell|roverrlandy|ruck: did they run ?14:26
rlandy|ruckmarios: k - merging that14:26
rlandy|ruckmarios: it will run once a day14:26
rlandy|ruckquiquell|rover: I'd have to check the logs on te-broker server14:27
quiquell|roverrlandy|ruck: porting reproducer to opendev14:27
rlandy|ruckbut did a manual kick now14:27
mariosrlandy|ruck: ack updated the task and assigned to you14:27
rlandy|ruckquiquell|rover: k- thanks for taking care of reproducer14:27
quiquell|roverrlandy|ruck: https://review.rdoproject.org/r/#/c/20371/14:27
quiquell|roversshnaidm, weshay|rover: ^14:27
sshnaidmquiquell|rover, not all moved to opendev.org14:33
sshnaidmquiquell|rover, is zuul-jobs under zuul/ now? not openstack/ ?14:33
quiquell|roversshnaidm: zuul14:35
quiquell|roverdid have to change it to fix14:35
sshnaidmquiquell|rover, and what with https://github.com/openstack/openstack-zuul-jobs ?14:37
quiquell|roversshnaidm: this is different jobs14:37
quiquell|roversshnaidm: you have zuul-jobs and openstack-zuul-jobs14:37
quiquell|roversshnaidm: they are different projects14:37
sshnaidmquiquell|rover, ack14:38
quiquell|roverrlandy|ruck, ykarel: test review for periodics is working fine https://review.rdoproject.org/r/#/c/20160/14:38
quiquell|roverovs issues feels related to failed to delete stack so migration is in better state now14:39
ykarelquiquell|rover, ack, master periodic also started let's see if we have promotion14:39
hubbot1All check jobs are working fine on master, stable/pike, master, stable/rocky, stable/rocky, stable/ocata.14:42
quiquell|roverykarel: would be nice14:42
quiquell|roverok droping now14:42
*** quiquell|rover is now known as quiquell|off14:42
rlandy|ruckquiquell|off: ykarel: tenant should be clean now of failed stacks/servers14:44
ykarelrlandy|ruck, ack14:46
quiquell|offrlandy|ruck: ack14:50
quiquell|offrlandy|ruck, sshnaidm: fixed https://review.rdoproject.org/r/#/c/20371/14:50
quiquell|offAnd now I drop let's x fingers for tomorrow14:50
quiquell|offrlandy|ruck, sshnaidm: Also upgraded the containers https://review.rdoproject.org/r/#/c/20381/14:51
sshnaidmquiquell|off, cool14:53
pandaweshay|rover: yes, current error is dlrn repo setup during building15:06
*** ykarel is now known as ykarel|away15:06
pandamaking progress at least15:07
panda(easter monday here)15:07
rlandy|ruckweshay|rover: do you remember why we decide against ssl_overcloud in fs053?15:07
weshay|roverrlandy|ruck 53 is browbeat ya?15:10
weshay|roverpanda sorry to disturb you.. forgot it's a holiday15:10
rlandy|ruckweshay|rover: yes - fs053 is failing a lot - getting rd of diff with fs00115:10
weshay|roverrlandy|ruck ya.. making fs001 or as close as possible +115:10
weshay|roverrlandy|ruck ideally a fs override and using fs001 directly as you have done w/ piddone would be ideal imho15:15
rlandy|ruckweshay|rover: ack - when we redo browbeat altogether15:15
rlandy|ruckright now just tryimg to get rid pf failures15:16
weshay|roverk15:16
rlandy|ruckweshay|rover: https://review.opendev.org/#/c/654481 - let's see if browbeat passes with ssl.15:21
rlandy|ruckaakarsh: ^^ fyi15:22
rlandy|ruckweshay|rover: I want to rekick rocky and stein phase 1 rdo_trunk_minimal as the bug is fixed.15:28
rlandy|ruckdo I just click 'build now' on https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-rocky-current-tripleo/?15:29
*** dsneddon has quit IRC15:31
weshay|roverrlandy|ruck you don't have to rekick the whole thing15:36
weshay|roverrlandy|ruck /me gets sec15:36
rlandy|ruckoh oops15:37
weshay|rovernp15:37
weshay|roveryou can click the failed multi-job15:37
weshay|roverand then click15:37
weshay|rover"resume build"15:37
weshay|roverrlandy|ruck and only the failed jobs will retrigger15:37
weshay|roverrlandy|ruck is stein tripleo rerunning?15:38
rlandy|ruckweshay|rover; k, doing that in stein15:38
rlandy|ruckyes15:38
rlandy|ruckit failed on the same bug15:38
weshay|roverah see it15:38
weshay|roverOH looks good15:38
weshay|roverhttps://review.rdoproject.org/r/#/c/20160/15:38
weshay|roverright?15:38
rlandy|ruckweshay|rover: also let's remove these stupid failing gates15:38
weshay|roverrlandy|ruck happy to15:39
rlandy|ruckweshay|rover: can we just delete them from jenkins?15:39
rlandy|ruckbj?15:39
weshay|roversure15:39
rlandy|rucklet's do this15:39
*** ykarel|away has quit IRC15:59
* marios out16:04
*** marios has quit IRC16:07
sshnaidmrlandy|ruck, where is browbeat role hosted? Need to fix it for opendev: The error was: 'dict object' has no attribute 'git.openstack.org/openstack/browbeat'16:08
rlandy|rucksshnaidm: from the comment I made on quiquell|off's patch?16:13
rlandy|ruckhe just added an x16:13
sshnaidmrlandy|ruck, no, I mean failing job.. preparing a patch for browbeat16:14
rlandy|ruckit should be on git.openstack.org/openstack/browbeat' moved to opendev16:14
rlandy|rucknot sure why it's different16:14
rlandy|ruckhttps://github.com/openstack/browbeat/16:15
rlandy|rucksshnaidm: ^^16:15
chandankumarrlandy|ruck: the projects which are not defined in governence repo (does not comes under any openstack team) and are renamed to x/project name16:15
rlandy|rucksshnaidm; asking rook16:15
sshnaidmrlandy|ruck, yeah, found it16:15
rlandy|ruckchandankumar: why would that project be undefined?16:16
rlandy|ruckwe have been working with it for some time16:16
rlandy|rucksshnaidm: you all set now?16:17
chandankumarrlandy|ruck: sorry you mean opendev.org/x/browbeat is undefined?16:19
sshnaidmrlandy|ruck, weshay|rover need to merge https://review.opendev.org/#/c/654557/ to make 053 job to pass16:21
weshay|roverk16:21
*** sshnaidm is now known as sshnaidm|afk16:22
*** tosky has joined #oooq16:22
rlandy|ruckchandankumar: ack - it should be under openstack16:24
weshay|roverrlandy|ruck ok.. merged16:24
rlandy|rucklike all other projects16:24
rlandy|ruckweshay|rover: k - bj again to delete in  ci-centos ?16:25
weshay|roveraye16:25
rlandy|ruckchandankumar: I'm just not sure why this repo is the only different one16:25
rlandy|rucksomewhere we must be missing a definition16:26
* rlandy|ruck need to ask rook16:26
*** gkadam has quit IRC16:30
chandankumarrlandy|ruck: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005438.html16:30
rfolcoweshay|rover, panda pre-planning ?16:31
sshnaidm|afkrlandy|ruck, weshay|rover periodic jobs in rdo: https://review.rdoproject.org/r/#/c/20388/16:31
rfolcoweshay|rover, panda no show ;(16:34
rfolcowait panda is there16:35
hubbot1All check jobs are working fine on master, stable/pike, master, stable/rocky, stable/rocky, stable/ocata.16:42
rlandy|ruckweshay|rover: https://review.opendev.org/#/c/654568/16:43
rlandy|ruckchandankumar: thanks for that ref16:46
chandankumarweshay|rover: rlandy|ruck regarding tomorrow perf meeting, anything to look before joining meeting?16:47
* chandankumar last looked browbeat back in early of 201716:47
weshay|roverrlandy|ruck https://review.rdoproject.org/r/2039416:53
rlandy|ruckchandankumar: it's been a while that I worked with per as well16:55
rlandy|ruckchandankumar: we want to move them out of jenkins entirely16:56
rlandy|ruckto zuul16:56
rlandy|rucklike we did with ha16:56
rlandy|ruckI think we just go and see what the deal is16:56
chandankumarrlandy|ruck: thanks for the info, sure :-)16:56
* chandankumar out now16:56
*** chandankumar is now known as raukadah16:56
rlandy|rucksshnaidm|afk: who has +2 here? https://review.opendev.org/#/c/654557/16:57
sshnaidm|afkrlandy|ruck, https://review.opendev.org/#/admin/groups/1425,members16:57
rlandy|rucksshnaidm|afk; k- will ping them16:59
rlandy|ruckaakarsh: ^^17:00
rlandy|rucksshnaidm|afk: will address this at the meeting tomorrow17:00
weshay|roverrfolco oh man.. I figured panda would not join17:03
weshay|roversorry rfolco panda17:03
rfolcoweshay|rover, he actually joined and I rescheduled to tomorrow17:04
rfolcoI forgot it was holiday17:04
weshay|roverI see.. sorry17:04
rfolcoeurope knows how to live17:05
weshay|roverright17:05
pandawell, wife is sick, I'm home, not doing anything, got bored of all this holiday thing17:05
pandaended up studying mostly.17:06
rfolcopanda, sorry to hear that17:06
rlandy|ruckGERRIT_HOST17:09
rlandy|ruckneeds update as well17:09
*** MisterS has quit IRC17:14
weshay|roverrlandy|ruck should I close tmate?17:19
rlandy|ruckweshay|rover: not yet - we have more changes17:20
rlandy|ruckweshay|rover: see tmate17:20
rlandy|ruckweshay|rover: plus - need to see where GERRIT_HOST is defined17:21
rlandy|ruckenv var17:21
weshay|roverk..  I see17:21
rlandy|ruckweshay|rover: see https://ci.centos.org/view/rdo/view/tripleo-gate/job/tripleo-quickstart-extras-gate-master-delorean-full-featureset052/3239/injectedEnvVars/17:21
rlandy|ruckweshay|rover: https://review.rdoproject.org/r/#/c/20395/17:28
rlandy|ruckweshay|rover: do you have access to edit GERRIT_HOST?17:28
rlandy|ruckweshay|rover: lastly there are a bunch of grafana updates needed17:29
rlandy|ruckwill do those after lunch17:29
weshay|roverrlandy|ruck I don't not any more17:29
weshay|roverthey killed most of my access17:29
rlandy|ruckweshay|rover: dmsimard?17:29
weshay|rovermaybe.. doubt it.. probably  in #centos-devel17:30
*** amoralej is now known as amoralej|off17:44
*** dsneddon has joined #oooq17:45
aakarshhey rlandy|ruck sorry don't use irc much. just saw pings.17:50
*** zbr is now known as zbr|pto17:56
*** ykarel|away has joined #oooq17:56
zbr|ptoweshay|rover: i will have a look at hubbot on wednesday evening, when I am back from london trip, now i try to prepare for the next two days.17:57
weshay|roverzbr|pto roger that :)17:57
weshay|rovergood luck17:57
zbr|ptothanks, i guess i need a little bit.17:58
*** holser_ has joined #oooq18:06
*** ykarel|away has quit IRC18:11
*** rnoriega has quit IRC18:11
*** holser_ has quit IRC18:11
*** rnoriega has joined #oooq18:12
rlandy|ruckaakarsh: we need someone to merge https://review.opendev.org/#/c/654557/18:16
aakarshI tagged the current cores rlandy|ruck and pinged them in internal channel #rhos-scale18:19
rlandy|ruckthanks18:19
weshay|roverrlandy|ruck I merged https://review.rdoproject.org/r/#/c/20395/18:21
rlandy|ruckweshay|rover: thanks18:22
aakarshgot it merged rlandy|ruck18:23
rlandy|ruckaakarsh: thanks18:23
aakarshI asked the folks to join this channel, but otherwise best way to reach out to them would be #rhos-scale on internal and ping tcw sai and zul18:24
aakarshoh wait we got zul already here, he's got a +2 on browbeat18:24
rlandy|ruckjoined thanks18:26
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.opendev.org/602248, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.opendev.org/56429118:42
zbr|ptoweshay|rover: i was relying on ansible-doc for getting info about modules, but it seems that it does not return the full man page, documents parameters,  but the return values are not returned.18:50
zbr|ptoansible-doc get_url - for example18:50
zbr|ptodocuments params ,but not the return values.18:50
zbr|ptolike status_code ....18:50
weshay|roverzbr|pto not worried about it now18:50
weshay|roverinfra rename blowing us up18:50
zbr|pto:(18:51
rlandy|ruckweshay|rover: https://review.opendev.org/#/c/654568/18:59
*** dsneddon has quit IRC19:17
weshay|roverrlandy|ruck http://logs.openstack.org/periodic/19:18
weshay|roverrlandy|ruck https://review.opendev.org/65461419:32
*** dsneddon has joined #oooq19:35
*** sshnaidm|afk has quit IRC20:16
*** rlandy|ruck is now known as rlandy|ruck|biab20:26
*** jfrancoa has quit IRC20:29
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.opendev.org/602248, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.opendev.org/56429120:42
*** zbr|pto has quit IRC21:03
*** mjturek has quit IRC21:03
*** fultonj has joined #oooq21:13
*** fmount has quit IRC21:18
*** fmount has joined #oooq21:22
*** aakarsh has quit IRC21:44
vinayknsweshay|rover: hey.!21:49
weshay|roverhellooo21:50
*** aakarsh has joined #oooq21:51
vinayknsweshay: do we have recommended approach of installing tripleo on baremetal nodes..?21:52
*** aakarsh has quit IRC21:57
vinayknsweshay|rover^^21:58
*** irclogbot_0 has quit IRC21:58
*** panda has quit IRC21:58
vinayknsis this the recommended approach https://docs.openstack.org/tripleo-docs/latest/install/environments/baremetal.html21:58
*** panda has joined #oooq22:00
weshay|rovervinaykns do you have hardware for that?  that is not virt, nor nodepool.. it's 5-10 machines22:00
*** rlandy|ruck|biab is now known as rlandy|ruck22:00
rlandy|ruckweshay|rover: back22:01
vinayknsweshay|rover: Yea I do have hardware for that..I have access to 5 machines22:01
weshay|roverAH HA.. vinaykns very nice22:01
*** irclogbot_3 has joined #oooq22:01
weshay|rovervinaykns so this is for telemetry.. ya?22:01
vinayknsyea to test SAF22:02
weshay|rovervinaykns yes .. that is the recommended approach22:02
weshay|rovervinaykns you should probably team up w/ at least one ci person22:02
weshay|roverwhile ur working it for support22:02
vinayknsweshay|rover: sure, that would be great..whom do you want me to work with..?22:03
weshay|rovervinaykns when do you think you'd start on it?22:04
vinayknsweshay|rover: asap I guess, today I want to have atleast UC installed22:05
weshay|rovervinaykns ok.. I'll set up a meeting soon.. not this week, I'll include you w/ the performance guys we're going to walk through this stuff22:06
vinayknsweshay|rover: I want to clarify that I'm not from perf team, like that miscommunication happened before22:07
*** dtrainor_ has joined #oooq22:09
*** vkapalav has joined #oooq22:11
*** irclogbot_3 has quit IRC22:11
*** kopecmartin has joined #oooq22:11
weshay|rovervinaykns ya I know :) sorry about that.. you work w/ prad22:12
weshay|rovervinaykns we are however bringing on the perf team for that bm setup22:12
vkapalavyea, right..I just want to give headsup22:12
vkapalavweshay|rover: aah got it22:13
weshay|roverwell.. now we've changed nicks22:13
weshay|roverlolz22:13
weshay|roverjust to mess w/ my head22:13
*** jrist- has joined #oooq22:13
vkapalavhaha..sorry abt that..22:14
vkapalavyeah, I guess I need to head out...will ping you later..!22:15
weshay|roverk22:15
weshay|rover:)22:15
*** kopecmartin|off has quit IRC22:16
*** fultonj has quit IRC22:16
*** jbadiapa has quit IRC22:16
*** vinaykns has quit IRC22:16
*** dtrainor has quit IRC22:16
*** jrist has quit IRC22:16
*** irclogbot_2 has joined #oooq22:18
*** irclogbot_2 has quit IRC22:19
*** irclogbot_0 has joined #oooq22:19
*** vkapalav has quit IRC22:20
*** sshnaidm has joined #oooq22:24
*** sshnaidm is now known as sshnaidm|afk22:24
rlandy|ruckweshay|rover: promoter ... http://38.145.34.55/centos7_stein.log22:29
* weshay|rover looks22:30
weshay|roverfun22:30
rlandy|ruckweshay|rover: was this from the testproject job?22:31
rlandy|ruckif so, makes sense22:31
weshay|roverhrm22:31
weshay|roverrlandy|ruck but I didn't merge the change to allow that run to pass22:31
rlandy|ruckwe never ran image build22:31
weshay|roverrlandy|ruck this is probably a openstack.org vs. opendev.org bug22:31
weshay|roversec22:31
rlandy|ruckweshay|rover: I don;t think so22:32
rlandy|ruckci-scripts/container-push/container-push.yml22:36
rlandy|ruckhttps://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/container-push.yml#L5022:38
rlandy|ruckweshay|rover: ^^22:39
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.opendev.org/602248, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.opendev.org/56429122:42
weshay|roverrlandy|ruck yup22:44
weshay|roverthat's the thing22:44
weshay|roverhow did we miss that?22:44
rlandy|ruckweshay|rover: I'm trying to figure out the right url ...22:46
rlandy|ruckhttps://opendev.org/openstack/tripleo-common/plain/container-images/overcloud_containers.yaml.j222:46
rlandy|ruckdoes not resolve22:46
weshay|roverrlandy|ruck https://opendev.org/openstack/tripleo-common/raw/branch/master/container-images/overcloud_containers.yaml.j222:46
weshay|roverrlandy|ruck they change the git system to something more modern as well22:47
rlandy|ruckweshay|rover: yep - the plain things didn;t work22:47
rlandy|ruckreview coming up22:47
weshay|roverrlandy|ruck rock22:47
weshay|roverhrm... how is stein promoting though? did we waive 35?22:48
weshay|roverrlandy|ruck oh.. promoter Trying to promote current-tripleo to current-tripleo-rdo22:49
weshay|rover:)22:49
rlandy|ruckweshay|rover: no - didn't22:49
rlandy|ruckis that in criteria?22:49
weshay|roverrlandy|ruck it's promoting triple -> rdo22:50
rlandy|ruckweshay|rover: https://review.rdoproject.org/r/#/c/20406/1/ci-scripts/container-push/container-push.yml22:52
*** aakarsh has joined #oooq22:55
weshay|rover+222:55
weshay|roverwaiting on ci22:55
rlandy|ruckweshay|rover: there are other references in ci-scripts/dlrnapi_trigger/test_dlrnapi_trigger.py- do we care about those?22:57
weshay|roverhrm22:58
weshay|roverrlandy|ruck ugh22:59
weshay|roverone more for the team?22:59
weshay|roverblue?22:59
weshay|roverrlandy|ruck if you need to go, I can just do it23:00
rlandy|ruckweshay|rover: np - I'm here - joining blue23:02
*** tosky has quit IRC23:09
*** vkapalav has joined #oooq23:46
vkapalavweshay|rover: ping.23:47
weshay|rovervkapalav ahoy23:48
vkapalavyeah you were telling about perf tests23:49
vkapalavyou want to do perf tests after deploying osp13..?23:49
rlandy|ruckok - now for downtream updates23:52
*** vkapalav is now known as vinaykns23:54
weshay|roverheh.. this guy keeps changing nicks23:54
weshay|rovervinaykns I feel like I'm playing tag23:55
weshay|roverlolz23:55
weshay|rovervinaykns nope23:55
weshay|roverdon't worry about perf23:55
vinayknsweshay|rover: lol..sorry about that23:57
vinayknsweshay|rover: okay, don't hesitate to contact me if something comes up..23:59
« Sunday, 2019-04-21 Index Tuesday, 2019-04-23 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!