| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 00:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 00:10 |
| *** lblanchard has joined #tripleo | 00:25 | |
| openstackgerrit | Merged openstack/paunch master: switch documentation job to new PTI https://review.openstack.org/597759 | 00:29 |
| openstackgerrit | Merged openstack/paunch master: add python 3.6 unit test job https://review.openstack.org/597760 | 00:48 |
| *** lblanchard has quit IRC | 01:06 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 01:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 01:10 |
| *** mrsoul has joined #tripleo | 01:22 | |
| *** mschuppert has quit IRC | 01:25 | |
| *** rcernin has quit IRC | 01:40 | |
| *** rcernin has joined #tripleo | 01:40 | |
| *** weshay is now known as weshay_afk | 01:49 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 02:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 02:10 |
| *** psachin has joined #tripleo | 02:41 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 03:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 03:10 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements stable/queens: import zuul job settings from project-config https://review.openstack.org/597838 | 03:36 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements stable/pike: import zuul job settings from project-config https://review.openstack.org/597820 | 03:36 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements stable/ocata: import zuul job settings from project-config https://review.openstack.org/597805 | 03:36 |
| *** psachin has quit IRC | 03:48 | |
| *** udesale has joined #tripleo | 03:50 | |
| *** psachin has joined #tripleo | 04:01 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 04:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 04:10 |
| *** skramaja has joined #tripleo | 04:17 | |
| *** ykarel has joined #tripleo | 04:22 | |
| openstackgerrit | Dan Macpherson proposed openstack/ansible-role-openstack-operations master: Adding Backup and Restore Operations https://review.openstack.org/604439 | 04:27 |
| *** Petersingh has joined #tripleo | 04:37 | |
| *** dtrainor has quit IRC | 04:51 | |
| *** shyamb has joined #tripleo | 04:52 | |
| *** Petersingh is now known as Petersingh|afk | 04:54 | |
| *** ramishra has joined #tripleo | 04:59 | |
| *** spsurya has joined #tripleo | 05:03 | |
| *** saneax has joined #tripleo | 05:06 | |
| *** shyamb has quit IRC | 05:09 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 05:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 05:10 |
| Tengu | hello there :) | 05:19 |
| *** shyamb has joined #tripleo | 05:23 | |
| *** Petersingh|afk is now known as Petersingh | 05:23 | |
| *** jaganathan has joined #tripleo | 05:27 | |
| *** shyamb has quit IRC | 05:33 | |
| *** jtomasek has joined #tripleo | 05:37 | |
| *** quique|rover|off is now known as quiquell|rover | 05:45 | |
| *** shyamb has joined #tripleo | 05:49 | |
| *** bugzy_ has joined #tripleo | 06:02 | |
| *** ajo has joined #tripleo | 06:04 | |
| *** pcaruana has joined #tripleo | 06:05 | |
| *** bugzy has quit IRC | 06:05 | |
| *** dtrainor has joined #tripleo | 06:05 | |
| *** apetrich has joined #tripleo | 06:05 | |
| *** ksambor has joined #tripleo | 06:07 | |
| *** boazel has quit IRC | 06:10 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 06:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 06:10 |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-heat-templates stable/rocky: Fix syntax for set_fact module. https://review.openstack.org/604681 | 06:13 |
| openstackgerrit | Yurii Prokulevych proposed openstack/python-tripleoclient stable/rocky: Fix typo in upgrade playbook's name. https://review.openstack.org/604685 | 06:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/tripleo-ci master: Use python3 for tox linting https://review.openstack.org/588587 | 06:13 |
| *** jfrancoa has joined #tripleo | 06:15 | |
| *** jfrancoa has quit IRC | 06:17 | |
| *** mschuppert has joined #tripleo | 06:18 | |
| quiquell|rover | Good morning | 06:22 |
| *** iranzo has joined #tripleo | 06:22 | |
| *** shyamb has quit IRC | 06:24 | |
| *** shyamb has joined #tripleo | 06:24 | |
| openstackgerrit | Pranali Deore proposed openstack/tripleo-heat-templates master: Pass image import conf file to command which runs glance server https://review.openstack.org/598560 | 06:28 |
| *** shyam89 has joined #tripleo | 06:29 | |
| *** shyamb has quit IRC | 06:32 | |
| *** chandankumar has joined #tripleo | 06:33 | |
| *** chandankumar is now known as chkumar|ruck | 06:34 | |
| chkumar|ruck | quiquell|rover: Good morning | 06:34 |
| *** gvrangan has joined #tripleo | 06:38 | |
| *** yprokule has joined #tripleo | 06:46 | |
| chkumar|ruck | quiquell|rover: fs020 master is failing at reinstalling the undercloud 2018-09-24 01:27:36 | fatal: [undercloud]: FAILED! => {"changed": false, "msg": "path /var/lib/config-data/mysql/etc/../usr/share/zoneinfo/UTC does not exist", "path": "/var/lib/config-data/mysql/etc/../usr/share/zoneinfo/UTC", "state": "absent"} | 06:48 |
| chkumar|ruck | http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/bff71b8/logs/undercloud/home/zuul/undercloud_reinstall.log.txt.gz | 06:48 |
| chkumar|ruck | quiquell|rover: is it a known issue? | 06:48 |
| quiquell|rover | chkumar|ruck: Did you read Wes's email ? | 06:48 |
| quiquell|rover | chkumar|ruck: Looks like gates are block at timeouts | 06:48 |
| chkumar|ruck | quiquell|rover: checking email | 06:48 |
| chkumar|ruck | tripleo-ci-centos-7-containers-multinode is timing out a lot looking into thatr | 06:49 |
| chkumar|ruck | quiquell|rover: yes, looking into timed out jobs | 06:55 |
| *** adrianreza has quit IRC | 06:56 | |
| *** jfrancoa has joined #tripleo | 06:57 | |
| quiquell|rover | chkumar|ruck: Trying to do some tooling arount that | 06:57 |
| quiquell|rover | brb | 06:58 |
| *** quiquell|rover is now known as quique|rover|brb | 06:58 | |
| *** rdopiera has joined #tripleo | 07:04 | |
| *** Petersingh is now known as Petersingh|afk | 07:05 | |
| *** rcernin has quit IRC | 07:06 | |
| *** shyam89 has quit IRC | 07:07 | |
| *** phuongnh has joined #tripleo | 07:08 | |
| jaosorior | quique|rover|brb, chkumar|ruck: Any patches you know of that might help with the gate issues? | 07:10 |
| *** florianf|afk is now known as florianf | 07:10 | |
| *** shyamb has joined #tripleo | 07:13 | |
| *** gfidente has joined #tripleo | 07:13 | |
| *** Petersingh|afk is now known as Petersingh | 07:14 | |
| *** aufi has joined #tripleo | 07:14 | |
| *** dciabrin has joined #tripleo | 07:14 | |
| chkumar|ruck | jaosorior: I am trying to find out the reason for timedout not sure which patches needs more priority | 07:17 |
| jaosorior | chkumar|ruck: got any logs from the timesouts so I can help out too? | 07:18 |
| *** shyamb has quit IRC | 07:18 | |
| chkumar|ruck | jaosorior: timedout one http://logs.openstack.org/21/597221/8/gate/tripleo-ci-centos-7-containers-multinode/59f728a/ | 07:19 |
| chkumar|ruck | jaosorior: passed one http://logs.openstack.org/20/597820/2/gate/tripleo-ci-centos-7-containers-multinode/66bbf17/ | 07:19 |
| chkumar|ruck | jaosorior: dashboard link for timedout http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen&from=now%2Fd&to=now%2Fd | 07:20 |
| *** shyamb has joined #tripleo | 07:20 | |
| jaosorior | chkumar|ruck: thanks | 07:21 |
| openstackgerrit | Merged openstack/tripleo-image-elements master: add python 3.6 unit test job https://review.openstack.org/596064 | 07:22 |
| *** holser_ has joined #tripleo | 07:23 | |
| *** pmannidi has quit IRC | 07:26 | |
| *** shardy has joined #tripleo | 07:27 | |
| *** ykarel is now known as ykarel|lunch | 07:27 | |
| jaosorior | chkumar|ruck: probably unrelated, but the failing case has a bunch more validation errors | 07:28 |
| jaosorior | and some funky error when trying to make sure that the kvm module is present | 07:28 |
| jaosorior | The undercloud install took double the time as well | 07:30 |
| *** quique|rover|brb is now known as quiquell|rover | 07:30 | |
| quiquell|rover | jaosorior: not yet | 07:31 |
| *** gfidente has quit IRC | 07:31 | |
| jaosorior | chkumar|ruck, quiquell|rover: damn... the one that passed, takes 20 min in doing the undercloud install, and is using instack-undercloud. The one that failed took 40 min in doing the undercloud install and is using the containerized undercloud :/ | 07:33 |
| chkumar|ruck | quiquell|rover: what is the usual time taken for containerized undercloud install? | 07:33 |
| *** shyamb has quit IRC | 07:35 | |
| quiquell|rover | chkumar|ruck: 20 min is very small | 07:35 |
| quiquell|rover | chkumar|ruck, jaosorior: Doing some tooling to gather upstream stats | 07:36 |
| *** yprokule has quit IRC | 07:36 | |
| jaosorior | ok | 07:36 |
| *** gfidente has joined #tripleo | 07:36 | |
| mschuppert | quiquell|rover: morning, any info when we expect a queens promotion? (related to the rootwrap issue) | 07:37 |
| chkumar|ruck | quiquell|rover: jaosorior here is my stats on a passed one it is taking http://logs.openstack.org/64/596064/3/gate/tripleo-ci-centos-7-containers-multinode/65e440c/logs/ara_oooq/ -> 46 mins | 07:37 |
| quiquell|rover | mschuppert: Friday we were expecting one, will tell you more today | 07:37 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Allow debootstrap to clenaup without a kernel https://review.openstack.org/604692 | 07:37 |
| mschuppert | quiquell|rover: ok, thx | 07:37 |
| chkumar|ruck | on timedout on rax cloud -> http://logs.openstack.org/06/588606/5/gate/tripleo-ci-centos-7-containers-multinode/ddfb143/logs/ara_oooq/ -> 1:00: 47 | 07:38 |
| chkumar|ruck | http://logs.openstack.org/12/600512/10/gate/tripleo-ci-centos-7-containers-multinode/c9d2dfa/logs/ara_oooq/ -> 1:5 mins | 07:38 |
| chkumar|ruck | http://logs.openstack.org/15/585015/7/gate/tripleo-ci-centos-7-containers-multinode/25fd842/logs/ara_oooq/ -> 55: 12 | 07:38 |
| chkumar|ruck | http://logs.openstack.org/21/597221/8/gate/tripleo-ci-centos-7-containers-multinode/59f728a/logs/ara_oooq/ -> 54: 12 | 07:38 |
| *** gvrangan has quit IRC | 07:42 | |
| bandini | oh we are still super red in CI. meh. | 07:42 |
| chkumar|ruck | quiquell|rover: on vexhost it is taking around 48 to 56 mins | 07:44 |
| *** dtantsur|afk is now known as dtantsur | 07:46 | |
| *** Petersingh is now known as Petersingh|lunch | 07:46 | |
| *** jpena|off is now known as jpena | 07:49 | |
| *** cylopez has joined #tripleo | 07:50 | |
| jaosorior | damn, in some of them, the undercloud installation takes more time than even the overcloud installation | 07:50 |
| *** fhubik has joined #tripleo | 07:51 | |
| *** akrivoka has joined #tripleo | 07:51 | |
| chkumar|ruck | 2018-09-22 20:21:49 | TASK [Run tripleo-container-image-prepare] ************************************* why this one is taking more than 25 mins | 07:51 |
| chkumar|ruck | http://logs.openstack.org/06/588606/5/gate/tripleo-ci-centos-7-containers-multinode/ddfb143/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-09-22_20_21_49 | 07:51 |
| *** tosky has joined #tripleo | 07:51 | |
| jaosorior | chkumar|ruck: that's where the container images for the undercloud are gotten | 07:52 |
| quiquell|rover | chkumar|ruck: Those were suppose to be fixed with promotions :-( | 07:52 |
| quiquell|rover | jaosorior: ^ | 07:52 |
| jaosorior | we gotta see what configuration is used for that | 07:52 |
| quiquell|rover | having promotion reduce the time it gets to update the container image, it gets a fresh one | 07:52 |
| jaosorior | might be that it's using dockerhub | 07:52 |
| chkumar|ruck | In passed one it is taking around 19 mins http://logs.openstack.org/64/596064/3/gate/tripleo-ci-centos-7-containers-multinode/65e440c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-09-24_04_55_09 | 07:52 |
| jaosorior | that's still a long time | 07:53 |
| *** rdo has joined #tripleo | 07:53 | |
| chkumar|ruck | http://logs.openstack.org/12/600512/10/gate/tripleo-ci-centos-7-containers-multinode/c9d2dfa/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-09-23_15_53_41 it is mre than 30 mins | 07:54 |
| chkumar|ruck | jaosorior: quiquell|rover tripleo-image-prepare step is one cause and we need to figure out others also | 07:54 |
| chkumar|ruck | where can i find the logs of tripleo-container-image prepare ansible runs logs? | 07:55 |
| quiquell|rover | chkumar|ruck: home/zuul/.logs | 07:55 |
| quiquell|rover | chkumar|ruck: this is a bash script at the end that dumps to a log file | 07:56 |
| *** jpich has joined #tripleo | 07:56 | |
| chkumar|ruck | it is only undercloud install log na? | 07:56 |
| chkumar|ruck | quiquell|rover: jaosorior I am opening a bug to track these info there | 07:58 |
| quiquell|rover | chkumar|ruck: sure | 07:58 |
| quiquell|rover | chkumar|ruck: I need some time for try to do some tooling | 07:58 |
| chkumar|ruck | quiquell|rover: sure, go ahead | 07:58 |
| quiquell|rover | chkumar|ruck: We need some batching to get numbers, it can be a big task or increase in a lot of small ones | 07:59 |
| quiquell|rover | chkumar|ruck, jaosorior: undercloud isntall has around 400 ansible task | 07:59 |
| chkumar|ruck | quiquell|rover: one way is to get ara.json from each job and compare the timedout one with the passed one accroding to duration | 08:00 |
| chkumar|ruck | that will give which specific task takes how much time | 08:00 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: Workflow for provisioning bare metals without Nova https://review.openstack.org/576856 | 08:01 |
| *** yprokule has joined #tripleo | 08:01 | |
| chkumar|ruck | but where shell script is invoked like undercloud install aprt there is hard to grab that info | 08:01 |
| quiquell|rover | chkumar|ruck: Yep that's what I am doing | 08:01 |
| jaosorior | not sure where those logs are | 08:02 |
| *** phuongnh has quit IRC | 08:06 | |
| *** jcoufal has joined #tripleo | 08:07 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Mount /usr/share/ceph-ansible into mistral-executor https://review.openstack.org/604357 | 08:07 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common master: Do not include ceph-ansible in the container image. https://review.openstack.org/604361 | 08:08 |
| *** yprokule has quit IRC | 08:14 | |
| shardy | hjensas: Hey, good morning | 08:20 |
| hjensas | shardy: good morning | 08:20 |
| shardy | hjensas: Hey do you happen to know where the masquerade rules are set up for the containerized undercloud? | 08:20 |
| shardy | https://github.com/openstack/instack-undercloud/blob/master/elements/undercloud-install/os-apply-config/var/opt/undercloud-stack/masquerade | 08:20 |
| shardy | e.g the equivalent of that | 08:20 |
| hjensas | shardy: yes, hang on I will get some links for you. | 08:21 |
| shardy | for me the nat from overcloud nodes is broken when not using network isolation, and I think it's because we're missing the previous rules | 08:21 |
| *** Petersingh|lunch is now known as Petersingh | 08:21 | |
| shardy | hjensas: thanks! | 08:21 |
| *** openstackgerrit has quit IRC | 08:22 | |
| hjensas | shardy: http://paste.openstack.org/show/730603/ | 08:25 |
| dtantsur | shardy: morning! I'm thinking of writing a short spec on the metalsmith work to make the final goal clearer, wdyt? | 08:25 |
| *** paramite has joined #tripleo | 08:26 | |
| dtantsur | also the updated patch is ready for testing (just note that it required metalsmith 0.7.0) | 08:26 |
| shardy | dtantsur: that sounds like a good idea, and perhaps we can also use the spec to help clarify the upgrade strategy as that's not yet fully defined | 08:27 |
| dtantsur | right yeah | 08:27 |
| dtantsur | shardy: I was thinking about this work over the weekend, and now I'm inclined to completely separate the baremetal provisioning step | 08:27 |
| dtantsur | even at the CLI level | 08:28 |
| shardy | hjensas: thanks! Will do some digging and try to figure out what's missing in my environment | 08:28 |
| dtantsur | then after it's done it's just the regular deployed-server workflow | 08:28 |
| shardy | dtantsur: agreed, but the question is how we handle existing heat/nova deployed servers in that model | 08:28 |
| dtantsur | yeah, it's a separate thing. probably a temporary upgrade workflow to disown them from nova. | 08:29 |
| dtantsur | this is the place where I'll need input from more people | 08:29 |
| dtantsur | because it will require dirty heat magic :) | 08:29 |
| shardy | dtantsur: yeah, and some heat interfaces to ensure the deletion policy means heat won't try to delete the server from nova when moving to the deployed server "dummy server" resource | 08:30 |
| *** shyamb has joined #tripleo | 08:30 | |
| dtantsur | yep | 08:30 |
| shardy | dtantsur: yeah, or if we can get nova to disown the ironic nodes and delete direct with nova, that would be enough | 08:30 |
| shardy | as heat treats a 404 like already deleted | 08:30 |
| * dtantsur thinks | 08:30 | |
| *** Petersingh is now known as Petersingh|afk | 08:30 | |
| dtantsur | dunno about nova, probably impossible | 08:31 |
| dtantsur | I think we'll have to do it on heat level, and make sure the ports don't get lost | 08:32 |
| *** janki has joined #tripleo | 08:33 | |
| *** janki has quit IRC | 08:34 | |
| *** janki has joined #tripleo | 08:34 | |
| *** holser_ has quit IRC | 08:35 | |
| *** holser_ has joined #tripleo | 08:36 | |
| janki | jaosorior, hey. some reviews for https://review.openstack.org/#/c/596987/ maybe??? | 08:37 |
| *** ykarel|lunch is now known as ykarel | 08:38 | |
| ykarel | chkumar|ruck, quiquell|rover are u looking for tripleo-container-image prepare? | 08:39 |
| ykarel | or u find it already | 08:39 |
| *** bogdando has joined #tripleo | 08:39 | |
| ykarel | chkumar|ruck, quiquell|rover if not u can check install_undercloud.log | 08:39 |
| quiquell|rover | ykarel: there is a review to split thos logs I think | 08:40 |
| quiquell|rover | ykarel: maybe it's good to get it merged | 08:40 |
| ykarel | quiquell|rover, don't know about that | 08:40 |
| *** derekh has joined #tripleo | 08:44 | |
| *** chem has joined #tripleo | 08:44 | |
| chkumar|ruck | ykarel: quiquell|rover jaosorior https://bugs.launchpad.net/tripleo/+bug/1794034 putting it on alert | 08:46 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 08:46 |
| jaosorior | chkumar|ruck: ack | 08:47 |
| ykarel | ack | 08:47 |
| quiquell|rover | chkumar|ruck: ack | 08:48 |
| ykarel | chkumar|ruck, but if i remember correctly it's taking 25+ minutes since long, so may be there isdfferent cause for timeout now | 08:48 |
| ykarel | quiquell|rover, jaosorior ^^ | 08:48 |
| ykarel | i mean before promotions | 08:48 |
| chkumar|ruck | ykarel: in a passed on is taking 15-19 mins | 08:49 |
| shardy | hjensas: thanks for the links, it seems like something in tripleoclient is broken because MasqueradeNetworks isn't set to the value from the undercloud.conf | 08:49 |
| shardy | http://paste.openstack.org/show/730605/ | 08:49 |
| ykarel | chkumar|ruck, so any job taking about 20 minutes not passing? | 08:50 |
| shardy | This wasn't noticeable before https://github.com/openstack/python-tripleoclient/commit/e202bdb01f0cd770629723e0e79fbdeafe88102b but now the overcloud deploy fails as it can't reach pool.ntp.org | 08:50 |
| ykarel | chkumar|ruck, not sure about now, but before freiday i have seen passes | 08:50 |
| chkumar|ruck | ykarel: yes | 08:50 |
| shardy | I'll raise a bug and see if I can spot the problem in tripleoclient | 08:51 |
| ykarel | chkumar|ruck, only one job failing? | 08:51 |
| chkumar|ruck | ykarel: I have one log from 21st sept http://logs.openstack.org/81/563881/2/gate/tripleo-ci-centos-7-containers-multinode/7620c42/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-09-21_17_31_54 | 08:51 |
| hjensas | shardy: hmm, what if you use the masquerade = true in [ctlplane-subnet] section of undercloud.conf ? | 08:51 |
| chkumar|ruck | here it is taking more than 30 mins | 08:51 |
| *** Petersingh|afk is now known as Petersingh | 08:51 | |
| chkumar|ruck | ykarel: from today's timedout one it is taking around 25-3-\0 | 08:51 |
| chkumar|ruck | there might be other task which is taking time | 08:51 |
| hjensas | shardy: see: http://paste.openstack.org/show/730606/ | 08:51 |
| chkumar|ruck | I am diggging in | 08:51 |
| *** openstackgerrit has joined #tripleo | 08:51 | |
| openstackgerrit | Pranali Deore proposed openstack/tripleo-heat-templates master: Pass image import conf file to command which runs glance server https://review.openstack.org/598560 | 08:51 |
| quiquell|rover | chkumar|ruck, ykarel: can be also multiple small tasks increasing, is not that simple | 08:52 |
| shardy | hjensas: Hmm, the quickstart generated undercloud.conf doesn't have that section at all | 08:52 |
| shardy | so that may be the problem | 08:52 |
| chkumar|ruck | ykarel: my comparison from pass and timedout one | 08:52 |
| quiquell|rover | chkumar|ruck, ykarel: Doing some tool to be able to browse them and compare between builds | 08:52 |
| shardy | the old format should still work though IMO | 08:52 |
| *** ratailor has joined #tripleo | 08:52 | |
| quiquell|rover | chkumar|ruck, ykarel: we need some statistics power | 08:52 |
| hjensas | shardy: looks like the deprecation was removed. https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/config/undercloud.py#L30 | 08:53 |
| ykarel | chkumar|ruck, ack good to look other things as well | 08:54 |
| shardy | hjensas: Hmm, what commit added it, I don't see it? | 08:56 |
| hjensas | shardy: https://github.com/openstack/instack-undercloud/blob/master/instack_undercloud/undercloud.py#L237 | 08:56 |
| hjensas | shardy: it was deprecated in instack-undercloud, queens ... | 08:57 |
| *** yprokule has joined #tripleo | 08:57 | |
| shardy | hjensas: ack, but should it be either marked deprecated in the tripleoclient code, or error when it is set as it's now not supported? | 08:57 |
| shardy | the issue is quickstart still writes the old config syntax | 08:57 |
| shardy | so we need to fix that, but it highlights an upgrade issue for existing users too | 08:58 |
| hjensas | shardy: yes, a pre_flight validation in tripleoclient would be good. | 09:00 |
| shardy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/undercloud.conf.j2#L150 | 09:00 |
| shardy | hjensas: yeah I think we probably need some validation, and that quickstart template needs rebasing - is there any patch already for that? | 09:00 |
| hjensas | shardy: I am not aware of a patch for that. I can try to get it done this week. | 09:01 |
| shardy | hjensas: ack thanks, I'll raise a bug so we can track both issues | 09:02 |
| shardy | hjensas: thanks for the help figuring it out! :) | 09:02 |
| hjensas | shardy: ack, we should backport the validation to Rocky as well. | 09:02 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Allow debootstrap to clenaup without a kernel https://review.openstack.org/604692 | 09:03 |
| *** openstackgerrit has quit IRC | 09:05 | |
| *** gfidente is now known as gfidente|afk | 09:05 | |
| quiquell|rover | chkumar|ruck, ykarel: Looking at this http://dashboard-ci.tripleo.org/d/-UEjGKFmz/exploration?orgId=1&var-influxdb_filter=result%7C%3D%7CTIMED_OUT&var-influxdb_filter=pipeline%7C%3D%7Ccheck&from=now-30d&to=now | 09:05 |
| quiquell|rover | It's histograms of TIMEDOUT_JOBS since 30 days | 09:06 |
| *** jcoufal has quit IRC | 09:07 | |
| quiquell|rover | Looks like it started to increase the week of 12th | 09:08 |
| shardy | hjensas: https://bugs.launchpad.net/tripleo/+bug/1794038 raised | 09:10 |
| openstack | Launchpad bug 1794038 in tripleo "undercloud masquerade_networks is now silently ignored" [High,Triaged] | 09:10 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 09:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 09:10 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 09:10 |
| *** jaosorior has quit IRC | 09:11 | |
| *** salmankhan has joined #tripleo | 09:15 | |
| *** openstackgerrit has joined #tripleo | 09:15 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: WIP: New workflow for standalone upgrade https://review.openstack.org/604706 | 09:15 |
| *** cylopez has quit IRC | 09:16 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: WIP: New featureset056 for standalone upgrade https://review.openstack.org/604707 | 09:20 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common stable/pike: Add container runtime packages for cron image https://review.openstack.org/604708 | 09:26 |
| *** shyamb has quit IRC | 09:26 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/puppet-tripleo master: Fix postrotate to notify holders of rotated logs https://review.openstack.org/596275 | 09:27 |
| hjensas | shardy: thanks. It should by default still masqurade the network set up as network_cidr with that quickstart template. | 09:29 |
| chkumar|ruck | In each job, tripleo validation takes around 35-45 mins also | 09:29 |
| shardy | hjensas: Hmm, but as you can see the MasqueradeNetworks parameter isn't set from the undercloud.conf masquerade_networks? | 09:30 |
| shardy | so AFAICS we don't include the puppet manifest | 09:30 |
| hjensas | shardy: yes, there is a bug here. I will dig a bit, we must have missed something when containerizing. | 09:33 |
| shardy | hjensas: ack thanks - http://paste.openstack.org/show/730609/ shows we definitely don't have the postrouting/forwarding rules | 09:34 |
| shardy | which makes sense given the hiera values | 09:34 |
| *** shyamb has joined #tripleo | 09:35 | |
| *** adrianreza has joined #tripleo | 09:36 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: WIP - Add nova file_backed_memory and memory_backing_dir support for qemu.conf https://review.openstack.org/604360 | 09:39 |
| *** skramaja has quit IRC | 09:42 | |
| *** skramaja has joined #tripleo | 09:42 | |
| chkumar|ruck | quiquell|rover: ykarel one more thing tripleo validation generally gets completed under 25-30 mins, 5 mins got increased there also | 09:42 |
| chkumar|ruck | around 5-10 | 09:42 |
| quiquell|rover | chkumar|ruck: What jobs are affected ? | 09:44 |
| chkumar|ruck | tripleo-ci-centos-7-containers-multinode and this tripleo-ci-centos-7-scenario001-multinode-oooq-container currently | 09:44 |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart master: [DNM] Testing perf for container updates with cached delorean-current https://review.openstack.org/604318 | 09:44 |
| chkumar|ruck | I am looking into scenario001 jobs | 09:44 |
| ykarel | chkumar|ruck, ack | 09:45 |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: Use external-(upgrade|update) commands for Ceph. https://review.openstack.org/604712 | 09:45 |
| *** shyamb has quit IRC | 09:46 | |
| Tengu | florianf akrivoka heya! I heard you might need to have a quick talk about the validation framework (and maybe in-flight validations) :). I'm back from PTO, ready to do things. | 09:46 |
| quiquell|rover | chkumar|ruck: what's the validation task ? | 09:51 |
| *** shyamb has joined #tripleo | 09:52 | |
| chkumar|ruck | quiquell|rover: tripleo-validations : Run validations tests through Mistral | 09:53 |
| chkumar|ruck | quiquell|rover: ^^ I am not sure about tripleo validation taking 5-10 times more time, I am not sure it adds any role to timedout | 09:57 |
| hjensas | shardy: So you need to set ctlplane_masquerade = True in some quickstart file to have it render undercloud.conf that enables masquerading. | 09:57 |
| quiquell|rover | chkumar|ruck: I am near to have something to show stuff give a min | 09:57 |
| hjensas | shardy: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/undercloud.conf.j2#L524 | 09:57 |
| chkumar|ruck | quiquell|rover: ack | 09:58 |
| shardy | hjensas: ah, thanks, I missed that | 10:01 |
| shardy | I guess that doesn't help any upgrading users though? | 10:01 |
| shardy | Its a shame that setting network_isolation: false isn't enough, as without both params the oc deploy is now broken | 10:04 |
| hjensas | shardy: users got deprecation warnings on Queens, so they should have already updated their config. But, I think it would be good to have a validation for any deprecated/removed parameter just to ensure we don't move on before the user did the appropriate updates to their config. | 10:04 |
| shardy | hjensas: yeah I think it's best to go from deprecated to broken vs silently ignored :) | 10:04 |
| shardy | and it's unfortunate that the quickstart defaults don't now render a valid config AFAICS | 10:05 |
| hjensas | shardy: I think quickstart is used for stable as well? So I guess people are in a hurry and make only the small changes to just make it work. | 10:07 |
| shardy | hjensas: yeah, be nice if we had some sane defaults though :) | 10:09 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 10:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 10:10 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 10:10 |
| chkumar|ruck | bogdando: therve where can i find the logs for update packages in containers? | 10:11 |
| chkumar|ruck | with related to this bug https://bugs.launchpad.net/tripleo/+bug/1794034 | 10:11 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 10:11 |
| chkumar|ruck | we have master promotion on sunday | 10:11 |
| janki | panda, hey. I rebased https://review.openstack.org/#/c/602196/ | 10:12 |
| therve | chkumar|ruck: No idea | 10:12 |
| panda | janki: yeah, I saw this was solved on another patch. Is this change needed anymore ? the .* should also match "no character" | 10:13 |
| janki | panda, which patch? | 10:13 |
| *** janki is now known as janki|lunch | 10:14 | |
| panda | janki|lunch: https://review.openstack.org/596061 | 10:14 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Refactor openshift services for composable roles https://review.openstack.org/599618 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Fix openshift new node detection https://review.openstack.org/600012 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Clean up previous osa inventory dir before deployment https://review.openstack.org/600028 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Configure haproxy for openshift infra https://review.openstack.org/601241 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Fix inventory files for newer openshift-ansible https://review.openstack.org/603577 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Deploy openshift all in one in scenario009 https://review.openstack.org/603780 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Add heat param for openshift prerequisites playbook https://review.openstack.org/604338 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Use openshift-ansible container instead of RPMs https://review.openstack.org/583868 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Introduce OpenShiftGlusterNodeVars heat param https://review.openstack.org/604724 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Make glusterfs the default sc when deploying with CNS https://review.openstack.org/604725 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Consolidate openshift-ansible global variables https://review.openstack.org/604726 | 10:15 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Remove unused networks from OpenShift roles https://review.openstack.org/604727 | 10:15 |
| mandre | boom | 10:15 |
| Tengu | mandre: you want to kill zuul, don't you? ;) | 10:15 |
| mandre | I wish we could start merging some of that stuff | 10:16 |
| * Tengu looks topic | 10:16 | |
| Tengu | nah. | 10:16 |
| chkumar|ruck | mandre: please have a look at this bug https://bugs.launchpad.net/tripleo/+bug/1794034 | 10:16 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 10:16 |
| chkumar|ruck | I am not sure why it is happening since yesterday we had a master promotion | 10:16 |
| mandre | Tengu: I'll keep hammering zuul until someone merges my patches :) | 10:16 |
| mandre | chkumar|ruck: I'll check if after lunch | 10:17 |
| Tengu | mandre: not sure that's the right thing to do, but you can try ;). | 10:17 |
| chkumar|ruck | mandre: ack, thanks :-) | 10:18 |
| *** Petersingh is now known as Petersingh|afk | 10:21 | |
| *** shyamb has quit IRC | 10:21 | |
| *** shyamb has joined #tripleo | 10:21 | |
| *** jaosorior has joined #tripleo | 10:22 | |
| *** fhubik is now known as fhubik|brb | 10:23 | |
| quiquell|rover | chkumar|ruck: There is something weird | 10:23 |
| *** ssbarnea|bkp has joined #tripleo | 10:23 | |
| quiquell|rover | I think there is something wrong with ara | 10:24 |
| quiquell|rover | chkumar|ruck: This http://logs.openstack.org/38/580238/8/gate/tripleo-ci-centos-7-containers-multinode/ba381f5/logs/ara_oooq_root/ | 10:25 |
| quiquell|rover | chkumar|ruck: is equals to http://logs.openstack.org/38/580238/8/gate/tripleo-ci-centos-7-containers-multinode/ba381f5/logs/ara_oooq_oc/ | 10:25 |
| *** Petersingh|afk has quit IRC | 10:25 | |
| *** waleedm has joined #tripleo | 10:25 | |
| chkumar|ruck | quiquell|rover: I am basically looking into ara_oooq, I never looked in to oc and uc ara logs | 10:30 |
| chkumar|ruck | quiquell|rover: is root for undercloud and OC for subnodes ? | 10:30 |
| quiquell|rover | chkumar|ruck: They show what's inside undercloud install and overcloud deploy | 10:30 |
| quiquell|rover | chkumar|ruck: It's like zooming in | 10:30 |
| quiquell|rover | https://review.openstack.org/#/c/597222/ | 10:31 |
| quiquell|rover | chkumar|ruck: But they look the same, something is not right yet | 10:31 |
| quiquell|rover | chkumar|ruck: undercloud has being for longer merged | 10:34 |
| *** shyamb has quit IRC | 10:35 | |
| derekh | I'm trying to get a new job with ironic in the overcloud, if anybody has time to review, all the required patches are depended on here https://review.openstack.org/#/c/582294/ | 10:35 |
| chkumar|ruck | quiquell|rover: if the job passed does it not generating the ara report? | 10:36 |
| chkumar|ruck | http://logs.openstack.org/91/564291/21/check/tripleo-ci-centos-7-scenario002-multinode-oooq/6c1b6fb/logs/ara_oooq_oc/ | 10:37 |
| chkumar|ruck | for example this one ^^ | 10:37 |
| quiquell|rover | chkumar|ruck: Something has failed there in the ara generation | 10:37 |
| quiquell|rover | chkumar|ruck: but we have undercloud | 10:37 |
| quiquell|rover | chkumar|ruck: Humm nop... | 10:37 |
| quiquell|rover | chkumar|ruck: Looks like collect logs failed there | 10:38 |
| chkumar|ruck | in in ara_ooq has full logs but root and oc has nothing except ara html template | 10:38 |
| *** saneax has quit IRC | 10:39 | |
| quiquell|rover | chkumar|ruck: When was the last time we didn't have so much timeouts ? | 10:43 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/rocky: Stop cap granting to empty pool when telemetry disabled https://review.openstack.org/604734 | 10:43 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Stop cap granting to empty pool when telemetry disabled https://review.openstack.org/604735 | 10:43 |
| *** dxiri has quit IRC | 10:44 | |
| *** paramite has quit IRC | 10:44 | |
| *** paramite has joined #tripleo | 10:44 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: WIP: Add necessary bits for N-1->N standalone upgrade. https://review.openstack.org/604736 | 10:46 |
| quiquell|rover | chkumar|ruck: at dashboard-ci oldest date is around 26th of july | 10:48 |
| quiquell|rover | chkumar|ruck: We have to compare jobs around 12th with ones this week | 10:53 |
| quiquell|rover | real good was 16h of Aughust | 10:54 |
| quiquell|rover | August | 10:54 |
| *** phuongnh has joined #tripleo | 10:55 | |
| *** jcoufal has joined #tripleo | 10:56 | |
| *** phuongnh has quit IRC | 10:56 | |
| quiquell|rover | chkumar|ruck: Old good one http://logs.openstack.org/38/592938/3/check/tripleo-ci-centos-7-containers-multinode/c6677d7/ | 10:56 |
| quiquell|rover | chkumar|ruck: Validations, looks like an issue | 10:59 |
| quiquell|rover | bogdando: I see a lot of increment in validation from august in the jobs | 11:02 |
| quiquell|rover | bogdando: Do you know if this is normal ? is all in the bug https://bugs.launchpad.net/tripleo/+bug/1794034 | 11:02 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 11:02 |
| *** janki|lunch is now known as janki | 11:10 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 11:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 11:10 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 11:10 |
| *** jpena is now known as jpena|lunch | 11:10 | |
| janki | panda, thanks for sharing it. I will remove dependency on zuul patch from one of my other patch. | 11:11 |
| janki | panda, also can I get workflow +1 for https://review.openstack.org/#/c/596987/ please | 11:13 |
| panda | janki: will the $bind_host ever have a port indication on it ? | 11:16 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: WIP: New workflow for standalone upgrade https://review.openstack.org/604706 | 11:16 |
| janki | panda, no. its just IP always. | 11:16 |
| jaosorior | panda: yeah, there is a specific entry for the port itself. Separate from the bind_host entry. | 11:16 |
| bogdando | quiquell|rover: that looks not normal at all :) | 11:17 |
| quiquell|rover | bogdando: Do you know who implemented validations or can guide me ? | 11:18 |
| bogdando | quiquell|rover: there is a validations DFG... | 11:18 |
| bogdando | not sure the source tho | 11:18 |
| quiquell|rover | bogdando: There is a DFG for that ? | 11:18 |
| quiquell|rover | chkumar|ruck: I am gathering 5 pages of zuul buils it will take time :-) | 11:19 |
| bogdando | akrivoka: ^^ hi, do you know something of the validations DFG? | 11:20 |
| bogdando | details | 11:20 |
| panda | janki: last annoying question: I see in http://ops4j.github.io/pax/web/SNAPSHOT/User-Guide.html org.ops4j.pax.web.listening.addresses may contain a list of addresses. Will it ever happen to have a list specified ? | 11:20 |
| quiquell|rover | akrivoka: Last comment at https://bugs.launchpad.net/tripleo/+bug/1794034 | 11:20 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 11:20 |
| jaosorior | florianf: ^^ | 11:20 |
| janki | panda, no. it would always be nust 1 IP. https://github.com/opendaylight/integration-packaging-puppet-opendaylight/blob/eff4bedd538d9e6015b94c762c3ad38effe03c28/manifests/config.pp#L204 and https://github.com/opendaylight/integration-packaging-puppet-opendaylight/blob/eff4bedd538d9e6015b94c762c3ad38effe03c28/manifests/params.pp#L14 | 11:22 |
| florianf | jaosorior: checking... | 11:23 |
| mandre | quiquell|rover, florianf: the fails job enters a retry loop in the validation, see http://logs.openstack.org/11/585911/3/gate/tripleo-ci-centos-7-containers-multinode/2a143a8/logs/undercloud/home/zuul/validations_post-deployment.log.txt.gz | 11:24 |
| quiquell|rover | mandre: Also passing one have increased a lot in time | 11:25 |
| *** udesale has quit IRC | 11:25 | |
| quiquell|rover | mandre: recent passing ones | 11:25 |
| quiquell|rover | mandre: ack thanks, let's compare | 11:25 |
| mandre | quiquell|rover: right, I'm seeing the passing one also has some retries | 11:26 |
| quiquell|rover | mandre: This is an Old passing one, with "good" times http://logs.openstack.org/38/592938/3/check/tripleo-ci-centos-7-containers-multinode/c6677d7/logs/ara_oooq/ | 11:26 |
| quiquell|rover | mandre: From august | 11:26 |
| mandre | quiquell|rover: could very well be related to the switch to containerized undercloud | 11:27 |
| quiquell|rover | mandre: I see more retries at actual success ones yep... | 11:28 |
| chkumar|ruck | quiquell|rover: cool | 11:29 |
| florianf | mandre, jaosorior, quiquell|rover: containerization would be my guess as well. although if the broken validations were the cause of that, we shouldn't see any success at all | 11:29 |
| quiquell|rover | rdopiera: Do you manage validations at tripleo ? | 11:30 |
| chkumar|ruck | quiquell|rover: mandre I have also seen somewhere validation job task is showing twice or thrice | 11:30 |
| rdopiera | quiquell|rover: no | 11:30 |
| chkumar|ruck | in the ara logs also | 11:30 |
| mandre | quiquell|rover, florianf: hmm the retries are not the problem, apparently this is just how we're polling for the result | 11:30 |
| quiquell|rover | mandre: sleep 1 | 11:31 |
| quiquell|rover | mandre: And also the overhead or runnign mistral again | 11:31 |
| quiquell|rover | mandre: Everything sums up | 11:31 |
| mandre | chkumar|ruck: yeah it's running different validations at different stages of the deployment | 11:32 |
| chkumar|ruck | quiquell|rover: may be therve can help on validation part | 11:32 |
| *** shyamb has joined #tripleo | 11:33 | |
| chkumar|ruck | sorry Tengu | 11:33 |
| quiquell|rover | mandre: Question is, why don't wee have this good times now http://logs.openstack.org/38/592938/3/check/tripleo-ci-centos-7-containers-multinode/c6677d7/logs/ara_oooq/ | 11:33 |
| quiquell|rover | mandre: ^ old august build | 11:33 |
| Tengu | hmm ? | 11:33 |
| *** saneax has joined #tripleo | 11:33 | |
| mandre | quiquell|rover: idk yet, I need to investigate what has changed | 11:34 |
| quiquell|rover | Tengu: hello, we are looking at issues with timeouts | 11:34 |
| quiquell|rover | Tengu: We found the validations are taking twice or thrice the time from august | 11:34 |
| Tengu | o_O | 11:34 |
| Tengu | what validations? | 11:34 |
| quiquell|rover | Tengu: https://bugs.launchpad.net/tripleo/+bug/1794034/comments/2 | 11:34 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 11:34 |
| Tengu | the undercloud preflight? | 11:34 |
| quiquell|rover | Tengu: when did we introduce that ? | 11:35 |
| Tengu | quiquell|rover: well, the undercloud preflight were added months ago. Like, really. | 11:35 |
| Tengu | quiquell|rover: I introduced only one more test, for the disk space. | 11:36 |
| janki | panda, maybe 1 more https://review.openstack.org/#/c/604069/ :) | 11:36 |
| chkumar|ruck | quiquell|rover: https://github.com/openstack/tripleo-quickstart/commit/9a71cc81763d7e90c16f533581ab5f8482b40b7c | 11:36 |
| Tengu | quiquell|rover: question is: is it the undercloud preflight validations, or some overcloud tingy? | 11:36 |
| chkumar|ruck | it was added around 10 months ago | 11:36 |
| quiquell|rover | Tengu: Don't know, I am new to validations, just comparing august job with actual one to find differences there | 11:37 |
| quiquell|rover | Tengu: Maybe you can help me there | 11:37 |
| quiquell|rover | Tengu: maybe actual validations times are good... but they have really increase from August | 11:38 |
| Tengu | quiquell|rover: trying to understand what validations we're talking about. | 11:39 |
| florianf | mandre, chkumar|ruck, quiquell|rover: Still wondering if this is connected to the fact that validations have been broken in a containerized undercloud. the patches to fix that have only been merged very recently in master (and one backport to rocky is still outstanding) | 11:39 |
| bogdando | Tengu: no that's not UC preflight | 11:39 |
| Tengu | http://logs.openstack.org/11/585911/3/gate/tripleo-ci-centos-7-containers-multinode/2a143a8/logs/ara_oooq/file/84fa95e8-9d04-495d-b1ba-58d5cf6025d1/#line-21 | 11:39 |
| Tengu | bogdando: that's what I think indeed. | 11:39 |
| *** saneax has quit IRC | 11:39 | |
| Tengu | so that's the overcloud validations. | 11:40 |
| quiquell|rover | florianf: Do you have the review of the fixes ? | 11:40 |
| Tengu | akrivoka: did you finish the swift container based validations? | 11:40 |
| *** quiquell|rover is now known as quique|rover|brb | 11:40 | |
| panda | janki: it looks good, but those files are out of my jurisdiction. I'm allowed to +2 on CI with some incursions on other repos when it's safe. | 11:41 |
| Tengu | chkumar|ruck, quiquell|rover: if akrivoka has finished to push/fetch validations in/from a swift container, that might explain. | 11:41 |
| florianf | quique|rover|brb: https://review.openstack.org/#/c/600102/ and https://review.openstack.org/#/c/598958/ | 11:41 |
| janki | panda, thank you taking a look :) | 11:41 |
| janki | mandre, hey. Can I get some reviews for https://review.openstack.org/#/c/604069/ | 11:42 |
| *** saneax has joined #tripleo | 11:42 | |
| florianf | quique|rover|brb: backports: https://review.openstack.org/#/c/603002/ and https://review.openstack.org/#/c/603031/ | 11:42 |
| *** saneax has quit IRC | 11:43 | |
| quique|rover|brb | florianf: The run of the reviews, have very good times at validations | 11:43 |
| quique|rover|brb | florianf: Like it had in august | 11:43 |
| *** saneax has joined #tripleo | 11:43 | |
| quique|rover|brb | http://logs.openstack.org/02/600102/1/check/tripleo-ci-centos-7-containers-multinode/1012829/logs/ara_oooq/ | 11:43 |
| quique|rover|brb | Search for "Run val" | 11:43 |
| *** bfournie has quit IRC | 11:43 | |
| janki | panda, should this be abandoned? https://review.openstack.org/#/c/602196/ | 11:43 |
| *** ansmith has quit IRC | 11:43 | |
| florianf | quique|rover|brb: so the fixes seem to have had an affect. maybe it helps if the last backport is merged? | 11:44 |
| *** quique|rover|brb is now known as quiquell|rover | 11:45 | |
| quiquell|rover | florianf: it's tripleo-common, we take it from DLRN (I think) | 11:45 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: WIP: Add necessary bits for N-1->N standalone upgrade. https://review.openstack.org/604736 | 11:45 |
| quiquell|rover | florianf: And it was merged like 1 week ago | 11:45 |
| *** shyamb has quit IRC | 11:45 | |
| quiquell|rover | florianf: We have to check it actual timeout jobs have the changes | 11:46 |
| *** rh-jelabarre has joined #tripleo | 11:46 | |
| *** shyamb has joined #tripleo | 11:47 | |
| florianf | quiquell|rover: right. | 11:48 |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates master: Add provision to specify java arguments to ODL https://review.openstack.org/604750 | 11:49 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: WIP: New workflow for standalone upgrade https://review.openstack.org/604706 | 11:50 |
| florianf | quiquell|rover: although I'm not sure I know how to do taht | 11:54 |
| florianf | *that | 11:54 |
| *** zul has joined #tripleo | 11:54 | |
| *** dprince has joined #tripleo | 11:54 | |
| quiquell|rover | florianf: Will do after lunch | 11:54 |
| florianf | quiquell|rover: thanks a lot! | 11:55 |
| *** sanjayu_ has joined #tripleo | 12:02 | |
| chkumar|ruck | quiquell|rover: CIX call | 12:02 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates stable/queens: Fix syntax for set_fact ansible task. https://review.openstack.org/604752 | 12:02 |
| *** saneax has quit IRC | 12:04 | |
| mandre | florianf, quiquell|rover: we have openstack-tripleo-validations package in mistral executor container in the failed job, I don't think this is it | 12:04 |
| mandre | http://logs.openstack.org/11/585911/3/gate/tripleo-ci-centos-7-containers-multinode/2a143a8/logs/undercloud/var/log/extra/docker/containers/mistral_executor/docker_info.log.txt.gz | 12:05 |
| mandre | florianf: btw, we have openstack-tripleo-validations package in mistral executor image but at the same time we bind-mount /usr/share/openstack-tripleo-validations from the undercloud fs | 12:06 |
| *** gfidente|afk is now known as gfidente | 12:06 | |
| mandre | florianf: the bind mount needs to be removed | 12:06 |
| *** amoralej is now known as amoralej|lunch | 12:09 | |
| florianf | mandre: "needs" or "should"? | 12:09 |
| florianf | or "could"? | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 12:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 12:10 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 12:10 |
| mandre | quiquell|rover, florianf, akrivoka: hmm the swift based validations appears like a potential culprit for increased time https://review.openstack.org/#/c/569513/ | 12:10 |
| *** trown|outtypewww is now known as trown | 12:11 | |
| chkumar|ruck | mandre: https://review.openstack.org/#/c/603804/ master one is merged, we need it for rocky | 12:12 |
| mandre | florianf: should :) otherwise the openstack-tripleo-validations package from the mistral-executor image is not used, but instead it's the one from the undercloud... and who knows for how long we're going to install openstack-tripleo-validation on the undercloud directly | 12:12 |
| *** psachin has quit IRC | 12:13 | |
| chkumar|ruck | quiquell|rover: https://bugs.launchpad.net/tripleo/+bug/1794034 do we need to add this also as a promotion blocker ? | 12:13 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 12:13 |
| florianf | mandre: OK, I'll test it and create a patch to fix it. But IIUC this is not causing buggy behavior right now... | 12:13 |
| *** jpena|lunch is now known as jpena | 12:14 | |
| akrivoka | mandre: Tengu: hi, yes the validations are being stored in swift now | 12:15 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: Activate ansible lint after deployment. https://review.openstack.org/604756 | 12:15 |
| mandre | florianf: depends on how the swift-based validations are implemented, it's just freaking slow to fetch each file from swift individually | 12:15 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Lint all the generated playbook after standalone deployment. https://review.openstack.org/604757 | 12:15 |
| chkumar|ruck | panda: marios https://review.openstack.org/#/c/603804/ need for rocky | 12:15 |
| *** zul has quit IRC | 12:15 | |
| mandre | chkumar|ruck: done! | 12:16 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: Activate ansible lint after deployment. https://review.openstack.org/604756 | 12:16 |
| chkumar|ruck | mandre: Thnaks :-) | 12:16 |
| *** ratailor has quit IRC | 12:16 | |
| florianf | mandre: but is that related to the bind-mount? | 12:17 |
| trown | flaper87: if you have a moment could you look at a few tripleo-common patches? We need https://review.openstack.org/#/c/602112/ and below, to merge the openshift validation patch | 12:17 |
| trown | also good morning :) | 12:17 |
| mandre | florianf: nope, different issue, my guess is that it's the swift based validations that cause the slow down | 12:17 |
| *** mmethot has quit IRC | 12:18 | |
| *** rfolco has joined #tripleo | 12:18 | |
| *** mmethot has joined #tripleo | 12:18 | |
| *** bfournie has joined #tripleo | 12:18 | |
| florianf | mandre: OK, got it then... :) | 12:18 |
| mandre | florianf: the bind mount will cause an error when we stop installing openstack-tripleo-validations on the undercloud, better stay on the safe side and remove the bind mount now before it breaks | 12:19 |
| Tengu | akrivoka: might it create some issues with load/timeout/others? | 12:19 |
| *** aufi has quit IRC | 12:19 | |
| *** leanderthal has joined #tripleo | 12:19 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Add f28 standalone job definition as non-voting https://review.openstack.org/604758 | 12:19 |
| florianf | mandre: right! that's what I assumed (glad that seems to be true...) | 12:20 |
| florianf | mandre: actually, there *is* some use to that. because that way you can add validations to the directory on the undercloud | 12:21 |
| flaper87 | trown: sure thing! I'm catching up with email after PTO so it might take me a bit | 12:21 |
| akrivoka | Tengu: it's certainly slower fetching files from swift so I guess it's possible | 12:21 |
| Tengu | quiquell|rover: -^ | 12:21 |
| chkumar|ruck | Tengu: is it possible disable it currently? | 12:21 |
| trown | flaper87: thanks! | 12:21 |
| bogdando | quiquell|rover, mandre: FWIW, https://review.openstack.org/#/c/604171/ may be the fix for that validation increase story | 12:22 |
| bogdando | just as well... | 12:22 |
| Tengu | akrivoka: might be good to get in touch with quiquell|rover - apparently validations are taking "far more time". | 12:22 |
| bogdando | looks very similar | 12:22 |
| bogdando | and there is retries for validations also, so looks the same issue | 12:22 |
| *** thrash|g0ne is now known as thrash | 12:22 | |
| mandre | bogdando: interesting | 12:24 |
| bogdando | yeah, I think those validations retries are stumbling upon itself | 12:25 |
| mandre | akrivoka: I'd suggest using a local cache for the validations and also bulk swift operations | 12:25 |
| *** Petersingh|afk has joined #tripleo | 12:26 | |
| bogdando | or at least it just makes it not to fail 8 times instead of 1 :) | 12:26 |
| bogdando | as we configured 8 retries | 12:26 |
| bogdando | so may be we should not retry at all | 12:26 |
| bogdando | for the validations | 12:26 |
| bogdando | makes it *now* to fail* (typo) | 12:27 |
| mandre | akrivoka: i do not think swift-based validations is our only issue, as bogdando pointed out there can also be an issue with how we setup the ssh connection | 12:28 |
| bogdando | or retry | 12:28 |
| bogdando | ansible | 12:28 |
| bogdando | slagle: ^^ | 12:28 |
| akrivoka | mandre: could you open a bug referencing those suggestions? | 12:29 |
| *** Petersingh|afk is now known as Petersingh | 12:29 | |
| slagle | bogdando: what? | 12:30 |
| quiquell|rover | akrivoka, mandre: The bug is https://bugs.launchpad.net/tripleo/+bug/1794034 | 12:30 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 12:30 |
| quiquell|rover | Going to rename it though | 12:31 |
| quiquell|rover | chkumar|ruck: Ok with renaming the timeout bug ? | 12:31 |
| chkumar|ruck | quiquell|rover: yup | 12:32 |
| bogdando | slagle: https://bugs.launchpad.net/tripleo/+bug/1794034/comments/2, I think that increase has something to the ansible retries recent changes | 12:32 |
| openstack | Launchpad bug 1794034 in tripleo "tripleo-container-image prepare step during undercloud install is taking more than 20 mins leading to timeout" [Critical,Triaged] | 12:32 |
| bogdando | not exactly like you described in https://review.openstack.org/#/c/604171/ but looks similar type of an issue | 12:33 |
| bogdando | may be we should not retry validations at all? | 12:33 |
| bogdando | with ansible I mean, slagle | 12:34 |
| slagle | bogdando: that patch didn't change anything related to retries | 12:34 |
| bogdando | ok | 12:34 |
| *** zul has joined #tripleo | 12:34 | |
| bogdando | please disregard then | 12:34 |
| slagle | bogdando: does the undercloud install even use the tripleo_common ansible action? | 12:34 |
| bogdando | yes | 12:34 |
| bogdando | not sure for the validations of overcloud tho | 12:35 |
| slagle | bogdando: ok. it shouldn't be going over ssh though for the undercloud install. it should just use connection:local | 12:36 |
| *** shyamb has quit IRC | 12:37 | |
| bogdando | slagle: right. The thing which may be wrong is retrying the failing validations 8 times | 12:37 |
| slagle | bogdando: where is that done? | 12:37 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-specs master: [WIP] Bare metal provisioning without Nova and Glance https://review.openstack.org/604761 | 12:37 |
| bogdando | so it takes those mentioned 20 min to fail 8 times now, and before it was 1 time amd ~4 min | 12:37 |
| quiquell|rover | bogdando, slagle: Validations times re good here https://review.openstack.org/#/c/600102/ | 12:37 |
| quiquell|rover | And it does not have slagle change | 12:37 |
| bogdando | that is my guess, needs to be verified, like if any recent changes to the ansible.cfg used with validaionts | 12:37 |
| bogdando | quiquell|rover: I think that change won't help for us, just related a little bit | 12:38 |
| slagle | bogdando: the 8 retries is for ssh. we shouldn't be going over ssh for the undercloud. and also, that bit isn't new | 12:38 |
| quiquell|rover | bogdando: But it has good times | 12:38 |
| quiquell|rover | :-/ | 12:38 |
| bogdando | um, slagle I thought ansible retries failing tasks 8 times | 12:38 |
| bogdando | I misunderstood it it seems | 12:39 |
| bogdando | folks, PTAL https://review.openstack.org/#/c/578799/ | 12:39 |
| bogdando | jaosorior, slagle: ^^ :) | 12:40 |
| slagle | bogdando: you might want to check the ansible configuration and verify what it's doing against the log | 12:40 |
| *** agopi has quit IRC | 12:41 | |
| *** jaosorior has quit IRC | 12:42 | |
| *** jcoufal has quit IRC | 12:43 | |
| quiquell|rover | mandre: Swift based validations has good times too at the review https://review.openstack.org/#/c/569513/ | 12:43 |
| *** jaosorior has joined #tripleo | 12:44 | |
| *** aufi has joined #tripleo | 12:45 | |
| mandre | flaper87, shardy: /me would loooove your reviews on https://etherpad.openstack.org/p/tripleo-openshift-patches, patches start to pile up :) | 12:46 |
| *** udesale has joined #tripleo | 12:46 | |
| shardy | mandre: ah thanks for the reminder, will take a look now | 12:46 |
| *** amoralej|lunch is now known as amoralej | 12:47 | |
| mandre | shardy, flaper87: I hope it won't take too much of your time :) thanks in advance | 12:48 |
| shardy | mandre: what's the status of scenario 009? | 12:49 |
| shardy | trying to see if we get it passing somewhere in this series :) | 12:49 |
| mandre | shardy: it's super green | 12:49 |
| mandre | haha | 12:49 |
| openstackgerrit | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Remove external_network_bridge Neutron option https://review.openstack.org/604764 | 12:49 |
| shardy | mandre: ah I was looking at the 3.10 series but I see the quickstart patches are green \o/ | 12:50 |
| shardy | nice :) | 12:50 |
| *** tzumainn has joined #tripleo | 12:50 | |
| mandre | shardy: I may have broken some things with the new patches I added this morning but otherwise it's been consistently green for a while | 12:50 |
| quiquell|rover | florianf: we are usin gthe latest tripleo-common so we have all the changes | 12:51 |
| shardy | mandre: ack - I was looking at the https://review.openstack.org/#/c/596820/ series where it's red, but I assume it's because we didn't yet land the osa pin to last good version patch | 12:51 |
| florianf | quiquell|rover: thanks for checking! looks like the problem is something else anyway... | 12:51 |
| quiquell|rover | florianf: Last comming to tripleo-common has good validation times http://logs.openstack.org/71/604171/2/check/tripleo-ci-centos-7-containers-multinode/c8422be/logs/ara_oooq/ | 12:52 |
| quiquell|rover | florianf: This is 21th | 12:52 |
| florianf | quiquell|rover: oh, that looks a lot better | 12:53 |
| mandre | shardy: wow this is old, Aug 28 - this tripleo-common series can be rebased if we absolutely want to prove it's green but the tht patches that depend on it prove they're good | 12:53 |
| slagle | bogdando: have a look at http://logs.openstack.org/06/588606/5/gate/tripleo-ci-centos-7-containers-multinode/ddfb143/logs/undercloud/home/zuul/install-undercloud.log.txt.gz | 12:53 |
| slagle | bogdando: doesnt that look like it's just the tripleo-modify-image stuff that's taking so long? | 12:54 |
| slagle | bogdando: how is this related to validations? | 12:54 |
| quiquell|rover | florianf: yep but we have a lot of todays job's that don't and they have the latest tripleo-common | 12:54 |
| florianf | quiquell|rover: that sounds like the problem isn't related to a specific fix? | 12:55 |
| quiquell|rover | florianf: don't know | 12:55 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: WIP Removes EXTRA_TAGS from toci_gate_test&quickstart.sh.j2 https://review.openstack.org/604768 | 12:55 |
| chkumar|ruck | quiquell|rover: Did we merge something related to post log stuff http://logs.openstack.org/15/585015/7/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/ad92baa/ | 12:59 |
| chkumar|ruck | once timedout happens undercloud logs are not getting collected | 13:00 |
| quiquell|rover | chkumar|ruck: Not that I know of | 13:00 |
| *** lblanchard has joined #tripleo | 13:00 | |
| bogdando | slagle: did you check https://bugs.launchpad.net/tripleo/+bug/1794034/comments/2 ? | 13:01 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] | 13:01 |
| bogdando | that also inputs into the time increase | 13:01 |
| *** zul has quit IRC | 13:01 | |
| bogdando | may be it worth opening another bug, quiquell|rover? | 13:01 |
| shardy | mandre: ack yup no worries I'll approve them then we can get the t-h-t patches merged | 13:02 |
| *** ansmith has joined #tripleo | 13:02 | |
| bogdando | slagle: oops, there is another bug, and the comment had moved into description | 13:02 |
| quiquell|rover | bogdando, slagle: Sorry about that, maybe we just need a generic timeout bug | 13:03 |
| quiquell|rover | validations looks like red herring, they timing is not consistent | 13:03 |
| *** agopi has joined #tripleo | 13:04 | |
| quiquell|rover | bogdando: Here validation takes up to 23 minutes http://logs.openstack.org/21/597221/8/gate/tripleo-ci-centos-7-containers-multinode/59f728a/logs/ara_oooq/ | 13:08 |
| quiquell|rover | that's normal ? | 13:08 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates stable/rocky: Persist ceph-ansible fetch_directory using config-download https://review.openstack.org/604772 | 13:09 |
| openstackgerrit | John Fulton proposed openstack/tripleo-common stable/rocky: Update swift_rings_backup workflow to also backup ceph fetch dir https://review.openstack.org/604773 | 13:09 |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates master: Add provision to specify java arguments to ODL https://review.openstack.org/604750 | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 13:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 13:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] | 13:10 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates stable/queens: Fix syntax for set_fact module. https://review.openstack.org/604774 | 13:10 |
| slagle | bogdando: where is the ansible log for the validations? | 13:12 |
| slagle | bogdando: i see this, http://logs.openstack.org/11/585911/3/gate/tripleo-ci-centos-7-containers-multinode/2a143a8/logs/undercloud/home/zuul/validations_pre-deployment.log.txt.gz | 13:12 |
| slagle | but it has no timestamps, so is not very useful | 13:12 |
| slagle | bogdando: where are these validations logging? | 13:12 |
| bogdando | slagle: http://logs.openstack.org/21/597221/8/gate/tripleo-ci-centos-7-containers-multinode/59f728a/logs/undercloud/home/zuul/validations_pre-deployment.log.txt.gz | 13:12 |
| *** paramite has quit IRC | 13:13 | |
| bogdando | executed from this place http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/playbooks/multinode-overcloud-prep.yml#n39 | 13:13 |
| slagle | bogdando: that's not all that useful. i mean the actual validations themselves, not just the mistral executions | 13:13 |
| bogdando | I do not know that yet | 13:13 |
| *** skramaja has quit IRC | 13:15 | |
| bogdando | gonna grep logs for 9d398576-dc6f-49b1-8a8b-0241d63a31ca id | 13:15 |
| bogdando | to see some time events | 13:15 |
| bogdando | and dbc1e673-57db-4580-9ddb-ff04065c4b08 probably, the 2nd one | 13:16 |
| bogdando | quiquell|rover: ^^ | 13:16 |
| quiquell|rover | bogdando: what job ? I have like have a docen of them in front of me | 13:17 |
| *** agopi is now known as agopi|ansible|tr | 13:17 | |
| bogdando | there is also 1161c0ac-8c5d-4afa-b7bf-6e034f1c26af. I hope logs contain all the neded timestamps to figure out why it took 23 min | 13:17 |
| bogdando | quiquell|rover: http://logs.openstack.org/21/597221/8/gate/tripleo-ci-centos-7-containers-multinode/59f728a/logs/ara_oooq/ | 13:17 |
| mschuppert | could I please get another +2 review for https://review.openstack.org/595001 | 13:20 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common stable/queens: Revert "Remove ceph-osd after deployment succeeded in containers" https://review.openstack.org/604779 | 13:22 |
| trown | thanks shardy | 13:24 |
| *** mcornea has joined #tripleo | 13:26 | |
| quiquell|rover | bogdando: I don't found dbc1e673-57db-4580-9ddb-ff04065c4b08 | 13:27 |
| quiquell|rover | I am suppose to look at containerized mistral ? | 13:28 |
| bogdando | quiquell|rover: I can see nothing weird in mistral.api log | 13:28 |
| quiquell|rover | slagle, bogdando: Make sense to deactivate validations and see if timeout is gone ? | 13:28 |
| quiquell|rover | slagle, bogdando: If we have timeouts at least we have remove a variable for investigation | 13:28 |
| bogdando | there is no huge leaps in timestamps, it just keeps doing those executions | 13:28 |
| quiquell|rover | bogdando: But the number of retries is kind of massive | 13:29 |
| bogdando | quiquell|rover: yeah, not sure how to debug those down | 13:29 |
| *** zul has joined #tripleo | 13:31 | |
| quiquell|rover | bogdando: Good one, only fiew retries, http://logs.openstack.org/24/567224/109/check/tripleo-ci-centos-7-containers-multinode/12fcbdd/logs/undercloud/home/zuul/validations_pre-deployment.log.txt.gz | 13:31 |
| quiquell|rover | bogdando: have to be the retries... | 13:31 |
| quiquell|rover | bogdando: What do I have to do If I want to disable validations ? | 13:32 |
| bogdando | quiquell|rover: set these two false http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/playbooks/multinode-overcloud-prep.yml#n48 | 13:33 |
| *** toure|gone is now known as toure | 13:33 | |
| slagle | quiquell|rover: don't know. probably need to get someone from validations involved | 13:34 |
| *** SteelyDan is now known as dansmith | 13:34 | |
| quiquell|rover | slagle: who are in the DFG ? | 13:34 |
| quiquell|rover | rdopiera ? | 13:35 |
| shardy | florianf: ^^ | 13:36 |
| quiquell|rover | florianf: Do you know how to debug retries at validations ? | 13:37 |
| openstackgerrit | John Fulton proposed openstack/tripleo-common master: Remove ceph-ansible Mistral workflow https://review.openstack.org/604783 | 13:37 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-specs master: [WIP] Bare metal provisioning without Nova and Glance https://review.openstack.org/604761 | 13:37 |
| dtantsur | shardy, slagle, a lot of text ^^^ more to come | 13:38 |
| Tengu | quiquell|rover: you get get florianf and akrivoka :) | 13:38 |
| bogdando | quiquell|rover: looking into tripleo-common ... https://review.openstack.org/#/c/445914/ | 13:39 |
| quiquell|rover | Tengu: that's more than enough | 13:39 |
| bogdando | you may want to ask that author :) | 13:39 |
| *** mjturek has joined #tripleo | 13:39 | |
| shardy | dtantsur: ack thanks! | 13:39 |
| Tengu | anyone "fluent" with the CI reproducer script running on a local libvirt host? | 13:39 |
| quiquell|rover | bogdando: thanks for the pointer | 13:40 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Fix error in Tenant network routes definition https://review.openstack.org/604784 | 13:42 |
| quiquell|rover | thrash: You there ? | 13:43 |
| thrash | quiquell|rover: Yes... | 13:43 |
| quiquell|rover | thrash: We are having longer duration than expected at tripleo validations | 13:44 |
| quiquell|rover | thrash: We see a lot of tretries | 13:44 |
| quiquell|rover | thrash: Do you know how to debug that ? | 13:44 |
| thrash | quiquell|rover: Is this the ansible validations? | 13:44 |
| bogdando | thrash: yeah, like 3:12:10 PM GMT+2 - slagle: bogdando: i see this, http://logs.openstack.org/11/585911/3/gate/tripleo-ci-centos-7-containers-multinode/2a143a8/logs/undercloud/home/zuul/validations_pre-deployment.log.txt.gz | 13:46 |
| bogdando | 3:12:23 PM GMT+2 - slagle: but it has no timestamps, so is not very useful | 13:46 |
| quiquell|rover | thrash: I think | 13:46 |
| quiquell|rover | thrash: A lot of retries there ^ | 13:46 |
| bogdando | so I traced down all those executions ID in mistral.api... but nothing interesting there | 13:47 |
| bogdando | so how could we to go deeper ?:) | 13:47 |
| bogdando | also, having so sparse debug records https://pastebin.com/9AcCR1Ze looks like we do not keep that mistral guy too busy :) | 13:48 |
| bogdando | it has tens of seconds for meditaion | 13:48 |
| openstackgerrit | Victor Pickard proposed openstack/tripleo-heat-templates master: Change log directory for OpenDaylight https://review.openstack.org/603907 | 13:50 |
| *** jaosorior has quit IRC | 13:55 | |
| *** bdodd has joined #tripleo | 13:55 | |
| *** beekneemech is now known as bnemec | 13:57 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: DNM: Disable tripleo validations https://review.openstack.org/604790 | 13:58 |
| quiquell|rover | bogdando: ^ | 13:58 |
| *** agopi|ansible|tr is now known as agopi|ansible | 13:58 | |
| quiquell|rover | bogdando: Let's see if we have timeouts there | 13:58 |
| quiquell|rover | bogdando: Will do some rechecks | 13:58 |
| bogdando | quiquell|rover: what do you mean? disabled validations produce nothing, even timeouts :) | 13:59 |
| quiquell|rover | bogdando: If I disable them and there is no timeouts after few runs, maybe give as confirmation of some issue there | 13:59 |
| quiquell|rover | bogdando: Wait that's old shit | 14:00 |
| *** dxiri has joined #tripleo | 14:01 | |
| *** raildo has joined #tripleo | 14:02 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Fix error in Tenant network routes definition https://review.openstack.org/604784 | 14:03 |
| *** ykarel is now known as ykarel|away | 14:03 | |
| bogdando | quiquell|rover: a known bug http://logs.openstack.org/75/596275/10/check/tripleo-ci-centos-7-containers-multinode/ab8229f/job-output.txt.gz#_2018-09-24_10_59_40_090971 ? | 14:03 |
| bogdando | perchance... | 14:03 |
| *** dxiri has quit IRC | 14:03 | |
| bogdando | never mind, a red herring | 14:03 |
| *** dxiri has joined #tripleo | 14:04 | |
| quiquell|rover | bogdando: This is just running the fallback, we are all good there | 14:05 |
| quiquell|rover | bandini: the rescue goes to qemu | 14:05 |
| quiquell|rover | bogdando: I mean | 14:05 |
| *** dhill_ has quit IRC | 14:06 | |
| *** dhill_ has joined #tripleo | 14:06 | |
| *** chkumar|ruck has quit IRC | 14:08 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 14:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 14:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] | 14:10 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Tag step plays https://review.openstack.org/599072 | 14:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Remove "when failed" from debug task names https://review.openstack.org/598221 | 14:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Tag tasks in in common tasks https://review.openstack.org/603250 | 14:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Convert with_dict tasks to use loop and be less chatty https://review.openstack.org/603251 | 14:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: puppet host https://review.openstack.org/603252 | 14:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: docker_puppet https://review.openstack.org/603253 | 14:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: docker_puppet_tasks https://review.openstack.org/603254 | 14:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: paunch configs https://review.openstack.org/603255 | 14:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Name unnamed tasks https://review.openstack.org/604460 | 14:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Fix tasks in check mode https://review.openstack.org/604461 | 14:11 |
| quiquell|rover | slagle: You are rewritting tht ^ ? | 14:12 |
| *** chandankumar has joined #tripleo | 14:14 | |
| *** ramishra has quit IRC | 14:14 | |
| *** chandankumar is now known as chkumar|ruck | 14:15 | |
| bogdando | debugging failing jobs getting harder these days ... any ideas what on earth went wrong here http://logs.openstack.org/75/596275/10/check/tripleo-ci-centos-7-containers-multinode/ab8229f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-24_12_03_35 or the ansible log itself http://logs.openstack.org/75/596275/10/check/tripleo-ci-centos-7-containers-multinode/ab8229f/logs/undercloud/var/lib/mistral/config-download-latest/ansib | 14:15 |
| *** quiquell|rover is now known as quique|rover|off | 14:15 | |
| *** ade_lee has joined #tripleo | 14:20 | |
| *** ratailor has joined #tripleo | 14:22 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-docs master: Document standalone/UC update Heat stack specifics https://review.openstack.org/578799 | 14:24 |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud stable/rocky: Include missing config classes https://review.openstack.org/604799 | 14:26 |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud stable/queens: Include missing config classes https://review.openstack.org/604800 | 14:27 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: use the debug callback (humananly_readable) https://review.openstack.org/604802 | 14:31 |
| florianf | quique|rover|off: back from a meeting | 14:33 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Inject undercloud user into the cloud image https://review.openstack.org/603069 | 14:33 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Fix environment vars not working with commands https://review.openstack.org/603360 | 14:33 |
| shardy | bandini: Hey just wanted to update you re https://bugs.launchpad.net/tripleo/+bug/1792613 - I started looking into it, but it's harder than I expected - we use the old bootstrap_nodeid in $many places :( | 14:35 |
| openstack | Launchpad bug 1792613 in tripleo "docker-puppet tasks can only run on the primary controller role" [High,Triaged] - Assigned to Steven Hardy (shardy) | 14:35 |
| openstackgerrit | Merged openstack/tripleo-docs master: Remove --container-registry-file parameter from upgrade docs https://review.openstack.org/570903 | 14:35 |
| shardy | bandini: so I need to unravel that then do the docker-puppet fix, otherwise stuff like composable upgrades and FFU will be broken in the exact same way | 14:36 |
| shardy | sigh | 14:36 |
| shardy | probably we should consider removing the global boostrap ID as it's clearly causing confusion | 14:36 |
| bandini | shardy: ah dang :/ thanks for looking into it in any case! | 14:36 |
| *** waleedm has quit IRC | 14:37 | |
| thrash | bogdando: quique|rover|off I don't see timeouts... | 14:37 |
| rascasoft | slagle, are you good with the env? Can I take it back? | 14:38 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: WIP: Add necessary bits for N-1->N standalone upgrade. https://review.openstack.org/604736 | 14:39 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Update standalone container prep steps https://review.openstack.org/604487 | 14:41 |
| bogdando | shardy: had we removed it, how to determine the single node for the cluster to execute docker puppet tasks?.. | 14:44 |
| slagle | rascasoft: yes, go ahead | 14:44 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: undercloud/stackrc: unset OS_* variables https://review.openstack.org/602367 | 14:44 |
| shardy | bogdando: it should be the first server where a service is enabled, even when the service spans more than one role | 14:44 |
| rascasoft | slagle, ack thanks | 14:44 |
| shardy | bogdando: not hard-coded to always run them e.g on "Controller" role | 14:44 |
| bogdando | shardy: "the first server where a service is enabled" totally makes sense, indeed | 14:45 |
| shardy | otherwise if you move Keystone to any other role, no users are created, because the tasks run on the wrong role, where no service_config_settings exist | 14:45 |
| shardy | bogdando: this is an unfortunate historical thing, we started with a global bootstrap_nodeid, then added e.g short_service_bootstrap_node, but never cleaned up and removed the old global ID | 14:46 |
| bogdando | :-< | 14:46 |
| shardy | thus, we've used the old global approach for a bunch of stuff re container bootstrap and upgrades | 14:46 |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient stable/rocky: Update default value for ntp servers https://review.openstack.org/604806 | 14:47 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Use int builtin filter for testing purpose https://review.openstack.org/604807 | 14:48 |
| *** Petersingh is now known as Petersingh|away | 14:57 | |
| *** Petersingh|away has quit IRC | 14:58 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-puppet-elements stable/queens: Remove ceph-osd from the overcloud baremetal image https://review.openstack.org/591486 | 14:59 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-puppet-elements stable/queens: Remove Openstack and Ceph packages https://review.openstack.org/603965 | 15:00 |
| *** chkumar|ruck is now known as chkumar|off | 15:02 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 15:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 15:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] | 15:10 |
| *** sanjayu_ has quit IRC | 15:10 | |
| *** fhubik|brb has quit IRC | 15:10 | |
| *** ksambor has quit IRC | 15:11 | |
| *** ratailor has quit IRC | 15:12 | |
| *** weshay_afk is now known as weshay | 15:23 | |
| *** iranzo has quit IRC | 15:31 | |
| *** leanderthal has quit IRC | 15:33 | |
| *** bugzy_ is now known as bugzy | 15:33 | |
| *** holser_ has quit IRC | 15:35 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Undercloud - Restart keepalived on update https://review.openstack.org/603587 | 15:35 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: WIP: New workflow for standalone upgrade https://review.openstack.org/604706 | 15:43 |
| *** sanjayu_ has joined #tripleo | 15:47 | |
| weshay | jistr, what log are you seeing this in? https://bugs.launchpad.net/tripleo/+bug/1794103 the journal? | 15:48 |
| openstack | Launchpad bug 1794103 in tripleo "Deployment with default roles data broken: container-image-prepare service not passing resource_registry to tripleo-container-image-prepare" [Critical,Triaged] | 15:48 |
| toure | fultonj ping | 15:48 |
| jistr | weshay: i saw it locally, it's in overcloud_deploy.log | 15:49 |
| jistr | weshay: i'm not sure if it affects any CI, mostly we use custom roles in CI i think | 15:49 |
| weshay | k.. that's what I'm checking | 15:49 |
| weshay | k | 15:50 |
| toure | fultonj just FYI found the problem and it is in the mistral_api service, I am still trying to find the root cause but it may be in the wsgi module | 15:50 |
| fultonj | toure: thanks for the update | 15:51 |
| toure | sure thing | 15:51 |
| weshay | reviews on https://review.openstack.org/#/c/604455 please | 15:54 |
| mwhahaha | weshay: sooooo how are things? :D (do we know whats up with timeouts yet) | 15:54 |
| *** dxiri has quit IRC | 15:55 | |
| weshay | quique and chandan mentioned validations and pulling containers.. | 15:55 |
| weshay | but you know | 15:55 |
| weshay | mwhahaha, sagi's patch to send the task data out is not merged | 15:55 |
| weshay | so | 15:55 |
| mwhahaha | if it's container downloads, has anyone ask infra about the health of the mirrors? | 15:56 |
| weshay | mwhahaha, quique put this place.. but I'm not clear on it yet.. it's new http://dashboard-ci.tripleo.org/d/-UEjGKFmz/exploration?orgId=1 | 15:56 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Inject undercloud user into the cloud image https://review.openstack.org/603069 | 15:56 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Fix environment vars not working with commands https://review.openstack.org/603360 | 15:56 |
| weshay | I just got in | 15:56 |
| weshay | haven't trolled irc logs | 15:56 |
| weshay | yet | 15:56 |
| *** tzumainn has left #tripleo | 15:57 | |
| *** tzumainn has joined #tripleo | 15:58 | |
| *** toure is now known as toure|biab | 16:00 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 16:00 |
| *** yprokule has quit IRC | 16:02 | |
| *** dxiri has joined #tripleo | 16:08 | |
| *** agopi|ansible is now known as agopi|training | 16:08 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 16:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 16:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] | 16:10 |
| bogdando | toure|biab: could that root cause be the case for https://bugs.launchpad.net/tripleo/+bug/1794034 as well? | 16:10 |
| *** bogdando has quit IRC | 16:15 | |
| *** janki has quit IRC | 16:20 | |
| *** aufi has quit IRC | 16:22 | |
| *** spotz is now known as spotz_ | 16:25 | |
| *** spotz_ is now known as spotz | 16:25 | |
| *** thrash is now known as thrash|biab | 16:26 | |
| *** rdopiera has quit IRC | 16:27 | |
| *** jpich has quit IRC | 16:50 | |
| *** trown is now known as trown|lunch | 16:50 | |
| *** paramite has joined #tripleo | 16:50 | |
| *** paramite has quit IRC | 16:52 | |
| *** Vorrtex has joined #tripleo | 16:55 | |
| *** thrash|biab is now known as thrash | 16:56 | |
| *** hamzy_ has joined #tripleo | 16:58 | |
| mwhahaha | weshay: do we know why the rdo jobs are flaky as well? | 16:58 |
| *** derekh has quit IRC | 16:59 | |
| *** hamzy has quit IRC | 16:59 | |
| weshay | mwhahaha, nope.. but it looks pretty shitty | 17:01 |
| dtantsur | slagle: finished a thought dump re distributed undercloud in the bottom of https://etherpad.openstack.org/p/tripleo-edge-squad-status | 17:07 |
| weshay | mwhahaha, all notes from the ruck/rover are always here https://review.rdoproject.org/etherpad/p/ruckrover-sprint19 | 17:07 |
| slagle | dtantsur: thanks! | 17:08 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 17:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 17:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] | 17:10 |
| *** udesale has quit IRC | 17:10 | |
| *** sshnaidm has joined #tripleo | 17:11 | |
| *** sshnaidm is now known as sshnaidm|pto | 17:13 | |
| *** sshnaidm|pto has quit IRC | 17:20 | |
| *** jpena is now known as jpena|off | 17:21 | |
| *** colonwq has quit IRC | 17:27 | |
| *** tosky has quit IRC | 17:30 | |
| *** sshnaidm|pto has joined #tripleo | 17:33 | |
| *** salmankhan has quit IRC | 17:37 | |
| openstackgerrit | Tong Liu proposed openstack/tripleo-heat-templates master: Add more NSX config parameters https://review.openstack.org/604858 | 17:39 |
| weshay | 17:48 | |
| *** akrivoka has quit IRC | 17:51 | |
| *** trown|lunch is now known as trown | 17:52 | |
| *** dtantsur is now known as dtantsur|afk | 17:56 | |
| *** toure|biab is now known as toure | 17:57 | |
| *** akhila has joined #tripleo | 17:58 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Handle non-existant plan when getting deployment status https://review.openstack.org/602753 | 18:04 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Don't fail tripleo-bootstrap on package installs https://review.openstack.org/603196 | 18:04 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Fix check mode for server deployments https://review.openstack.org/600182 | 18:04 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Check mode support for hiera deployments https://review.openstack.org/600183 | 18:04 |
| *** florianf has quit IRC | 18:07 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 18:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 18:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] | 18:10 |
| beagles | mwhahaha: are we still not workflowing on master? | 18:14 |
| mwhahaha | beagles: i mean you could, but it's not likely to land anytime soon | 18:15 |
| mwhahaha | beagles: currently we're +20 hours in the gate | 18:15 |
| beagles | mwhahaha: yeah, just saw that | 18:15 |
| beagles | wow | 18:15 |
| mwhahaha | was +38 yesterday | 18:15 |
| mwhahaha | so yea | 18:15 |
| mwhahaha | we need to figure out the timeout stuff | 18:15 |
| weshay | quique|rover|off, chkumar|off for tomorrow ^^ | 18:17 |
| toure | thrash ping | 18:21 |
| weshay | mwhahaha, there are actually are not that many gate failures in the last 24hr | 18:35 |
| weshay | only 5 | 18:35 |
| mwhahaha | only | 18:35 |
| weshay | it's been more recently | 18:35 |
| weshay | 8-10 | 18:36 |
| weshay | it should be zero I know | 18:36 |
| mwhahaha | yea cause no one was approving stuff yesterday :D | 18:36 |
| weshay | just saying | 18:36 |
| openstackgerrit | Honza Pokorny proposed openstack/tempest-tripleo-ui master: Add basic project structure https://review.openstack.org/575730 | 18:36 |
| *** sshnaidm|pto has quit IRC | 18:36 | |
| beagles | slagle: if you have a sec https://review.openstack.org/#/c/597038/ - I realize we probs aren't workflowing atm but I'd like to get these teed up as there is an interested party chomping at the bit for a fix | 18:37 |
| *** ykarel|away has quit IRC | 18:39 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: clean up gentoo overlay provisioning https://review.openstack.org/604873 | 18:39 |
| weshay | nhicher++ | 18:46 |
| mwhahaha | faaaaailure in the gate | 18:47 |
| weshay | geez | 18:50 |
| weshay | yup | 18:50 |
| weshay | https://review.openstack.org/#/c/563881/ | 18:51 |
| weshay | yay.. it has logs | 18:51 |
| mwhahaha | dat multinode | 18:51 |
| weshay | http://logs.openstack.org/81/563881/2/gate/tripleo-ci-centos-7-containers-multinode/4b768e7/logs/ara_oooq_oc/ | 18:52 |
| mwhahaha | so 27 mins to download the containers | 18:52 |
| weshay | 1 hr 4min to install the undercloud | 18:53 |
| weshay | :) | 18:53 |
| mwhahaha | http://logs.openstack.org/81/563881/2/check/tripleo-ci-centos-7-containers-multinode/80b990d/logs/ara_oooq_oc/ | 18:53 |
| mwhahaha | so that's the passing check job | 18:53 |
| mwhahaha | 20 mins to pull containers | 18:53 |
| weshay | passing job was 52 min in the undercloud | 18:54 |
| weshay | mwhahaha, we don't have any infra folks online atm | 18:54 |
| mwhahaha | so in the gate the overcloud deploy took +17 mins | 18:55 |
| mwhahaha | everything was slower | 18:55 |
| mwhahaha | both vexxhost | 18:56 |
| mwhahaha | both same az | 18:56 |
| mwhahaha | http://logs.openstack.org/81/563881/2/gate/tripleo-ci-centos-7-containers-multinode/4b768e7/zuul-info/inventory.yaml vs http://logs.openstack.org/81/563881/2/check/tripleo-ci-centos-7-containers-multinode/80b990d/zuul-info/inventory.yaml | 18:56 |
| *** panda has quit IRC | 18:56 | |
| mwhahaha | so to me that points to the mirror | 18:57 |
| mwhahaha | hmm | 18:57 |
| mwhahaha | mnaser: you around? | 18:57 |
| mnaser | mwhahaha: sup | 18:58 |
| mwhahaha | mnaser: so we're running into timeouts and the latest failure was in vexxhost, trying to determine if the mirrors are overloaded or something | 18:58 |
| mnaser | you know | 18:58 |
| mwhahaha | see -^ | 18:58 |
| mnaser | so infra-root runs mirrors, we don't | 18:59 |
| *** dxiri has quit IRC | 18:59 | |
| mnaser | but i'm wondering if ipv6 is a factor | 18:59 |
| mnaser | like for some reason it tries to pull stuff over ipv6 and infra mirrors isnt configured to pull right over ipv6? | 18:59 |
| mwhahaha | mnaser: right but wanted to make sure there aren't any uplink issues or stuff | 18:59 |
| mnaser | mwhahaha: nothing that i know of. it'd be good to get infra-root involved and digging into it | 19:00 |
| fungi | our mirrors in regions which have ipv6 support are set up with v6 global addresses and aaaa records | 19:00 |
| mwhahaha | yea | 19:00 |
| mwhahaha | i'm not seeing anything funny in the cacti graphs either :/ | 19:01 |
| *** amoralej is now known as amoralej|off | 19:02 | |
| mwhahaha | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=4557&rra_id=all | 19:02 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: ensure tripleo-validations are turned off upstream https://review.openstack.org/604878 | 19:03 |
| mwhahaha | the interface traffic seems to be ok (at least not pegging on anything) | 19:03 |
| mwhahaha | fungi: did we ever get caching stats from the mirrors recorded somewhere? | 19:04 |
| *** bugzy has quit IRC | 19:04 | |
| fungi | clarkb probably remembers, i'm not sure. may want to move this to #openstack-infra | 19:05 |
| mwhahaha | k | 19:05 |
| *** bugzy has joined #tripleo | 19:09 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 19:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 19:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] | 19:10 |
| *** tosky has joined #tripleo | 19:11 | |
| mwhahaha | weshay: are we accidently nuking the container-image-prepare log? http://logs.openstack.org/81/563881/2/gate/tripleo-ci-centos-7-containers-multinode/4b768e7/logs/ara_oooq_oc/result/7c70e575-709c-44cb-9242-20619186115b/ it seems to be set to ../install-undercloud.log but it's not appending? | 19:11 |
| *** shardy has quit IRC | 19:12 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: ensure tripleo-validations are turned off upstream https://review.openstack.org/604878 | 19:12 |
| weshay | mwhahaha, ^ this should help | 19:13 |
| weshay | mwhahaha, /me looks at the log | 19:13 |
| openstackgerrit | Michael Bayer proposed openstack/tripleo-specs master: Global Galera Database https://review.openstack.org/600555 | 19:13 |
| *** sai_p has joined #tripleo | 19:15 | |
| *** jfrancoa has quit IRC | 19:15 | |
| *** pcaruana has quit IRC | 19:17 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/rocky: Add override_ansible_cfg https://review.openstack.org/604879 | 19:18 |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient stable/rocky: Add --override-ansible-cfg https://review.openstack.org/604881 | 19:20 |
| *** dxiri has joined #tripleo | 19:20 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/rocky: Add workflow for config-download export https://review.openstack.org/604884 | 19:22 |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient stable/rocky: Use workflow config_download_export https://review.openstack.org/604885 | 19:22 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Add debug to container iamge prepare https://review.openstack.org/604886 | 19:22 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: DNM: force debug to container image prepare https://review.openstack.org/604887 | 19:22 |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient stable/rocky: Add --override-ansible-cfg https://review.openstack.org/604881 | 19:22 |
| mwhahaha | so did we break rocky with the --override-ansible-cfg? | 19:25 |
| mwhahaha | slagle, weshay: did we screw up oooq-extras so we shouldn't be passing in custom_ansible.cfg for rocky yet? | 19:25 |
| weshay | hrm | 19:27 |
| mwhahaha | weshay: https://logs.rdoproject.org/06/588606/5/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/cae47fe/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-24_18_28_17 | 19:28 |
| weshay | yuppers https://logs.rdoproject.org/06/588606/5/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/cae47fe/logs/undercloud/home/zuul/overcloud-deploy.sh | 19:30 |
| weshay | that merged last friday I though | 19:31 |
| weshay | I thought | 19:31 |
| weshay | wtf http://logs.openstack.org/22/597222/15/check/tripleo-ci-centos-7-containers-multinode-rocky/ee8ec26/job-output.txt.gz | 19:32 |
| * weshay flips a table | 19:32 | |
| *** holser_ has joined #tripleo | 19:33 | |
| * toure hands weshay a snikers bar | 19:34 | |
| weshay | mwhahaha, weird | 19:35 |
| weshay | https://review.openstack.org/#/c/597222/15/roles/overcloud-deploy/templates/overcloud-deploy.sh.j2 | 19:35 |
| weshay | mwhahaha, your job log is maste | 19:36 |
| weshay | r | 19:36 |
| mwhahaha | weshay: oh well that's even worse? | 19:36 |
| openstackgerrit | wes hayutin proposed openstack/python-tripleoclient master: Remove instack-undercloud and --use-heat https://review.openstack.org/588606 | 19:36 |
| weshay | mwhahaha, probably just needs a rebase | 19:36 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Rely on path for CLI calls when possible https://review.openstack.org/604891 | 19:36 |
| mwhahaha | weshay: yes likely | 19:37 |
| weshay | mwhahaha, don't scare me like that faker | 19:37 |
| *** panda has joined #tripleo | 19:37 | |
| * weshay sends a snickers bar to mwhahaha | 19:43 | |
| slagle | mwhahaha: no idea. i mean...i just backported it today | 20:03 |
| slagle | so, black magic or rocky ci not testing rocky? | 20:03 |
| mwhahaha | slagle: i misread it, it was a master job on an out of date tripleoclient change | 20:06 |
| * weshay flips mwhahaha | 20:09 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 20:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 20:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] | 20:10 |
| *** raildo_ has joined #tripleo | 20:11 | |
| *** raildo_ has quit IRC | 20:12 | |
| *** raildo has quit IRC | 20:13 | |
| *** vinaykns has joined #tripleo | 20:15 | |
| *** agopi|training has quit IRC | 20:15 | |
| *** salmankhan has joined #tripleo | 20:22 | |
| openstackgerrit | Ian Main proposed openstack/python-tripleoclient master: WIP: Speed up plan operations by using a tarball in swift. https://review.openstack.org/581141 | 20:22 |
| openstackgerrit | Ian Main proposed openstack/tripleo-common master: WIP: Remove ProcessTemplatesAction as base class https://review.openstack.org/599188 | 20:24 |
| openstackgerrit | Ian Main proposed openstack/tripleo-common master: WIP: Use process-templates.py and cache templates to speed up plan ops. https://review.openstack.org/581153 | 20:25 |
| *** lblanchard has quit IRC | 20:28 | |
| *** dprince has quit IRC | 20:30 | |
| gouthamr | hi, i asked in #rdo, but can anyone here help me find information about container image promotions? I made some changes on kolla, want to know when they'll show up on tripleo images | 20:31 |
| *** akhila has quit IRC | 20:32 | |
| mwhahaha | gouthamr: so we had a promotion over the weekend, when did the kolla changes merge? | 20:33 |
| *** sshnaidm|pto has joined #tripleo | 20:34 | |
| gouthamr | mwhahaha: Sep 6, about three weeks ago | 20:34 |
| mwhahaha | so it should be available now | 20:34 |
| gouthamr | mwhahaha: awesome, ty.. but for future reference, how do these promotions work? :) my google-foo didn't bring up any docs | 20:35 |
| mwhahaha | the promotion process runs multiple times a day but it depends on the status of the jobs on if it actually goes through | 20:35 |
| mwhahaha | when we have blocking issues, it can lead to some delay | 20:35 |
| mwhahaha | currently we're 2 days old because of some issues | 20:35 |
| gouthamr | mwhahaha: oh... so this was out-of-the-norm.. | 20:35 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Calculate ARA metrics for overcloud https://review.openstack.org/604900 | 20:35 |
| mwhahaha | gouthamr: https://ci.centos.org/view/rdo/view/promotion-pipeline/ | 20:35 |
| mwhahaha | wait that might be the old one | 20:36 |
| * mwhahaha can never remember where the dashboard is | 20:36 | |
| openstackgerrit | Bob Fournier proposed openstack/os-net-config master: Restart ivs/nvfswitch after config file is updated https://review.openstack.org/604901 | 20:38 |
| mwhahaha | https://dashboards.rdoproject.org/rdo-dev there it is | 20:39 |
| mwhahaha | not sure what teh difference with master vs tripleo is on that one. always thought they were the same | 20:39 |
| openstackgerrit | Bob Fournier proposed openstack/os-net-config master: Restart ivs/nvfswitch after config file is updated https://review.openstack.org/604901 | 20:39 |
| weshay | gouthamr, mwhahaha http://cistatus.tripleo.org/promotion/ | 20:41 |
| weshay | https://review.rdoproject.org/zuul/status.html | 20:41 |
| weshay | mwhahaha, https://ci.centos.org/view/rdo/view/promotion-pipeline/ is the RDO promotion | 20:42 |
| weshay | mwhahaha, I don't think anyone uses https://dashboards.rdoproject.org/rdo-dev | 20:42 |
| mwhahaha | maybe we should? :D | 20:43 |
| weshay | mwhahaha, I use the internal one | 20:43 |
| weshay | if they want me to do everything twice.. they can pay me double | 20:43 |
| mwhahaha | i would enjoy the proper rate of 10 times | 20:43 |
| weshay | lolz | 20:43 |
| weshay | +10 | 20:44 |
| gouthamr | weshay++ | 20:49 |
| gouthamr | mwhahaha++ | 20:49 |
| gouthamr | :) | 20:49 |
| gouthamr | ty! | 20:49 |
| mwhahaha | weshay: shouldn't we promote rdo before tripleo? not sure why those are different | 20:51 |
| weshay | mwhahaha, if you want to block tripleo on packstack and puppet jobs.. sure | 20:51 |
| weshay | but I know you don't | 20:52 |
| weshay | so | 20:52 |
| *** ansmith has quit IRC | 20:53 | |
| *** Vorrtex has quit IRC | 20:57 | |
| *** slaweq has quit IRC | 21:01 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Undercloud - Restart keepalived on update https://review.openstack.org/603587 | 21:02 |
| *** trown is now known as trown|outtypewww | 21:05 | |
| *** iranzo has joined #tripleo | 21:05 | |
| *** sshnaidm|pto has quit IRC | 21:10 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 21:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 21:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 21:10 |
| *** PIutNQlSvf has left #tripleo | 21:11 | |
| *** rh-jelabarre has quit IRC | 21:13 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/rocky: Stop cap granting to empty pool when telemetry disabled https://review.openstack.org/604734 | 21:20 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Stop cap granting to empty pool when telemetry disabled https://review.openstack.org/604735 | 21:20 |
| *** dsneddon has quit IRC | 21:24 | |
| *** dsneddon has joined #tripleo | 21:24 | |
| *** Ng__ has joined #tripleo | 21:32 | |
| *** ChanServ sets mode: +v Ng__ | 21:32 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: docker_puppet https://review.openstack.org/603253 | 21:33 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: docker_puppet_tasks https://review.openstack.org/603254 | 21:33 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: paunch configs https://review.openstack.org/603255 | 21:33 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Name unnamed tasks https://review.openstack.org/604460 | 21:33 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Fix tasks in check mode https://review.openstack.org/604461 | 21:33 |
| *** jtomasek has quit IRC | 21:37 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: docker_puppet https://review.openstack.org/603253 | 21:38 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: docker_puppet_tasks https://review.openstack.org/603254 | 21:38 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: check mode: paunch configs https://review.openstack.org/603255 | 21:38 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Name unnamed tasks https://review.openstack.org/604460 | 21:38 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Fix tasks in check mode https://review.openstack.org/604461 | 21:38 |
| *** salmankhan has quit IRC | 21:39 | |
| *** sshnaidm|pto has joined #tripleo | 21:40 | |
| *** agopi|training has joined #tripleo | 21:43 | |
| *** bfournie has quit IRC | 21:44 | |
| *** gfidente has quit IRC | 21:47 | |
| *** ansmith has joined #tripleo | 21:49 | |
| *** thrash is now known as thrash|g0ne | 21:58 | |
| *** dxiri has quit IRC | 21:59 | |
| *** mcornea has quit IRC | 22:06 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 22:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 22:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 22:10 |
| *** vinaykns has quit IRC | 22:12 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-quickstart-extras master: Rebase undercloud.conf template https://review.openstack.org/604922 | 22:14 |
| openstackgerrit | Harald Jensås proposed openstack/python-tripleoclient master: Undercloud Validations - Deprecated (replaced/removed) opts https://review.openstack.org/604923 | 22:14 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 22:14 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 22:16 |
| *** holser_ has quit IRC | 22:16 | |
| *** timothyb89 has joined #tripleo | 22:21 | |
| *** timothyb89 has left #tripleo | 22:21 | |
| *** iranzo has quit IRC | 22:29 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/tripleo-heat-templates stable/pike: DNM - testing scenario004 https://review.openstack.org/602148 | 22:33 |
| *** panda is now known as panda|off | 22:38 | |
| *** rcernin has joined #tripleo | 22:45 | |
| *** dxiri has joined #tripleo | 22:51 | |
| *** owalsh_ has joined #tripleo | 22:53 | |
| *** mmedvede_ has joined #tripleo | 23:00 | |
| *** owalsh has quit IRC | 23:00 | |
| *** fpan has quit IRC | 23:00 | |
| *** weshay has quit IRC | 23:00 | |
| *** bandini has quit IRC | 23:00 | |
| *** mmedvede has quit IRC | 23:00 | |
| *** mcarden has quit IRC | 23:00 | |
| *** kambiz has quit IRC | 23:00 | |
| *** mmedvede_ is now known as mmedvede | 23:00 | |
| *** fpan has joined #tripleo | 23:02 | |
| *** tosky has quit IRC | 23:03 | |
| *** sshnaidm|pto has quit IRC | 23:04 | |
| *** dxiri has quit IRC | 23:07 | |
| *** dxiri has joined #tripleo | 23:07 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1794034 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 23:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 23:10 |
| openstack | Launchpad bug 1794034 in tripleo "Validation tasks have increase their duration" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 23:10 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: podman: create/delete systemd unit files when configured with 'restart=always' https://review.openstack.org/600849 | 23:37 |
| openstackgerrit | Dan Macpherson proposed openstack/ansible-role-openstack-operations master: Adding Backup and Restore Operations https://review.openstack.org/604439 | 23:39 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Stop hardcoding 'docker' and make it more generic https://review.openstack.org/601290 | 23:43 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: WIP - podman: setup systemd dependencies https://review.openstack.org/601421 | 23:45 |
| *** mjturek has quit IRC | 23:52 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/rocky: undercloud/stackrc: unset OS_* variables https://review.openstack.org/604936 | 23:53 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-quickstart-extras master: Rebase undercloud.conf template https://review.openstack.org/604922 | 23:56 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Allow to run bootstrap containers in privileged mode. https://review.openstack.org/600533 | 23:59 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Set proper setype for service directories https://review.openstack.org/600534 | 23:59 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Allow to deactivate SELinux separation for selected containers https://review.openstack.org/600535 | 23:59 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Support podman when tagging container for Pacemaker https://review.openstack.org/604180 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!