| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 00:00 |
|---|---|---|
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 00:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 00:10 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Refactor openshift services for composable roles https://review.openstack.org/599618 | 00:14 |
| *** lblanchard has joined #tripleo | 00:25 | |
| *** hamzy has joined #tripleo | 00:25 | |
| *** rlandy has quit IRC | 00:36 | |
| openstackgerrit | Merged openstack/tripleo-common master: Tag openshift images for Infra service https://review.openstack.org/603050 | 00:51 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix openshift new node detection https://review.openstack.org/600012 | 00:51 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Add CephOSD service to roles/Standalone.yaml https://review.openstack.org/603758 | 00:51 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/rocky: Start websocket client before workflows https://review.openstack.org/605499 | 00:51 |
| openstackgerrit | Merged openstack/instack-undercloud stable/rocky: Include missing config classes https://review.openstack.org/604799 | 00:51 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Tag step plays https://review.openstack.org/599072 | 00:54 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Remove "when failed" from debug task names https://review.openstack.org/598221 | 00:54 |
| openstackgerrit | Merged openstack/tripleo-common master: Handle non-existant plan when getting deployment status https://review.openstack.org/602753 | 00:54 |
| *** tzumainn has quit IRC | 00:55 | |
| openstackgerrit | Merged openstack/tripleo-common stable/rocky: Add override_ansible_cfg https://review.openstack.org/604879 | 00:57 |
| *** phuongnh has joined #tripleo | 01:04 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 01:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 01:10 |
| *** shardy has quit IRC | 01:10 | |
| *** shardy has joined #tripleo | 01:18 | |
| *** phuongnh has quit IRC | 01:30 | |
| *** dmacpher_ has joined #tripleo | 01:33 | |
| *** dmacpher has quit IRC | 01:35 | |
| *** itlinux has joined #tripleo | 01:44 | |
| *** itlinux has quit IRC | 01:44 | |
| *** zzzeek has quit IRC | 01:48 | |
| *** zzzeek has joined #tripleo | 01:49 | |
| *** mrsoul has quit IRC | 01:55 | |
| *** mschuppert has quit IRC | 01:56 | |
| *** mschuppert has joined #tripleo | 01:57 | |
| *** mburned is now known as mburned_out | 02:00 | |
| *** jamesdenton has joined #tripleo | 02:07 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 02:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 02:10 |
| *** ykarel has joined #tripleo | 02:18 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Support podman when tagging container for Pacemaker https://review.openstack.org/604180 | 02:24 |
| *** boazel has joined #tripleo | 02:26 | |
| EmilienM | chkumar|off: http://logs.openstack.org/17/600517/35/check/tripleo-ci-centos-7-undercloud-containers/f15eb65/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-09-27_23_39_00 | 02:27 |
| EmilienM | 2018-09-27 23:39:00 | mkdir: cannot create directory '/home/zuul/tempest': Permission denied | 02:27 |
| EmilienM | chkumar|off: I think we're close | 02:27 |
| *** ykarel has quit IRC | 02:32 | |
| *** jhebden has quit IRC | 02:44 | |
| *** jhebden has joined #tripleo | 02:51 | |
| *** skramaja has joined #tripleo | 02:53 | |
| *** phuongnh has joined #tripleo | 02:53 | |
| *** med_ has quit IRC | 02:57 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 03:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 03:10 |
| *** ykarel has joined #tripleo | 03:13 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Remove toci_jobtype definition from v3 jobs https://review.openstack.org/593863 | 03:13 |
| *** lblanchard has quit IRC | 03:28 | |
| *** psachin has joined #tripleo | 03:30 | |
| openstackgerrit | Merged openstack/tripleo-common master: Update swift_rings_backup workflow to also backup ceph fetch dir https://review.openstack.org/597221 | 03:31 |
| openstackgerrit | Merged openstack/tripleo-validations master: Add new nova-event-callback validation https://review.openstack.org/513333 | 03:31 |
| *** sanjayu_ has joined #tripleo | 03:48 | |
| *** iranzo has joined #tripleo | 03:51 | |
| openstackgerrit | Merged openstack/python-tripleoclient stable/rocky: Fix typo in upgrade playbook's name. https://review.openstack.org/604685 | 03:57 |
| openstackgerrit | Merged openstack/tripleo-common master: Make ODL healthcheck IPv6 compatible https://review.openstack.org/596987 | 03:57 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Fix syntax for set_fact module. https://review.openstack.org/604774 | 03:57 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Expose IronicImageDownloadSource as a parameter https://review.openstack.org/603796 | 03:57 |
| openstackgerrit | Merged openstack/tripleo-common master: Don't fail tripleo-bootstrap on package installs https://review.openstack.org/603196 | 03:57 |
| *** jaganathan has joined #tripleo | 04:00 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 04:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 04:10 |
| Tengu | hello there :) | 04:47 |
| *** ramishra has joined #tripleo | 05:09 | |
| *** udesale has joined #tripleo | 05:09 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 05:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 05:10 |
| *** ykarel_ has joined #tripleo | 05:11 | |
| *** itlinux has joined #tripleo | 05:12 | |
| *** ykarel has quit IRC | 05:14 | |
| *** ykarel__ has joined #tripleo | 05:15 | |
| *** ykarel_ has quit IRC | 05:18 | |
| *** ykarel_ has joined #tripleo | 05:20 | |
| *** ykarel__ has quit IRC | 05:23 | |
| *** ykarel__ has joined #tripleo | 05:24 | |
| *** ykarel_ has quit IRC | 05:27 | |
| *** ykarel_ has joined #tripleo | 05:29 | |
| *** ykarel__ has quit IRC | 05:32 | |
| Tengu | if anyone in here could add the missing cr+2 on that one it would be great :) https://review.openstack.org/#/c/600534/ | 05:35 |
| *** quiquell|off is now known as quiquell | 05:41 | |
| quiquell | Tengu: good morning | 05:45 |
| *** chkumar|off is now known as chandankumar | 05:46 | |
| chandankumar | quiquell: Tengu jaosorior \o/ | 05:46 |
| quiquell | chandankumar: o/ | 05:46 |
| *** Petersingh has joined #tripleo | 05:47 | |
| Tengu | hello quiquell, chandankumar and jaosorior :)) | 05:49 |
| Tengu | jaosorior: hey, I'm pretty sure you're in a good mind for some cr+2 :) https://review.openstack.org/#/c/600534/ please? :) | 05:49 |
| chandankumar | Tengu: just one question https://review.openstack.org/#/c/600534/ does this changes is not needed for tempest container? | 05:52 |
| quiquell | Tengu, jaosorior, chandankumar: Proper handling of connection close with zaqar https://review.openstack.org/#/c/605387/ | 05:53 |
| Tengu | chandankumar: good question - I didn't do anything with tempest testing on that. It's for the plain deploy itself in fact. | 05:53 |
| quiquell | Good for debugging ^ | 05:53 |
| chandankumar | Tengu: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/tempest.yaml#L55 | 05:53 |
| Tengu | chandankumar: hmm yep, needed. | 05:53 |
| chandankumar | Tengu: let this patch get's merged | 05:54 |
| chandankumar | Tengu: I need to update tempest.yaml also with some changes | 05:54 |
| chandankumar | I will take care in that | 05:54 |
| Tengu | chandankumar: ok, cool :) | 05:54 |
| Tengu | chandankumar: I will probably need a second pass on the whole t-h-t - I've mainly worked out the issues I got while deploying the undercloud with podman+selinux | 05:54 |
| Tengu | chandankumar: so if you can take care of that directory... also, take care of the creation with setype | 05:55 |
| Tengu | chandankumar: https://review.openstack.org/#/c/600534/12/docker/services/ironic-api.yaml@158 for example with a loop | 05:55 |
| chandankumar | Tengu: yup, sure | 05:56 |
| Tengu | great :) | 05:56 |
| Tengu | quiquell: reading your patch :). Debug is good | 05:56 |
| quiquell | Tengu: yeah, you are sensible of this after a harsh rover session | 05:57 |
| Tengu | quiquell: no kidding ;) | 05:58 |
| *** gfidente has joined #tripleo | 06:07 | |
| quiquell | chandankumar, Tengu: Do you know why this review that has all is not merged ? | 06:08 |
| quiquell | https://review.openstack.org/#/c/594511/ | 06:08 |
| quiquell | It's like stuck at openstack/triple-docs | 06:09 |
| Tengu | quiquell: probably because of CI issues - someone needs to -w // w+1 | 06:09 |
| jaosorior | chandankumar, quiquell: Seems we have a gigantic queue in zuul again | 06:10 |
| jaosorior | and infra is starting to call us out on it | 06:10 |
| jaosorior | any idea why? | 06:10 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 06:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 06:10 |
| quiquell | jaosorior: But I see stuff in wrong queues https://review.openstack.org/#/c/594511/ | 06:10 |
| jaosorior | Also, they say we do spend too much time collecting logs...and asked if we can trim that time down | 06:10 |
| quiquell | jaosorior: hummm we introduces one stuff there, about gatering ARA (I was suspecting it will cost us) | 06:10 |
| quiquell | jaosorior: You mean collect logs or post actions ? | 06:11 |
| *** pcaruana has joined #tripleo | 06:11 | |
| *** ksambor has joined #tripleo | 06:17 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: Add httpd and mod_ssl packages to octavia api image https://review.openstack.org/603220 | 06:19 |
| quiquell | Tengu: I have end up at one of your patches at tht https://github.com/openstack/tripleo-heat-templates/commit/623790385292acf4cb4f357a8d089e9d08d4d212 | 06:22 |
| quiquell | Tengu: we are exercising rocky->master containerized undercloud upgrade | 06:22 |
| quiquell | Tengu: and looks like xinetd service does not exists | 06:22 |
| *** dsneddon has quit IRC | 06:23 | |
| *** verdurin has quit IRC | 06:23 | |
| *** holser_ has joined #tripleo | 06:25 | |
| *** anande has joined #tripleo | 06:27 | |
| jaosorior | quiquell: any idea what's up with these http://status.openstack.org/elastic-recheck/data/others.html ? | 06:27 |
| quiquell | jaosorior: zuul post timedout | 06:29 |
| quiquell | jaosorior: but time difference is very small, something is broken at timeout config | 06:30 |
| Tengu | quiquell: xinetd was used previously for 1-2 services. | 06:30 |
| Tengu | quiquell: the goal there is to allow to remove that deprecated service | 06:30 |
| Tengu | care to explain your issue? | 06:30 |
| quiquell | Tengu: last comment https://review.openstack.org/#/c/590774 | 06:31 |
| quiquell | Tengu: looks like xinted service is not present at rocky | 06:31 |
| chandankumar | Post queue has 92 hrs waiting time | 06:31 |
| Tengu | quiquell: that makes sense in fact. | 06:31 |
| *** jfrancoa has joined #tripleo | 06:31 | |
| jaosorior | quiquell: is that timeout config something we set in tripleo? | 06:31 |
| jaosorior | * in tripleo-ci or quickstart | 06:31 |
| Tengu | quiquell: so the code I produced should be a bit different I guess so that it's kicking ONLY if we're <rocky ? | 06:32 |
| quiquell | jaosorior: Wait there is something I don't understand | 06:32 |
| Tengu | quiquell: unless we can do a "ignore_errors: true" in there? | 06:32 |
| quiquell | Tengu: But the tht put's rocky in the top, is not enough ? | 06:32 |
| * quiquell is a total noob on tht | 06:32 | |
| *** holser_ has quit IRC | 06:33 | |
| Tengu | quiquell: not sure it's used for that in fact. | 06:33 |
| Tengu | I think it's more a validation thing in order to ensure we use the right template version for the current deploy. | 06:33 |
| quiquell | jaosorior: Ahh wait... the minutes are the same the hours not... yepp is a clear timeout config is ok | 06:33 |
| Tengu | quiquell: answered your comment - guess the "ignore_errors" directive is the right thing. | 06:33 |
| quiquell | Tengu: yep is kind of cleanup, if not there we are good too, that's it ? | 06:34 |
| Tengu | quiquell: exactly | 06:34 |
| quiquell | Tengu: cool thanks, will try to fix | 06:34 |
| quiquell | Tengu: maybe is better to check for existence | 06:35 |
| Tengu | quiquell: hmm yeah why not. using "systemd" ansible module to get state | 06:36 |
| jaosorior | gfidente: Seen this issue before http://logs.openstack.org/58/601558/3/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/f6a3788/logs/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz#_2018-09-28_06_25_08_448 ? | 06:36 |
| Tengu | quiquell: do you take care of that? or do you want me to fix it? | 06:36 |
| jaosorior | gfidente: this commit targetted for queens just hit it https://review.openstack.org/#/c/601558/ | 06:36 |
| quiquell | jaosorior: This could be the issue https://review.openstack.org/#/c/580238/ | 06:37 |
| quiquell | Tengu: going to try to take care, so I deep into tht | 06:37 |
| quiquell | s/deep/dig/ | 06:37 |
| Tengu | quiquell: fine :). you should just edit that xinetd service file | 06:37 |
| Tengu | it's plain ansible. no real magic ;) | 06:37 |
| quiquell | Tengu: let's see what we find after fix this... like trains in the station | 06:38 |
| Tengu | hehe | 06:38 |
| *** mrsoul has joined #tripleo | 06:38 | |
| Tengu | quiquell: I know that - had that same kind of thing while working on podman+selinux integration :D | 06:38 |
| Tengu | well, and STILL finding things. | 06:38 |
| Tengu | like the modprobe done within containers -.- | 06:38 |
| quiquell | Tengu: now that you mention containers, I have question | 06:39 |
| * Tengu hides | 06:39 | |
| quiquell | Tengu: is possible to store containers at images ? | 06:39 |
| Tengu | gni? | 06:39 |
| Tengu | don't understand your question | 06:39 |
| Tengu | do you mean generate image from a running/deployed container? | 06:39 |
| quiquell | Tengu: yep | 06:40 |
| Tengu | yep, we can | 06:40 |
| Tengu | at least with docker | 06:40 |
| quiquell | Tengu: so it's like having RPM installed but instead of that we have containers "installed) | 06:40 |
| Tengu | we probably can do the same with either podman or buildah | 06:40 |
| quiquell | Tengu: will try some proto, maybe need your help | 06:40 |
| Tengu | quiquell: this in order to prevent the whole bootstrap of the containers? | 06:40 |
| quiquell | Tengu: to try to reduce times | 06:41 |
| jaosorior | quiquell: well, as far as I can tell from the failures here http://status.openstack.org/elastic-recheck/data/others.html , it's still timeouts... maybe we did increase the time by collecting more logs, but I think those are quite useful... do we have anything else that we could cut time on, or is there any other cause for the timeouts that we're still dealing with? | 06:41 |
| Tengu | quiquell: well, that will eat space, and when you fetch the images, it will take network bandwidth. A delicate balance. | 06:41 |
| quiquell | Tengu: ... in case of non-containerize, this space is consume by installed RPMs | 06:42 |
| Tengu | yeah, but you might get mutliple containers with the same packages | 06:42 |
| quiquell | Tengu: so have to be similar but with the overhead of docker registry (I can be missing a lot of stuff) | 06:42 |
| quiquell | Tengu: don't agree, depends on the layers | 06:42 |
| Tengu | at least same package base | 06:42 |
| quiquell | Tengu: layers are shared if they are the same | 06:42 |
| Tengu | need to control that when you generate image from a running container - not sure if it works 100% the same. | 06:43 |
| quiquell | Tengu: hummm that's right, is very difficult to make it right... | 06:43 |
| Tengu | quiquell: you might want to ping EmilienM when he's connected for some discussion. I think he knows a bit more than me about all that. | 06:44 |
| quiquell | jaosorior: Humm you are right... RUN times out too :-/ | 06:44 |
| *** jtomasek has joined #tripleo | 06:44 | |
| quiquell | Tengu: cool thanks | 06:44 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Add podman support to validate-tempest role https://review.openstack.org/605356 | 06:47 |
| *** holser_ has joined #tripleo | 06:49 | |
| *** holser_ has quit IRC | 06:49 | |
| *** holser_ has joined #tripleo | 06:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/tripleo-ci master: Enable featureset override https://review.openstack.org/594511 | 06:50 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-heat-templates master: Set proper setype for tempest service directories https://review.openstack.org/605980 | 06:54 |
| chandankumar | Tengu: ^^ | 06:54 |
| Tengu | chandankumar: \o/ | 06:55 |
| openstackgerrit | hanish proposed openstack/puppet-tripleo master: Implements: liquidio-containerization https://review.openstack.org/605981 | 06:55 |
| chandankumar | I need to wait for Bodgan to come I need to make some more changes to tempest container | 06:55 |
| chandankumar | mandre: Hello | 06:56 |
| chandankumar | mandre: In tempest container, I want to have three volumes auto mounted from tempest kolla images directory | 06:57 |
| chandankumar | mandre: one is /var/log/tempest, tempest workspace and data directory and all these directory should be owned by tempest user | 06:57 |
| chandankumar | mandre: It can be handled on tht side but I donot want to do that, Is it possible to handle directly on kolla tempest dockerfile side? | 06:58 |
| openstackgerrit | hanish proposed openstack/tripleo-heat-templates master: Implements: liquidio-containerization https://review.openstack.org/605982 | 07:01 |
| *** florianf|afk has quit IRC | 07:01 | |
| *** shardy has quit IRC | 07:01 | |
| *** shardy has joined #tripleo | 07:02 | |
| Tengu | chandankumar: you might want to use "mkdir -p {{ tempest_dir }}" in case it already exists or need to create a tree. | 07:03 |
| Tengu | chandankumar: (for https://review.openstack.org/#/c/605356/4/roles/validate-tempest/templates/configure-tempest.sh.j2 ) | 07:04 |
| *** quiquell is now known as quiquell|brb | 07:04 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Add podman support to validate-tempest role https://review.openstack.org/605356 | 07:04 |
| openstackgerrit | hanish proposed openstack/tripleo-heat-templates master: Implements: liquidio-containerization https://review.openstack.org/605982 | 07:05 |
| chandankumar | Tengu: thanks, I think I need to get rid of these dir, it can be handled on tht side or may be on kolla tempest dockerfile itself | 07:05 |
| Tengu | chandankumar: t-h-t might be the right place indeed. I don't really know tempest though, can't judge more. | 07:06 |
| chandankumar | Tengu: yup | 07:07 |
| Tengu | anyway, replacing bash scripts by ansible is always a good move :) | 07:08 |
| chandankumar | Tengu: yes long term place is to stub all these shell scripts here https://github.com/openstack/openstack-ansible-os_tempest | 07:08 |
| chandankumar | into ansible | 07:09 |
| *** dtrainor has quit IRC | 07:09 | |
| chandankumar | *plan | 07:09 |
| Tengu | \o/ | 07:10 |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 07:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 07:10 |
| *** f2 has joined #tripleo | 07:11 | |
| *** f2 is now known as florianf | 07:11 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Improve nova statedir ownership logic https://review.openstack.org/587066 | 07:11 |
| *** rcernin has quit IRC | 07:12 | |
| *** rdopiera has joined #tripleo | 07:14 | |
| *** ssbarnea|bkp has quit IRC | 07:16 | |
| *** cylopez has joined #tripleo | 07:20 | |
| gfidente | jaosorior ah no, looking into it now | 07:23 |
| gfidente | I guess this will affect rocky and master cause we use the same version for all branches | 07:23 |
| *** gkadam has joined #tripleo | 07:23 | |
| *** cylopez has left #tripleo | 07:25 | |
| *** amoralej|off is now known as amoralej | 07:26 | |
| gfidente | jaosorior I don't get it though, it's not happening for all runs? | 07:26 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: WIP - Fix stackviz https://review.openstack.org/605419 | 07:27 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-quickstart master: FS038: enable tempest run https://review.openstack.org/599178 | 07:27 |
| jaosorior | gfidente: not sure, I just saw it though | 07:28 |
| jaosorior | gfidente: and it was for queens too | 07:28 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Fix overcloud ARA data collection https://review.openstack.org/605678 | 07:29 |
| *** Petersingh is now known as Petersingh|lunch | 07:32 | |
| *** quiquell|brb is now known as quiquell | 07:35 | |
| *** jpena|off is now known as jpena | 07:43 | |
| gfidente | jaosorior so I am not sure, the copy module for that task is wrapped into a custom action so there could be issues with params mangling in the wrapper | 07:43 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: Ignore errors at xinetd stop/uninstall https://review.openstack.org/605989 | 07:43 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Allow a containerized logrotate to access docker https://review.openstack.org/605349 | 07:43 |
| gfidente | but then in http://tripleo.org/cistatus.html I see that both scenario001 and 004 are green and pretty stable | 07:43 |
| *** phuongnh has quit IRC | 07:44 | |
| gfidente | it's pretty complicated to add a debug ling in the wrapper action because tripleo/ci won't consume it unless it's built into an rpm | 07:45 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Switch previous release of master from 'queens' to 'rocky' https://review.openstack.org/590774 | 07:45 |
| quiquell | Tengu: ^ | 07:46 |
| *** AJaeger has joined #tripleo | 07:47 | |
| quiquell | jaosorior: The timeouts are at rocky ? | 07:47 |
| AJaeger | tripleo team, jaosorior, you have only *2* changes open for python3-first: both are stable changes for paunch, see https://review.openstack.org/597831 and https://review.openstack.org/597848 - and both fail ;( | 07:48 |
| *** bogdando has joined #tripleo | 07:48 | |
| AJaeger | What do you want to do to get those merged? | 07:48 |
| AJaeger | openstack-tox-py27 is failing in both cases | 07:49 |
| bandini | any takers for a simple cherry-pick ? https://review.openstack.org/#/c/601077/ | 07:49 |
| *** leanderthal has joined #tripleo | 07:50 | |
| *** tosky has joined #tripleo | 07:50 | |
| Tengu | quiquell: hmm. I doubt this will avoid the xinetd thingy. well, let's see what ci spits. | 07:50 |
| quiquell | Tengu: ack, ideally I want to check if the package is installed, but want to unblock at least to see next issues | 07:50 |
| jaosorior | AJaeger: I'll check it out | 07:54 |
| quiquell | jaosorior: I don't see to much timeouts in the gates, also found that at the merge for master timeout is recent | 07:54 |
| *** assassin has joined #tripleo | 07:54 | |
| quiquell | jaosorior: at rocky | 07:54 |
| *** quiquell is now known as quiquell|brb | 07:55 | |
| AJaeger | thanks, jaosorior | 07:55 |
| *** holser_ has quit IRC | 07:55 | |
| *** ratailor has joined #tripleo | 07:56 | |
| *** jpich has joined #tripleo | 07:58 | |
| *** holser_ has joined #tripleo | 07:58 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 08:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/pike: GATE CHECK for TripleO https://review.openstack.org/602248 | 08:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 08:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 08:00 |
| *** quiquell|brb is now known as quiquell | 08:05 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 08:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 08:10 |
| *** anande has quit IRC | 08:17 | |
| quiquell | jaosorior: I am starting to see POST_FAILURES with POST timeouts | 08:17 |
| quiquell | marios ^ | 08:18 |
| *** sai_p has quit IRC | 08:26 | |
| mandre | chandankumar: hi! I'm back | 08:26 |
| mandre | chandankumar: so you want to change ownership of some directories mounted in the container but do not want to do it in the script that starts your tempest container? | 08:27 |
| chandankumar | mandre: yes correct | 08:28 |
| *** ykarel__ has joined #tripleo | 08:28 | |
| chandankumar | mandre: since the same container can be used by other distribution | 08:28 |
| therve | quiquell: Do you know what's the failure with "gating_repo.tar.gz: No such file or directory" ? | 08:28 |
| chandankumar | mandre: with in that direction write and read permission should happen | 08:29 |
| quiquell | therve: do you have a log around ? | 08:29 |
| therve | quiquell: https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-extras-gate-master-delorean-full-featureset052-719/console.txt.gz | 08:30 |
| *** Petersingh|lunch is now known as Petersingh | 08:30 | |
| mandre | chandankumar: hmm, the rule we've been following so far is that the tool using the container should set the right perms on the directories it uses | 08:30 |
| chandankumar | mandre: for example https://github.com/openstack/kolla/blob/master/docker/tempest/extend_start.sh#L4 -> it should point to /var/log/tempest | 08:30 |
| *** ykarel_ has quit IRC | 08:31 | |
| mandre | chandankumar: I've always hated these mkdir in extend_start and proposed to remove them | 08:31 |
| *** dtrainor has joined #tripleo | 08:31 | |
| mandre | chandankumar: that's kolla-ansible specific and should be created in kolla-ansible | 08:32 |
| mandre | we do not use this path in tripleo | 08:32 |
| chandankumar | mandre: I want to carry minimal stuff in tht and handle all stuff in kolla container | 08:32 |
| mandre | the dir is there but we don't care about it | 08:32 |
| chandankumar | mandre: tempest is not currently consumed in kolla-ansible | 08:32 |
| mandre | chandankumar: one more reason to remove everything that's in https://github.com/openstack/kolla/blob/master/docker/tempest/extend_start.sh | 08:33 |
| mandre | chandankumar: what is the problem with setting the right perms in tht? | 08:33 |
| chandankumar | mandre: nothing | 08:34 |
| chandankumar | mandre: our long term plan with validate-tempest role is to replace validate-tempest role with ohttps://github.com/openstack/openstack-ansible-os_tempest and consume it in tripleo , openstack-ansible and kolla-ansible | 08:34 |
| chandankumar | that's why I donot wanted to keep stuff in tht | 08:35 |
| mandre | chandankumar: well, in that case, fixing the perms should go in your openstack-ansible-os_tempest role, shouldn't it? | 08:36 |
| *** anande has joined #tripleo | 08:36 | |
| chandankumar | mandre: yup, | 08:36 |
| mandre | chandankumar: I suppose tempest is a special beast and we could make an exception | 08:36 |
| mandre | but if it's possible to fix the perms where the container is used I much prefer it | 08:36 |
| chandankumar | mandre: let me see what can I do | 08:37 |
| *** derekh has joined #tripleo | 08:37 | |
| quiquell | therve: It's missing checking compressed_gating_repo conditional | 08:38 |
| quiquell | therve: looks like it's not generating any gating_repo | 08:38 |
| *** ratailor has quit IRC | 08:40 | |
| *** anande has quit IRC | 08:41 | |
| *** ratailor has joined #tripleo | 08:41 | |
| therve | quiquell: OK not sure what that means :). What generate this? | 08:42 |
| quiquell | therve: To be able to test changes at projects that are installed through RPMs we have to create a special yum repos with new RPMs containing those changes. | 08:44 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Add OS::TripleO::Services::Rhsm to OpenShift roles https://review.openstack.org/605999 | 08:44 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Use Timesync service instead of Ntp https://review.openstack.org/606000 | 08:44 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Let openshift-ansible configure the firewall https://review.openstack.org/606001 | 08:44 |
| quiquell | therve: the build-test-packages role do that inspecting the zuul/jenkins changes, and register it at a variable that have to be checked before use it. | 08:45 |
| therve | quiquell: So the variable is present, but why the file isn't? | 08:45 |
| quiquell | therve: The variable is not present, we are not checking it at the failing task | 08:45 |
| quiquell | therve: a 'whenÂ' is missing | 08:46 |
| therve | OK I trust you on this :) | 08:46 |
| quiquell | therve: Then you are screw :-P | 08:46 |
| therve | I see that --extra-vars artg_compressed_gating_repo=/home/stack/gating_repo.tar.gz | 08:46 |
| quiquell | therve: but don't know why this is failing now | 08:46 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Use glusterfs for registry when deploying with CNS https://review.openstack.org/605825 | 08:46 |
| quiquell | therve: yep that's it | 08:47 |
| quiquell | therve: do you have the review that trigger this job ? | 08:47 |
| quiquell | therve: is weird that we don't have gating_repo, has to be over tq tqe | 08:48 |
| therve | quiquell: https://review.openstack.org/#/c/604979/ | 08:48 |
| quiquell | therve: a ok this is a change at tqe, tqe is not installed with RPM so no gating_repo is generated | 08:48 |
| *** arxcruz is now known as arxcruz|doctor | 08:49 | |
| quiquell | therve: Then I don't know why the failing task is missing the when statment | 08:49 |
| therve | quiquell: Shouldn't that affect all changes then? | 08:49 |
| therve | tqe ones that is | 08:49 |
| quiquell | therve: tqe, tq and possible tripleo-ci | 08:49 |
| quiquell | therve: let me check if the task is new | 08:50 |
| quiquell | therve: this is weird then when statement is here http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/libvirt/setup/undercloud/tasks/main.yml#n51 | 08:52 |
| quiquell | therve: something is artifically generating the variable | 08:52 |
| therve | quiquell: Does the "artg_" prefix matter? | 08:52 |
| *** shyamb has joined #tripleo | 08:52 | |
| quiquell | therve: where do you see --extra-vars artg_compressed_gating_repo ? | 08:53 |
| therve | quiquell: The quickstart call in that file | 08:54 |
| therve | quiquell: bash quickstart.sh --working-dir /home/jenkins/workspace/tripleo-quickstart-extras-gate-master-delorean-full-featureset052/ --no-clone --bootstrap --extra-vars artg_compressed_gating_repo=/home/stack/gating_repo.tar.gz --playbook build-test-packages.yml --tags all --teardown all --release centosci/master 172.19.2.99 | 08:54 |
| quiquell | therve: that's wrong | 08:54 |
| quiquell | therve: this is centos.org don't know where the script lives | 08:57 |
| therve | ? | 08:58 |
| quiquell | ykarel__: Do you know where is the script that do this https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-extras-gate-master-delorean-full-featureset052-719/console.txt.gz | 08:58 |
| therve | quiquell: https://github.com/openstack/tripleo-quickstart/blob/master/ci-scripts/full-deploy.sh#L61-L71 ? | 08:59 |
| ykarel__ | quiquell, see ci-config | 08:59 |
| *** ykarel__ is now known as ykarel | 08:59 | |
| ykarel | rdo-infra/ci-config | 08:59 |
| quiquell | therve, ykarel: we are still using full-deploy.sh ? | 09:00 |
| openstackgerrit | Udi Kalifon proposed openstack/tempest-tripleo-ui master: Selenium infra https://review.openstack.org/605424 | 09:00 |
| ykarel | quiquell, yes atleast in phase 1 it's used, not sure about other places | 09:00 |
| therve | Maybe not :) | 09:00 |
| therve | Oh ok | 09:00 |
| *** tosky has quit IRC | 09:02 | |
| quiquell | therve: this is standalone | 09:02 |
| *** tosky has joined #tripleo | 09:03 | |
| therve | OK I have no idea how all this works :D | 09:03 |
| quiquell | therve: Looks like it have fails forever for standalone on tq, tqe changes... | 09:04 |
| *** ykarel is now known as ykarel|lunch | 09:04 | |
| therve | Sounds worth fixing. Or to get rid of the job. | 09:05 |
| quiquell | therve: let me verify where do we call the build-test-packages | 09:05 |
| marios|rover | quiquell: ack | 09:06 |
| marios|rover | quiquell: which jobs/info? | 09:06 |
| marios|rover | quiquell: nm i see one in grafana | 09:07 |
| quiquell | marios|rover: http://dashboard-ci.tripleo.org/d/FEdraO0ik/jobs-exploration?orgId=1&var-influxdb_filter=result%7C%3D%7CPOST_FAILURE | 09:08 |
| *** salmankhan has joined #tripleo | 09:08 | |
| quiquell | therve: the ansible fact is cached so no need to pass it over quickstart.sh calls I think | 09:08 |
| quiquell | therve: we can totally remove it | 09:08 |
| quiquell | therve: weird thing it's going to fail at tq/tqe changes at all calls | 09:08 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 09:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 09:10 |
| therve | quiquell: Last success for this build is 2 weeks ago... | 09:12 |
| quiquell | therve: depends on the kind of change, if it's for example at THT it's going to work | 09:13 |
| quiquell | therve: But it's at tq,tqe is not | 09:13 |
| *** Petersingh is now known as Petersingh|afk | 09:13 | |
| therve | Sure talking about https://ci.centos.org/job/tripleo-quickstart-extras-gate-master-delorean-full-featureset052/ | 09:14 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Remove standalone quickstart.sh gating_repo var https://review.openstack.org/606012 | 09:16 |
| quiquell | therve: ^ | 09:16 |
| therve | Thanks! | 09:16 |
| quiquell | therve: add a Depends-On to see if it works now | 09:16 |
| quiquell | therve: worth checking also at tht dummy change for example, can you do that for me ? | 09:16 |
| *** rdo has quit IRC | 09:17 | |
| therve | Sorry I don't know what you mean | 09:17 |
| therve | It's non-voting right? | 09:18 |
| therve | I'd rather make another recheck if possible | 09:18 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Add standalone upgrade role and playbook. https://review.openstack.org/604736 | 09:22 |
| *** Petersingh|afk is now known as Petersingh | 09:23 | |
| shyamb | Hi | 09:23 |
| shyamb | Overcloud deployment for queens is failing even if I don't change anything | 09:24 |
| shyamb | It's taking 2-3 retries to get a successful deployment | 09:24 |
| shyamb | shardy: Tengu: jaosorior: | 09:24 |
| shyamb | RHOSP10 was quite stable and consistent but RHOSP13 is not same | 09:25 |
| shardy | shyamb: fails how? | 09:25 |
| shardy | and on what platform? | 09:25 |
| shyamb | http://paste.openstack.org/show/731080/ | 09:25 |
| shyamb | rhel7 platform | 09:26 |
| shyamb | shardy: Error doesn't look consistent across deployments | 09:26 |
| shardy | shyamb: Ok, probably need more information to figure out why - openstack stack failures list overcloud --long as a start but probably you'll need to look at the logs to work out what's up with haproxy | 09:28 |
| shyamb | shardy: ok | 09:31 |
| shyamb | but if I don't change anything in the command or overcloud nodes, things should work as it is | 09:32 |
| shyamb | in that case if deployment fails, I am not getting motivation to go ahead and debug the issue | 09:32 |
| *** akrivoka has joined #tripleo | 09:33 | |
| shardy | shyamb: :\ | 09:39 |
| shardy | If you're not prepared to even try to debug it, why bother asking for help here? | 09:39 |
| shardy | sigh | 09:39 |
| *** Petersingh is now known as Petersingh|afk | 09:41 | |
| shyamb | shardy: I debugged my issues | 09:43 |
| shyamb | and fixed many | 09:43 |
| shyamb | but if I don't change anything in the command or overcloud and it's working on next retry | 09:43 |
| shyamb | why should I debug it | 09:44 |
| shyamb | My concern is why should it work on retry | 09:44 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: WIP: read job variables from deploy playbooks https://review.openstack.org/606017 | 09:45 |
| *** ykarel|lunch is now known as ykarel | 09:48 | |
| jaosorior | gfidente: should I raise a bug? That affected a job in pike as well. | 09:48 |
| shardy | shyamb: my point is if you don't capture why it failed the first time, we have zero chance of fixing the underlying issue | 09:49 |
| gfidente | jaosorior I think an issue in github for ceph-ansible | 09:49 |
| jaosorior | I see | 09:49 |
| gfidente | jaosorior can you paste me a link to the pike error? because in pike we're using a different version of ceph-ansible | 09:49 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: DNM: To test job variables https://review.openstack.org/606020 | 09:50 |
| shyamb | shardy: Next time, I will capture it | 09:50 |
| shardy | shyamb: thanks | 09:51 |
| jaosorior | gfidente: oh, it was a failure; but a different(now that I'm digging into the ceph ansible logs) | 09:51 |
| jaosorior | gfidente: http://logs.openstack.org/08/604708/1/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/f07f96c/logs/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz#_2018-09-28_08_05_48_545 | 09:51 |
| jaosorior | and | 09:51 |
| jaosorior | gfidente:http://logs.openstack.org/08/604708/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e0c9113/logs/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz#_2018-09-28_08_26_15_870 | 09:51 |
| *** jtomasek has quit IRC | 09:56 | |
| *** shyamb has quit IRC | 09:56 | |
| *** shyamb has joined #tripleo | 09:58 | |
| *** Petersingh|afk is now known as Petersingh | 10:03 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates stable/pike: Do not disable ipv6 on loopback interface for epmd https://review.openstack.org/606026 | 10:07 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 10:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 10:10 |
| *** AJaeger has left #tripleo | 10:14 | |
| *** Petersingh has quit IRC | 10:22 | |
| *** Petersingh_ has joined #tripleo | 10:22 | |
| *** Petersingh_ is now known as Petersingh|afk | 10:23 | |
| quiquell | ykarel: rings any bell -> http://logs.openstack.org/74/590774/16/check/tripleo-ci-centos-7-undercloud-upgrades/4f31e68/logs/undercloud/home/zuul/undercloud_reinstall.log.txt.gz#_2018-09-28_09_23_52 ? | 10:25 |
| ykarel | quiquell, yes this is to do with the undercloud validation during reinstall | 10:26 |
| ykarel | i remeber there are already patches for it | 10:26 |
| ykarel | to fix it | 10:26 |
| quiquell | ykarel: I remember we were fixing something similar last ruck/rovering | 10:26 |
| ykarel | yes | 10:26 |
| quiquell | ykarel: jpena maybe ? | 10:26 |
| ykarel | nope | 10:26 |
| ykarel | jistr, jfrancoa | 10:27 |
| *** shyamb has quit IRC | 10:27 | |
| quiquell | ykarel: Let's see if they heard the siren song | 10:27 |
| quiquell | jillr, jaosorior: Are you there guys ? | 10:28 |
| ykarel | quiquell, https://review.openstack.org/#/c/603523/ | 10:28 |
| jaosorior | quiquell: I'm around, what's up/ | 10:28 |
| ykarel | and rocky backport: https://review.openstack.org/#/c/605815/,not merged, quiquell in which release u saw that error | 10:28 |
| quiquell | ykarel: master, but we need a promotion | 10:29 |
| quiquell | jaosorior: was calling jfrancoa sorry | 10:29 |
| ykarel | quiquell, ack | 10:29 |
| quiquell | jaosorior: my fingers are like "manojo de pollas" sometimes | 10:29 |
| jaosorior | hahaha fair enough | 10:30 |
| *** abishop has joined #tripleo | 10:30 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Fix overcloud ARA data collection https://review.openstack.org/605678 | 10:30 |
| quiquell | jaosorior: btw, gates are good now ? | 10:33 |
| jaosorior | still big zuul queues. However, haven't noticed many timeouts | 10:33 |
| jaosorior | so it's better | 10:33 |
| quiquell | jaosorior: ack | 10:34 |
| jfrancoa | quiquell: ykarel: right that's the patch to fix that issue. I proposed a different one but I abandoned it in favor of that one | 10:34 |
| quiquell | jfrancoa: I suppose we have to wait a promotion to have it | 10:34 |
| quiquell | jfrancoa: Or do we have already promoted ? | 10:35 |
| quiquell | jaosorior, jfrancoa: To have overcloud ARA correctly collected https://review.openstack.org/#/c/605678 | 10:36 |
| quiquell | much needed to debug timeouts | 10:36 |
| jfrancoa | quiquell: no idea, I guess it's that according to the log. | 10:36 |
| quiquell | jfrancoa: you where near | 10:37 |
| jaosorior | quiquell: ack, thanks | 10:37 |
| *** sri_ has quit IRC | 10:50 | |
| quiquell | Tengu: tht change worked | 10:54 |
| Tengu | quiquell: what change? | 10:57 |
| *** dtantsur|afk is now known as dtantsur | 10:57 | |
| Tengu | being thanked while I did nothing: feels weird :D | 10:57 |
| quiquell | Tengu: https://review.openstack.org/#/c/605989/ | 11:00 |
| *** shyamb has joined #tripleo | 11:01 | |
| Tengu | ah, that one. ok :D | 11:01 |
| *** Petersingh|afk is now known as Petersingh | 11:02 | |
| *** med_ has joined #tripleo | 11:02 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: DNM: To test job variables https://review.openstack.org/606020 | 11:04 |
| *** hjensas has joined #tripleo | 11:05 | |
| *** rfolco has quit IRC | 11:07 | |
| *** jpena is now known as jpena|lunch | 11:07 | |
| *** ssbarnea|bkp has joined #tripleo | 11:08 | |
| quiquell | ssbarnea|bkp: you there ? | 11:08 |
| ssbarnea | quiquell: yes. | 11:09 |
| ssbarnea | quiquell: can I help with something? | 11:09 |
| *** Aelia has joined #tripleo | 11:09 | |
| quiquell | ssbarnea: #oooq | 11:10 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 11:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,Triaged] - Assigned to Jiří Stránský (jistr) | 11:10 |
| *** jjoyce has quit IRC | 11:15 | |
| *** ratailor has quit IRC | 11:17 | |
| *** jjoyce has joined #tripleo | 11:17 | |
| Aelia | hello | 11:22 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-specs master: Validation Framework specifications https://review.openstack.org/589169 | 11:22 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: move tripleo-ci release files inside CentOS-7 folder https://review.openstack.org/605642 | 11:23 |
| Tengu | florianf: care to re-check? I just addressed slagle comment regarding namespace (re: validation framework) | 11:23 |
| Aelia | I have something really weird happening after I deployed successfully a ceph node on pike containerized. The OSD containers fail because /dev/vde1 does not exist. I can see no reference to that in the ceph-install-workflow.log on undercloud ... | 11:24 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Add podman support to validate-tempest role https://review.openstack.org/605356 | 11:24 |
| Tengu | florianf: guess we have now a spec that meets all the requirements and should make ppl happy :) | 11:24 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Switch fs027 to deploy with podman https://review.openstack.org/600517 | 11:24 |
| Tengu | florianf: sorry for taking so long addressing that - had "some" other things on the desk ^^' | 11:24 |
| Tengu | gfidente: care to check with Aelia possible issue on ceph-ansible? | 11:25 |
| Aelia | ceph-ansible created a gpt partition table, but no partitions were created | 11:25 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Create soft links for tripleo-ci release files https://review.openstack.org/605643 | 11:26 |
| *** rfolco has joined #tripleo | 11:26 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/rocky: Handle non-existant plan when getting deployment status https://review.openstack.org/606039 | 11:29 |
| fultonj | Aelia: did you clean your disks before depoying? | 11:29 |
| fultonj | Aelia: which task did ceph-ansible fail on as per ceph-install-workflow.log? | 11:30 |
| Aelia | fultonj: qcow2 images created specifically for this test. the deployment succeded no error reported. (I am testing on VMs with vbmc for ironic) | 11:31 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: In process-templates script write output files to provided dir when using base path https://review.openstack.org/605736 | 11:31 |
| *** udesale has quit IRC | 11:32 | |
| florianf | Tengu: Done | 11:32 |
| florianf | Tengu: \o/ | 11:32 |
| Tengu | florianf: great, thanks! | 11:32 |
| Tengu | slagle: if you have a minute just to validate one last time? https://review.openstack.org/589169 | 11:32 |
| fultonj | Aelia: so you'll want to debug directly on the ceph containers as described in https://hub.docker.com/r/ceph/daemon/ | 11:33 |
| openstackgerrit | Udi Kalifon proposed openstack/tempest-tripleo-ui master: Selenium infra https://review.openstack.org/605424 | 11:33 |
| florianf | akrivoka: woudl you like to have a last look as well: https://review.openstack.org/#/c/589169/ | 11:33 |
| florianf | *would | 11:33 |
| Aelia | fultonj: I have a cluster working, I scaled up with one new node. | 11:33 |
| Aelia | but the OSDs containers on the new node are not starting. | 11:34 |
| fultonj | Aelia: so you were adding a new node with N OSDs to an existing cluster | 11:34 |
| fultonj | did all OSDs fail? | 11:34 |
| fultonj | on the new node | 11:34 |
| Aelia | fultonj: yes, and yes. the reason it fails, is that /dev/vd{b,c,d,e} do not have the partitions expected by the container | 11:35 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Merge new params - nic-config templates https://review.openstack.org/605807 | 11:35 |
| Aelia | this is the only thing logged by the container before terminating and being restarted by systemd | 11:35 |
| Aelia | fultonj: 2 lines of log only -> "2018-09-28 11:27:54 /entrypoint.sh: static: does not generate config" and "mount: special device /dev/vde1 does not exist" | 11:36 |
| fultonj | you should be able to 'sgdisk -Z /dev/sdX' for each X and redo | 11:36 |
| fultonj | you need to ensure the disk is clean | 11:36 |
| fultonj | sounds like somewhere in the middle something ahppened which is now getting in the way | 11:37 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Tag tasks in in common tasks https://review.openstack.org/603250 | 11:37 |
| fultonj | or see Zap a device | 11:37 |
| fultonj | under https://hub.docker.com/r/ceph/daemon/ | 11:37 |
| fultonj | Aelia: you can have systemd stop restarting the container | 11:37 |
| fultonj | restart=true --> false in the unit file | 11:37 |
| *** Petersingh is now known as Petersingh|afk | 11:38 | |
| fultonj | there should be a prepare container run and then it activates them | 11:38 |
| *** lblanchard has joined #tripleo | 11:38 | |
| fultonj | if you look at that URL ^ | 11:39 |
| fultonj | you'll see "Deploy an OSD | 11:39 |
| fultonj | " | 11:39 |
| Aelia | fultonj: ok I have used the sgdisk -Z method. Will try to deploy again. | 11:39 |
| fultonj | describing what ceph-ansibel coordinates to get your OSD ready | 11:39 |
| fultonj | somewhere in that process things went wrong | 11:39 |
| fultonj | you can try to do it manually to find it or have ceph-ansible do it again | 11:40 |
| *** agopi|brb is now known as agopi | 11:40 | |
| *** ssbarnea|bkp has quit IRC | 11:40 | |
| fultonj | if it fails repeatedly you'll need to follow the aove in deatils and see ceph-ansible tasks to see what's going and where it's getting stuck | 11:40 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-heat-templates stable/pike: Always lowercase role name https://review.openstack.org/598588 | 11:40 |
| *** shyamb has quit IRC | 11:41 | |
| *** shyamb has joined #tripleo | 11:41 | |
| *** panda|off is now known as panda | 11:42 | |
| *** Petersingh|afk has quit IRC | 11:42 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-heat-templates stable/ocata: Always lowercase role name https://review.openstack.org/598589 | 11:42 |
| akrivoka | florianf: ack, looking | 11:42 |
| Tengu | akrivoka: thanks :) | 11:42 |
| Aelia | fultonj: but what I find strange is that in the ceph-ansible logs I have this -> 2018-09-28 11:26:16,237 p=21839 u=mistral | ok: [10.27.100.12] => (item=/dev/vdb) => {"changed": false, "cmd": "parted --script /dev/vdb print | egrep -sq '^ 1.*ceph'", "delta": "0:00:00.036621", "end": "2018-09-28 09:26:16.071592", "failed_when_result": false, "item": "/dev/vdb", "msg": "non-zero return code", "rc": 1, | 11:44 |
| Aelia | "start": "2018-09-28 09:26:16.034971", "stderr": "Error: /dev/vdb: unrecognised disk label", "stderr_lines": ["Error: /dev/vdb: unrecognised disk label"], "stdout": "", "stdout_lines": []} | 11:44 |
| Aelia | so apparently before the ceph-ansible run, there was no partition table on the disk, it was created by ceph-ansible ... | 11:44 |
| *** yolanda has joined #tripleo | 11:44 | |
| fultonj | "parted --script /dev/vdb print | egrep -sq '^ 1.*ceph'" | 11:44 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/paunch master: Add support for --cap-add to add capabilities https://review.openstack.org/606042 | 11:46 |
| Aelia | fultonj: after that I have an action with "cmd": ["parted", "-s", "/dev/vdb", "mklabel", "gpt"] for vdb | 11:46 |
| Aelia | fultonj: but no action to create any partition on vdb | 11:46 |
| fultonj | are you colocating? | 11:46 |
| fultonj | the journal or using a sep journal disk? | 11:47 |
| fultonj | https://github.com/ceph/ceph-ansible/blob/824ec6d256fc23794d69dd82f789fb05ef5c7bb6/roles/ceph-osd/tasks/check_gpt.yml#L11 | 11:47 |
| *** assassin has quit IRC | 11:47 | |
| akrivoka | Tengu: florianf: looks good! | 11:47 |
| Tengu | bogdando: you create hobbit capacity? :D | 11:47 |
| Tengu | akrivoka: good news :) | 11:47 |
| bogdando | Tengu: :) | 11:47 |
| Aelia | fultonj: should be at least the others are. But for this one I am overriding variables using https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/node_specific_hieradata.html | 11:48 |
| fultonj | Aelia: cool | 11:48 |
| Aelia | the only variable I override is the "devices" to set other block devices than the other nodes | 11:48 |
| fultonj | yep | 11:49 |
| fultonj | Aelia: i really think your disks are not clean | 11:49 |
| holser_ | Tengu, quiquell - my concern is line 59 | 11:50 |
| holser_ | 58 | 11:50 |
| holser_ | where we need to put when: remove_xinetd_pkg|bool | 11:50 |
| holser_ | that's all | 11:50 |
| fultonj | what does lsblk return? | 11:50 |
| Tengu | holser_: ah, that one. well. probably no need to ignore errors on the package itself. | 11:50 |
| Aelia | fultonj: well the deployment is ongoing now so I will keep you informed | 11:51 |
| Tengu | holser_: heee... nope, there we can ignore the error - it fails it the service isn't defined. | 11:51 |
| Tengu | holser_: the service won't be defined if the package is removed. | 11:51 |
| Aelia | fultonj: I already destroyed the partition tables with " sgdisk -Z " | 11:51 |
| fultonj | good | 11:51 |
| fultonj | i think that will help | 11:52 |
| holser_ | Tengu agree | 11:52 |
| quiquell | Tengu, holser_: So just at service is enough ? | 11:52 |
| fultonj | Aelia: i basically clean my nodes w/ ironic between deployments | 11:52 |
| Tengu | holser_: so 2 ways to do things: either ignore_errors, or do a pre-detection using "systemd" module, that will provide the state (defined, running, and so on) | 11:52 |
| Tengu | quiquell: yep | 11:52 |
| holser_ | Tengu we detect on upgrades | 11:52 |
| holser_ | on systemd level | 11:52 |
| holser_ | let me show the sample | 11:52 |
| quiquell | Tengu: ack will do | 11:53 |
| *** mrsoul has quit IRC | 11:54 | |
| *** mschuppert has quit IRC | 11:54 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: Ignore errors at xinetd stop/uninstall https://review.openstack.org/605989 | 11:55 |
| quiquell | holser_: ^ | 11:55 |
| Tengu | holser_: more over, we actually want to deactivate that service whatever is the step - the removal is optional though | 11:56 |
| *** assassin has joined #tripleo | 11:56 | |
| Tengu | holser_: this is a cleanup step, and it's already well used for now containerized services. | 11:56 |
| *** mburned_out is now known as mburned | 11:57 | |
| holser_ | well if we have error ... for instance it was not stopped | 11:57 |
| holser_ | for some reason | 11:57 |
| holser_ | playbook will continue rather than killing it for sure | 11:58 |
| Tengu | holser_: yeah, so this joins the other solution I proposed in the comment :). | 11:58 |
| Tengu | use systemd in order to detect service presence, and do whatever is needed. | 11:58 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/tripleo-ci master: provider: Add vexxhost https://review.openstack.org/596432 | 11:58 |
| Tengu | that said - xinetd is a simple service, and it's already empty - i.e. there isn't any custom service running in it, as rsync and the other one were already removed. | 11:58 |
| Tengu | but indeed, I could have done that in a more.... "now you shut the f**k up and die already" way :) | 11:59 |
| holser_ | Tengu I love your last sentence ... | 11:59 |
| holser_ | that's my point | 11:59 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: fedora28 standalone job definition https://review.openstack.org/595374 | 11:59 |
| Tengu | holser_: so, quiquell might use the systemd module, if service is present then stop/disable it with the current code, and drop the "ignore_errors". | 12:00 |
| holser_ | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/iscsid.yaml#L124-L145 | 12:00 |
| *** mmethot has joined #tripleo | 12:00 | |
| *** EmilienM is now known as EvilienM | 12:00 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: fedora28 standalone job definition https://review.openstack.org/595374 | 12:00 |
| Tengu | that way: we trigger the stop/disable IFF the service is loaded, and it fails if systemd can't kill it properly | 12:00 |
| holser_ | systemd will do the magic | 12:00 |
| Tengu | eeewwww | 12:00 |
| Tengu | don't use "command" please X( | 12:00 |
| *** mschuppert has joined #tripleo | 12:01 | |
| jaosorior | zzzeek: around? | 12:01 |
| Tengu | holser_: https://docs.ansible.com/ansible/latest/modules/systemd_module.html#systemd-module so "systemd: name: xinetd" with a register, and you should get its status. | 12:02 |
| jaosorior | zzzeek: I took a read at the latest version of the global galera spec. It looks good overall; I only left one request in the patch. If that's addressed, It's +2 from my side. | 12:02 |
| *** shyamb has quit IRC | 12:02 | |
| *** shyamb has joined #tripleo | 12:03 | |
| EvilienM | chandankumar: thanks for taking care of https://review.openstack.org/#/c/600517/ | 12:03 |
| EvilienM | chandankumar: nice work on https://review.openstack.org/#/c/605356/ | 12:03 |
| EvilienM | chandankumar: I haven't figured the issues that we saw yesterday, I couldn't reproduce and it seems to be random and environmental. We'll see. | 12:04 |
| *** jpena|lunch is now known as jpena | 12:05 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Use cap sysadmin for Neutron/OVN agents https://review.openstack.org/606045 | 12:05 |
| jaosorior | florianf: could you check this out https://review.openstack.org/#/c/602007/ ? | 12:06 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: [WIP] Enable full tempest api and scenario tests for basic services https://review.openstack.org/606046 | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 12:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 12:10 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/rocky: Don't fail tripleo-bootstrap on package installs https://review.openstack.org/606047 | 12:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/rocky: Don't fail tripleo-bootstrap on package installs https://review.openstack.org/606047 | 12:12 |
| quiquell | EvilienM: was thinking about reducing jobs time, how feasible would be to have images with local docker container on them ? | 12:12 |
| EvilienM | quiquell: we talked about it 2 days ago, it takes a very big image that infra is unlikely willing to store | 12:13 |
| openstackgerrit | Bob Fournier proposed openstack/tripleo-heat-templates master: In process-templates script write output files to provided dir when using base path https://review.openstack.org/605736 | 12:13 |
| quiquell | EvilienM: even sharing layers ? | 12:13 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/rocky: Tag step plays https://review.openstack.org/606048 | 12:14 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/rocky: Remove "when failed" from debug task names https://review.openstack.org/606049 | 12:14 |
| EvilienM | quiquell: http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2018-09-26.log.html#t2018-09-26T20:41:17 | 12:14 |
| openstackgerrit | John Fulton proposed openstack/tripleo-common stable/rocky: Update swift_rings_backup workflow to also backup ceph fetch dir https://review.openstack.org/604773 | 12:14 |
| *** dprince has joined #tripleo | 12:15 | |
| *** rh-jelabarre has joined #tripleo | 12:15 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/rocky: Remove "when failed" from debug task names https://review.openstack.org/606049 | 12:15 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates stable/rocky: Persist ceph-ansible fetch_directory using config-download https://review.openstack.org/604772 | 12:15 |
| fultonj | gfidente: do you mind voting on those two ^ ? | 12:16 |
| fultonj | (again) | 12:16 |
| fultonj | clean cherry picks | 12:16 |
| gfidente | fultonj ack done | 12:17 |
| fultonj | thanks | 12:17 |
| *** rfolco has quit IRC | 12:18 | |
| openstackgerrit | Merged openstack/tripleo-specs master: Remove the redundant word https://review.openstack.org/594799 | 12:19 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/rocky: Tag tasks in in common tasks https://review.openstack.org/606051 | 12:19 |
| florianf | jaosorior: yes, taking a look | 12:21 |
| Aelia | fultonj: ceph-ansible has finished, and I am exactly in the same state as before ... | 12:21 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: fedora28 standalone job definition https://review.openstack.org/595374 | 12:21 |
| fultonj | Aelia: lsblk | 12:21 |
| Aelia | fultonj: "vdb 252:16 0 50G 0 disk" no partition | 12:22 |
| fultonj | want to put that in a pasteing? | 12:22 |
| fultonj | pastebin | 12:22 |
| fultonj | lsblk | curl -F 'f:1=<-' ix.io | 12:22 |
| fultonj | send me output of ^ | 12:22 |
| *** weshay is now known as weshay_ruck | 12:23 | |
| Aelia | fultonj: I used a gist -> https://gist.github.com/dabelenda/2ecf1b21a90a50ef8572763374f2e0e7 | 12:23 |
| fultonj | Aelia: are you running with CephAnsibleVerbosity set to any value >0 ? | 12:24 |
| *** rlandy has joined #tripleo | 12:24 | |
| Aelia | fultonj: not overriden in my environment files | 12:26 |
| *** skramaja has quit IRC | 12:26 | |
| marios | gfidente: ping when you're ready thanks | 12:26 |
| * marios there | 12:26 | |
| jaosorior | florianf: thanks | 12:27 |
| fultonj | Aelia: https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html#override-ansible-run-options | 12:27 |
| fultonj | for next time you run it ^ | 12:27 |
| fultonj | "the ceph-ansible parameters that are passed as overrides as described in this document, are stored on the undercloud in a directory that matches the pattern /tmp/ansible-mistral-action*" | 12:27 |
| Aelia | fultonj: ah -> CephAnsiblePlaybookVerbosity: 1 | 12:28 |
| fultonj | Aelia: ok, good | 12:28 |
| fultonj | ls /tmp/ansible-mistral-action* | 12:28 |
| *** ykarel_ has joined #tripleo | 12:28 | |
| fultonj | Aelia: look in the ceph-ansible inventory and make sure the node override is passing the correct disk list | 12:29 |
| fultonj | Aelia: ensure you're looking at the latest one (ls -lhtr) | 12:29 |
| *** med_ has quit IRC | 12:31 | |
| openstackgerrit | Merged openstack/tripleo-specs master: Validation Framework specifications https://review.openstack.org/589169 | 12:31 |
| *** ykarel has quit IRC | 12:31 | |
| Aelia | fultonj: I updated the gist showing the override of devices is ok | 12:31 |
| *** ykarel_ is now known as ykarel | 12:31 | |
| Tengu | so, see you folks. Happy weekend, see you on Monday ;). | 12:32 |
| fultonj | Aelia: ceph-ansible must have done something with OSD tasks on those disks | 12:32 |
| fultonj | in the run log, you should be able to trace the tasks | 12:32 |
| fultonj | realabive to the OSD role | 12:33 |
| EvilienM | bogdando, bandini: for the After vs Wants thing, I think I can do it on a separated patch, maybe we can go ahead with https://review.openstack.org/#/c/600849/ | 12:33 |
| fultonj | relative* | 12:33 |
| Aelia | the log containing vdb for example is really short I will put it into the gist too | 12:34 |
| *** agopi is now known as agopi|brb | 12:34 | |
| *** jcoufal has joined #tripleo | 12:34 | |
| *** tzumainn has joined #tripleo | 12:34 | |
| Aelia | fultonj: done | 12:34 |
| bandini | EvilienM: I am fine with that | 12:35 |
| EvilienM | bandini: I will iterate on this code during the milestone | 12:35 |
| EvilienM | bandini: but as is, it worked for the undercloud | 12:35 |
| EvilienM | it worked (tm) | 12:35 |
| bandini | :D | 12:36 |
| fultonj | Aelia: you want to see more context around that | 12:36 |
| fultonj | which task was doing this? | 12:36 |
| *** Petersingh|afk has joined #tripleo | 12:36 | |
| fultonj | less the log and look for 14:15:34,547 | 12:36 |
| fultonj | then compare that task via the ceph-ansible code for the version of it you're using | 12:37 |
| fultonj | with the output | 12:37 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/tripleo-heat-templates stable/pike: Add a way to override base path when file driver is used https://review.openstack.org/601286 | 12:37 |
| *** raildo has joined #tripleo | 12:38 | |
| *** agopi|brb has quit IRC | 12:39 | |
| Aelia | fultonj: I changed a bit the grep command to add -B4 | 12:39 |
| Aelia | this is sufficient for in this case to show the TASK names | 12:39 |
| Aelia | fultonj: gist updates | 12:39 |
| Aelia | s/s/d/ | 12:39 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: Add a fact checking xinetd service present https://review.openstack.org/605989 | 12:41 |
| fultonj | is Sdc on your other servers? | 12:41 |
| quiquell | holser_: ^ this is it ? | 12:41 |
| fultonj | while Vdc is on the new one? | 12:41 |
| Aelia | fultonj: yes all other servers are in "Sdc" and only the new one has "Vdc" | 12:42 |
| *** trown|outtypewww is now known as trown | 12:43 | |
| fultonj | Aelia: http://paste.openstack.org/show/731095/ | 12:43 |
| *** Petersingh|afk is now known as Petersingh | 12:43 | |
| openstackgerrit | Kamil Sambor proposed openstack/python-tripleoclient master: Add fixture to replace multiple mocks https://review.openstack.org/600415 | 12:43 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Support podman when tagging container for Pacemaker https://review.openstack.org/604180 | 12:43 |
| *** artom has quit IRC | 12:44 | |
| fultonj | Aelia: http://ix.io/1nKP | 12:44 |
| fultonj | with a little cleaning and jq | 12:44 |
| Aelia | fultonj: ok, but I am not sure what you want to show me there :D | 12:45 |
| fultonj | TASK [ceph-osd : systemd start osd container | 12:45 |
| fultonj | Aelia: run the ExecStart of that on your system with the problem | 12:46 |
| fultonj | /usr/share/ceph-osd-run.sh vdb | 12:46 |
| fultonj | Aelia: you need to debug on the container itself | 12:46 |
| fultonj | I need to get back to my patch now | 12:47 |
| *** raildo has quit IRC | 12:47 | |
| fultonj | but you'll need to foucus on the container failing to do what it needs to do | 12:47 |
| Aelia | ok... but the container is starting, it fails immediately after though | 12:47 |
| fultonj | Aelia: right, find out why | 12:48 |
| fultonj | docker ps -a | 12:48 |
| fultonj | did the prepare contianer finish correctly? | 12:48 |
| Aelia | it says /dev/vdb1 does not exist. | 12:48 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: Add a fact checking xinetd service present https://review.openstack.org/605989 | 12:48 |
| fultonj | was the prepare container unable to make it? | 12:49 |
| *** shyamb has quit IRC | 12:49 | |
| fultonj | or why was it unable to | 12:49 |
| fultonj | Aelia: as per Deploy an OSD | 12:49 |
| fultonj | from https://hub.docker.com/r/ceph/daemon/ | 12:49 |
| fultonj | there's a prepare option | 12:50 |
| fultonj | run that manually to see what it's hitting | 12:50 |
| holser_ | quiquell +! | 12:50 |
| holser_ | +! | 12:50 |
| fultonj | Aelia: disable restart always in systemd too | 12:50 |
| fultonj | it will make troubleshooting harder | 12:50 |
| quiquell | holser_: testing it here https://review.openstack.org/#/c/590774 | 12:51 |
| Aelia | Error response from daemon: No such container: expose_partitions_vdb | 12:52 |
| Aelia | fultonj: I updated the gist with the complete output of /usr/share/ceph-osd-run.sh vdb | 12:53 |
| fultonj | 08:49 fultonj: Aelia: as per Deploy an OSD | 12:54 |
| fultonj | 08:49 fultonj: from https://hub.docker.com/r/ceph/daemon/ | 12:54 |
| fultonj | 08:50 fultonj: there's a prepare option | 12:54 |
| fultonj | 08:50 fultonj: run that manually to see what it's hitting | 12:54 |
| holser_ | quiquell quick question ... will Depends-On: https://review.openstack.org/#/c/605989/ work? | 12:54 |
| holser_ | I thought we need to put Change-ID | 12:54 |
| fultonj | not the container option to start the OSD, but the container option to prepare it | 12:55 |
| fultonj | it should be making that partition, it seems to have failed so you need to find out why | 12:55 |
| openstackgerrit | Daniel Alvarez proposed openstack/tripleo-heat-templates master: Configure http/https on OVN Metadata service to talk to Nova https://review.openstack.org/605406 | 12:55 |
| bcafarel | http://logs.openstack.org/75/596275/10/check/puppet-openstack-unit-4.8-centos-7/de27998/job-output.txt.gz#_2018-09-28_09_14_59_562301 cri installation failure (2.15.1 requires ruby 2.3), is that a known issue? | 12:57 |
| bcafarel | seen in https://review.openstack.org/#/c/596275/ gate issue, but quick launchpad search turns up empty | 12:58 |
| bcafarel | previous checks passed but they were installing cri 2.6.1 | 12:58 |
| jaosorior | alright folks, I'm off. Have a good weekend everyone! | 12:59 |
| marios | happy friday jaosorior | 12:59 |
| quiquell | holser_: It's better the full url, you can have more than one review with the same Change-Id | 13:00 |
| *** jaosorior has quit IRC | 13:00 | |
| holser_ | good to know... thanks a lot | 13:00 |
| *** raildo has joined #tripleo | 13:00 | |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient master: Filter messages not from waiting execution https://review.openstack.org/605520 | 13:01 |
| holser_ | indeed, sometimes we have same review but different branches | 13:01 |
| holser_ | same change-id... | 13:01 |
| slagle | thrash: fyi, https://review.openstack.org/#/c/605520/ it seems to work now. but I had to fix ~100 tests | 13:01 |
| slagle | thrash: i'm not sure if that's good or bad :) | 13:01 |
| slagle | cuz all the tests assume all messages are going through, with no matching on the execution id | 13:02 |
| quiquell | holser_: more than welcome, we are supose to us it now, as infra guys suggets | 13:02 |
| *** raildo_ has joined #tripleo | 13:03 | |
| *** boazel has quit IRC | 13:04 | |
| fultonj | running standalone-edge.sh I hit this http://paste.openstack.org/show/731098 | 13:04 |
| *** raildo has quit IRC | 13:05 | |
| EvilienM | quiquell: I guess if we wanted to do it, we would have to run the jobs in our cloud, to store the image in our side | 13:05 |
| EvilienM | quiquell: but again even with the container layers, it'll take a lot of space | 13:05 |
| EvilienM | quiquell: have you deployed an undercloud with a local registry? go do it and tell me how many GB container takes | 13:06 |
| quiquell | EvilienM: matbu is my here :-) | 13:06 |
| quiquell | s/here/hero/ | 13:06 |
| weshay_ruck | lol | 13:06 |
| *** rfolco has joined #tripleo | 13:06 | |
| quiquell | EvilienM: Will check | 13:07 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: In process-templates script write output files to provided dir when using base path https://review.openstack.org/605736 | 13:07 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Merge new params - nic-config templates https://review.openstack.org/605807 | 13:07 |
| *** raildo_ has quit IRC | 13:08 | |
| *** raildo has joined #tripleo | 13:08 | |
| quiquell | EvilienM: thanks for the irc snippet is gold | 13:08 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 13:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 13:10 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Pass execution_id to tripleo.ansible-playbook. https://review.openstack.org/606064 | 13:11 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Fail multiple executions of config-download of the same plan https://review.openstack.org/606065 | 13:11 |
| *** assassin has left #tripleo | 13:12 | |
| *** psachin has quit IRC | 13:13 | |
| *** agopi|brb has joined #tripleo | 13:14 | |
| *** agopi|brb is now known as agopi|afk | 13:14 | |
| *** chem has quit IRC | 13:15 | |
| *** chem has joined #tripleo | 13:15 | |
| Aelia | fultonj: I managed to bootstrap correctly the OSD if I try to execute manually: /usr/bin/docker run -it --rm --net=host --privileged=true --pid=host --memory=3g --cpu-quota=100000 -v /dev:/dev -v /etc/localtime:/etc/localtime:ro -v /var/lib/ceph:/var/lib/ceph -v /etc/ceph:/etc/ceph -e OSD_TYPE=prepare -e OSD_FILESTORE=1 -e OSD_DMCRYPT=0 -e CLUSTER=ceph -e OSD_DEVICE=/dev/vdc -e | 13:16 |
| *** zul has quit IRC | 13:16 | |
| Aelia | CEPH_DAEMON=OSD_CEPH_DISK_PREPARE --name=ceph-osd-overcloud-cephstorage-2-vdc docker.io/ceph/daemon:v3.0.7-stable-3.0-jewel-centos-7-x86_64 | 13:16 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: WIP: extra volumes map https://review.openstack.org/602721 | 13:16 |
| Aelia | fultonj: and after that the container runs correctly with systemctl start $service | 13:16 |
| fultonj | Aelia: nice | 13:17 |
| *** zul has joined #tripleo | 13:17 | |
| dalvarez | beagles: mwhahaha can you guys please give some love to https://review.openstack.org/#/c/568858/ ? | 13:17 |
| dalvarez | thanks a lot | 13:17 |
| fultonj | Aelia: so it's good you're up and running, i assume you can apply same process to other OSDs | 13:18 |
| fultonj | i wonder why when ansible presumably did the same it didn't work out | 13:18 |
| Aelia | fultonj: something is weird -> # ceph status -> 2018-09-28 13:17:59.019785 7f52f4cdb700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory | 13:18 |
| Aelia | on the new ceph node, where as it works on older ceph nodes. | 13:18 |
| fultonj | there's a ceph-ansible option to copy keys | 13:19 |
| fultonj | might have had a default change | 13:20 |
| fultonj | you can put keys there if you need to run 'ceph -s' on the osd | 13:20 |
| fultonj | i normally only run that command from ceph mon | 13:20 |
| Aelia | ok | 13:20 |
| *** ramishra has quit IRC | 13:22 | |
| *** amoralej is now known as amoralej|lunch | 13:23 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Add standalone upgrade role and playbook. https://review.openstack.org/604736 | 13:24 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: New workflow for standalone upgrade. https://review.openstack.org/604706 | 13:25 |
| *** derekh has quit IRC | 13:26 | |
| *** derekh has joined #tripleo | 13:26 | |
| *** arxcruz|doctor is now known as arxcruz | 13:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack/ansible-role-container-registry master: Remove release-openstack-server https://review.openstack.org/606073 | 13:27 |
| openstackgerrit | Andreas Jaeger proposed openstack/ansible-role-redhat-subscription master: Remove release-openstack-server https://review.openstack.org/606074 | 13:28 |
| *** artom has joined #tripleo | 13:29 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: New workflow for standalone upgrade. https://review.openstack.org/604706 | 13:29 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: DNM: To test job variables https://review.openstack.org/606020 | 13:29 |
| openstackgerrit | Andreas Jaeger proposed openstack/ansible-role-tripleo-cookiecutter master: Remove release-openstack-server https://review.openstack.org/606075 | 13:30 |
| *** toure|gone is now known as toure | 13:30 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Switch Heat Launcher to use Podman instead of Docker when containerized https://review.openstack.org/606077 | 13:31 |
| openstackgerrit | Andreas Jaeger proposed openstack/ansible-role-tripleo-modify-image master: Remove release-openstack-server https://review.openstack.org/606079 | 13:31 |
| *** holser_ has quit IRC | 13:42 | |
| *** zzzeek has quit IRC | 13:43 | |
| *** artom has quit IRC | 13:43 | |
| *** artom has joined #tripleo | 13:44 | |
| *** zzzeek has joined #tripleo | 13:45 | |
| *** mcornea has joined #tripleo | 13:47 | |
| *** zzzeek has quit IRC | 13:48 | |
| *** amoralej|lunch is now known as amoralej | 13:49 | |
| fultonj | EvilienM: before you ran standalone-edge.sh did you preconfigure the IP on your host or let TripleO do it for you? ; it's not doing it for me, so I think i need to preconfigure it | 13:49 |
| *** zzzeek has joined #tripleo | 13:49 | |
| EvilienM | fultonj: I didn't configure networking | 13:49 |
| fultonj | EvilienM: ok, so export IP=192.168.0.12 was configured by tripleo, thanks | 13:50 |
| fultonj | i need to figure out why it's not happening forme | 13:50 |
| *** artom has quit IRC | 13:52 | |
| *** Vorrtex has joined #tripleo | 13:52 | |
| EvilienM | fultonj: wait, no I think the IP was configured when I deployed sorry | 13:52 |
| fultonj | EvilienM: np, i have snapshots :) | 13:53 |
| fultonj | thanks i'll go do that | 13:53 |
| *** agopi|afk is now known as agopi | 13:53 | |
| weshay_ruck | marios, fyi.. the gate failures seem to be less timeouts and more failures | 13:54 |
| weshay_ruck | http://logs.openstack.org/57/595357/2/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/ebbc2f9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 13:54 |
| weshay_ruck | based on that nodepool nodes | 13:54 |
| marios | weshay_ruck: ack theres a mix. tried digging into some of the timeouts earlier but didn't find something its all during overcloud deploy afaics | 13:55 |
| marios | weshay_ruck: looking | 13:55 |
| marios | weshay_ruck: yesi was looking at this http://logs.openstack.org/24/567224/111/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/58802fa/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz same issue looke like | 13:56 |
| thrash | slagle: yeah... Not sure what to say about that one. lol | 13:57 |
| *** ade_lee has joined #tripleo | 14:00 | |
| *** artom has joined #tripleo | 14:01 | |
| mwhahaha | weshay_ruck, marios: that would indicate ceph-ansible problems or something | 14:02 |
| mwhahaha | weshay_ruck, marios: cause it's happening on scenario001/004. pretty sure that's screwed the whole gate | 14:03 |
| mwhahaha | EvilienM: -^ fyi | 14:03 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Use subnodes groups for multinode roles and templates https://review.openstack.org/606087 | 14:03 |
| marios | mwhahaha: ack yeah i recall the workflowtask are/were used for the ceph-ansible calls | 14:03 |
| marios | mwhahaha: weshay_ruck i am finishing up some status and will try dig in a bit more there | 14:03 |
| * fultonj looking at logs from ^ | 14:03 | |
| fultonj | http://logs.openstack.org/24/567224/111/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/58802fa/logs/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz#_2018-09-28_10_36_33_265 | 14:04 |
| slagle | thrash: the API is difficult to use. as a consumer, what knowledge do I have that when I make an API call I need to open a zaqar websocket and start acting on messages? | 14:05 |
| fultonj | gfidente: ^ fyi | 14:05 |
| slagle | thrash: and likewise start ignoring messages not from the API call I made? | 14:05 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/pike: [Pike only] Pass DeployIdentifier in upgrade tasks. https://review.openstack.org/606089 | 14:05 |
| *** Petersingh is now known as Petersingh|away | 14:06 | |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-heat-templates master: WIP - Telemetry Framework https://review.openstack.org/605724 | 14:06 |
| gfidente | fultonj yeah I think there could be something wrong in the config_action module but it seems to be there unchanged from april | 14:06 |
| *** Petersingh|away has quit IRC | 14:06 | |
| mwhahaha | did we get a new version of ansible in rdo? | 14:06 |
| fultonj | ansible-2.4.4 | 14:06 |
| fultonj | http://logs.openstack.org/24/567224/111/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/58802fa/logs/undercloud/var/log/yum.log.txt.gz | 14:06 |
| mwhahaha | uh | 14:07 |
| mwhahaha | that's old | 14:07 |
| mwhahaha | wtf | 14:07 |
| gfidente | mwhahaha yeah was thinking about some diffs in ansible itself | 14:07 |
| mwhahaha | oh that's queens | 14:07 |
| mwhahaha | that job is for queens | 14:07 |
| gfidente | mwhahaha still queens should be using 2.5 | 14:07 |
| mwhahaha | no i'm pretty sure we were on 2.4 in queens | 14:08 |
| gfidente | mwhahaha right but I mean, we should be using 2.5 | 14:08 |
| mwhahaha | not necessarily | 14:08 |
| cgoncalves | can we recheck changes for which verification failed for unit tests? | 14:08 |
| gfidente | or I'll settle for 2.6 then | 14:08 |
| mwhahaha | so here's a previous successful run, http://logs.openstack.org/24/567224/110/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/264c07a/logs/undercloud/var/log/extra/rpm-list.txt.gz | 14:09 |
| mwhahaha | which was on 2.4.4 | 14:09 |
| mwhahaha | so what changed | 14:09 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 14:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 14:10 |
| mwhahaha | same version of ceph-ansible | 14:10 |
| mwhahaha | weird | 14:10 |
| *** dxiri has quit IRC | 14:10 | |
| mwhahaha | it was likely a backport in tripleo-common | 14:12 |
| thrash | slagle: that's fair. Probably need some docs first of all. | 14:12 |
| fultonj | FWIW i have a queens undercloud deployed by oooq with ceph-ansible working ceph-ansible-3.1.0.0-0.rc21 using ansible 2.4.4 | 14:13 |
| bogdando | is it a known blocker From puppet-openstack-unit-4.8-centos-7: | 14:14 |
| bogdando | 2018-09-27 20:49:15.971220 | centos-7 | Gem::InstallError: cri requires Ruby version ~> 2.3. | 14:14 |
| bogdando | 2018-09-27 20:49:15.971352 | centos-7 | An error occurred while installing cri (2.15.1), and Bundler cannot continue. | 14:14 |
| bogdando | 2018-09-27 20:49:15.971420 | centos-7 | Make sure that `gem install cri -v '2.15.1' --source 'https://rubygems.org/'` | 14:14 |
| bogdando | 2018-09-27 20:49:15.971450 | centos-7 | succeeds before bundling. ? | 14:14 |
| slagle | thrash: i was thinking if we should go back to not using a global queue. at least it would eliminate this current issue. I'm not sure about UI implications though | 14:14 |
| mwhahaha | i wonder if the mistral container has a different version of ansible | 14:14 |
| bogdando | https://review.openstack.org/#/c/596275/ has it for a while | 14:14 |
| slagle | thrash: in reality though, the UI shouldn't have to be relying on other API callers all having used the same queue | 14:14 |
| fultonj | mwhahaha: good thought, but i thought we said this was a queens job | 14:14 |
| slagle | thrash: the needed state/info should be within the API responses themselves | 14:15 |
| fultonj | this is the master gate | 14:15 |
| fultonj | ? | 14:15 |
| mwhahaha | fultonj: it is, but i'm not sure how this changed in the last 2 days | 14:15 |
| mwhahaha | fultonj: the logs i'm looking at are for a test ci job for queens | 14:15 |
| mwhahaha | https://review.openstack.org/#/c/567224/111 | 14:15 |
| mwhahaha | it broke sometime on the 26th | 14:16 |
| *** chem has quit IRC | 14:16 | |
| thrash | slagle: There is no requirement, really. The CLI could pass whatever queue name it wants. | 14:16 |
| *** chem has joined #tripleo | 14:16 | |
| mwhahaha | fultonj: and ansible/ceph-ansible are the same versions on the host so it makes me think a container thing maybe | 14:16 |
| thrash | slagle: but I agree... We shouldn't be relying on zaqar for the response. It should be in the output of the workflow itself. | 14:16 |
| cgoncalves | answering to my own question: no. pending merge of https://review.openstack.org/#/c/605350/ | 14:17 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/puppet-tripleo master: Fix wrapper containers for podman w/o sockets https://review.openstack.org/606095 | 14:17 |
| mwhahaha | cgoncalves: yea puppet unit tests are still screwed | 14:17 |
| fultonj | how do we look inside the mistral_executor container for that job then? | 14:18 |
| mwhahaha | fultonj: i don't think we can, you could manually pull down the container and look ig uess | 14:19 |
| mwhahaha | we don't capture the contents of the containers | 14:19 |
| cgoncalves | mwhahaha, the depends-on of ^ merged, but ^ is not queued at CI. recheck? | 14:19 |
| fultonj | do we know the contianer version that's running on the UC? | 14:20 |
| *** bnemec is now known as beekneemech | 14:20 | |
| mwhahaha | cgoncalves: ykarel cherry-picked the depends on which blocks it since those haven't merged | 14:20 |
| mwhahaha | fultonj: oh this is queens, it's not containerized | 14:20 |
| cgoncalves | noooo :/ | 14:20 |
| fultonj | that's what was bending my mind | 14:20 |
| fultonj | i figured i must not understand something | 14:20 |
| *** bogdando has quit IRC | 14:20 | |
| *** holser_ has joined #tripleo | 14:21 | |
| therve | slagle: What's the issue exactly? | 14:21 |
| * mwhahaha sighs | 14:22 | |
| mwhahaha | too many problems | 14:22 |
| mwhahaha | i think we need to purge the gate and let ci settle, it's far too delayed | 14:22 |
| gfidente | that original_basename seems to have transformed into _original_basename | 14:22 |
| slagle | therve: https://bugs.launchpad.net/tripleo/+bug/1794277 | 14:22 |
| openstack | Launchpad bug 1794277 in tripleo "openstack overcloud failures|status sometimes shows incorrect output ( from deployment process) " [Medium,In progress] - Assigned to James Slagle (james-slagle) | 14:22 |
| gfidente | https://github.com/ceph/ceph-ansible/blob/v3.1.6/plugins/actions/_v2_config_template.py#L638 | 14:23 |
| slagle | therve: we can't be adding new functionality to tripleoclient that make workflow calls that are intended to be used as other ongoing workflows | 14:23 |
| cgoncalves | post queue has an astonishing 444 jobs pending | 14:23 |
| slagle | therve: due to the existing model we have where everything uses a single global "tripleo" zaqar queue | 14:23 |
| therve | slagle: Right. I think your filtering idea is good for that no? | 14:23 |
| slagle | therve: we could override that and go back to have each worfklow call generate a new queue with unique uuid. | 14:24 |
| therve | Oh I didn't know that was the original design | 14:24 |
| slagle | therve: yea i think my fix is ok for tripleclient | 14:24 |
| fultonj | gfidente: ceph-ansible-3.1.6-1.el7.noarch against queens? | 14:24 |
| fultonj | yeah i gues that makes sense | 14:24 |
| slagle | therve: what i was pontificating about is more about the usablity of the API in general | 14:24 |
| gfidente | fultonj yes that is correct | 14:24 |
| slagle | therve: how is any consumer supposed to know how to use this outsdie of tripleoclient | 14:25 |
| slagle | and perhaps they aren't | 14:25 |
| therve | slagle: Well, it could be documented | 14:25 |
| slagle | therve: as a user, i make an API call by starting a workflow. now what? | 14:25 |
| slagle | i'd be lost | 14:25 |
| slagle | so that's why i feel odd about fixing this in tripleoclient | 14:26 |
| therve | OK I see | 14:26 |
| slagle | therve: and my patch breaks assumptions in about 100 unit tests :) | 14:26 |
| slagle | so was wondering if I had done the right thing or not :) | 14:26 |
| mwhahaha | weshay_ruck, EvilienM: i'm going to purge the gate unless there are any objections. are there any patches i should leave in? | 14:26 |
| slagle | i also patched the code so that it would ignore the dummy ID value that unit tests use | 14:26 |
| slagle | figured i'd get a -1 for that though :) | 14:27 |
| EvilienM | mwhahaha: I'm fine | 14:27 |
| slagle | *almost | 14:27 |
| therve | Not sure it broke assumptions or assumptions weren't right in the first place | 14:27 |
| slagle | therve: yes, exactly | 14:27 |
| *** vkapalav has joined #tripleo | 14:28 | |
| therve | slagle: And why did we switch to a single queue? | 14:28 |
| gfidente | so there must be a problem with the ansible version because up to 2.5 the param was original_basename and from 2.6 it became _original_basename | 14:28 |
| weshay_ruck | mwhahaha, oh snap you never do that | 14:28 |
| gfidente | somehow the version of ansible in the pike and queens jobs is bumped to include that change | 14:28 |
| slagle | therve: that's what i'm after as well. will need to check with jtomasek probably to see if it was a UI requirement | 14:28 |
| weshay_ruck | mwhahaha, marios what patches are you clearing it for? I don't have any one my list | 14:28 |
| weshay_ruck | marios, do you? | 14:28 |
| marios | weshay_ruck: what are we clearing? /me readsback (but no i don't have a list of patches :) ) | 14:29 |
| mwhahaha | marios: the gate, we need to understand the current state of things and land patches that'll stop the resets in the gate | 14:29 |
| *** agopi is now known as agopi|afk | 14:29 | |
| marios | mwhahaha: ack i am just coming back here but was there a solution to that scen4 worfklow tasks thing | 14:30 |
| mwhahaha | marios: is that affecting master? or just queens? | 14:30 |
| therve | slagle: We could cheat and post to 2 queues as well | 14:30 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: WIP: read job variables at deploy playbooks https://review.openstack.org/606017 | 14:30 |
| mwhahaha | seems like it's also affecting pike | 14:31 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: DNM: To test job variables https://review.openstack.org/606020 | 14:31 |
| slagle | therve: heh, yea, that may very well work | 14:31 |
| *** quiquell is now known as quiquell|off | 14:31 | |
| gfidente | marios I think it's the version of ansible on the nodes | 14:31 |
| gfidente | making scenario4 failing in queens | 14:31 |
| marios | mwhahaha: queens examples so far | 14:31 |
| *** mjturek has joined #tripleo | 14:32 | |
| marios | http://logs.openstack.org/24/567224/111/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/58802fa/logs/undercloud/var/log/extra/errors.txt.gz and http://logs.openstack.org/57/595357/2/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/ebbc2f9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz but let me also dig for master | 14:32 |
| mwhahaha | marios: i just saw a pike patch fail on scenario001/004 | 14:32 |
| *** cmurphy has left #tripleo | 14:32 | |
| marios | https://review.openstack.org/#/c/603275/6 green here mwhahaha | 14:32 |
| mwhahaha | marios: https://review.openstack.org/#/c/604708/ | 14:33 |
| marios | ack gfidente do we have a bug yet? weshay_ruck i'll file it if we dont | 14:34 |
| mwhahaha | http://logs.openstack.org/08/604708/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e0c9113/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-28_08_26_31 | 14:34 |
| gfidente | marios don't have a bug yet but I am pretty sure about the root cause | 14:34 |
| gfidente | marios in 2.5 the copy module accepted original_basename parameter https://github.com/ansible/ansible/blob/stable-2.5/lib/ansible/modules/files/copy.py#L272 | 14:34 |
| gfidente | marios in 2.6 it doesn't anymore https://github.com/ansible/ansible/blob/stable-2.6/lib/ansible/modules/files/copy.py#L286 | 14:34 |
| marios | ack mwhahaha so Q/P | 14:35 |
| gfidente | mwhahaha fultonj ^^ | 14:35 |
| marios | gfidente: does it make sense same root for P too? | 14:35 |
| mwhahaha | marios: i wonder if quickstart is installing a newer version? | 14:35 |
| gfidente | marios for pike haven't checked, will do | 14:35 |
| gfidente | so pike is showing a different problem | 14:36 |
| gfidente | http://logs.openstack.org/48/602248/4/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/52a00ae/logs/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz#_2018-09-28_11_46_31_847 | 14:36 |
| gfidente | but I think same root cause | 14:36 |
| *** dxiri has joined #tripleo | 14:36 | |
| gfidente | in pike we have slighly different workflow and ceph-ansible version | 14:37 |
| weshay_ruck | http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?panelId=61&fullscreen&orgId=1 | 14:37 |
| gfidente | though nothing changed recently in either | 14:37 |
| gfidente | in both | 14:37 |
| gfidente | in either? | 14:38 |
| weshay_ruck | mwhahaha, that is a lot canned air man.. there be dust bunny dragons? | 14:38 |
| gfidente | in neither? | 14:38 |
| gfidente | whatever | 14:39 |
| *** artom has quit IRC | 14:39 | |
| mwhahaha | weshay_ruck: cats | 14:39 |
| marios | https://bugs.launchpad.net/tripleo/+bug/1795009 weshay_ruck gfidente fyi | 14:40 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,Triaged] - Assigned to Giulio Fidente (gfidente) | 14:40 |
| marios | gfidente: its yours btw congrats | 14:40 |
| marios | you sound like you know what you're doing | 14:40 |
| gfidente | marios and why did you assign it to me! | 14:40 |
| gfidente | WTF | 14:40 |
| marios | :) | 14:40 |
| gfidente | I know why it's failing | 14:40 |
| gfidente | not how to fix it | 14:41 |
| marios | gfidente: ack re-assigning | 14:41 |
| marios | gfidente: can you please add a comment there? | 14:41 |
| gfidente | marios yes sorry with link | 14:41 |
| *** ksambor has quit IRC | 14:43 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Allow to actually disable heat-native https://review.openstack.org/606100 | 14:43 |
| mwhahaha | marios, gfidente: is the ansible installed via quickstart accidently upgrading ansible on the undercloud or something? | 14:44 |
| gfidente | marios so trying to be serious, there seems to be something updating further ansible at some point | 14:44 |
| gfidente | yeah what mwhahaha said | 14:44 |
| *** DirectorN00b has joined #tripleo | 14:45 | |
| DirectorN00b | omg, i'm so happy I found this channel :-) | 14:45 |
| gfidente | we're so happy to see you | 14:46 |
| *** gfidente is now known as gfidenteN00b | 14:46 | |
| DirectorN00b | lol | 14:46 |
| marios | mwhahaha: well the ansible version on the failing pike seems to be still 2.4 afaics ? http://logs.openstack.org/08/604708/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e0c9113/logs/undercloud/var/log/yum.log.txt.gz | 14:47 |
| DirectorN00b | Any of you guys use the "plan" aspect of director/triopleo? I am trying to figure out a few things. Even basic things like can I use multiple plans against my infrastructure to test out different template/env configs? | 14:47 |
| mwhahaha | marios: yea but doesn't quickstart pip install ansible? | 14:47 |
| mwhahaha | marios: http://logs.openstack.org/08/604708/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e0c9113/logs/undercloud/var/log/extra/pip.txt.gz | 14:47 |
| mwhahaha | marios: ansible 2.6.4 is pip installed | 14:48 |
| mwhahaha | globally | 14:48 |
| mwhahaha | so did we break the venv in quickstart? | 14:48 |
| weshay_ruck | mwhahaha, qs should be doing it in a virtenv | 14:48 |
| mwhahaha | weshay_ruck: yes it *should* be :D but things never do what we think they do | 14:48 |
| marios | mwhahaha: ack i see the 2.6 indeed | 14:48 |
| marios | mwhahaha: looking for any recent commit (but why queens only not master that is strange) | 14:49 |
| mwhahaha | marios: it's likely a quickstart or quickstart-extras change in master that would affect this | 14:49 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-common master: Add wrapper for openshift-ansible docker command https://review.openstack.org/605399 | 14:49 |
| mwhahaha | marios: actually i wonder if it's coming from the image | 14:50 |
| mwhahaha | marios: because quickstart is installing 2.5.x | 14:50 |
| mwhahaha | i think we need to pip remove ansible | 14:52 |
| mwhahaha | in prep | 14:52 |
| DirectorN00b | Also, is rhel's "Director" just a red wrapper for tripleo? OR is is a fork? Or... | 14:52 |
| mwhahaha | marios, gfidenteN00b: http://logs.openstack.org/24/567224/109/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8b476ee/logs/undercloud/var/log/extra/pip.txt.gz | 14:53 |
| mwhahaha | so we used to have 2.4.4 | 14:53 |
| mwhahaha | so something is upgrading it | 14:53 |
| marios | mwhahaha: or pin version in requirements and let pip so its thing | 14:53 |
| mwhahaha | we shouldn't be using the pip version | 14:53 |
| marios | mwhahaha: oh i see | 14:53 |
| mwhahaha | so something is pip installing ansible on the image | 14:54 |
| DirectorN00b | I'm trying to figure if it's behind and some bugs are still lingering, or whether it's me doing something. | 14:54 |
| DirectorN00b | I think this might be to blame, but not sure... https://review.openstack.org/#/c/530225/ | 14:54 |
| weshay_ruck | http://logs.openstack.org/08/604708/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e0c9113/job-output.txt.gz#_2018-09-28_06_41_40_658878 | 14:54 |
| mwhahaha | weshay_ruck: so i don't think it's quickstart because quickstart is install 2.5.7 | 14:55 |
| weshay_ruck | yup yup.. | 14:55 |
| weshay_ruck | mwhahaha, it could be infra's ansible? | 14:55 |
| mwhahaha | yes | 14:56 |
| mwhahaha | or even the images themselves | 14:56 |
| marios | mwhahaha: weshay_ruck ack so we are already pinning in requrements | 14:56 |
| mwhahaha | since we're getting a version newer than we're expecting anywhere, i'd probably check the images first | 14:56 |
| weshay_ruck | marios, yes.. we always have | 14:57 |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-heat-templates master: WIP - Telemetry Framework https://review.openstack.org/605724 | 14:57 |
| weshay_ruck | mwhahaha, what about updating the undercloud deployment ansible.cg | 14:58 |
| weshay_ruck | cfg | 14:58 |
| weshay_ruck | to use a specific known install of ansible | 14:58 |
| mwhahaha | to do what? | 14:58 |
| weshay_ruck | like the overcloud | 14:58 |
| openstackgerrit | Udi Kalifon proposed openstack/tempest-tripleo-ui master: Selenium infra https://review.openstack.org/605424 | 14:58 |
| mwhahaha | we need to figure out where this newer version is coming from | 14:58 |
| marios | weshay_ruck: wondering where we can land some band-aid for now to pip remove it? like in https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/run-v3.yaml | 14:58 |
| mwhahaha | marios: yes that would be a stop gap | 14:59 |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Add wrapper for openshift-ansible docker command https://review.openstack.org/605399 | 14:59 |
| weshay_ruck | mwhahaha, this is kind of a nightmare | 14:59 |
| weshay_ruck | we also have osp shipping a different version of ansible | 14:59 |
| mwhahaha | yes i am aware | 14:59 |
| * weshay_ruck wonders if this is another reason to request a tripleo-centos image | 15:00 | |
| openstackgerrit | David Vallee Delisle proposed openstack/tripleo-heat-templates master: Validate that a detected ceph-disk is member of a cluster before considering that we need ceph-osd package https://review.openstack.org/606105 | 15:00 |
| mwhahaha | happy friday | 15:00 |
| gfidenteN00b | weshay_ruck funny thing ceph-ansible did wnt to upgrade ansible! | 15:01 |
| gfidenteN00b | weshay_ruck so we're serving great testing here | 15:01 |
| weshay_ruck | and then that | 15:01 |
| weshay_ruck | if only there was a company that could package linux binaries into a useful format | 15:01 |
| weshay_ruck | and sell support for that | 15:01 |
| weshay_ruck | some kind of package management | 15:02 |
| *** Vorrtex has quit IRC | 15:02 | |
| * weshay_ruck installs an ansible flatpak | 15:02 | |
| gfidenteN00b | it's a bit like running oooq in a container and install in the container image whatever version of ansible you want | 15:02 |
| gfidenteN00b | but honestly | 15:02 |
| gfidenteN00b | this is terrible on the ansible side | 15:02 |
| weshay_ruck | mwhahaha, marios if we start uninstalling infra's version of ansible then some infra tasks in post could fail | 15:03 |
| gfidenteN00b | breaking compatibility at will | 15:03 |
| gfidenteN00b | every minor update | 15:03 |
| weshay_ruck | mwhahaha, marios seems like this may require.. hrm... so kind of coordination | 15:03 |
| mwhahaha | weshay_ruck: they shouldn't be running ansible on the host itself, they run it from zuul | 15:03 |
| mwhahaha | weshay_ruck: so i don't think that's an issue | 15:04 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Allow to actually disable heat-native https://review.openstack.org/606100 | 15:05 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Use local ansible connection for libvirt repro https://review.openstack.org/605013 | 15:09 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Don't merge /etc/collectd.d https://review.openstack.org/603123 | 15:09 |
| *** artom has joined #tripleo | 15:09 | |
| openstackgerrit | Russell Bryant proposed openstack/tripleo-docs master: Update standalone doc title. https://review.openstack.org/606109 | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 15:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795009 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 15:10 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 15:10 |
| *** Vorrtex has joined #tripleo | 15:11 | |
| openstackgerrit | Daniel Alvarez proposed openstack/tripleo-heat-templates master: Configure http/https on OVN Metadata service to talk to Nova https://review.openstack.org/605406 | 15:12 |
| *** bugzy has joined #tripleo | 15:14 | |
| *** mwhahaha changes topic to "Welcome to Rocky | CI Status: RED, DO NOT WORKFLOW OR RECHECK (unless explicitly for CI fixing) https://docs.openstack.org/tripleo-docs/latest/" | 15:15 | |
| *** chem has quit IRC | 15:16 | |
| *** chem has joined #tripleo | 15:16 | |
| *** bugzy_ has quit IRC | 15:17 | |
| *** dtrainor has quit IRC | 15:21 | |
| *** dtrainor has joined #tripleo | 15:21 | |
| *** iranzo has quit IRC | 15:23 | |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: WIP: test remove pip ansible as workaround for scen1/4 https://review.openstack.org/606116 | 15:23 |
| marios | weshay_ruck: incase you want to do that and cos i'm almost eod here | 15:24 |
| marios | ^^^ | 15:24 |
| marios | but not sure if that is too early | 15:24 |
| marios | i.e. do we need to do that in toci_gate_test/quickstart? | 15:24 |
| marios | not sure where/when the pip install is happening | 15:24 |
| *** Vorrtex has quit IRC | 15:27 | |
| *** Vorrtex has joined #tripleo | 15:28 | |
| *** leanderthal has quit IRC | 15:28 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Allow to actually disable heat-native https://review.openstack.org/606100 | 15:29 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/queens: WIP testing the depends-on for +bug/1795009 workaround https://review.openstack.org/606118 | 15:32 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Switch Heat Launcher to use Podman instead of Docker when containerized https://review.openstack.org/606077 | 15:32 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Fix quickstart undercloud selinux configuration https://review.openstack.org/602703 | 15:32 |
| *** ykarel is now known as ykarel|away | 15:35 | |
| *** AJaeger has joined #tripleo | 15:36 | |
| *** chem has quit IRC | 15:37 | |
| AJaeger | https://review.openstack.org/#/q/topic:update-zuul+projects:openstack/ansible-role are some reviews for a few ansible-role repos that have wrong Zuul setup, they use by error a release job in-repo. That one should be in project-config. Could you put them on your review list and merge once you unfreeze, please? | 15:37 |
| *** chem has joined #tripleo | 15:39 | |
| *** zul has quit IRC | 15:41 | |
| *** boazel has joined #tripleo | 15:44 | |
| *** dxiri has quit IRC | 15:46 | |
| *** holser_ has quit IRC | 15:52 | |
| *** jfrancoa has quit IRC | 15:55 | |
| weshay_ruck | marios, mwhahaha ansible is built into the centos image in the python module path | 15:56 |
| weshay_ruck | 9/20 centos image has __version__ = '2.4.2.0' | 15:57 |
| * weshay_ruck will download the latest and see but cetainly it's at 2.6 | 15:57 | |
| EvilienM | 2.4.2.0 ? | 15:57 |
| EvilienM | mhh too old | 15:58 |
| marios | weshay_ruck: tanks | 15:58 |
| dtantsur | hi folks! can someone please check this backport? https://review.openstack.org/#/c/601613/ | 15:58 |
| marios | thanks even weshay_ruck | 15:58 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 16:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 16:00 |
| weshay_ruck | mwhahaha, I thought you were going to kill the queue | 16:00 |
| mwhahaha | i did except for the stuff that would be useful in ci | 16:04 |
| mwhahaha | let me check what's left | 16:04 |
| mwhahaha | i might have missed something | 16:04 |
| mwhahaha | yea some stuff has snuck in afterwards | 16:05 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: Add 2h timeout when waiting for websocket messages on package_update https://review.openstack.org/604325 | 16:05 |
| * mwhahaha slaps EvilienM's hand for approving stuff | 16:05 | |
| EvilienM | ah | 16:06 |
| weshay_ruck | ha ha | 16:06 |
| EvilienM | that's what happens when I try to be nice | 16:06 |
| weshay_ruck | he's such a yes man | 16:06 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Allow to actually disable heat-native https://review.openstack.org/606100 | 16:07 |
| * mwhahaha slaps gfidenteN00b's hand for approving stuff | 16:07 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795009 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 16:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 16:10 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 16:10 |
| weshay_ruck | mwhahaha, ignore me if you are in a mtg... which bug is address the ansible version, 1795009? | 16:12 |
| gfidenteN00b | mwhahaha that's bribe | 16:13 |
| gfidenteN00b | mwhahaha not just approving stuff | 16:13 |
| weshay_ruck | NOOOB | 16:14 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Remove artificial constrains around notification drivers https://review.openstack.org/606126 | 16:14 |
| *** sanjayu_ has quit IRC | 16:15 | |
| mwhahaha | gfidenteN00b: https://bugs.launchpad.net/tripleo/+bug/1795009 | 16:16 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 16:16 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/tripleo-ci master: provider: Add vexxhost https://review.openstack.org/596432 | 16:17 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/tripleo-ci master: provider: Add vexxhost https://review.openstack.org/596432 | 16:25 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: New workflow for standalone upgrade. https://review.openstack.org/604706 | 16:25 |
| *** zul has joined #tripleo | 16:27 | |
| *** ykarel_ has joined #tripleo | 16:29 | |
| *** akrivoka has quit IRC | 16:29 | |
| *** shyamb has joined #tripleo | 16:30 | |
| *** ykarel|away has quit IRC | 16:30 | |
| *** ykarel__ has joined #tripleo | 16:31 | |
| *** ykarel_ has quit IRC | 16:31 | |
| chandankumar | EvilienM: sorry podman tempest need some more changes | 16:32 |
| EvilienM | chandankumar: no prob | 16:32 |
| openstackgerrit | Ben Nemec proposed openstack/os-collect-config master: Don't ignore SIGPIPE https://review.openstack.org/606133 | 16:34 |
| chandankumar | mwhahaha: EvilienM: https://review.openstack.org/#/c/605980/ | 16:34 |
| chandankumar | some fixes related to selinux part | 16:35 |
| *** shyamb has quit IRC | 16:35 | |
| EvilienM | chandankumar: lgtm | 16:35 |
| *** shyamb has joined #tripleo | 16:35 | |
| openstackgerrit | Russell Bryant proposed openstack/python-tripleoclient master: Fix misspelling in deployment complete message. https://review.openstack.org/606134 | 16:36 |
| *** dxiri has joined #tripleo | 16:36 | |
| chandankumar | mwhahaha: EvilienM http://logs.openstack.org/46/606046/1/check/tripleo-ci-centos-7-standalone/968ace6/logs/undercloud/home/zuul/tempest.log.txt.gz | 16:38 |
| chandankumar | only 18 failed tests on full tempest with standalone | 16:38 |
| *** dxiri has quit IRC | 16:38 | |
| *** salmankhan has quit IRC | 16:38 | |
| chandankumar | If i make them passing it will be a good replacement for any job taking 1 hr 30 mins | 16:38 |
| chandankumar | EvilienM: https://review.openstack.org/606046 | 16:39 |
| *** dxiri has joined #tripleo | 16:39 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Merge new params - nic-config templates https://review.openstack.org/605807 | 16:41 |
| *** gkadam has quit IRC | 16:42 | |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| *** rdopiera has quit IRC | 16:48 | |
| *** thrash is now known as thrash|f00dz | 16:49 | |
| DirectorN00b | Hi all. Not to sound repetative, but is redhat director just tripleo? I'm trying to get familiar with Director, but jut need to understand a bit of the basics. | 16:49 |
| DirectorN00b | (ie: overcloud plan methods) | 16:50 |
| chem | weshay_ruck: I've got an error during repo setup where https doesn't work because it point to an http scheme | 16:51 |
| chem | weshay_ruck: but first, hi :) | 16:51 |
| weshay_ruck | ? | 16:51 |
| weshay_ruck | HI! | 16:51 |
| weshay_ruck | hachem | 16:51 |
| weshay_ruck | chem, link? | 16:51 |
| *** jpich has quit IRC | 16:52 | |
| chem | weshay_ruck: https://mirror.regionone.rdo-cloud.rdoproject.org:8080/rdo/centos7/0f/e2/0fe2e39140ff038ce66f43a478fc792e8a271fe2_b2d2686b/delorean.repo | 16:52 |
| weshay_ruck | ovb job? | 16:52 |
| chem | weshay_ruck: no standalone-upgrade testing | 16:52 |
| weshay_ruck | why did it get the rdo mirror | 16:53 |
| weshay_ruck | that's odd | 16:53 |
| chem | weshay_ruck: reproducer script in rdo | 16:53 |
| weshay_ruck | OH | 16:53 |
| chem | weshay_ruck: the thing is that curl http://... work fine | 16:53 |
| chem | weshay_ruck: no "s" | 16:54 |
| weshay_ruck | hrm | 16:54 |
| weshay_ruck | k | 16:54 |
| weshay_ruck | pokes | 16:54 |
| * weshay_ruck pokes around | 16:54 | |
| *** derekh has quit IRC | 16:55 | |
| weshay_ruck | chem, hrm..http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/nodepool-setup/templates/mirror_info.sh.j2#n66 | 16:56 |
| weshay_ruck | chem, however our release files have https | 16:56 |
| chem | so we substitue but switch from http to https right ? | 16:57 |
| chem | weshay_ruck: ^ | 16:57 |
| *** jpena is now known as jpena|off | 16:57 | |
| weshay_ruck | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L44 | 16:58 |
| chem | weshay_ruck: yeah that was I had in mind, so it happen there .. I'll try to hardcode http there and let you know | 16:59 |
| weshay_ruck | chem, I wonder if we exposed a bug by including only the tasks | 16:59 |
| weshay_ruck | chem ya.. I think the config may need to change from https to http | 16:59 |
| weshay_ruck | chem, is that working as expected btw? | 16:59 |
| weshay_ruck | the include_role: task: foo.yml | 16:59 |
| chem | weshay_ruck: yeah it seems it point to master and all | 17:00 |
| *** ykarel__ has quit IRC | 17:00 | |
| *** shyamb has quit IRC | 17:03 | |
| *** jaganathan has quit IRC | 17:06 | |
| chem | weshay_ruck: well, it's confusing and I need to go, I'll look at the result in the ci later on | 17:08 |
| weshay_ruck | k | 17:08 |
| *** psachin has joined #tripleo | 17:09 | |
| *** gfidenteN00b has quit IRC | 17:10 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 17:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795009 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 17:10 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 17:10 |
| *** trown is now known as trown|lunch | 17:11 | |
| DirectorN00b | So, "openstack overcloud plan delete this-plan" --> Cannot delete a plan that has an associated stack. | 17:19 |
| EvilienM | chandankumar: ack, I would show https://review.openstack.org/#/c/606046/ to mwhahaha as well | 17:19 |
| DirectorN00b | openstack stack list shows a stack name associated with CREATE_FAILED | 17:19 |
| DirectorN00b | Can I just "stack delete" this and then remove the plan? | 17:20 |
| slagle | DirectorN00b: yes, although openstack overcloud delete will delete both | 17:20 |
| DirectorN00b | slagle: It's not a new overcloud, just a new plan. Can I mix plans with a single overcloud? | 17:21 |
| slagle | no | 17:21 |
| DirectorN00b | (or am I confusing terminologies) | 17:21 |
| slagle | openstack overcloud delete <name> -- will delete both a stack and plan with that <name> | 17:22 |
| DirectorN00b | slagle: Oh. So, I cannot use one plan against an overcloud, and then when it fails, create a new plan and try to execute that against the overcloud with modified template information? | 17:22 |
| DirectorN00b | okay, I am executing the overcloud delete for that plan. | 17:23 |
| slagle | you can also just re-run the same deployment command and will update both the existing plan, and then try and update the stack | 17:24 |
| DirectorN00b | Though that tends to suggest deleting of the nodes already deployed :-) | 17:24 |
| *** jtomasek has joined #tripleo | 17:24 | |
| DirectorN00b | slagle: I am struggling with that bit. I can create a plan with --template, but then when I ant to update those templates in the plan, I cannot see how to do this. | 17:24 |
| DirectorN00b | I mean, I can update env stuff easy (just re-include with parameters) but the templates once modified... | 17:25 |
| slagle | "openstack overcloud deploy" updates both the plan and stack | 17:25 |
| slagle | it will save the updated templates in the plan, then do a stack-update with Heat | 17:26 |
| DirectorN00b | I thiought I tried that, and when I exported it, the templates were the same | 17:26 |
| DirectorN00b | okay, that plan has gone now, thanks. | 17:27 |
| DirectorN00b | So, if I go and change a template now (/usr/share/openstack-tripleo-heat-templates) and then tell the plan to export (container save) to a local folder here, then that should have the updated template details? | 17:28 |
| DirectorN00b | (Sorry if i'm getting confused. I'm still trying to wrap my head around what is going on) | 17:28 |
| slagle | no, you'd have to actually run openstack overcloud deploy after making the change for the plan to get updated | 17:29 |
| slagle | then if you did export, you should see the change | 17:29 |
| DirectorN00b | Ah, I See. | 17:30 |
| DirectorN00b | okay, let me have a tinker with that - very much apprieciated feedback!! | 17:30 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Add podman support to validate-tempest role https://review.openstack.org/605356 | 17:31 |
| DirectorN00b | slagle: Why do I have to deploy it before I can see the change it has made? on reflection that seems a little backward :-/ | 17:37 |
| slagle | DirectorN00b: you can use openstack overcloud deploy with --update-plan-only if you only want to update the plan and skip the stack update | 17:42 |
| DirectorN00b | slagle: oooooh That's very good to know, thankyou :-) | 17:47 |
| *** dsneddon has joined #tripleo | 17:47 | |
| *** dsneddon has quit IRC | 17:47 | |
| DirectorN00b | Seems logical you'd want to see the effects of what you are changing before pressing the "push to all nodes as an update!" button :-) | 17:48 |
| *** dsneddon has joined #tripleo | 17:49 | |
| *** agopi|afk is now known as agopi | 17:51 | |
| DirectorN00b | ERROR configuring gnocci. | 17:57 |
| DirectorN00b | I think this might be to blame, but not sure... https://review.openstack.org/#/c/530225/ | 17:57 |
| DirectorN00b | So I'll get rid of that, and then redeploy. | 17:57 |
| EvilienM | don't blame me | 17:57 |
| EvilienM | gnocci doesn't exist, it's gnocchi | 17:58 |
| openstackgerrit | Merged openstack/ansible-role-tripleo-modify-image master: Remove compare_host_packages strategy https://review.openstack.org/600273 | 17:59 |
| DirectorN00b | It's been a long day :-) | 18:00 |
| DirectorN00b | Also, no blame cultre here, I'm just trying to learn :-) | 18:00 |
| * DirectorN00b points finger and scowles | 18:01 | |
| DirectorN00b | It fails on a few things. Not sure of it's what I have configured(/not configured) or whether it's a bug. Trial and error now. | 18:01 |
| weshay_ruck | mwhahaha, EvilienM seems like queens is installing ansible 2.6.4 | 18:02 |
| weshay_ruck | and master is installing 2.5.4 | 18:02 |
| weshay_ruck | sweet | 18:02 |
| EvilienM | DirectorN00b: no worries :-) | 18:02 |
| weshay_ruck | sorry.. 2.5.2 | 18:02 |
| *** TheJulia is now known as needssleep | 18:02 | |
| *** raildo has quit IRC | 18:06 | |
| *** raildo has joined #tripleo | 18:06 | |
| *** jcoufal has quit IRC | 18:08 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795009 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 18:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 18:10 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 18:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Set proper setype for tempest service directories https://review.openstack.org/605980 | 18:11 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch fs027 to deploy with podman https://review.openstack.org/600517 | 18:12 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Add podman support to validate-tempest role https://review.openstack.org/605356 | 18:12 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch fs027 to deploy with podman https://review.openstack.org/600517 | 18:12 |
| EvilienM | chandankumar: fixed order ^ | 18:12 |
| chandankumar | EvilienM: thanks! | 18:14 |
| *** med_ has joined #tripleo | 18:15 | |
| *** chem has quit IRC | 18:18 | |
| *** chem has joined #tripleo | 18:18 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-upgrade master: remove ansible from triplo-upgrade requirements https://review.openstack.org/606156 | 18:19 |
| *** thrash|f00dz is now known as thrash | 18:19 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-upgrade stable/rocky: remove ansible from triplo-upgrade requirements https://review.openstack.org/606157 | 18:19 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-upgrade stable/queens: remove ansible from triplo-upgrade requirements https://review.openstack.org/606158 | 18:19 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-upgrade stable/pike: remove ansible from triplo-upgrade requirements https://review.openstack.org/606159 | 18:20 |
| *** trown|lunch is now known as trown | 18:37 | |
| EvilienM | thrash: do you think we could get +A on https://review.openstack.org/#/c/605633/ today? | 18:38 |
| thrash | EvilienM: Let me try | 18:38 |
| *** med_ has quit IRC | 19:00 | |
| * mwhahaha sighs | 19:00 | |
| mwhahaha | no on reads their email | 19:00 |
| mwhahaha | anyway did we figure out where the newer ansible is coming from yet? | 19:01 |
| AJaeger | tripleo cores: https://review.openstack.org/606075 and https://review.openstack.org/606074 don't use ansible - could you review those to help with Zuul job setup, please? | 19:02 |
| AJaeger | (https://review.openstack.org/606079 and https://review.openstack.org/606073 are the same change for repos that need ansible in testing, so won't ask for +A now;) | 19:02 |
| AJaeger | thanks, mwhahaha | 19:04 |
| mwhahaha | np i'll try and get the others later | 19:04 |
| AJaeger | thanks | 19:04 |
| openstackgerrit | Merged openstack/ansible-role-redhat-subscription master: Remove release-openstack-server https://review.openstack.org/606074 | 19:10 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 19:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795009 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 19:10 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 19:10 |
| openstackgerrit | Merged openstack/ansible-role-tripleo-cookiecutter master: Remove release-openstack-server https://review.openstack.org/606075 | 19:13 |
| *** florianf is now known as florianf|afk | 19:14 | |
| *** Chaserjim has joined #tripleo | 19:16 | |
| *** chem has quit IRC | 19:19 | |
| *** chem has joined #tripleo | 19:19 | |
| *** artom has quit IRC | 19:22 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Allow pinning of ara in undercloud-setup https://review.openstack.org/606179 | 19:23 |
| mwhahaha | weshay_ruck, marios, EvilienM -^ fix for scenario001/004 | 19:24 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Add sample designate environment for ha https://review.openstack.org/584026 | 19:25 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Split designate envs https://review.openstack.org/584532 | 19:25 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Add /v2 suffix to Designate uris https://review.openstack.org/585882 | 19:25 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Set correct project name for designate-neutron integration https://review.openstack.org/585902 | 19:25 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Don't configure BIND to listen on localhost https://review.openstack.org/606180 | 19:25 |
| thrash | EvilienM: No cores available... | 19:26 |
| thrash | I'll send an email to d0ugal and apetrich about it. | 19:26 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-quickstart master: Run Designate tempest test in scenario003 https://review.openstack.org/571321 | 19:26 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart master: Pin older versions of ara for pike/queens https://review.openstack.org/606181 | 19:28 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Unpin quickstart undercloud ara version https://review.openstack.org/606182 | 19:29 |
| *** artom has joined #tripleo | 19:30 | |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Add scenario010 to the check queue https://review.openstack.org/587015 | 19:34 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Fix skip of octavia-undercloud Ansible role https://review.openstack.org/591413 | 19:35 |
| weshay_ruck | ah | 19:42 |
| * weshay_ruck looks | 19:42 | |
| weshay_ruck | mwhahaha, why would that lead to diff versions across branches? | 19:42 |
| weshay_ruck | still we need to pin that, but I don't think that's it | 19:43 |
| mwhahaha | weshay_ruck: because we have a sufficient version in rocky+ | 19:43 |
| mwhahaha | 0.16.1 requires ansible 2.4.5 | 19:43 |
| mwhahaha | we have 2.4.4 in queens/pike | 19:43 |
| mwhahaha | in rocky+ it's already there | 19:43 |
| mwhahaha | weshay_ruck: it's likely because we just switched it on recently | 19:44 |
| mwhahaha | weshay_ruck: because i fyou check 2 days agao, ara was not installed http://logs.openstack.org/24/567224/109/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8b476ee/logs/undercloud/var/log/extra/pip.txt.gz | 19:44 |
| mwhahaha | weshay_ruck: but it is installed now http://logs.openstack.org/24/567224/111/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/448effe/logs/undercloud/var/log/extra/pip.txt.gz | 19:45 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Download CentOS-based amphora image if not present https://review.openstack.org/591997 | 19:45 |
| weshay_ruck | ok dumb question | 19:46 |
| weshay_ruck | why does ara install ansible? https://github.com/openstack/ara/blob/master/requirements.txt | 19:46 |
| weshay_ruck | I see | 19:46 |
| weshay_ruck | ha | 19:46 |
| mwhahaha | https://github.com/openstack/ara/blob/master/requirements.txt#L4 | 19:46 |
| weshay_ruck | looking right at it | 19:46 |
| weshay_ruck | dammit | 19:46 |
| weshay_ruck | that is like a circular dependency | 19:47 |
| mwhahaha | weshay_ruck: so can you add a line item to get ara pacakged in rdo | 19:47 |
| mwhahaha | dmsimard pasted the rpm specs in #rdo | 19:47 |
| mwhahaha | cause we shouldn't be system pip installing anything | 19:47 |
| mwhahaha | because as we all know, it breaks things | 19:47 |
| *** dprince has quit IRC | 19:47 | |
| weshay_ruck | aye | 19:48 |
| * mwhahaha looks at everyone who touched that ansible role to install pip/setuptools/ara | 19:48 | |
| weshay_ruck | mwhahaha, thanks | 19:50 |
| weshay_ruck | that landed 4 months ago .. dang | 19:51 |
| mwhahaha | but we didn't turn it on until recently | 19:52 |
| weshay_ruck | hrm.. I don't think that's right | 19:52 |
| mwhahaha | maybe it wasn't working until recently? | 19:52 |
| weshay_ruck | what do you mean turn it on.. ara has been capturing the undercloud tasks for a while | 19:53 |
| weshay_ruck | actually I think the overcloud work broke the undercloud ara work | 19:53 |
| weshay_ruck | tbh | 19:53 |
| mwhahaha | oh maybe it's because 0.16.0 was just published | 19:53 |
| mwhahaha | when was that | 19:53 |
| mwhahaha | cause 0.15.0 was fine, but that wouldn't explain http://logs.openstack.org/24/567224/109/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8b476ee/logs/undercloud/var/log/extra/pip.txt.gz | 19:53 |
| weshay_ruck | mwhahaha, well you can't get too mad | 19:54 |
| mwhahaha | http://logs.openstack.org/24/567224/109/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8b476ee/job-output.txt.gz#_2018-09-26_23_00_55_083912 | 19:54 |
| mwhahaha | it was always failing | 19:54 |
| mwhahaha | ignore_errors: true | 19:54 |
| * mwhahaha sighs | 19:54 | |
| weshay_ruck | it was working at one point | 19:54 |
| weshay_ruck | you +2 | 19:55 |
| weshay_ruck | :) | 19:55 |
| * weshay_ruck looks for rpm reviews | 19:55 | |
| weshay_ruck | there was duress re: timeouts | 19:56 |
| weshay_ruck | as usual | 19:56 |
| dpeacock | Right - I'm bailing - need to get my folks to the airport. Have a good weekend folks. :-) | 20:02 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/queens: DNM: ci test https://review.openstack.org/606191 | 20:04 |
| mwhahaha | weshay_ruck: -^ that should test the whole thing, we'll see | 20:04 |
| fultonj | EvilienM: earlier we were talking about how you preconfigured your IP on your edge cloud node | 20:04 |
| weshay_ruck | k | 20:05 |
| weshay_ruck | thanks mwhahaha | 20:05 |
| fultonj | i'm hitting http://paste.openstack.org/show/731116 because my role IP map is empty | 20:05 |
| fultonj | http://paste.openstack.org/show/731120/ | 20:05 |
| DirectorN00b | Notice: heira(): cannot load backend module_data: cannot load such file -- heira/backend/module_data_backend <--- anyone know what this is alluding to? | 20:06 |
| fultonj | but tripleo seems to know about it as my HostsEntry was correctly populated with the IP e exported | 20:06 |
| fultonj | http://paste.openstack.org/show/731121/ | 20:07 |
| fultonj | s/tripleo/my second heat stack | 20:07 |
| *** openstackgerrit has quit IRC | 20:07 | |
| DirectorN00b | slagle: I have redeployed the overcloud, and then container save'd the config it's using, but the template does not have the modifications I made :-( | 20:08 |
| fultonj | slagle: ^ ? | 20:08 |
| *** Chaserjim has quit IRC | 20:10 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 20:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795009 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 20:10 |
| DirectorN00b | From our discussion before (perhaps I misunderstood?) But I changed the heat-base.yaml template at /usr/share/opentack-triple-heat-templates, re-overcloud deployed | 20:10 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 20:10 |
| *** psachin has quit IRC | 20:10 | |
| DirectorN00b | I then saed out, as I thought you said after the deployment (or the --update-plan-only) it will modify the plan? | 20:10 |
| DirectorN00b | s/saed/saved | 20:11 |
| DirectorN00b | Now checking the templates in the rendered save output, I see the code still there that I commented out :-/ | 20:11 |
| *** tzumainn has quit IRC | 20:12 | |
| DirectorN00b | Also, seems --update-plan-only isn't a valid argument :-( | 20:13 |
| mwhahaha | DirectorN00b: what version? --update-plan-only on tripleo deployhas been a thing for like 2 years | 20:16 |
| mwhahaha | https://github.com/openstack/python-tripleoclient/blame/master/tripleoclient/v1/overcloud_deploy.py#L657 | 20:16 |
| *** mjturek has quit IRC | 20:18 | |
| *** openstackgerrit has joined #tripleo | 20:20 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: break out release config by distro type https://review.openstack.org/602387 | 20:20 |
| DirectorN00b | Maybe i#m not using it correctly? Or maybe because this is a redhat director version it's different? "opentack overcloud deploy mycloud --update-plan-only" ? | 20:21 |
| *** rlandy is now known as rlandy|brb | 20:27 | |
| DirectorN00b | Looks like i'm using version 7.6.x or so. | 20:29 |
| bandini | mwhahaha: throw some cluebones my way on https://bugs.launchpad.net/tripleo/+bug/1795027 (see #4 and #5)? | 20:36 |
| openstack | Launchpad bug 1795027 in tripleo "redis is installed by default in the containerized undercloud" [Medium,Triaged] | 20:36 |
| *** agopi is now known as agopi|brb | 20:37 | |
| mwhahaha | bandini: you figured it out? | 20:37 |
| mwhahaha | bandini: zaqar environment enables redis but we configure zaqar with swift | 20:37 |
| bandini | mwhahaha: yeah I know why it happens, not sure how to best fix it | 20:37 |
| bandini | exactly | 20:37 |
| mwhahaha | oh, well | 20:37 |
| mwhahaha | yea | 20:37 |
| bandini | I could add an environments/service/zaqar-noredis.yaml and use that? | 20:38 |
| bandini | seems a bit ugly though? | 20:38 |
| mwhahaha | in the past we had an undercloud-<service>.yaml which is silly | 20:38 |
| mwhahaha | bandini: go patch it out in undercloud_paramers.yaml | 20:38 |
| mwhahaha | bandini: because that comes from python-tripleoclient | 20:38 |
| * bandini looks | 20:39 | |
| mwhahaha | hrm maybe not | 20:39 |
| mwhahaha | that just seems to be parameters_defaults | 20:39 |
| mwhahaha | bandini: do we have a disable-redis.yaml somewhere? | 20:40 |
| mwhahaha | we could add that to the end of the deploy command | 20:40 |
| mwhahaha | that would probably be a more explicit thing easier to follow thing | 20:40 |
| bandini | let me see, I don't think we have it | 20:40 |
| *** agopi|brb has quit IRC | 20:41 | |
| bandini | nope | 20:41 |
| bandini | mwhahaha: I am starting to feel that 'environments/services/undercloud-zaqar.yaml' is almost the less horrible option? | 20:42 |
| mwhahaha | i really wanted to get rid of those undercloud-* ones | 20:42 |
| bandini | I see | 20:43 |
| mwhahaha | maybe we just pull the redis out of zaqar.yaml and into a redis.yaml | 20:43 |
| mwhahaha | alternatively zaqar-swift-backend.yaml that doesn't have redis enabled | 20:43 |
| * mwhahaha shakes his fist at redis and zaqar | 20:44 | |
| bandini | I think the latter is preferable as it does not break existing file users | 20:44 |
| bandini | lol | 20:44 |
| * bandini tries | 20:44 | |
| mwhahaha | so THT/environments/services/zaqar-swift-backend.yaml that enables zaqar but disables redis and then we just change out the file in tripleoclient | 20:45 |
| bandini | right | 20:45 |
| mwhahaha | at least it doesn't have the undercloud- in the name :D | 20:45 |
| bandini | :) | 20:45 |
| bandini | CLEAR WIN | 20:45 |
| *** rlandy|brb is now known as rlandy | 20:49 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: turn off named prior to validation for scen003 https://review.openstack.org/606198 | 20:52 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Add a zaqar-swift-backend environment file https://review.openstack.org/606200 | 20:53 |
| openstackgerrit | Michele Baldessari proposed openstack/python-tripleoclient master: Zaqar on the containerized undercloud should not use Redis https://review.openstack.org/606201 | 20:53 |
| weshay_ruck | mwhahaha, ok.. so https://review.openstack.org/#/c/606180/ > https://review.openstack.org/606198 | 20:53 |
| weshay_ruck | ? | 20:53 |
| *** artom has quit IRC | 20:54 | |
| mwhahaha | weshay_ruck: named is used by designate | 20:54 |
| mwhahaha | weshay_ruck: so yea use ben's patch | 20:54 |
| weshay_ruck | k.. I briefly looked at the tempest tests.. wasn't sure if that was validated | 20:54 |
| mwhahaha | it's not actually blocking anythign but is showing up in the openstack health reports (elastic search is triggering) | 20:55 |
| weshay_ruck | ya.. just trying get the gate to be healthier | 20:56 |
| DirectorN00b | Bah, I need to see what the version of this are. REdhat drives me insane. | 20:56 |
| DirectorN00b | Subscriptions and stuff, and blah blah. | 20:56 |
| DirectorN00b | Thanks for input today. I will be back asking more dumb questions soon enough :-) | 20:57 |
| *** panda has quit IRC | 20:57 | |
| *** panda has joined #tripleo | 20:58 | |
| *** shardy has quit IRC | 21:00 | |
| *** mmethot has quit IRC | 21:00 | |
| *** mmethot has joined #tripleo | 21:00 | |
| openstackgerrit | David Vallee Delisle proposed openstack/tripleo-heat-templates master: Validate that a detected ceph-disk is member of a cluster before considering that we need ceph-osd package https://review.openstack.org/606105 | 21:02 |
| *** raildo has quit IRC | 21:04 | |
| *** mmethot has quit IRC | 21:05 | |
| *** Vorrtex has quit IRC | 21:07 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795009 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 21:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 21:10 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 21:10 |
| *** rfolco has quit IRC | 21:18 | |
| openstackgerrit | Merged openstack/os-net-config stable/rocky: Restart ivs/nvfswitch after config file is updated https://review.openstack.org/605668 | 21:22 |
| *** dtrainor_ has joined #tripleo | 21:22 | |
| *** dsneddon has quit IRC | 21:22 | |
| *** dsneddon has joined #tripleo | 21:23 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Unpin quickstart undercloud ara version https://review.openstack.org/606182 | 21:24 |
| *** dtrainor has quit IRC | 21:24 | |
| *** agopi|brb has joined #tripleo | 21:28 | |
| *** slaweq has quit IRC | 21:28 | |
| * mwhahaha flips tables over failure in the gate again | 21:41 | |
| mwhahaha | argh it was my own fault for restoring changes too | 21:42 |
| *** vkapalav has quit IRC | 21:42 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795009 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 22:10 |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 22:10 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 22:10 |
| *** mcornea has quit IRC | 22:12 | |
| *** artom has joined #tripleo | 22:13 | |
| *** EvilienM is now known as EmilienM | 22:20 | |
| *** toure is now known as toure|gone | 22:24 | |
| *** panda is now known as panda|off | 22:26 | |
| *** rlandy has quit IRC | 22:27 | |
| *** boazel has quit IRC | 22:29 | |
| *** tosky has quit IRC | 22:42 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1715374 | 23:10 |
| openstack | Launchpad bug 1715374 in tripleo "Reloading compute with SIGHUP prenvents instances to boot" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792560 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795009 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 23:10 |
| openstack | Launchpad bug 1795009 in tripleo "the scenario001/004 multinode-oooq-container job is failing for Queens and Pike for overcloud deployment error "overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:" " [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 23:10 |
| *** dtrainor__ has joined #tripleo | 23:17 | |
| *** dtrainor_ has quit IRC | 23:20 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/queens: DNM: ci test https://review.openstack.org/606191 | 23:36 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart master: Pin older versions of ara for pike/queens https://review.openstack.org/606181 | 23:40 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/queens: DNM: ci test https://review.openstack.org/606191 | 23:42 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!