| *** mhenkel has quit IRC | 00:04 | |
| *** dsneddon_ has joined #tripleo | 00:06 | |
| *** ayoung has quit IRC | 00:08 | |
| *** thrash has quit IRC | 00:09 | |
| *** trown|outtypewww has quit IRC | 00:09 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 00:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| *** bfournie has joined #tripleo | 00:11 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Install dracut-generic-config package https://review.openstack.org/411540 | 00:16 |
| openstackgerrit | Mikhail S Medvedev proposed openstack/diskimage-builder: Fix bootloader element on ppc https://review.openstack.org/411541 | 00:16 |
| *** trown has joined #tripleo | 00:19 | |
| *** thrash has joined #tripleo | 00:20 | |
| *** thrash has quit IRC | 00:20 | |
| *** thrash has joined #tripleo | 00:20 | |
| EmilienM | dsneddon: are we still on track for https://blueprints.launchpad.net/tripleo/+spec/tripleo-lldp-validation in ocata? | 00:31 |
| *** rwsu has quit IRC | 00:32 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove swapfile from undercloud https://review.openstack.org/410323 | 00:38 |
| *** rwsu has joined #tripleo | 00:44 | |
| *** limao has joined #tripleo | 00:46 | |
| *** ayoung has joined #tripleo | 00:53 | |
| *** saneax is now known as saneax-_-|AFK | 00:58 | |
| openstackgerrit | jeck proposed openstack/puppet-tripleo: [TrivialFix] Fix typo error in comment https://review.openstack.org/411550 | 01:07 |
| *** ctayal has quit IRC | 01:09 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 01:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| *** george_goh has quit IRC | 01:11 | |
| *** ctayal has joined #tripleo | 01:12 | |
| *** ctayal has quit IRC | 01:14 | |
| openstackgerrit | jeck proposed openstack/puppet-tripleo: [TrivialFix] Fix typo error in comment https://review.openstack.org/411551 | 01:16 |
| *** mhenkel has joined #tripleo | 01:19 | |
| *** bana_k has quit IRC | 01:21 | |
| *** george_goh has joined #tripleo | 01:21 | |
| *** mhenkel has quit IRC | 01:24 | |
| *** sshnaidm is now known as sshnaidm|afk | 01:58 | |
| *** jeckersb is now known as jeckersb_gone | 01:59 | |
| *** jeckersb_gone is now known as jeckersb | 02:00 | |
| *** ctayal has joined #tripleo | 02:06 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 02:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| *** ctayal_ has joined #tripleo | 02:12 | |
| dsneddon_ | EmilienM: I think we've got good progress on reviews for LLDP data collection. If it's not fully polished by Ocata, at least most of the building blocks will be done. | 02:13 |
| *** ctayal has quit IRC | 02:13 | |
| dsneddon_ | EmilienM: Enough to call the blueprint completed, I would guess. | 02:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Install dracut-generic-config package https://review.openstack.org/411540 | 02:19 |
| *** fzdarsky_ has joined #tripleo | 02:24 | |
| *** fzdarsky|afk has quit IRC | 02:28 | |
| openstackgerrit | zhangyanxian proposed openstack/puppet-tripleo: Fix typo in endpoint.pp https://review.openstack.org/411588 | 02:39 |
| openstackgerrit | zhangyanxian proposed openstack/puppet-tripleo: Fix typo in endpoint.pp https://review.openstack.org/411588 | 02:39 |
| *** TSCHAK_ has quit IRC | 02:44 | |
| *** sai-out is now known as sai | 02:58 | |
| *** yamahata has quit IRC | 02:58 | |
| *** cwolferh has quit IRC | 03:01 | |
| *** thrash is now known as thrash|g0ne | 03:05 | |
| *** fragatina has quit IRC | 03:07 | |
| *** fragatina has joined #tripleo | 03:08 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 03:10 |
| *** mhenkel has joined #tripleo | 03:10 | |
| *** fragatina has quit IRC | 03:12 | |
| *** fragatina has joined #tripleo | 03:12 | |
| *** ctayal_ has quit IRC | 03:12 | |
| *** mhenkel has quit IRC | 03:14 | |
| *** fragatin_ has joined #tripleo | 03:15 | |
| *** fragatina has quit IRC | 03:15 | |
| *** fragatin_ has quit IRC | 03:18 | |
| *** sudipto has joined #tripleo | 03:19 | |
| *** fragatina has joined #tripleo | 03:19 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-docs: Reflect default usage of image build https://review.openstack.org/409839 | 03:19 |
| *** fragatina has quit IRC | 03:24 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore https://review.openstack.org/365449 | 03:31 |
| *** yamahata has joined #tripleo | 03:44 | |
| *** ramishra has quit IRC | 03:47 | |
| *** limao has quit IRC | 03:57 | |
| *** limao has joined #tripleo | 03:57 | |
| *** limao has quit IRC | 04:01 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 04:10 |
| *** dmacpher is now known as dmacpher-afk | 04:26 | |
| *** udesale has joined #tripleo | 04:51 | |
| *** bana_k has joined #tripleo | 05:03 | |
| *** ooolpbot has joined #tripleo | 05:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 05:11 |
| *** ooolpbot has quit IRC | 05:11 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 05:11 |
| *** prateek has joined #tripleo | 05:13 | |
| *** links has joined #tripleo | 05:21 | |
| *** Vijayendra has joined #tripleo | 05:24 | |
| *** dsneddo__ has joined #tripleo | 05:25 | |
| *** fragatina has joined #tripleo | 05:26 | |
| *** dsneddon_ has quit IRC | 05:29 | |
| *** fragatina has quit IRC | 05:36 | |
| *** fragatina has joined #tripleo | 05:36 | |
| *** masco has joined #tripleo | 05:36 | |
| *** bana_k has quit IRC | 05:38 | |
| *** ctayal has joined #tripleo | 05:42 | |
| *** ramishra has joined #tripleo | 05:43 | |
| *** saneax-_-|AFK is now known as saneax | 05:44 | |
| *** jaosorior has joined #tripleo | 06:02 | |
| *** limao has joined #tripleo | 06:04 | |
| *** dmacpher-afk is now known as dmacpher | 06:05 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 06:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| *** pgadiya has joined #tripleo | 06:12 | |
| *** rcernin has quit IRC | 06:13 | |
| *** bana_k has joined #tripleo | 06:24 | |
| *** ctayal_ has joined #tripleo | 06:29 | |
| *** dsneddo__ has quit IRC | 06:29 | |
| *** ctayal has quit IRC | 06:30 | |
| *** rcernin has joined #tripleo | 06:37 | |
| *** lmiccini has joined #tripleo | 06:37 | |
| bandini | morning | 06:40 |
| jaosorior | bandini: sup dude | 06:42 |
| *** rcernin has quit IRC | 06:43 | |
| jaosorior | alee: should we merge this https://review.openstack.org/#/c/408783/ ? | 06:43 |
| *** ctayal_ has quit IRC | 06:44 | |
| *** matbu is now known as matbu|halfpto | 06:51 | |
| *** rcernin has joined #tripleo | 06:54 | |
| *** yprokule has joined #tripleo | 07:01 | |
| *** ramishra has quit IRC | 07:01 | |
| *** abehl has joined #tripleo | 07:05 | |
| *** ramishra has joined #tripleo | 07:06 | |
| *** paramite has joined #tripleo | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 07:10 |
| *** dsneddon_ has joined #tripleo | 07:14 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo: Initial pacemaker remote profile support https://review.openstack.org/400967 | 07:16 |
| *** mhenkel has joined #tripleo | 07:16 | |
| *** leanderthal|afk is now known as leanderthal | 07:18 | |
| *** dsneddon_ has quit IRC | 07:19 | |
| *** bnemec has quit IRC | 07:21 | |
| *** abregman has joined #tripleo | 07:23 | |
| d0ugal | Ng: did you manage to figure it out? | 07:30 |
| *** jpena|off is now known as jpena | 07:35 | |
| *** bnemec has joined #tripleo | 07:37 | |
| *** dsariel has quit IRC | 07:39 | |
| *** abregman has quit IRC | 07:39 | |
| *** aufi has joined #tripleo | 07:39 | |
| *** derekh has joined #tripleo | 07:40 | |
| derekh | sshnaidm|afk: panda|Zz bnemec EmilienM new problem today, the instances nodepool is starting wont boot, and it keeps trying over and over | 07:42 |
| jaosorior | X_X\ | 07:42 |
| derekh | console log shows this http://chunk.io/f/188ecaea89224868bd091e9c9f13bca5 | 07:43 |
| derekh | I'm thinking nodepool has just switched onto a new image with problems (maybe a switch from 7.2 too 7.3) | 07:44 |
| derekh | to avoid the cloud being DOS'd I've run this "iptables -I INPUT -s 23.253.73.160 -j DROP" | 07:44 |
| jaosorior | A start job is running for dev-disk... | 07:44 |
| jaosorior | and it gets stuck there? | 07:44 |
| derekh | this blocks out infra while we figure it out | 07:44 |
| *** mcornea has joined #tripleo | 07:45 | |
| *** rasca has joined #tripleo | 07:45 | |
| derekh | jaosorior: I havn't looked at any details | 07:45 |
| derekh | jaosorior: getting kids ready for school etc... I'll be back later | 07:45 |
| derekh | mainly I jumped in to tell people about the iptables rule so they wouldn't start debuging that | 07:46 |
| jaosorior | derekh: thanks | 07:46 |
| derekh | ttyl | 07:47 |
| *** derekh has quit IRC | 07:47 | |
| *** iranzo has joined #tripleo | 07:54 | |
| *** iranzo has joined #tripleo | 07:54 | |
| *** tobias-fiberdata has quit IRC | 07:55 | |
| *** pcaruana has joined #tripleo | 07:56 | |
| *** florianf has joined #tripleo | 07:57 | |
| *** tesseract has joined #tripleo | 07:57 | |
| *** tesseract is now known as Guest31304 | 07:58 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/tripleo-ci: Update my feed to use the tripleo tag https://review.openstack.org/411687 | 08:06 |
| openstackgerrit | Dougal Matthews proposed openstack-infra/tripleo-ci: Use a more specific feed for my blog https://review.openstack.org/411687 | 08:07 |
| *** fragatina has quit IRC | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 08:10 |
| *** shardy has joined #tripleo | 08:16 | |
| *** arxcruz has quit IRC | 08:20 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 08:21 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add hook to generate metadata from service profiles https://review.openstack.org/411339 | 08:26 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals https://review.openstack.org/411340 | 08:26 |
| jaosorior | shardy: this is the hook that I talked about yesterday https://review.openstack.org/#/c/411339/3 and this is an implementation of it https://review.openstack.org/#/c/411340/4 | 08:27 |
| *** fzdarsky_ is now known as fzdarsky | 08:29 | |
| *** bana_k has quit IRC | 08:31 | |
| *** amoralej|off is now known as amoralej | 08:31 | |
| *** rlandy|bbl is now known as rlandy | 08:31 | |
| *** rlandy has quit IRC | 08:31 | |
| *** jaosorior has quit IRC | 08:32 | |
| *** ramishra has quit IRC | 08:32 | |
| *** jaosorior has joined #tripleo | 08:33 | |
| *** flepied has joined #tripleo | 08:33 | |
| *** pblaho has quit IRC | 08:42 | |
| *** hogepodge has quit IRC | 08:42 | |
| *** udesale has quit IRC | 08:44 | |
| *** hogepodge has joined #tripleo | 08:44 | |
| *** spredzy has joined #tripleo | 08:44 | |
| *** flepied has quit IRC | 08:49 | |
| *** ccamacho|out is now known as ccamacho | 08:55 | |
| *** jaosorior has quit IRC | 08:55 | |
| *** dsneddon_ has joined #tripleo | 08:55 | |
| Ng | d0ugal: actually no, I decided to sleep instead of keep bashing at it :D | 08:56 |
| Ng | d0ugal: so if you have any hints, I would be very grateful! | 08:56 |
| *** ohamada has joined #tripleo | 08:58 | |
| *** udesale has joined #tripleo | 09:00 | |
| *** lucas-afk is now known as lucasagomes | 09:04 | |
| *** jbadiapa has joined #tripleo | 09:06 | |
| *** dsneddo__ has joined #tripleo | 09:06 | |
| *** dsnedd___ has joined #tripleo | 09:08 | |
| *** dsneddon_ has quit IRC | 09:09 | |
| *** hewbrocca_afk is now known as hewbrocca | 09:10 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 09:10 |
| *** tobias_fiberdata has joined #tripleo | 09:10 | |
| *** dsneddo__ has quit IRC | 09:12 | |
| *** dsneddon_ has joined #tripleo | 09:14 | |
| *** dsneddo__ has joined #tripleo | 09:16 | |
| d0ugal | Ng: Let me check I understand the issue. You are making changes to tripleoclient but then when you run "openstack overcloud deploy" (or another command) the changes don't seem to be there? | 09:16 |
| d0ugal | Ng: Which command are you working on? | 09:16 |
| Ng | d0ugal: I'm adding a new command, and trying to get it to show up in the openstack command list | 09:17 |
| d0ugal | Ng: ah, did you add it here? https://github.com/openstack/python-tripleoclient/blob/master/setup.cfg#L57 | 09:17 |
| Ng | d0ugal: yep | 09:18 |
| shardy | Ng: have you done pip install . in the tripleoclient git tree and/or done a delorean build of the tripleoclient tree? | 09:18 |
| *** dsnedd___ has quit IRC | 09:18 | |
| Ng | shardy: I did a setup.py install | 09:18 |
| d0ugal | Yeah, that ^ | 09:18 |
| shardy | setup.py install isn't enough to register the osc plugins IIRC | 09:18 |
| d0ugal | it should be :) | 09:18 |
| shardy | last time I tried, it wasn't | 09:18 |
| d0ugal | ah, interesting. I wonder why. | 09:18 |
| Ng | aha! | 09:18 |
| *** dsneddon_ has quit IRC | 09:19 | |
| Ng | as always, thanks! | 09:19 |
| *** lucasagomes is now known as lucas-brb | 09:19 | |
| *** flepied has joined #tripleo | 09:19 | |
| shardy | Ng: np, shout if that doesn't work :) | 09:19 |
| Ng | shardy: will do. I must owe you approx one metric brewery of beers by now ;) | 09:20 |
| *** dsneddon_ has joined #tripleo | 09:21 | |
| d0ugal | I think we all do | 09:21 |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci: Update packates after configuring openstack repos https://review.openstack.org/411725 | 09:24 |
| shardy | hehe, no worries, happy to help :) | 09:24 |
| *** yamahata has quit IRC | 09:24 | |
| *** dsneddo__ has quit IRC | 09:24 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci: Update packages after configuring openstack repos https://review.openstack.org/411725 | 09:25 |
| shardy | Can anyone give an update on the status of the broken ovb ha jobs? | 09:27 |
| d0ugal | The last update I seen was from bnemec on https://bugs.launchpad.net/tripleo/+bug/1649742 | 09:28 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 09:28 |
| *** athomas has joined #tripleo | 09:28 | |
| shardy | Ok https://review.openstack.org/#/c/411514 seems to be the partial workaround for that | 09:29 |
| shardy | lets hope that passes the gate this time | 09:29 |
| d0ugal | shardy: doesn't look like it will | 09:30 |
| d0ugal | I was just looking at the status in zuul | 09:30 |
| shardy | ugh | 09:31 |
| * shardy thinks we should focus on multinode testing from now on | 09:32 | |
| *** panda|Zz is now known as panda | 09:33 | |
| panda | shardy: we don't have a root cause yet, we are speculating that the te client is unable to receive the exit of the test runner, but we don't know why, and why only on ha jobs | 09:35 |
| *** gfidente has joined #tripleo | 09:35 | |
| *** gfidente has quit IRC | 09:35 | |
| *** gfidente has joined #tripleo | 09:35 | |
| shardy | panda: Ok, any ideas how we debug further? | 09:37 |
| shardy | the ha job enables network isolation, but it sounds like the problem is a layer below that? | 09:37 |
| shardy | unless our net-iso stuff breaks the network somehow | 09:38 |
| *** zoli|gone is now known as zoli | 09:38 | |
| *** derekh has joined #tripleo | 09:41 | |
| derekh | <shardy>Can anyone give an update on the status of the broken ovb ha jobs? | 09:42 |
| derekh | shardy: we got a new problem thismorning | 09:42 |
| derekh | shardy: all slaves that nodepool was bringing up are failing to boot | 09:43 |
| shardy | derekh: Ok. Wow it's been a rough week for CI :( | 09:44 |
| derekh | shardy: ya, I think this might be related to the switch to centos 7.3 , that will be the 3rd 7.3 related thing... | 09:45 |
| d0ugal | dang | 09:45 |
| derekh | shardy: and we had a rh1 problem on monday | 09:45 |
| *** dsneddon_ has quit IRC | 09:45 | |
| *** dsneddon has quit IRC | 09:46 | |
| *** dsneddon_ has joined #tripleo | 09:46 | |
| *** gfidente has quit IRC | 09:47 | |
| *** dsariel has joined #tripleo | 09:48 | |
| *** shinobu__ has joined #tripleo | 09:49 | |
| *** Vijayendra_ has joined #tripleo | 09:52 | |
| *** gfidente has joined #tripleo | 09:53 | |
| *** Vijayendra has quit IRC | 09:54 | |
| Ng | shardy: pip install worked btw :) | 09:57 |
| shardy | Ng: excellent :) | 09:58 |
| *** chem has joined #tripleo | 09:59 | |
| d0ugal | I really want to know why they are different :) | 09:59 |
| *** pgadiya has quit IRC | 10:01 | |
| derekh | sshnaidm|afk: panda EmilienM bnemec yup, that latest problem is related to CentOS 7.3 https://bugzilla.redhat.com/show_bug.cgi?id=1405238 | 10:05 |
| openstack | bugzilla.redhat.com bug 1405238 in util-linux "findmnt --target behaviour changed in 7.3, shows all mount-points in chroot" [Unspecified,New] - Assigned to kzak | 10:05 |
| derekh | infra have the same problem on other clouds | 10:05 |
| panda | derekh: nobody tested this before upgrading :( | 10:07 |
| panda | ? | 10:07 |
| *** milan has joined #tripleo | 10:07 | |
| *** chem has quit IRC | 10:08 | |
| *** chem has joined #tripleo | 10:08 | |
| derekh | panda: nodepool just builds a new image and uploads it to the clouds it uses, there isn't any test on it that I know of, in theory infra can just flick a switch to switch back to a old image if needed | 10:08 |
| derekh | panda: but nobody on #infra at the moment to do it | 10:08 |
| derekh | brb | 10:09 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 10:10 |
| *** dsneddon_ has quit IRC | 10:11 | |
| *** pgadiya has joined #tripleo | 10:13 | |
| *** akrivoka has joined #tripleo | 10:14 | |
| *** dsariel has quit IRC | 10:15 | |
| sshnaidm|afk | derekh, panda EmilienM bnemec https://review.openstack.org/#/c/410470/6 - not ideal, but could be also solution for "testenv timeouts" | 10:16 |
| panda | sshnaidm|afk: isn't the job marked as failure when client reaches time out ? | 10:18 |
| *** openstackgerrit has quit IRC | 10:18 | |
| panda | sshnaidm|afk: seems to be working | 10:21 |
| *** dsneddon has joined #tripleo | 10:26 | |
| *** ealcaniz has joined #tripleo | 10:26 | |
| sshnaidm|afk | panda, no, it passed there | 10:28 |
| *** limao has quit IRC | 10:29 | |
| *** zoli is now known as zoli|lunch | 10:30 | |
| *** lucas-brb is now known as lucasagomes | 10:31 | |
| *** arxcruz has joined #tripleo | 10:32 | |
| *** florianf has quit IRC | 10:36 | |
| *** b00tcat has quit IRC | 10:36 | |
| *** numans has quit IRC | 10:38 | |
| *** florianf has joined #tripleo | 10:41 | |
| *** numans has joined #tripleo | 10:45 | |
| panda | is there a single component that installs firewall rules on the undercloud, or every element takes care of it ? | 10:48 |
| *** saibarspeis has joined #tripleo | 10:51 | |
| shardy | panda: they're managed by puppet, via the same puppet-tripleo manifest we use for the overcloud | 10:51 |
| shardy | https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.pp#L66 | 10:51 |
| *** openstackgerrit has joined #tripleo | 10:51 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci: Update packages after configuring openstack repos https://review.openstack.org/411725 | 10:51 |
| shardy | https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L646 | 10:51 |
| shardy | panda: ^^ | 10:51 |
| panda | shardy: ok, and this https://github.com/openstack/tripleo-image-elements/blob/master/elements/geard/os-refresh-config/pre-configure.d/97-gearman-iptables is only used to build the overcloud image ? | 10:53 |
| *** pblaho has joined #tripleo | 10:55 | |
| *** prateek has quit IRC | 10:59 | |
| *** b00tcat has joined #tripleo | 11:03 | |
| *** b00tcat has quit IRC | 11:03 | |
| *** b00tcat has joined #tripleo | 11:03 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 11:10 |
| *** milan has quit IRC | 11:16 | |
| *** chem has quit IRC | 11:18 | |
| *** tosky has joined #tripleo | 11:19 | |
| *** chem has joined #tripleo | 11:19 | |
| shardy | panda: not sure about that one tbh - there's a bunch of element stuff which is no longer used, but we've not done the work to figure it all out and remove it | 11:20 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/tripleo-quickstart: VirtualBMC support for tripleo-quickstart https://review.openstack.org/399704 | 11:25 |
| *** prateek has joined #tripleo | 11:27 | |
| shardy | hrm, has something changed in how we write overcloudrc recently? | 11:31 |
| shardy | mine is broken as it's pointing directly to the controller not the vip | 11:31 |
| *** jaosorior has joined #tripleo | 11:38 | |
| jaosorior | shardy: hey, just saw your review on the metadata hook | 11:39 |
| *** zoli|lunch is now known as zoli | 11:40 | |
| shardy | jaosorior: yeah just had a couple of questions | 11:41 |
| jaosorior | shardy: So, I initially had it in the service profiles since I was using the servername, but it turned out that's not necessary as we can assume that from the data nova gives out. However, regarding this https://review.openstack.org/#/c/411340/4/extraconfig/nova_metadata/krb-service-principals.yaml the input parameters are actually specific to servers and roles | 11:41 |
| jaosorior | shardy: for instance https://review.openstack.org/#/c/411340/4/puppet/services/aodh-api.yaml | 11:41 |
| jaosorior | that will end up giving telling the stack to pass that metadata for instances that deploy that service | 11:42 |
| jaosorior | and that's needed, since we have composable services, and we can't sure that all the services are deployed together | 11:43 |
| jaosorior | I haven't merged TLS for rabbitMQ, but that will be the same case as the apache services | 11:43 |
| jaosorior | say that we have a role that is dedicated to just the message broker, then that set of nodes would get the relevant metadata for the rabbitmq service (and thus will be able to set up the relevant service principals) | 11:44 |
| *** jkilpatr has quit IRC | 11:44 | |
| shardy | jaosorior: Yeah I get the per-service part, I'm just not clear why we need S::TripleO::ServerMetadataHook yet in every role | 11:45 |
| jaosorior | and, for instance, the computes, don't need to get the metadata for apache, since it doesn't need it | 11:45 |
| jaosorior | shardy: I could move it to services.yaml | 11:45 |
| jaosorior | shardy: would that be a better place? | 11:45 |
| *** thrash|g0ne is now known as thrash | 11:46 | |
| *** dsneddon has quit IRC | 11:48 | |
| shardy | jaosorior: I guess I'm wondering why the per-service metadata you have in krb-service-principals can't go in a service template, then we don't need the new hook at all, only the wiring of the metadata_settings into each role | 11:49 |
| shardy | perhaps there's some later patch which will make that clear tho :) | 11:49 |
| jaosorior | shardy: uhm... how could it be in a service template? | 11:50 |
| *** milan|afk has joined #tripleo | 11:51 | |
| jaosorior | so basically krb-service-principals parses what the metadata_settings in the roles gives, having attempted to make it slightly generic so this wouldn't be the only use case. It subsequently outputs it in a format that the vendordata plugin expects | 11:52 |
| jaosorior | shardy: in the service templates, we don't have visibility of all the services that are enabled per-role, so I wouldn't be able to do it there | 11:52 |
| EmilienM | hello | 11:53 |
| shardy | jaosorior: So there are other data formats we expect to support besides this one? | 11:53 |
| *** dsneddon has joined #tripleo | 11:54 | |
| jaosorior | shardy: well, vendordata plugins are not a limited use-case. When we were discussing those in the summit it turned out a bunch of deployers were using those for very different things. So I thought it would be a good thing to support; that's why I attempted to make it generic | 11:54 |
| EmilienM | sshnaidm|afk, panda, derekh: /me reading backlog | 11:55 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Postci tests trial and error https://review.openstack.org/411189 | 11:55 |
| jaosorior | shardy: I could ditch the attempt to make it generic and just parse the output though, then I would rename it to something more specific to this vendordata plugin | 11:56 |
| shardy | jaosorior: Ok, fair enough. I guess I started looking at the hard-coded references to eg haproxy and mysql and that made me think of defining the data with those services | 11:56 |
| derekh | EmilienM: tldr is that nodepool uploaded a new centos image that doesn't work (another centos 7.3 problem) https://bugs.launchpad.net/tripleo/+bug/1650503 | 11:56 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 11:56 |
| shardy | jaosorior: I'm OK with the current approach provided we actually need it to be generic and pluggable, I just wanted to clarify | 11:57 |
| EmilienM | derekh: I guess it's to early to have infra folks reverting the image | 11:57 |
| shardy | thanks for the additional info | 11:57 |
| derekh | EmilienM: while we wait for somebody in infra to revert back to the old image I've blocked nodepools access to rh1 | 11:57 |
| jaosorior | shardy: well, nobody has requested it yet, maybe I over-thought it. | 11:57 |
| EmilienM | derekh: I guess everything is blocked now | 11:57 |
| derekh | EmilienM: yup, I asked nobody there to do it yet | 11:57 |
| derekh | EmilienM: yup, jobs a just queueing up and not running | 11:57 |
| jaosorior | (talking about actual BZs) | 11:57 |
| jaosorior | but who knows | 11:57 |
| EmilienM | derekh: let me try to ping people | 11:58 |
| EmilienM | derekh: I'll be that guy today | 11:58 |
| panda | derekh: in the meantime, is there something we could do to fix the 7.3 image ? | 11:58 |
| *** akrivoka has quit IRC | 11:58 | |
| derekh | panda: I don't think so, its infras image we can't instruct nodepool to use a different one | 12:00 |
| derekh | panda: infra ppl were working on a fix last night and deleted a broken image, but I guess forgot nodepool would build and upload a new one | 12:01 |
| jaosorior | shardy: anyway, thanks for taking a look dude, I really appreciate your feedback; change the names as you suggested and parse the output in services.yaml, that way there's less changes in the actual roles templates | 12:04 |
| *** tobias-fiberdata has joined #tripleo | 12:05 | |
| *** tobias_fiberdata has quit IRC | 12:06 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 12:10 |
| *** akrivoka has joined #tripleo | 12:13 | |
| *** dtantsur|afk is now known as dtantsur | 12:16 | |
| *** ramishra has joined #tripleo | 12:16 | |
| *** jkilpatr has joined #tripleo | 12:16 | |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart-extras: Add introspection with retries option https://review.openstack.org/403677 | 12:18 |
| *** chem has quit IRC | 12:23 | |
| *** ohamada has quit IRC | 12:24 | |
| *** ohamada has joined #tripleo | 12:24 | |
| *** chem has joined #tripleo | 12:24 | |
| *** paramite_ has joined #tripleo | 12:27 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/tripleo-quickstart: VirtualBMC support for tripleo-quickstart https://review.openstack.org/399704 | 12:27 |
| *** b00tcat has quit IRC | 12:27 | |
| *** pkovar has joined #tripleo | 12:29 | |
| *** pgadiya has quit IRC | 12:31 | |
| *** jeckersb is now known as jeckersb_gone | 12:33 | |
| *** lucasagomes is now known as lucas-hungry | 12:35 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Implement stack update as mistral actions https://review.openstack.org/379516 | 12:39 |
| *** links has quit IRC | 12:42 | |
| jaosorior | bandini: do you know if we're running cinder-api with pacemaker? | 12:43 |
| *** ohamada has quit IRC | 12:43 | |
| *** ohamada_ has joined #tripleo | 12:43 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: WIP -- Add pacemaker ansible module of composable upgrade https://review.openstack.org/403397 | 12:44 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Adds a step0 for pre upgrade-init checks https://review.openstack.org/408631 | 12:44 |
| EmilienM | jaosorior: pcs status? | 12:44 |
| marios | matbu|halfpto: sorry i rebased your one too //review.openstack.org/403397 - rebased onto shardys https://review.openstack.org/#/c/411310/ which removes the upgrade init from steps | 12:45 |
| *** tremble has joined #tripleo | 12:45 | |
| *** zoli is now known as zoliXXL | 12:46 | |
| *** dprince has joined #tripleo | 12:46 | |
| *** ansmith has joined #tripleo | 12:46 | |
| *** pgadiya has joined #tripleo | 12:47 | |
| jaosorior | shardy: is there a way for me to merge two lists via Heat? or should I use yaql for that? | 12:47 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Adds a step0 for pre upgrade-init checks https://review.openstack.org/408631 | 12:48 |
| bandini | jaosorior: only cinder volume and cinder backup (of the cinder ones) | 12:49 |
| bandini | jaosorior: if you now look at master https://github.com/openstack/puppet-tripleo/tree/master/manifests/profile/pacemaker you see which profiles still exist. we removed the non-used ones | 12:50 |
| dprince | d0ugal: got a minute to chat about https://review.openstack.org/#/c/410970/ | 12:50 |
| d0ugal | dprince: sure | 12:50 |
| jaosorior | bandini: ah, ok, seems that cinder-api is still in t-h-t | 12:50 |
| d0ugal | dprince: I hadn't seen your response - reading. | 12:50 |
| dprince | d0ugal: well... we might should wait for rbrady-afk but he can catch up I guess | 12:50 |
| bandini | jaosorior: right we should remove the corresponding tht ones as well. I'll do | 12:50 |
| d0ugal | dprince: what/where is the undercloud installer? This is the Heat undercloud installer right? not instack? | 12:51 |
| dprince | d0ugal: yes | 12:51 |
| dprince | d0ugal: https://review.openstack.org/#/c/351351/ | 12:51 |
| dprince | d0ugal: https://etherpad.openstack.org/p/tripleo-composable-undercloud | 12:51 |
| dprince | d0ugal: probably more than you'd like to read right now | 12:52 |
| d0ugal | dprince: thanks, I'll take a look later. I was aware of the work but hadn't been following it as much as I probably should have. | 12:52 |
| dprince | d0ugal: I think I found a similar example to what I'm trying to do though. It is similar to the tarball pattern | 12:52 |
| dprince | d0ugal: I think we still use the tripleo_common.utils tarball function in python-tripleoclient | 12:52 |
| dprince | d0ugal: and we also consume it via a Mistral workflow | 12:53 |
| dprince | d0ugal: I think this is a fine pattern for things that have to happen locally | 12:53 |
| d0ugal | dprince: Yeah, i wanted to remove that use | 12:53 |
| d0ugal | dprince: :) | 12:53 |
| dprince | d0ugal: but how can you? other than duplicating it? | 12:53 |
| d0ugal | dprince: I hadn't gotten that far yet | 12:54 |
| dprince | d0ugal: here is a paste file of my tripleo-undercloud-passwords.yaml file (that works today) http://paste.openstack.org/show/592619/ | 12:54 |
| dprince | d0ugal: I need pretty much all of those | 12:55 |
| *** dmarlin_ has quit IRC | 12:55 | |
| dprince | d0ugal: and this function would be a shame to duplicate. Keeping in mind that instack already has its own logic for this we'd essentially have 3 different password generators in TripleO | 12:55 |
| dprince | d0ugal: in the short term anyway... | 12:55 |
| dprince | d0ugal: the point of this is all feedback loops, and eliminating old elements, and eventually instack too | 12:56 |
| dprince | d0ugal: in short, I think we will have some low level functions for things like passwords, tarballs that need to get used in multiple projects | 12:56 |
| d0ugal | I think the problem is that it is totally unclear what tripleo-common is. If it is going to be a shared library then we should move the workflows stuff out into it's own repo | 12:57 |
| dprince | d0ugal: and it would be a good idea if we could share them | 12:57 |
| *** b00tcat has joined #tripleo | 12:57 | |
| dprince | d0ugal: I totally support that | 12:57 |
| d0ugal | At the moment it walks this weird line between being everything and nothing | 12:57 |
| dprince | d0ugal: but the functions we are debating here I think should probably stay in tripleo-common then | 12:57 |
| d0ugal | dprince: Agreed. | 12:58 |
| *** ealcaniz has quit IRC | 12:58 | |
| *** ansmith has quit IRC | 12:58 | |
| d0ugal | Moving everything out of tripleo-common sounds like hard work :) | 12:59 |
| d0ugal | and I don't really know where we draw the line, so I am not sure it will ever become clearer | 12:59 |
| dprince | d0ugal: I didn't say it had to happen. But in the meantime I don't think it is that bad to reuse some code from tripleo_common.utils externally either | 12:59 |
| d0ugal | dprince: yeah, I guess. I don't like it but I don't have a better solution for you today. | 12:59 |
| *** matbu|halfpto is now known as matbu | 13:00 | |
| dprince | d0ugal: I suppose we could document our limited use intent for the utils code in the readme | 13:00 |
| *** weshay_bbiab is now known as weshay | 13:01 | |
| d0ugal | dprince: it would be nice to rename "generate_overcloud_passwords" | 13:01 |
| d0ugal | (i.e. remove "overcloud") | 13:01 |
| dprince | d0ugal: I'd be happy to do that as part of this patch | 13:02 |
| *** morazi has joined #tripleo | 13:02 | |
| d0ugal | dprince: cool, that would clear the usage up a bit at least too | 13:02 |
| d0ugal | dprince: so you actually want SnmpdReadonlyUserPassword to be a random password? | 13:03 |
| d0ugal | because that is where it will be initially created? | 13:03 |
| dprince | d0ugal: the first time it is generated random is fine | 13:04 |
| dprince | d0ugal: thereafter this file would get loaded and it would get re-used | 13:04 |
| d0ugal | dprince: right, makes sense. | 13:04 |
| * d0ugal considers starting a larger question about what tripleo-common should be on openstack-dev | 13:05 | |
| *** fultonj has joined #tripleo | 13:05 | |
| openstackgerrit | Flavio Percoco proposed openstack/python-tripleoclient: Add container support to undercloud https://review.openstack.org/372671 | 13:05 |
| dprince | shardy: hey, I got one for you. Any ideas on how we might make deployed_server run firstboot scripts | 13:07 |
| EmilienM | flaper87: you got a sec to talk about https://review.openstack.org/#/c/372671/ ? | 13:07 |
| dprince | shardy: I'd like the capability to share docker initialization and perhaps OVS initialization across the overcloud/undercloud efforst | 13:07 |
| EmilienM | dprince: same for you ^ | 13:07 |
| openstackgerrit | Flavio Percoco proposed openstack/python-tripleoclient: Add container support to undercloud https://review.openstack.org/372671 | 13:08 |
| EmilienM | dprince, flaper87: before we go further, I would like to see a tripleo CI job similar to our "undercloud-only" job but with containers. It would be something like "undercloud-only-container" or something. | 13:08 |
| EmilienM | dprince, flaper87: and I wouldn't be in favor to review this patch until we have this job in place. | 13:09 |
| *** dmarlin_ has joined #tripleo | 13:09 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras: Add validation parameters to overcloud deploy https://review.openstack.org/404726 | 13:09 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras: Replace hardcoded stack user by ansible_user https://review.openstack.org/404800 | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 13:10 |
| flaper87 | EmilienM: yup | 13:10 |
| EmilienM | dprince, flaper87: I could help you to create the job ubt i'll need your help too | 13:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add hook to generate metadata from service profiles https://review.openstack.org/411339 | 13:11 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals https://review.openstack.org/411340 | 13:11 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Introduce role-specific nova-server-metadata https://review.openstack.org/410545 | 13:11 |
| dprince | EmilienM: ++ on the jobs. I think we need 2 of them | 13:11 |
| dprince | EmilienM: one for containers and one for baremetal | 13:11 |
| dprince | EmilienM: both will be useful, especially if we engage scenarios alongside of them I think | 13:12 |
| EmilienM | dprince: and it's super fast, the undercloud-only take 20 or 30 min max | 13:12 |
| EmilienM | so it's not a big deal to add one (it's single node) | 13:12 |
| dprince | flaper87: see my question to shardy above about reusing the firstboot scripts. You will likely care about my bootstrapping madness | 13:13 |
| dprince | EmilienM: fast is nice :) | 13:13 |
| flaper87 | dprince: need to read backlog | 13:13 |
| flaper87 | EmilienM: we're working on CI, it's not like it's not important it's just taken a bit more time because master was broken for containers | 13:13 |
| flaper87 | I agree we shouldn't merge this w/o ci | 13:14 |
| flaper87 | and I don't expect it to land before next year anyway | 13:14 |
| EmilienM | flaper87: I can bootstrap it on project-config | 13:14 |
| dprince | flaper87: because you said that I'm gonna land your patch right now | 13:14 |
| flaper87 | dprince: rofl | 13:15 |
| flaper87 | dprince: mine depends on yours | 13:15 |
| flaper87 | :D | 13:15 |
| dprince | flaper87: its fine. I'll land that too and just squash them into one big commit | 13:15 |
| *** rbrady-afk is now known as rbrady | 13:16 | |
| flaper87 | dprince: and ninja-approve all the things | 13:16 |
| flaper87 | EmilienM: so, I'm good with that, I just think it's too early to setup a CI for this undercloud thing | 13:16 |
| flaper87 | it's not going to land now, really | 13:16 |
| EmilienM | flaper87: what about experimental pipeline? | 13:16 |
| flaper87 | there are things still changing | 13:16 |
| flaper87 | EmilienM: that could work | 13:16 |
| EmilienM | but how do you test your code? | 13:17 |
| dprince | EmilienM: container aside. I'm game for a CI job for the baremetal version | 13:17 |
| EmilienM | dprince: it's good to be game for that, we need to actually do it :D | 13:17 |
| flaper87 | EmilienM: experimental sounds good if we want to kickstart something | 13:17 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add hook to generate metadata from service profiles https://review.openstack.org/411339 | 13:17 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals https://review.openstack.org/411340 | 13:17 |
| dprince | and I'm optimistic on the containers one too. Once we get the bootstrapping issues sorted out I think it'll come fairly quickly | 13:17 |
| EmilienM | dprince: I talked with weshay yesterday and the missing thing is the oooq patch in tripleo-ci so we can enable the container ci job | 13:17 |
| EmilienM | flaper87: I'll bootstrap something | 13:18 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: deployment updates for containerized compute https://review.openstack.org/400986 | 13:18 |
| flaper87 | EmilienM: cool, I'll use that to help adding more CIs | 13:18 |
| EmilienM | dprince: I'll be optimistic when I'll see it working in CI | 13:18 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates: WIP: Configure Kernel Args and Tuned and then reboot for Compute https://review.openstack.org/411797 | 13:19 |
| skramaja | shardy: ^ | 13:19 |
| skramaja | shardy: i am still testing, will update by next week... | 13:20 |
| *** jcoufal has joined #tripleo | 13:20 | |
| *** masco has quit IRC | 13:20 | |
| flaper87 | dprince: what exactly did you say to shardy ? | 13:21 |
| * flaper87 can't find the logs | 13:21 | |
| dprince | flaper87: the question was if he had any ideas about how we might use firstboot scripts (user_data) with deployed_server to bootstrap things like docker and OVS | 13:22 |
| dprince | flaper87: I would like to bootstrap these things consistently across the undercloud/overcloud efforts as we need them | 13:22 |
| dprince | flaper87: for the overcloud, now that we are using overcloud-full though we could use packages (or elements) for some of it I guess | 13:23 |
| jpena | just fyi: https://review.openstack.org/411800 should fix the tripleo CI, amoralej told me it was failing in https://review.openstack.org/411725 | 13:25 |
| *** pradk has joined #tripleo | 13:25 | |
| weshay | EmilienM, I'll ping panda and trown and see if we can get it merged today | 13:26 |
| EmilienM | weshay: merge what? all CI is broken now | 13:26 |
| weshay | ugh.. /me looks | 13:26 |
| panda | ... and it gets worse each day | 13:26 |
| weshay | ha still? | 13:27 |
| flaper87 | dprince: mmh, I'd rather not depend on things coming from overcloud-full. I don't think we do that now but just saying | 13:27 |
| flaper87 | dprince: also, +1 for consistency | 13:27 |
| *** dr_gogeta86 has joined #tripleo | 13:29 | |
| *** dr_gogeta86 has joined #tripleo | 13:29 | |
| *** fragatina has joined #tripleo | 13:29 | |
| *** pgadiya has quit IRC | 13:30 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals https://review.openstack.org/411340 | 13:31 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras: prep-network: add support to IPv6 topology https://review.openstack.org/400837 | 13:32 |
| *** jayg|g0n3 is now known as jayg | 13:32 | |
| *** rlandy has joined #tripleo | 13:33 | |
| panda | weshay: ha still yes, we have a workaround, but infra images were updated to 7.3 and they are not booting | 13:33 |
| shardy | dprince: Hey, sorry been grabbing lunch - can we configure the server so that the user_data gets put into a local datasource which cloud-init then reads? | 13:33 |
| dprince | shardy: perhaps, if that works | 13:34 |
| shardy | http://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html | 13:34 |
| dprince | shardy: I was thinking a bit more streamlined with having a mechanism to pass-thru the shell scripts into a Heat script element which runs via os-collect-config directly | 13:35 |
| shardy | something like that perhaps - I'm thinking we could use a SoftwareDeployment that runs very early to bootstrap the cloud-init data and re-run cloud-init? | 13:35 |
| shardy | dprince: Sure, that just won't work with e.g cloud-config yaml etc | 13:36 |
| shardy | we could have occ collect the user_data, write it, then run cloud-init via an o-r-c script tho I guess | 13:36 |
| dprince | shardy: yes, that might work | 13:37 |
| shardy | kind of a weird way to run cloud-init but it might work | 13:37 |
| openstackgerrit | Merged openstack/tripleo-validations: Don't rely on overcloudrc https://review.openstack.org/400800 | 13:37 |
| dprince | shardy: weird is okay w/ me | 13:37 |
| shardy | hehe :) | 13:37 |
| shardy | maybe we could juggle the service start ordering, so cloud-init actually runs after the o-r-c script | 13:38 |
| jaosorior | trown: hey, could you check this out https://review.openstack.org/#/c/401452/ ? | 13:38 |
| trown | jaosorior: sure | 13:39 |
| *** numans has quit IRC | 13:39 | |
| pradk | gfidente, Hi | 13:40 |
| pradk | gfidente, is there any more info you need from us regarding https://bugs.launchpad.net/tripleo/+bug/1646506 ? I did a fresh deploy yesterday with ceph and aodh/gnocchi works fine for me | 13:41 |
| openstack | Launchpad bug 1646506 in tripleo "Gnocchi / Aodh fail to work when RBD backend is enabled" [High,Confirmed] | 13:41 |
| pradk | gfidente, I'm wondering if this is a timing issue? | 13:41 |
| weshay | panda, ah ya.. saw that early this morning | 13:42 |
| *** numans has joined #tripleo | 13:42 | |
| pradk | gfidente, may be in this ci scenario we could try configuring ceph earlier ? | 13:43 |
| dprince | flaper87: the heat noauth middleware patch landed. So that is one less patch :) | 13:43 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras: Install gcc to use upstream PXE https://review.openstack.org/410293 | 13:44 |
| *** jpena is now known as jpena|lunch | 13:45 | |
| *** amoralej is now known as amoralej|lunch | 13:47 | |
| flaper87 | dprince: yeah, noticed this morning | 13:47 |
| *** rhallisey has joined #tripleo | 13:49 | |
| *** lucas-hungry is now known as lucasagomes | 13:49 | |
| *** fultonj has quit IRC | 13:49 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: deployment updates for containerized compute https://review.openstack.org/400986 | 13:52 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: add the containers prep role to the quickstart-extras https://review.openstack.org/400983 | 13:52 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras: Create tasks to install, run stackviz and collect logs https://review.openstack.org/400782 | 13:55 |
| *** fultonj has joined #tripleo | 13:56 | |
| *** dsneddon has quit IRC | 13:57 | |
| *** [1]cdearborn has joined #tripleo | 13:59 | |
| *** ealcaniz has joined #tripleo | 13:59 | |
| *** chlong has quit IRC | 14:00 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras: Baremetal undecloud role playbook fixes https://review.openstack.org/411336 | 14:00 |
| *** sudipto has quit IRC | 14:01 | |
| *** numans has quit IRC | 14:02 | |
| *** numans has joined #tripleo | 14:06 | |
| *** Goneri has joined #tripleo | 14:07 | |
| *** Vijayendra_ has quit IRC | 14:07 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 14:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 14:10 |
| *** Vijayendra_ has joined #tripleo | 14:15 | |
| *** almondjoy has joined #tripleo | 14:15 | |
| *** zoliXXL is now known as zoli|brb | 14:16 | |
| *** almondjoy has quit IRC | 14:19 | |
| *** almondjoy has joined #tripleo | 14:20 | |
| *** trown is now known as trown|brb | 14:22 | |
| *** bogdando has quit IRC | 14:23 | |
| thrash | d0ugal: fwiw, tripleo-workflows would be just fine with me. :) | 14:23 |
| EmilienM | rook: ^ see the 2 alerts | 14:24 |
| rook | at 0710 | 14:24 |
| rook | looking | 14:24 |
| *** ealcaniz has quit IRC | 14:27 | |
| *** Vijayendra_ has quit IRC | 14:27 | |
| social | anyone deploying rdo master and able to do introspection? | 14:27 |
| *** tzumainn has joined #tripleo | 14:27 | |
| jaosorior | derekh: any idea what's upw itht his http://logs.openstack.org/14/411514/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/3136b48/console.html#_2016-12-16_14_29_04_708997 ? | 14:30 |
| jaosorior | seems commits now are failing in seconds O_o | 14:30 |
| d0ugal | thrash: interesting, thanks - maybe I'll propose that. I do think it came up in the past. /cc rbrady | 14:30 |
| derekh | jaosorior: trying to find it at the moment | 14:30 |
| derekh | jaosorior: thats a good way to clear our queue ;-) | 14:31 |
| jaosorior | hahaha well, true that | 14:31 |
| *** lblanchard has joined #tripleo | 14:31 | |
| *** prateek has quit IRC | 14:31 | |
| derekh | jaosorior: found it, one sec | 14:34 |
| derekh | jaosorior: I was beaten to it https://review.openstack.org/#/c/411800/ | 14:36 |
| *** amoralej|lunch is now known as amoralej | 14:36 | |
| *** prateek has joined #tripleo | 14:37 | |
| matbu | shardy: the CI is broken ? | 14:37 |
| *** trown|brb is now known as trown | 14:37 | |
| matbu | i was looking at your changes | 14:37 |
| *** prateek has quit IRC | 14:38 | |
| *** prateek has joined #tripleo | 14:39 | |
| shardy | matbu: Yeah, ovb was already broken but now the multinode job seems broken too :( | 14:39 |
| shardy | http://tripleo.org/cistatus.html | 14:39 |
| shardy | Not figured out why yet | 14:39 |
| EmilienM | shardy: I'm looking at it now | 14:39 |
| derekh | Ok, first order of business, recheck this https://review.openstack.org/#/c/411514/ as soon as this lands https://review.openstack.org/#/c/411800/ | 14:39 |
| derekh | then we're ok, right? | 14:40 |
| EmilienM | it's in puppet catalog | 14:40 |
| EmilienM | derekh: yes | 14:41 |
| EmilienM | we should | 14:41 |
| EmilienM | pradk, amoralej, jpena|lunch: http://logs.openstack.org/87/411687/2/check/gate-tripleo-ci-centos-7-nonha-multinode/25c5d50/logs/var/log/undercloud_install.txt.gz#_2016-12-16_08_57_01_000 | 14:41 |
| shardy | derekh: Yeah that's one issue, but there's also a problem with the undercloud install now | 14:41 |
| EmilienM | amoralej: is it the thing you're trying to fix with yum update? | 14:41 |
| EmilienM | shardy: the multinode problem is a 7.3 thing again | 14:41 |
| amoralej | https://review.openstack.org/#/c/411725/ | 14:41 |
| derekh | bummer | 14:41 |
| amoralej | yes | 14:41 |
| EmilienM | shardy: we need to run yum update before running puppet | 14:41 |
| shardy | EmilienM: :( | 14:42 |
| EmilienM | derekh: https://review.openstack.org/#/c/411725/ too | 14:42 |
| *** Vijayendra has joined #tripleo | 14:42 | |
| shardy | EmilienM: Ok, I guess that should be easy enough for multinode | 14:42 |
| amoralej | but we need to get https://review.openstack.org/#/c/411800/ | 14:42 |
| openstackgerrit | Andrey Shestakov proposed openstack/diskimage-builder: Fix dhcp-all-interfaces for ubuntu-minimal xenial https://review.openstack.org/407725 | 14:42 |
| *** tremble has quit IRC | 14:42 | |
| amoralej | i can take a more conservative approach and update only mariadb-libs until new package is provided | 14:43 |
| pradk | EmilienM, yep i'm hitting the same locally on centos 7.3 | 14:43 |
| amoralej | mariadb-libs issue pradk? | 14:43 |
| amoralej | using tripleo.sh? | 14:43 |
| pradk | yea packaging file conflict | 14:43 |
| pradk | yep | 14:43 |
| *** jeckersb_gone is now known as jeckersb | 14:43 | |
| EmilienM | it sounds like we need amoralej's patch first to fix multinode jobs (voting) | 14:43 |
| EmilienM | the other ones are for ovb | 14:44 |
| EmilienM | (non-voting) | 14:44 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: pin ansible-lint at v3.4.7 until we resolve issues with 3.4.8 https://review.openstack.org/411843 | 14:44 |
| EmilienM | amoralej: I'm doing recheck. Can you address trown's comment by patching doc too (later when you can) | 14:45 |
| amoralej | i'm on it | 14:45 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Poll nodes list while nodes are in progress state https://review.openstack.org/395637 | 14:45 |
| *** tosky has quit IRC | 14:45 | |
| amoralej | but i think that's not the right bug... | 14:46 |
| *** tremble has joined #tripleo | 14:46 | |
| *** tremble has quit IRC | 14:46 | |
| *** tremble has joined #tripleo | 14:46 | |
| EmilienM | amoralej: you mean it won't help to fix the mariadb thing? | 14:46 |
| trown | amoralej: it is a different symptom of the same problem | 14:46 |
| trown | amoralej: or to put it differently, the solution to the mariadb thing also fixes that bug... | 14:47 |
| amoralej | i'm not sure, without more logs.... | 14:47 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: pin ansible-lint at v3.4.7 until we resolve issues with 3.4.8 https://review.openstack.org/411843 | 14:47 |
| *** bogdando has joined #tripleo | 14:47 | |
| amoralej | yes, i understand the point | 14:47 |
| *** liverpooler has joined #tripleo | 14:47 | |
| amoralej | but i trust you trown :) | 14:47 |
| trown | :), the bug I pointed to is a differenent side effect of 7.3 that only effects trying to deploy with a 7.2 image | 14:48 |
| trown | but it would also be solved by updating packages earlier | 14:48 |
| amoralej | trown, i'll put it as related-bug ok? | 14:49 |
| trown | amoralej: sure that makes sense | 14:49 |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci: Update packages after configuring openstack repos https://review.openstack.org/411725 | 14:50 |
| *** noslzzp has quit IRC | 14:51 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 14:51 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Add basic pre/post upgrade sanity checks https://review.openstack.org/411846 | 14:51 |
| shardy | matbu: ^^ FYI I added some basic smoke tests since we can't yet run the pingtest | 14:51 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Add basic pre/post upgrade sanity checks https://review.openstack.org/411846 | 14:52 |
| matbu | shardy: nice /me looks | 14:53 |
| *** flepied has quit IRC | 14:53 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Fix blank overcloud credentials https://review.openstack.org/408138 | 14:54 |
| *** Vijayendra has quit IRC | 14:54 | |
| EmilienM | shardy: your stuff is cool for smoke test but we already did something similar for undercloud smoke test. Could we maybe try to use the same things? (fwiw I prefer your code now) | 14:55 |
| marios | matbu: have you come across this before " "ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.\n" - it failed on step0 but not how i wanted :) (I put the cluster down again). at least this time the ansible is running, but seems to be a nit with one of the tasks i added in step0 | 14:56 |
| EmilienM | shardy: maybe we could have some bash functions for smoke test and re-use them for undercloud-only job and overcloud upgrade job | 14:57 |
| matbu | marios: yep, probablye because the module is not well configure | 14:57 |
| *** paramite_ has quit IRC | 14:57 | |
| matbu | marios: check your /etc/ansible/ansible.cfg (the line library=) | 14:57 |
| *** paramite has quit IRC | 14:58 | |
| matbu | marios: then make sure the modules are in the specified path | 14:58 |
| *** jpena|lunch is now known as jpena | 14:58 | |
| marios | matbu: ah you mean you suspect it might be the pacemaker_cluster.py/resource from the new module... k will check thanks | 14:58 |
| jrist | jtomasek, florianf - so the ids should be what format? for automation? | 14:58 |
| matbu | marios: yes at 99% :) | 14:58 |
| jrist | jtomasek, florianf - you're not happy with underscore namespaced? | 14:59 |
| marios | matbu: heh cool thanks man | 14:59 |
| *** milan|afk has quit IRC | 14:59 | |
| *** coolsvap has quit IRC | 14:59 | |
| *** milan has joined #tripleo | 15:00 | |
| florianf | jrist: I'd prefer if the component name would be part of the id, potentially separated by a dot. That would be similar to how the namespacing is done with i18n. | 15:00 |
| jrist | ah | 15:00 |
| florianf | jrist: AFAIK dots are valid characters in id attributes. | 15:01 |
| jrist | ok | 15:01 |
| jrist | this is new to me :) | 15:01 |
| shardy | EmilienM: definitely, I'll look at refactoring so they use the same code | 15:01 |
| EmilienM | shardy: nothing urgent, just fyi. I can take this task, as I wrote this undercloud check thing | 15:01 |
| shardy | EmilienM: Cool, yeah I just wanted to keep this separate initially, as I was thinking there's some chance we'll delete this code when we get the pingtest working before/after upgrades | 15:02 |
| shardy | but obviously we can keep it if it's still useful | 15:02 |
| shardy | EmilienM: I'll add sanity checks for the other services so we can prove the upgrade patches, then we can look at refactoring, if that's OK with you? | 15:03 |
| marios | matbu: so we have a chicken/egg situation i mean we need to install those ansible modules as the (new) upgrade init shardy moved it, so it happens before the steps run | 15:04 |
| marios | matbu: ie.. in upgrade init we can copy over those ansible modules from your review to the nodes | 15:05 |
| marios | matbu: (for now/testing ) | 15:05 |
| marios | shardy: i just noticed/saw /var/lib/heat-config/heat-config-ansible/3b0f9dc7-0a9f-4a8a-99bb-7c0e5ad957b8_playbook.yaml for first time pretty cool to see it composed :) | 15:05 |
| matbu | marios: hm right, but we would need also to install the heat ansible hook | 15:05 |
| matbu | marios: so i'm wondering how we can put that on the nodes | 15:06 |
| marios | shardy: well actually, its not composed... its the steps which give the order and they are done via the dependencies | 15:06 |
| matbu | marios: it could also be pushed by heat itself | 15:07 |
| marios | shardy: so we'd need to process that to get it intoan execution order right i mean to run it independently (I know that is low priority right now) | 15:07 |
| *** trown is now known as trown|afk | 15:08 | |
| marios | matbu: yeah i believe the idea of shardy moving the upgrade init was so we can do things like that (the heat-ansible-*) | 15:08 |
| marios | matbu: well actually it was for installing the new hiera hook | 15:08 |
| marios | matbu: but same applies so we could deliver those ansible modules there. | 15:09 |
| matbu | marios: yep | 15:09 |
| *** flepied has joined #tripleo | 15:09 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 15:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals https://review.openstack.org/411340 | 15:10 |
| *** paramite_ has joined #tripleo | 15:12 | |
| shardy | marios, matbu: Yeah I *had* to move it because of the hiera hook, but we can add other pre-upgrade dependencies in there too | 15:12 |
| EmilienM | shardy: yes | 15:13 |
| shardy | DeployArtifactURLs can also be used to deliver things like ansible modules if needed | 15:13 |
| shardy | marios: re the steps, yeah we'd have to generate another playbook which runs the one heat drives step by step | 15:13 |
| shardy | marios: that would be pretty easy, but I've not done it yet | 15:13 |
| *** b00tcat has quit IRC | 15:13 | |
| shardy | I have driven the steps manually via ansible-playbook on the CLI tho, works well | 15:14 |
| shardy | combined with the tripleo-validations dynamic inventory | 15:14 |
| *** tremble has quit IRC | 15:15 | |
| shardy | For now I just see it as a useful-for-debugging thing tho, at least until we have the full heat-driven upgrade working | 15:15 |
| *** jaosorior has quit IRC | 15:15 | |
| *** derekh has quit IRC | 15:15 | |
| *** tremble has joined #tripleo | 15:15 | |
| *** paramite has joined #tripleo | 15:16 | |
| *** b00tcat has joined #tripleo | 15:16 | |
| *** b00tcat has quit IRC | 15:16 | |
| *** b00tcat has joined #tripleo | 15:16 | |
| marios | shardy: yeah well we can use it to deliver matbu modules from https://review.openstack.org/#/c/403397/11/ansible/library/pacemaker_cluster.py and _resource.py as one immediate eexample... we can put those into the default module path whatever that is matbu is there such a location? | 15:16 |
| marios | shardy: i rebased my step0 onto yours since the init is gone now... i made it part of the main loop and started at 0 https://review.openstack.org/#/c/408631/5/puppet/major_upgrade_steps.j2.yaml | 15:16 |
| d0ugal | thrash, toure, rbrady: I do think I agree with jtomasek's comment on https://review.openstack.org/#/c/404736/ | 15:17 |
| d0ugal | but I need to think about it a bit more. | 15:17 |
| *** prateek has quit IRC | 15:17 | |
| marios | shardy: it got me thinking how we might be able to get extra steps in... something like ExtraSteps param e.g. [2.21, 3.11, 3.12, 4.1, 5.6] ... i was trying to come up with a way of dynamically including those in that loop | 15:17 |
| shardy | marios: the problem is we don't know the steps defined in the templates when j2 runs | 15:18 |
| marios | shardy: e.g. make the 2.21 depends on the 2 etc ... | 15:18 |
| shardy | we'd have to use e.g ResourceChain instead | 15:18 |
| marios | shardy: yeah the idea is that you add a new step | 15:18 |
| *** zoli|brb is now known as zoli | 15:18 | |
| marios | shardy: liek i just added 1.1 and then you would add that into the extrasteps... so it would be hardwired | 15:18 |
| shardy | which I did consider, but this seemed cleaner | 15:18 |
| *** zoli is now known as zoliXXL | 15:18 | |
| rbrady | d0ugal: I don't see jtomasek's comment on that review | 15:19 |
| shardy | marios: Yeah, possibly - I'll give some thought to how we might do that | 15:19 |
| marios | shardy: but still i couldnt' come up with a way of including that in the for loop (e.g. added 1.1 so it would then make it depend on 1 etc etc ) | 15:19 |
| shardy | marios: FWIW I went with this approach because it's simple, and it should hopefully tend to discourage adding $many additional steps, as there's obviously cost associated with each one | 15:20 |
| d0ugal | jtomasek, thrash, toure, rbrady: sorry, I meant https://review.openstack.org/#/c/405531/ | 15:20 |
| marios | shardy: the use case was for things like migrations | 15:20 |
| marios | shardy: which it may depend at which point we want to run it | 15:20 |
| shardy | marios: Yeah, you can't do it via a heat parameter, we'd need to add it as some input data to jinja2 with the current model | 15:20 |
| shardy | or, completely rework how the steps are run (which might be possible, but I'd rather not do that right now) | 15:21 |
| *** chlong has joined #tripleo | 15:21 | |
| marios | shardy: heh no lets not do that | 15:22 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: add collectd composable service https://review.openstack.org/411048 | 15:22 |
| shardy | marios: if we wanted to get really fancy we'd do some sort of preview create that gave us the flattened list of tags without creating the stack, then feed those into j2 as the steps | 15:23 |
| * shardy will think more about it | 15:23 | |
| marios | shardy: if we got that we could even go as far as getting the tasks out too right I mean for runinng stand-alone (well subject to processing) | 15:23 |
| marios | shardy: but i have no idea how we would do that shardy :) | 15:24 |
| marios | shardy: i mean the preview create | 15:24 |
| shardy | The thing I like about the current implementation is it's simple and easy to understand (relatively ;) | 15:24 |
| marios | shardy: could we add all the upgrade tasks to the output for the service? is that sane | 15:24 |
| *** dtantsur is now known as dtantsur|afk | 15:24 | |
| shardy | marios: They are already output from all the services, and combined via the ResourceChains - we'd just have to add an output for each role's playbook of tasks to overcloud.j2.yaml | 15:26 |
| marios | shardy: sorry thats actually where their defined... i mean some way of extracting them without running them | 15:26 |
| *** jeckersb is now known as jeckersb_gone | 15:26 | |
| shardy | I already did exactly that for testing | 15:26 |
| shardy | so the missing part is seeing if stack-preview would actually give us that data | 15:26 |
| shardy | I'll try it | 15:26 |
| *** abehl has quit IRC | 15:27 | |
| shardy | but relative to all the other work we have to do, this is probably a lower priority? | 15:27 |
| *** sanagikoki has joined #tripleo | 15:27 | |
| shardy | e.g if we have to finesse the steps a bit for ocata, it's still *way* more hackable than the bash script approach :) | 15:27 |
| *** jeckersb_gone is now known as jeckersb | 15:30 | |
| shardy | marios: also, for each service, isn't the task listing enough to serialize e.g a migration task with some other thing, even when they're the same step? | 15:30 |
| shardy | I thought ansible walked the talks in order, but maybe I'm mistaken | 15:30 |
| shardy | s/talks/tasks/ | 15:31 |
| *** flepied has quit IRC | 15:32 | |
| *** brault has joined #tripleo | 15:37 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo: add support for collectd https://review.openstack.org/411047 | 15:41 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Add Octavia API service definitions https://review.openstack.org/411872 | 15:42 |
| *** hjensas has quit IRC | 15:44 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Introduce Octavia implememtation services https://review.openstack.org/411874 | 15:46 |
| *** tobias-fiberdata has quit IRC | 15:46 | |
| *** dsneddon has joined #tripleo | 15:46 | |
| *** hjensas has joined #tripleo | 15:47 | |
| EmilienM | amoralej: your patch is still failing, /me reading logs | 15:48 |
| amoralej | i was checking it also, something configuring ovs | 15:48 |
| EmilienM | http://logs.openstack.org/25/411725/4/check/gate-tripleo-ci-centos-7-nonha-multinode/6260e1c/console.html#_2016-12-16_15_34_26_358151 | 15:49 |
| *** flepied has joined #tripleo | 15:49 | |
| EmilienM | it looks like undercloud can't ping overcloud | 15:49 |
| EmilienM | switching on infra channel | 15:50 |
| EmilienM | #openstack-infra | 15:50 |
| *** dsneddon has quit IRC | 15:51 | |
| EmilienM | bnemec, derekh, have you seen that before? ^ | 15:51 |
| *** hjensas has quit IRC | 15:54 | |
| EmilienM | I think all multinodes in OpenStack Infra are now broken | 15:55 |
| EmilienM | or at least in tripleo | 15:55 |
| marios | shardy: sorry was on scrum, reading back | 15:55 |
| openstackgerrit | Martin André proposed openstack/tripleo-docs: Fix OpenStack client invocation https://review.openstack.org/411878 | 15:56 |
| marios | shardy: oh i see so yeah if we can explicitly define the order like that with same step number but defined in series that would be great | 15:56 |
| *** arxcruz has quit IRC | 15:56 | |
| *** bana_k has joined #tripleo | 15:57 | |
| EmilienM | amoralej: I'm doing recheck for now, since we fixed the " issue in JJB | 15:57 |
| amoralej | last run was after fixing " , but go ahead | 15:58 |
| shardy | marios: ack - OK lets try that, and btw the preview approach does also work | 15:58 |
| shardy | http://paste.openstack.org/show/592653/ | 15:58 |
| shardy | so we have that as an option if we need it | 15:58 |
| pradk | yum update does seem to resolve mariadb issue, thx for the tip | 15:58 |
| pradk | amoralej, ^^ | 15:58 |
| amoralej | cool | 15:58 |
| *** bana_k has quit IRC | 15:59 | |
| marios | shardy: awesome with a bit of login in tripleo-common we could parse that and put thm into step order too | 15:59 |
| marios | shardy: s/login/logic | 15:59 |
| shardy | marios: Yep we probably could | 16:00 |
| EmilienM | amoralej: the patch is maybe merged but not applied on all zuul nodes | 16:00 |
| EmilienM | amoralej: puppet runs periodically | 16:00 |
| amoralej | it'd have given a different error EmilienM | 16:00 |
| amoralej | i'm sure it was merged | 16:00 |
| *** yprokule has quit IRC | 16:01 | |
| EmilienM | amoralej: the ovs issue is new now | 16:01 |
| amoralej | yep | 16:01 |
| bnemec | Of course it is. | 16:02 |
| bnemec | This week. Good grief. | 16:02 |
| *** Guest31304 has quit IRC | 16:02 | |
| *** aufi has quit IRC | 16:02 | |
| yolanda | hi, i tried upgrading tripleoclient and i'm getting an error: | 16:03 |
| yolanda | Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL. | 16:03 |
| yolanda | SSL exception connecting to https://192.168.24.2:13000/v2.0/tokens: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",) | 16:03 |
| yolanda | anyone knows how to skip it? | 16:03 |
| bnemec | yolanda: Sounds like an ssl cert wasn't installed correctly for some reason. | 16:06 |
| bnemec | How did you configure ssl on the undercloud? | 16:06 |
| yolanda | bnemec, so what i did is to deploy an undercloud with ooq. But then, i upgraded tripleoclient to consume latest patchsets, and wanted to redeploy overcloud again | 16:06 |
| *** saneax is now known as saneax-_-|AFK | 16:07 | |
| yolanda | the initial deploy worked, it started to fail as soon as i upgraded tripleoclient | 16:07 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-quickstart: Create directories with root https://review.openstack.org/384892 | 16:09 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 16:10 |
| EmilienM | we should add ovs now ^ | 16:10 |
| EmilienM | I'm filling a bug | 16:10 |
| *** mcornea has quit IRC | 16:10 | |
| yolanda | bnemec, i remember that this was something related with requests | 16:11 |
| yolanda | but i don't remember the way to disable that ssl check, i just want to do a test uploading images | 16:11 |
| bnemec | yolanda: I'm not sure. You could look for a --insecure option or something in osc. | 16:12 |
| yolanda | i exported OS_INSECURE=true, but no luck | 16:14 |
| yolanda | i will try redeploying and instead of upgrading with pip, just add the bits in tripleoclient i need... | 16:14 |
| yolanda | with the tripleoclient upgrade , there were lots of dependencies upgrades as well, so there may be conflicts | 16:15 |
| *** udesale has quit IRC | 16:16 | |
| EmilienM | https://bugs.launchpad.net/tripleo/+bug/1650612 | 16:17 |
| openstack | Launchpad bug 1650612 in tripleo "nodepool subnodes can't ping each others" [Critical,Triaged] | 16:17 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: Add doc templating to validate-tempest role https://review.openstack.org/411888 | 16:19 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: [WIP] - Add basic documentation for overcloud-upgrade role https://review.openstack.org/409328 | 16:21 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Revert "Fix pip-and-virtualenv to work with python3" https://review.openstack.org/411892 | 16:22 |
| amoralej | Same error in recheck EmilienM | 16:23 |
| *** rcernin has quit IRC | 16:23 | |
| EmilienM | damn | 16:23 |
| *** pkovar has quit IRC | 16:24 | |
| EmilienM | amoralej: we have no way to debug easily, I'm about to ask access to one node on #openstack-infra | 16:25 |
| *** pcaruana has quit IRC | 16:25 | |
| amoralej | i'm trying to understand how this multinode jobs work | 16:25 |
| amoralej | ok | 16:25 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ubuntu-precise support to dib-python https://review.openstack.org/411898 | 16:27 |
| *** chlong has quit IRC | 16:29 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-puppet-elements: Octavia integration https://review.openstack.org/411902 | 16:32 |
| bnemec | EmilienM: I wonder if something changed in the firewall on 7.3. We seem to be having a lot of weird connectivity issues with stuff that used to work. | 16:32 |
| EmilienM | bnemec: see on infra channel, it looks like transient | 16:33 |
| EmilienM | bnemec: I see a lot of jobs passing now | 16:34 |
| EmilienM | I'll do recheck again on amoralej's patch | 16:34 |
| amoralej | hopefully | 16:34 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo: add support for collectd https://review.openstack.org/411047 | 16:37 |
| *** cwolferh has joined #tripleo | 16:37 | |
| bnemec | EmilienM: slagle used to have a multinode dev environment on one of the rh clouds. Maybe we could set one up with 7.3 and see if we can reproduce these problems? | 16:38 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common: Heat-agents container: set up trunk repos correctly https://review.openstack.org/411908 | 16:38 |
| EmilienM | bnemec: yeah we could, jeblair also proposed to give access to the infra resource | 16:39 |
| EmilienM | bnemec: let me see if I can get it first otherwise, let's use slagle's env | 16:40 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Update packages after configuring openstack repos https://review.openstack.org/411725 | 16:40 |
| EmilienM | amoralej: just kickoff a new CI job ^ by changing commit messagfe | 16:41 |
| amoralej | yes, i see, i have an alternative in which i only update mariadb-libs if we think that's the issue | 16:42 |
| bnemec | What mariadb-libs problem are we fixing here? Jobs are passing without this workaround. | 16:43 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Increase func testing for ubuntu-minimal element https://review.openstack.org/411910 | 16:43 |
| amoralej | bnemec, there is a conflict between the mariadb-libs package in CentOS 7.3 and the one in RDO repos | 16:44 |
| amoralej | i've seeing jobs failing because of that issue | 16:44 |
| bnemec | amoralej: How are jobs passing then? | 16:44 |
| * amoralej checking logs | 16:44 | |
| amoralej | it can pass if it's updated before trying to deploy mariadb-server or if mariadb-libs are not in the base image | 16:45 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common: Heat-agents container: set up trunk repos correctly https://review.openstack.org/411908 | 16:47 |
| bnemec | Which should always be the case. This kind of thing is exactly why we have a forced yum update early in the undercloud install. | 16:47 |
| *** rhallisey has quit IRC | 16:47 | |
| EmilienM | https://review.openstack.org/#/c/411340/ is a good example of multinode job working fine now | 16:48 |
| *** akrivoka has quit IRC | 16:48 | |
| *** akrivoka has joined #tripleo | 16:48 | |
| EmilienM | do we really need this workaround? | 16:48 |
| *** tremble has quit IRC | 16:50 | |
| *** chlong has joined #tripleo | 16:53 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Add Octavia API service definitions https://review.openstack.org/411872 | 16:53 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Introduce Octavia implememtation services https://review.openstack.org/411874 | 16:54 |
| *** b00tcat has quit IRC | 16:55 | |
| EmilienM | bnemec: 411514 is about to move in gate soon :D | 16:57 |
| bnemec | EmilienM: As soon as the stupid non-voting job finishes. :-) | 16:58 |
| bnemec | Of course, it's been there before so I'm not counting my chickens here. | 16:58 |
| * EmilienM googles "counting chickens" | 16:59 | |
| *** lmiccini has quit IRC | 16:59 | |
| *** b00tcat has joined #tripleo | 16:59 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: [WIP] - Add basic documentation for overcloud-upgrade role https://review.openstack.org/409328 | 17:00 |
| bnemec | This is less than promising: http://logs.openstack.org/33/411733/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/930da6b/console.html#_2016-12-16_16_40_27_440521 | 17:01 |
| EmilienM | bnemec: social reported an issue with introspection this morning | 17:02 |
| EmilienM | social: have you found out? | 17:02 |
| rlandy | dprince: https://bugs.launchpad.net/heat-templates/+bug/1650625 ... what are your thoughts? currently we have a failing bond-with-vlans job in the promotion pipeline and a rejected proposed review to fix it. | 17:02 |
| openstack | Launchpad bug 1650625 in Heat Templates "Relative path to run-os-net-config will fail for custom nic-configs saved in a different directory" [Undecided,New] | 17:02 |
| EmilienM | bnemec: I would be surprises it comes from upstream since we didn't promote tripleo CI for 9 days | 17:03 |
| amoralej | EmilienM, bnemec, the reason why some jobs are passing is because they are still using 7.2 image | 17:03 |
| * bnemec is concerned that we are stuck in quicksand and the more we fight it the the more it sucks us down. | 17:04 | |
| *** rcernin has joined #tripleo | 17:04 | |
| amoralej | weird, we have different images in different jobs | 17:05 |
| EmilienM | let's see what cloud providers have what | 17:06 |
| amoralej | osic is 7.2 | 17:07 |
| EmilienM | amoralej: where do you see the centos version? | 17:07 |
| amoralej | kernel version | 17:07 |
| amoralej | http://logs.openstack.org/40/411340/8/check/gate-tripleo-ci-centos-7-nonha-multinode/966cbe7/console.html#_2016-12-16_15_13_13_099677 | 17:07 |
| amoralej | 3.10.0-327 is 7.2 | 17:07 |
| amoralej | http://logs.openstack.org/09/409809/1/gate/gate-tripleo-ci-centos-7-nonha-multinode/43ed9c4/console.html#_2016-12-16_08_20_53_132555 | 17:08 |
| amoralej | 7.3 | 17:08 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650612 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1650612 in tripleo "nodepool subnodes can't ping each others" [Critical,Triaged] | 17:10 |
| *** jpena is now known as jpena|off | 17:13 | |
| *** panda is now known as panda|off | 17:14 | |
| *** zoliXXL is now known as zoli|gone | 17:15 | |
| *** yamahata has joined #tripleo | 17:17 | |
| *** saibarspeis has quit IRC | 17:19 | |
| *** arxcruz has joined #tripleo | 17:20 | |
| *** saibarspeis has joined #tripleo | 17:20 | |
| *** athomas has quit IRC | 17:20 | |
| *** bana_k has joined #tripleo | 17:25 | |
| EmilienM | amoralej: sounds like it should be synced now | 17:27 |
| openstackgerrit | Merged openstack/diskimage-builder: Add ubuntu-precise support to dib-python https://review.openstack.org/411898 | 17:27 |
| amoralej | anyway, i think we should merge the fix mariadb-libs to get ready for when it cames back EmilienM | 17:27 |
| EmilienM | amoralej: we'll see how CI works on this one | 17:28 |
| amoralej | if it fails, i'll send an update to update only mariadb-libs, i'm pretty sure it will work | 17:28 |
| EmilienM | ok | 17:28 |
| *** akrivoka has quit IRC | 17:29 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates: WIP: Add custom role for realtime compute https://review.openstack.org/411925 | 17:30 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Start func testing on centos-minimal again https://review.openstack.org/411926 | 17:31 |
| mwhahaha | do we have abug for the bariadb-libs thing yet? | 17:31 |
| *** dsneddon has joined #tripleo | 17:35 | |
| *** ayoung has quit IRC | 17:35 | |
| *** hewbrocca is now known as hewbrocca_afk | 17:36 | |
| rasca | guys my deploy on master just failed with "openstack overcloud deploy: error: unrecognized arguments: --neutron-bridge-mappings" with previous versions this always worked. When this was changed? | 17:36 |
| bnemec | rasca: That's been deprecated for multiple releases and was removed in https://github.com/openstack/python-tripleoclient/commit/3ab40d4ea4b320bcfe85bc1f3c1ded987c65e5d0 | 17:39 |
| bnemec | You need to pass those parameters in an env file now. | 17:39 |
| *** dsneddon has quit IRC | 17:40 | |
| rasca | bnemec, so it is sufficient to use NeutronBridgeMappings: "datacentre:br-ex,floating:br-floating" in my env? | 17:40 |
| bnemec | rasca: I believe so. | 17:40 |
| EmilienM | mwhahaha: not afik | 17:42 |
| rasca | bnemec, ok, many thanks, I'll give it a try, but you think this will be retro compatible? | 17:42 |
| rasca | bnemec, I mean if I don't declare the option in newton or mitaka will it work? | 17:42 |
| rasca | (I know it sounds like a silly question) | 17:42 |
| bnemec | rasca: You've been able to define these in an env file at least since they were deprecated in liberty, so it should work fine. | 17:43 |
| rasca | bnemec, ok, I will test it and let you know, many thanks Ben | 17:43 |
| bnemec | The cli params were a mistake in the kilo-based downstream release that tripleoclient originated in. | 17:43 |
| rasca | that's good to know. | 17:43 |
| rasca | bnemec, and it also answer a question I made to shardy and EmilienM a week ago | 17:44 |
| *** milan has quit IRC | 17:44 | |
| rasca | but I need to test it first | 17:44 |
| * EmilienM afk lunch break | 17:44 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-common: Don't require mistralclient for password gen https://review.openstack.org/410970 | 17:45 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Start func testing on centos-minimal again https://review.openstack.org/411926 | 17:49 |
| *** [1]cdearborn has quit IRC | 17:50 | |
| *** rhallisey has joined #tripleo | 17:57 | |
| *** dsneddon has joined #tripleo | 17:58 | |
| *** dprince has quit IRC | 18:02 | |
| *** dsneddon has quit IRC | 18:02 | |
| *** ohamada_ has quit IRC | 18:07 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650612 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1650612 in tripleo "nodepool subnodes can't ping each others" [Critical,Triaged] | 18:10 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Start func testing on centos-minimal again https://review.openstack.org/411926 | 18:16 |
| *** flepied has quit IRC | 18:18 | |
| EmilienM | bnemec: I see quite a lot of multinode jobs passing :D http://tripleo.org/cistatus.html | 18:22 |
| amoralej | EmilienM, https://review.openstack.org/#/c/411725/ passed | 18:23 |
| *** lucasagomes is now known as lucas-afk | 18:23 | |
| bnemec | EmilienM: There's only one I really care about: telnet://146.20.105.108:19885 :-) | 18:23 |
| EmilienM | amoralej: do we actually need it? | 18:24 |
| EmilienM | bnemec: which one is that? | 18:24 |
| amoralej | we'll need it if 7.3 images are pushed before we fix mariadb | 18:25 |
| bnemec | EmilienM: The ovb workaround | 18:25 |
| EmilienM | amoralej: ok +2 then, bnemec you ok too? | 18:25 |
| bnemec | If that passes then it merges. If not, we waste another half day. | 18:25 |
| EmilienM | bnemec: ssl? | 18:25 |
| bnemec | Yeah | 18:25 |
| amoralej | if you prefer, i can test a less aggressive update to just fix mariadb-libs | 18:26 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/tripleo-quickstart: VirtualBMC support for tripleo-quickstart https://review.openstack.org/399704 | 18:27 |
| EmilienM | amoralej: it's ok for me as it is now | 18:27 |
| EmilienM | amoralej: I'll let bnemec approve it if he's fine | 18:27 |
| amoralej | ok | 18:28 |
| EmilienM | amoralej: did we report a bug in launchpad for this problem? | 18:28 |
| amoralej | Not sure | 18:28 |
| amoralej | lemme check | 18:28 |
| amoralej | i didn't | 18:28 |
| bnemec | I'm not crazy about merging a fix for a problem that may or may not happen depending on the state of packages when we move to 7.3. | 18:31 |
| *** trown|afk is now known as trown | 18:31 | |
| bnemec | It also needs to be documented if it ends up being required. | 18:31 |
| EmilienM | bnemec: it sounds that no matter 7.2 or 7.3, we need to get latest version of this package | 18:32 |
| bnemec | EmilienM: Not if jobs are passing right now. | 18:32 |
| amoralej | no EmilienM, it depends on the version of mariadb-libs in base image | 18:32 |
| EmilienM | bnemec: we could hold it for now, and use it if mariadb is not working anymore | 18:33 |
| amoralej | that's why we don't need it with 7.2 images | 18:33 |
| bnemec | I'm also still not sure why the yum update in the undercloud install doesn't take care of this. | 18:33 |
| EmilienM | it probably happens too late | 18:33 |
| EmilienM | let me check | 18:33 |
| openstackgerrit | Merged openstack/tripleo-quickstart: pin ansible-lint at v3.4.7 until we resolve issues with 3.4.8 https://review.openstack.org/411843 | 18:33 |
| amoralej | we need to have the RDO repos enabled | 18:33 |
| *** ctayal has joined #tripleo | 18:33 | |
| EmilienM | amoralej: it should be the case when puppet run yum update | 18:34 |
| EmilienM | I think it's an orchestration thing in the catalog | 18:34 |
| EmilienM | amoralej: see https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.pp#L26 | 18:34 |
| amoralej | we have a yum update in undercloud install? | 18:34 |
| amoralej | then we need to move that to the very beginning of the undercloud install, IMO | 18:35 |
| EmilienM | amoralej: yes, https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.pp#L39-L44 | 18:35 |
| amoralej | we are doint it at the end | 18:35 |
| amoralej | iiuc | 18:35 |
| amoralej | is this used for undercloud upgrades EmilienM? | 18:37 |
| amoralej | if so, it make sense to do it at the end | 18:37 |
| EmilienM | amoralej: no, everytime undercloud is deployed or updated | 18:37 |
| EmilienM | technically speaking, everytime puppet is run on the undercloud | 18:38 |
| amoralej | but for updates too, right?, mmm doing a full update at the beginning is not a good idea for update case | 18:38 |
| bnemec | Sigh, I'm pretty sure the undercloud ssl patch has hit the multinode hang bug. | 18:39 |
| trown | amoralej: EmilienM bnemec that is exactly the bug I filed yesterday | 18:39 |
| trown | namely that the undercloud install should happen before any service configuration | 18:40 |
| trown | err. undercloud upgrade rather | 18:40 |
| EmilienM | trown: yeah, I assigned it to myself I think, it's on my todo | 18:40 |
| trown | https://bugs.launchpad.net/tripleo/+bug/1650374 | 18:40 |
| openstack | Launchpad bug 1650374 in tripleo "[instack-undercloud] package upgrade should happen before service configuration" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 18:40 |
| bnemec | Umm, it does, doesn't it? | 18:40 |
| bnemec | Oh, this is a new install, not an upgrade. | 18:41 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 18:41 |
| trown | bnemec: it does in the file... but not when actually running puppet.,, there are no dependencies setup | 18:41 |
| EmilienM | bnemec: I'm wondering if we should run yum update before running puppet | 18:41 |
| *** dprince has joined #tripleo | 18:41 | |
| EmilienM | I've never been a fan of running yum in a exec from puppet :D | 18:41 |
| EmilienM | maybe we could just run that command just *before* running puppet | 18:41 |
| trown | if we move it out, we just need to maintain the option to not do it at all (which currently is wired in via hiera) | 18:42 |
| bnemec | That's what it used to do, although I think it was in an o-r-c script back then. | 18:42 |
| amoralej | but running a "yum update -y" before running puppet it's good for first deployment | 18:42 |
| amoralej | i have doubts for the update | 18:42 |
| amoralej | for update we may need orchestration to update service by service | 18:43 |
| amoralej | no? | 18:43 |
| bnemec | No :-) | 18:43 |
| EmilienM | trown: yes, definitly | 18:43 |
| bnemec | We take all the services down as part of the update because otherwise things tend to go badly. | 18:43 |
| amoralej | oooook, i see | 18:43 |
| bnemec | Several of the services have issues when they get restarted in the post scripts of the package install. | 18:43 |
| amoralej | then full upgrade at the beginning should be ok | 18:44 |
| bnemec | Which doesn't happen if they're not running in the first place. | 18:44 |
| EmilienM | if you guys are ok, I can work on a patch that 1) run "yum update" befre running puppet 2) make it optional using the same "update_packages" parameter | 18:44 |
| *** yamahata has quit IRC | 18:44 | |
| trown | so we can make an update o-r-c script in https://github.com/openstack/instack-undercloud/tree/master/elements/puppet-stack-config/os-refresh-config/configure.d guess we would control it via ENV var? | 18:44 |
| EmilienM | trown: yes, +1 | 18:44 |
| trown | EmilienM: ya I think that would be a good solution to https://bugs.launchpad.net/tripleo/+bug/1650374 | 18:45 |
| openstack | Launchpad bug 1650374 in tripleo "[instack-undercloud] package upgrade should happen before service configuration" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 18:45 |
| amoralej | then i could abandon my patch too | 18:45 |
| trown | EmilienM: solving it via puppet would be a lot of dependencies... I was looking at it yesterday and we would need a Service tag for each service | 18:45 |
| EmilienM | trown: right, it sounds too complex | 18:46 |
| amoralej | are all repos enabled at that time? | 18:46 |
| trown | ya | 18:46 |
| amoralej | then, it should work fine | 18:47 |
| *** fragatina has quit IRC | 18:47 | |
| *** gfidente is now known as gfidente|afk | 18:51 | |
| *** saibarspeis has quit IRC | 18:54 | |
| *** flepied has joined #tripleo | 18:54 | |
| *** egafford has joined #tripleo | 18:54 | |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Run `yum update -y` before Puppet run https://review.openstack.org/411957 | 18:58 |
| EmilienM | trown, amoralej: that's a quick PoC ^ | 18:58 |
| EmilienM | trown, amoralej: any feedback is welcome :) | 18:59 |
| trown | EmilienM: before we updated openstack packages first then the rest... I don't totally understand why we needed that, but we lose that ability with your patch | 19:01 |
| trown | also not sure there is a good solution for that without puppet involved... but just wanted to point it out | 19:03 |
| EmilienM | trown: we wanted to update the undercloud everytime puppet run | 19:05 |
| EmilienM | I think it's fine now | 19:05 |
| EmilienM | but I'll let instack experts looking :D i'm sure it can be better | 19:05 |
| trown | ya, I am not sure putting it in python there is good... for one `tox -e py27` is running yum update on my machine :P | 19:06 |
| EmilienM | trown: how would you handle it? | 19:07 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650612 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 19:10 |
| trown | EmilienM: commented in the review, but I think we can make a script like https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/os-refresh-config/configure.d/50-puppet-stack-config in the same directory like "40-yum-update-before-puppet" | 19:10 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1650612 in tripleo "nodepool subnodes can't ping each others" [Critical,Triaged] | 19:10 |
| mwhahaha | well the yum update was after the openstack package updates which would capture at least the openstack service restarts. But it also assumed that the dependency versions would magically get updated as well which we saw not to be the case yesterday with the python2-cryptography thing and ironic | 19:10 |
| *** rbrady is now known as rbrady-afk | 19:13 | |
| *** dsneddon has joined #tripleo | 19:16 | |
| *** florianf has quit IRC | 19:21 | |
| *** dsneddon_ has joined #tripleo | 19:21 | |
| *** yamahata has joined #tripleo | 19:21 | |
| *** dsneddon has quit IRC | 19:24 | |
| weshay | jistr++ | 19:32 |
| *** florianf has joined #tripleo | 19:36 | |
| *** yamahata has quit IRC | 19:36 | |
| *** yamahata has joined #tripleo | 19:36 | |
| EmilienM | trown: ok looking | 19:37 |
| *** owalsh has quit IRC | 19:40 | |
| *** rhallisey has quit IRC | 19:41 | |
| *** amoralej is now known as amoralej|off | 19:48 | |
| *** florianf has quit IRC | 19:54 | |
| *** fragatina has joined #tripleo | 19:55 | |
| *** rhallisey has joined #tripleo | 19:59 | |
| *** chem has quit IRC | 20:06 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 20:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650612 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1650612 in tripleo "nodepool subnodes can't ping each others" [Critical,Triaged] | 20:10 |
| *** dsariel has joined #tripleo | 20:20 | |
| *** paramite has quit IRC | 20:23 | |
| *** paramite_ has quit IRC | 20:24 | |
| *** ayoung has joined #tripleo | 20:29 | |
| *** rhallisey has quit IRC | 20:33 | |
| *** jtomasek has quit IRC | 20:33 | |
| *** b00tcat has quit IRC | 20:34 | |
| larsks | During an update operation, is it normal for a stack to show state UPDATE_IN_PROGRESS when all the resources in that stack show either UPDATE_COMPLETE or CREATE_COMPLETE? | 20:40 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Run `yum update -y` before Puppet run https://review.openstack.org/411957 | 20:44 |
| *** chlong has quit IRC | 20:48 | |
| *** jcoufal has quit IRC | 20:48 | |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Include nova::compute::libvirt::qemu from the libvirt profile https://review.openstack.org/411984 | 20:49 |
| *** leanderthal is now known as leanderthal|afk | 20:51 | |
| *** fragatina has quit IRC | 20:53 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Increase libvirt/qemu.conf max_files and max_processes https://review.openstack.org/411987 | 20:56 |
| *** jcoufal has joined #tripleo | 21:01 | |
| *** jcoufal has quit IRC | 21:01 | |
| *** b00tcat has joined #tripleo | 21:01 | |
| *** dsneddon has joined #tripleo | 21:07 | |
| *** liverpooler has quit IRC | 21:07 | |
| *** dprince has quit IRC | 21:08 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 21:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 21:10 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart-extras: Add modify-images role https://review.openstack.org/411995 | 21:20 |
| *** rlandy has quit IRC | 21:21 | |
| *** ctayal has quit IRC | 21:22 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Fetch images in a standalone role https://review.openstack.org/408760 | 21:24 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Run `yum update -y` before Puppet run https://review.openstack.org/411957 | 21:25 |
| *** dsneddo__ has joined #tripleo | 21:25 | |
| *** dsnedd___ has joined #tripleo | 21:27 | |
| *** dsneddon_ has quit IRC | 21:28 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: Fetch images in a standalone role https://review.openstack.org/408760 | 21:28 |
| *** dsneddo__ has quit IRC | 21:30 | |
| *** jayg is now known as jayg|g0n3 | 21:31 | |
| *** trozet has quit IRC | 21:31 | |
| *** gfidente|afk has quit IRC | 21:31 | |
| *** trozet has joined #tripleo | 21:46 | |
| *** trozet has quit IRC | 21:49 | |
| *** fultonj has quit IRC | 21:58 | |
| *** scorcoran has joined #tripleo | 22:00 | |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Run `yum update -y` before Puppet run https://review.openstack.org/411957 | 22:01 |
| *** trown is now known as trown|outtypewww | 22:05 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 22:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650503 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 22:10 |
| *** fzdarsky is now known as fzdarsky|afk | 22:10 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: add the containers prep role to the quickstart-extras https://review.openstack.org/400983 | 22:11 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: config for containerized-compute https://review.openstack.org/393348 | 22:11 |
| *** bana_k has quit IRC | 22:12 | |
| *** tzumainn has quit IRC | 22:12 | |
| *** ctayal has joined #tripleo | 22:14 | |
| *** lblanchard has quit IRC | 22:16 | |
| EmilienM | bnemec: looks like introspection is broken or something | 22:18 |
| EmilienM | http://logs.openstack.org/14/411514/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/7030340/console.html#_2016-12-16_21_21_37_089760 | 22:18 |
| EmilienM | bnemec: but gate-tripleo-ci-centos-7-ovb-ha passed | 22:19 |
| *** bana_k has joined #tripleo | 22:19 | |
| * EmilienM checking centos version | 22:19 | |
| *** jeckersb is now known as jeckersb_gone | 22:19 | |
| *** scorcoran is now known as scorcoran_biab | 22:19 | |
| bnemec | EmilienM: Yeah, I noticed a similar error in nonha this morning. | 22:20 |
| EmilienM | it might be centos 7.2 vs 7.3? | 22:20 |
| *** rasca has quit IRC | 22:21 | |
| bnemec | EmilienM: Could be. I don't grok mistral errors very well though. | 22:21 |
| EmilienM | bnemec: looks like both are 7.3 | 22:23 |
| bnemec | Only one error in the logs, and it's the same one in the console: http://logs.openstack.org/14/411514/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/7030340/logs/undercloud/var/log/mistral/engine.txt.gz#_2016-12-16_21_21_35_093 | 22:23 |
| EmilienM | indeed | 22:23 |
| EmilienM | thrash: are you still around ? :D | 22:23 |
| EmilienM | I don't see anything in ironic-inspector | 22:24 |
| EmilienM | neither in ironic-conductor | 22:24 |
| bnemec | Ditto | 22:25 |
| bnemec | I tried to debug a similar error once and basically got nowhere. | 22:25 |
| EmilienM | I'm filing a (new) bug with an alert | 22:25 |
| bnemec | https://review.openstack.org/#/c/411902/1 passed nonha on newton, so it must be something to do with master. | 22:27 |
| EmilienM | if you look at logstash, it only happens to gate-tripleo-ci-centos-7-ovb-nonha | 22:27 |
| bnemec | That's the only job where we run introspection | 22:27 |
| EmilienM | what is different between ha & nonha here | 22:27 |
| EmilienM | ah :D | 22:27 |
| *** jkilpatr has quit IRC | 22:27 | |
| bnemec | It takes a while, so we put it on the shortest job. | 22:27 |
| * EmilienM looking latest patches in Ironic | 22:28 | |
| EmilienM | bnemec: what is weird is that we haven't got promotion for a while so how a patch in ironic or ironic-inspector could break us | 22:29 |
| EmilienM | we need to find the latest sucessful job and compare packages | 22:29 |
| * EmilienM clicks | 22:29 | |
| bnemec | Could be we merged something that accidentally broke this. Anything that went in this week couldn't have had a recent introspection pass. | 22:30 |
| EmilienM | tonight at 1.17am it worked | 22:30 |
| EmilienM | on https://review.openstack.org/#/c/411541/1 | 22:31 |
| * EmilienM runs diff now | 22:31 | |
| EmilienM | I'm afraid it was centos 7.2 | 22:31 |
| *** weshay is now known as weshay_lata | 22:32 | |
| EmilienM | bnemec: https://www.diffchecker.com/t4XiBVfM | 22:33 |
| EmilienM | it sunds like DIB | 22:33 |
| bnemec | Or puppet-mistral | 22:34 |
| *** toure has quit IRC | 22:34 | |
| bnemec | Those are the only two that seem relevant though. | 22:34 |
| EmilienM | mwhahaha: https://github.com/openstack/puppet-mistral/commit/bf3625d5af5aabd5b2d3679f2ae61d63153cb2a4 | 22:35 |
| EmilienM | that! | 22:35 |
| EmilienM | it *could* be related | 22:35 |
| bnemec | Weird. Why did that package even change today though? The patch merged three days ago. | 22:35 |
| EmilienM | bnemec: indeed | 22:36 |
| *** toure has joined #tripleo | 22:36 | |
| bnemec | Maybe dlrn was having 7.3 issues too. :-) | 22:36 |
| EmilienM | we should maybe switch puppet-mistral to use ovb-nonha | 22:37 |
| EmilienM | not sure | 22:37 |
| mwhahaha | what | 22:38 |
| bnemec | Might be good. The introspection workflow seems to be a common breakage. | 22:38 |
| bnemec | And the rest are pretty much the same between ha and nonha. | 22:38 |
| EmilienM | mwhahaha: we're investigating http://logs.openstack.org/14/411514/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/7030340/console.html#_2016-12-16_21_21_37_089760 | 22:38 |
| EmilienM | bnemec: when things are stable again, i'll add it :D | 22:38 |
| *** flepied has quit IRC | 22:40 | |
| mwhahaha | not sure how switching out something that was backwards compatible would break the data input into a work flow | 22:42 |
| * mwhahaha shrugs | 22:42 | |
| mwhahaha | this week has proven anything is possible | 22:42 |
| *** scorcoran_biab is now known as scorcoran | 22:42 | |
| EmilienM | lol indeed, and it's time for a break :) | 22:42 |
| EmilienM | I want to do recheck on https://review.openstack.org/#/c/411541/ and see if it pass again :D | 22:45 |
| EmilienM | bnemec: looking at logstash it's somthing that happened today | 22:46 |
| EmilienM | i'll call it a week | 22:47 |
| EmilienM | see you folks | 22:47 |
| *** scorcoran is now known as scorcoran_afk | 22:48 | |
| *** jkilpatr has joined #tripleo | 22:50 | |
| *** fragatin_ has joined #tripleo | 22:51 | |
| *** fragatin_ has quit IRC | 22:53 | |
| bnemec | o/ | 22:54 |
| *** fragatina has joined #tripleo | 22:56 | |
| *** fragatina has quit IRC | 23:00 | |
| openstackgerrit | Merged openstack/tripleo-common: Remove remaining vendor plugins from default image YAML https://review.openstack.org/409809 | 23:04 |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud: Add cell_v2 simple_cell_setup https://review.openstack.org/412006 | 23:07 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1609688 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1609688 in tripleo "CI: nonha jobs fails in introspection step because of mistral error" [Critical,Confirmed] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles) | 23:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 23:10 |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud: Add cell_v2 simple_cell_setup https://review.openstack.org/412006 | 23:12 |
| *** saneax-_-|AFK is now known as saneax | 23:12 | |
| *** b00tcat has quit IRC | 23:12 | |
| *** b00tcat has joined #tripleo | 23:15 | |
| *** b00tcat has quit IRC | 23:15 | |
| *** b00tcat has joined #tripleo | 23:15 | |
| *** trozet has joined #tripleo | 23:15 | |
| *** shinobu__ has quit IRC | 23:20 | |
| *** mhenkel has quit IRC | 23:20 | |
| *** mhenkel has joined #tripleo | 23:21 | |
| *** shinobu__ has joined #tripleo | 23:23 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo: Add cell_v2 setup for nova https://review.openstack.org/412007 | 23:24 |
| mwhahaha | hrm has anyone actually tried 'openstack overcloud deploy --templates' lately? The last few times i've just run that (and only that) it's timed out | 23:30 |
| *** almondjoy has quit IRC | 23:32 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-puppet-elements: Octavia integration https://review.openstack.org/412012 | 23:48 |
| *** scorcoran_afk has quit IRC | 23:52 | |
| *** chatter has joined #tripleo | 23:54 | |
| chatter | hey guys | 23:55 |
| mwhahaha | hi | 23:55 |
| chatter | allah is doing | 23:55 |
| chatter | sun is not doing allah is doing | 23:55 |
| chatter | to accept islam say that i bear witness that there is no deity worthy of worship except allah and muhammad peace be upon him is his slave and messenger | 23:55 |
| chatter | hey | 23:56 |
| *** chatter has left #tripleo | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!