| *** rhallisey has quit IRC | 00:00 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: [WIP] role definitions https://review.openstack.org/445687 | 00:02 |
|---|---|---|
| *** funzo has joined #tripleo | 00:03 | |
| *** marst has joined #tripleo | 00:04 | |
| *** funzo has quit IRC | 00:08 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1672730 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1672730 in tripleo "instack-undercloud pep8 failures due to broken dib-lint" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 00:10 |
| *** lblanchard has joined #tripleo | 00:28 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 00:30 |
| *** rwsu has quit IRC | 00:31 | |
| *** ansmith has quit IRC | 00:35 | |
| openstackgerrit | Leehom Li proposed openstack/diskimage-builder master: Fix problem unable disable iptables service https://review.openstack.org/419703 | 00:40 |
| *** limao has joined #tripleo | 00:44 | |
| *** funzo has joined #tripleo | 01:04 | |
| stevebaker | weshay: is this with a recent pull of dprince/undercloud_containers? | 01:05 |
| EmilienM | stevebaker: hey, I have a quick question for you | 01:05 |
| stevebaker | EmilienM: yo | 01:05 |
| EmilienM | stevebaker: do we mount the policy.json files for all projects, like we do with .conf? example: /etc/nova/policy.json | 01:06 |
| EmilienM | I think yet but I want to be sure | 01:06 |
| *** dhill_ has quit IRC | 01:08 | |
| stevebaker | EmilienM: I'm not sure, but I though policy files were treated as lib, not config. so they are installed via package at image build time | 01:08 |
| EmilienM | stevebaker: yes they are but I want to modify them with Puppet | 01:08 |
| EmilienM | stevebaker: I'm going to send a PoC, can you review it please? | 01:09 |
| *** funzo has quit IRC | 01:09 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Allow to configure /etc/nova/policy.json https://review.openstack.org/445700 | 01:10 |
| EmilienM | stevebaker: this is super WIP but maybe you can see if I miss something obvious | 01:10 |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1672730 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1672730 in tripleo "instack-undercloud pep8 failures due to broken dib-lint" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 01:10 |
| stevebaker | EmilienM: in that case, same as the conf file. | 01:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Allow to configure /etc/nova/policy.json https://review.openstack.org/445700 | 01:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Allow to configure /etc/nova/policy.json https://review.openstack.org/445700 | 01:11 |
| EmilienM | stevebaker: is it correct? https://review.openstack.org/#/c/445700/3/docker/services/nova-api.yaml | 01:12 |
| EmilienM | oops I did a typo again | 01:12 |
| stevebaker | EmilienM: That looks fine, but the changes will be even smaller once Slower's changes to remove kolla_config land, like https://review.openstack.org/#/c/443967/1/docker/services/glance-api.yaml | 01:12 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Allow to configure /etc/nova/policy.json https://review.openstack.org/445700 | 01:12 |
| stevebaker | EmilienM: then we're just mounting all of /etc/foo, so if there is a policy file in there it will be used | 01:12 |
| EmilienM | stevebaker: ah even better | 01:13 |
| EmilienM | thanks! | 01:13 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Allow to configure /etc/nova/policy.json https://review.openstack.org/445700 | 01:14 |
| openstackgerrit | Merged openstack/instack-undercloud master: Align stars to fix CI https://review.openstack.org/445226 | 01:18 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Disable cinder v1 api from tempest https://review.openstack.org/445516 | 01:18 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add undercloud user to docker group https://review.openstack.org/445428 | 01:18 |
| *** dmacpher has joined #tripleo | 01:19 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - test nova/policy.json https://review.openstack.org/445702 | 01:22 |
| *** fragatin_ has joined #tripleo | 01:44 | |
| *** fragatina has quit IRC | 01:48 | |
| *** fragatin_ has quit IRC | 01:49 | |
| *** cdearborn has quit IRC | 01:59 | |
| *** funzo has joined #tripleo | 02:05 | |
| *** yamahata has quit IRC | 02:08 | |
| *** funzo has quit IRC | 02:10 | |
| *** fragatina has joined #tripleo | 02:11 | |
| *** fragatina has quit IRC | 02:16 | |
| *** lblanchard has quit IRC | 02:25 | |
| openstackgerrit | Merged openstack/os-net-config master: H803 hacking have been deprecated https://review.openstack.org/413933 | 02:35 |
| *** fzdarsky_ has joined #tripleo | 02:38 | |
| *** rlandy has quit IRC | 02:40 | |
| *** fzdarsky has quit IRC | 02:42 | |
| *** tzumainn has quit IRC | 02:50 | |
| *** bkopilov has quit IRC | 02:58 | |
| *** flepied has quit IRC | 03:04 | |
| *** funzo has joined #tripleo | 03:06 | |
| *** funzo has quit IRC | 03:10 | |
| *** masterjcool has quit IRC | 03:20 | |
| *** masterjcool has joined #tripleo | 03:32 | |
| *** psahoo has joined #tripleo | 03:43 | |
| *** links has joined #tripleo | 03:56 | |
| *** funzo has joined #tripleo | 04:06 | |
| *** jaganathan has joined #tripleo | 04:09 | |
| *** funzo has quit IRC | 04:11 | |
| *** fragatina has joined #tripleo | 04:15 | |
| *** fragatina has quit IRC | 04:16 | |
| *** fragatina has joined #tripleo | 04:17 | |
| *** udesale has joined #tripleo | 04:18 | |
| *** pmannidi_ has joined #tripleo | 04:25 | |
| *** fragatina has quit IRC | 04:26 | |
| *** pmannidi has quit IRC | 04:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/440124 | 04:29 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-validations master: Updated from global requirements https://review.openstack.org/445155 | 04:29 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements master: Add remove-machine-id element https://review.openstack.org/445173 | 04:32 |
| *** oaken_chris has quit IRC | 04:35 | |
| *** agurenko has quit IRC | 04:36 | |
| *** saneax-_-|AFK is now known as saneax | 04:41 | |
| *** gkadam has joined #tripleo | 04:45 | |
| *** bkopilov has joined #tripleo | 05:05 | |
| *** funzo has joined #tripleo | 05:07 | |
| *** funzo has quit IRC | 05:11 | |
| *** cwolferh has quit IRC | 05:13 | |
| *** udesale__ has joined #tripleo | 05:17 | |
| *** udesale has quit IRC | 05:21 | |
| *** agurenko has joined #tripleo | 05:26 | |
| *** ukalifon has joined #tripleo | 05:26 | |
| *** prateek has joined #tripleo | 05:32 | |
| *** prateek has quit IRC | 05:32 | |
| *** prateek has joined #tripleo | 05:33 | |
| *** ukalifon has quit IRC | 05:34 | |
| *** udesale has joined #tripleo | 05:35 | |
| *** udesale__ has quit IRC | 05:37 | |
| *** udesale__ has joined #tripleo | 05:44 | |
| *** udesale has quit IRC | 05:46 | |
| *** dparkes has quit IRC | 05:47 | |
| *** yprokule has joined #tripleo | 05:55 | |
| *** fragatina has joined #tripleo | 05:55 | |
| *** yamahata has joined #tripleo | 05:59 | |
| *** funzo has joined #tripleo | 06:08 | |
| *** funzo has quit IRC | 06:13 | |
| *** ealcaniz has joined #tripleo | 06:28 | |
| *** dsneddon has quit IRC | 06:29 | |
| *** dsneddon has joined #tripleo | 06:32 | |
| *** flepied has joined #tripleo | 06:35 | |
| *** lmiccini has joined #tripleo | 06:48 | |
| *** fragatina has quit IRC | 06:50 | |
| *** jaosorior has joined #tripleo | 06:52 | |
| *** mcornea has joined #tripleo | 06:54 | |
| *** dparkes has joined #tripleo | 06:57 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 06:58 |
| *** saneax is now known as saneax-_-|AFK | 07:00 | |
| *** karthiks_afk is now known as karthiks | 07:01 | |
| *** cmyster has quit IRC | 07:02 | |
| *** jtomasek has quit IRC | 07:07 | |
| *** funzo has joined #tripleo | 07:09 | |
| *** tremble has joined #tripleo | 07:10 | |
| *** tremble has joined #tripleo | 07:10 | |
| *** dbecker has joined #tripleo | 07:12 | |
| *** flepied has quit IRC | 07:12 | |
| *** funzo has quit IRC | 07:13 | |
| *** fragatina has joined #tripleo | 07:14 | |
| *** skramaja has quit IRC | 07:18 | |
| *** skramaja has joined #tripleo | 07:18 | |
| *** cmyster has joined #tripleo | 07:26 | |
| *** cmyster has quit IRC | 07:26 | |
| *** cmyster has joined #tripleo | 07:26 | |
| *** gkadam is now known as gkadam-lunch | 07:29 | |
| *** skramaja_ has joined #tripleo | 07:32 | |
| *** skramaja has quit IRC | 07:33 | |
| *** pgadiya has joined #tripleo | 07:34 | |
| *** abehl has joined #tripleo | 07:34 | |
| *** skramaja_ is now known as skramaja | 07:36 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 07:42 |
| *** flepied has joined #tripleo | 07:42 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 07:43 |
| *** tesseract has joined #tripleo | 07:44 | |
| *** flepied has left #tripleo | 07:45 | |
| *** jlinkes has joined #tripleo | 07:45 | |
| *** flepied has joined #tripleo | 07:45 | |
| *** jlinkes has quit IRC | 07:49 | |
| *** jlinkes_ has joined #tripleo | 07:49 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 07:52 |
| openstackgerrit | Merged openstack/tripleo-validations master: Updated from global requirements https://review.openstack.org/445155 | 07:53 |
| *** jaganathan is now known as jaganathan_lunch | 07:58 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK https://review.openstack.org/430277 | 08:00 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #tripleo | 08:00 | |
| *** shardy_afk is now known as shardy | 08:00 | |
| *** jaganathan_lunch has quit IRC | 08:02 | |
| *** pcaruana has joined #tripleo | 08:05 | |
| *** dsariel has joined #tripleo | 08:06 | |
| *** aufi has joined #tripleo | 08:09 | |
| *** funzo has joined #tripleo | 08:10 | |
| *** masco has joined #tripleo | 08:13 | |
| *** udesale has joined #tripleo | 08:13 | |
| *** stendulker has joined #tripleo | 08:13 | |
| *** saneax-_-|AFK is now known as saneax | 08:13 | |
| *** funzo has quit IRC | 08:14 | |
| *** udesale__ has quit IRC | 08:14 | |
| *** florianf has joined #tripleo | 08:17 | |
| *** gchamoul|afk is now known as gchamoul | 08:19 | |
| openstackgerrit | kavitha h r proposed openstack/tripleo-image-elements master: Prepare for using standard python tests https://review.openstack.org/445811 | 08:19 |
| *** salmankhan has joined #tripleo | 08:26 | |
| *** cylopez has joined #tripleo | 08:27 | |
| *** salmankhan has quit IRC | 08:31 | |
| *** yamahata has quit IRC | 08:31 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Cleanup no longer used upgrade files https://review.openstack.org/445816 | 08:31 |
| *** d0ugal has joined #tripleo | 08:32 | |
| *** jaganathan_lunch has joined #tripleo | 08:33 | |
| *** salmankhan has joined #tripleo | 08:34 | |
| *** limao has quit IRC | 08:36 | |
| *** amoralej|off is now known as amoralej | 08:37 | |
| *** limao has joined #tripleo | 08:37 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: [WIP] Testing overlayfs for docker https://review.openstack.org/445819 | 08:37 |
| *** salmankhan has quit IRC | 08:39 | |
| *** jpich has joined #tripleo | 08:39 | |
| *** jaganathan_lunch is now known as jaganathan | 08:45 | |
| *** jpena|off is now known as jpena | 08:46 | |
| *** ccamacho has joined #tripleo | 08:48 | |
| *** pmannidi_ has quit IRC | 08:51 | |
| *** dbecker has quit IRC | 08:52 | |
| *** Guest4569 has joined #tripleo | 08:55 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Clean up oooq jobs https://review.openstack.org/445628 | 08:55 |
| *** pgadiya has quit IRC | 08:59 | |
| *** zoli|gone is now known as zoli | 09:00 | |
| *** b00tcat has joined #tripleo | 09:00 | |
| zoli | good morning | 09:00 |
| *** flepied has quit IRC | 09:00 | |
| *** percevalbot has joined #tripleo | 09:00 | |
| *** bogdando has joined #tripleo | 09:01 | |
| *** prateek_ has joined #tripleo | 09:02 | |
| *** Vijayendra has quit IRC | 09:03 | |
| *** udesale__ has joined #tripleo | 09:04 | |
| *** prateek has quit IRC | 09:04 | |
| *** Guest4569 has quit IRC | 09:04 | |
| *** karimb has joined #tripleo | 09:05 | |
| *** nijaba has quit IRC | 09:06 | |
| jaosorior | mandre, sshnaidm: the gate-tripleo-ci-centos-7-ovb-containers-oooq-nv job has been timing out a lot lately. Is it broken? | 09:06 |
| *** udesale has quit IRC | 09:06 | |
| sshnaidm | jaosorior, can you show the log? | 09:08 |
| jaosorior | sshnaidm: http://logs.openstack.org/91/445391/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/8534dfc/console.html | 09:09 |
| *** ckyriakidou has joined #tripleo | 09:09 | |
| sshnaidm | jaosorior, it's not "broken", it timeouted | 09:10 |
| *** funzo has joined #tripleo | 09:10 | |
| jaosorior | sshnaidm: right, but it times out pretty constantly | 09:11 |
| *** dsariel has quit IRC | 09:11 | |
| sshnaidm | seems like performance regression in CI | 09:11 |
| jaosorior | sshnaidm: oh, haha just saw it pass :P | 09:11 |
| *** nijaba has joined #tripleo | 09:11 | |
| *** nijaba has quit IRC | 09:11 | |
| *** nijaba has joined #tripleo | 09:11 | |
| jaosorior | jistr: you around? | 09:11 |
| *** pgadiya has joined #tripleo | 09:15 | |
| *** funzo has quit IRC | 09:15 | |
| *** lucas-afk is now known as lucasagomes | 09:17 | |
| *** jtomasek has joined #tripleo | 09:18 | |
| *** flepied has joined #tripleo | 09:19 | |
| *** bogdando has quit IRC | 09:19 | |
| *** Vijayendra has joined #tripleo | 09:19 | |
| *** bogdando has joined #tripleo | 09:19 | |
| mandre | sshnaidm: do you have an idea how we can disable tempest on the containers oooq ovb job? | 09:20 |
| sshnaidm | mandre, yeah | 09:20 |
| sshnaidm | mandre, although the problem is not with tempest, but in performance regression.. | 09:21 |
| sshnaidm | mandre, seems like deploy started to take much more time | 09:21 |
| mandre | sshnaidm: agreed, the job is taking longer as we've added more containers | 09:22 |
| sshnaidm | mandre, I'm afraid it's right for all ovb jobs, not only containers | 09:22 |
| sshnaidm | mandre, but as temporary solution, I'll submit tempest disable | 09:23 |
| mandre | sshnaidm: for the containers, we're using a slow docker graph driver by default | 09:23 |
| mandre | sshnaidm: here https://review.openstack.org/#/c/445819/, I'm trying to find how much we can gain by using a more performant one | 09:23 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/445836 | 09:25 |
| mandre | sshnaidm: afaik tempest adds around 14 min to the job, and since we're really close to the limit disabling it will help see more jobs passing until we fix the performance problem | 09:25 |
| *** cmyster has quit IRC | 09:25 | |
| sshnaidm | mandre, ok | 09:25 |
| *** cmyster has joined #tripleo | 09:27 | |
| *** cmyster has quit IRC | 09:27 | |
| *** cmyster has joined #tripleo | 09:27 | |
| *** jaosorior is now known as jaosorior_lunch | 09:28 | |
| chandankumar | sshnaidm: how we are running tempest with containers oooq ovb jobs? | 09:28 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Disable tempest in container job https://review.openstack.org/445840 | 09:30 |
| sshnaidm | mandre, jaosorior_lunch ^^ | 09:30 |
| sshnaidm | chandankumar, like with all others, validate-tempest role | 09:30 |
| *** gkadam-lunch is now known as gkadam | 09:34 | |
| pliu | EmilienM, ping | 09:34 |
| shardy | pliu: he probably won't be around for another 2-3 hours | 09:35 |
| *** hewbrocca_afk is now known as hewbrocca | 09:35 | |
| pliu | shardy, thanks. Do you know who can give workflow at tripleo-puppet-elements? | 09:37 |
| shardy | pliu: any tripleo-core, what is the link? | 09:37 |
| pliu | https://review.openstack.org/#/c/439574/ | 09:37 |
| shardy | pliu: done! | 09:40 |
| *** dbecker has joined #tripleo | 09:40 | |
| *** pgadiya has quit IRC | 09:40 | |
| pliu | shardy, much appreciate. | 09:41 |
| *** derekh has joined #tripleo | 09:43 | |
| snecklifter | Morning #tripleo, anyone know why heat hosts has .local appended everywhere? | 09:43 |
| snecklifter | Not seen this behaviour before | 09:43 |
| snecklifter | I have dhcp_domain set in neutron | 09:44 |
| snecklifter | scratch that, just worked it out | 09:44 |
| jaosorior_lunch | snecklifter: localdomain? | 09:44 |
| jaosorior_lunch | oh, ok | 09:44 |
| shardy | snecklifter: what do you have CloudDomain set to? | 09:44 |
| * snecklifter needs to remember to drink coffee before work | 09:45 | |
| shardy | hehe | 09:45 |
| snecklifter | sorry folks, thanks for responding, sorry for the noise | 09:45 |
| * shardy remembers he also needs coffee | 09:45 | |
| snecklifter | shardy: yes, clouddomain was the prob | 09:45 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Introduce nodes assignment using NodePicker component https://review.openstack.org/445526 | 09:48 |
| *** panda|zZ is now known as panda | 09:50 | |
| openstackgerrit | Sanjay Upadhyay proposed openstack/tripleo-puppet-elements master: Adding DIB elements for skydive on the controller https://review.openstack.org/443564 | 09:51 |
| *** pgadiya has joined #tripleo | 09:54 | |
| *** mrunge_ has quit IRC | 09:54 | |
| *** amoralej has quit IRC | 09:54 | |
| *** jpena has quit IRC | 09:55 | |
| *** mrunge_ has joined #tripleo | 09:59 | |
| *** jpena|off has joined #tripleo | 09:59 | |
| *** amoralej has joined #tripleo | 10:00 | |
| *** nyechiel has joined #tripleo | 10:02 | |
| *** gfidente has joined #tripleo | 10:04 | |
| *** salmankhan has joined #tripleo | 10:05 | |
| *** limao has quit IRC | 10:05 | |
| *** limao has joined #tripleo | 10:05 | |
| *** jpena|off is now known as jpena | 10:06 | |
| sshnaidm | panda, I guess it's bootstrap nodes script which breaks branch jobs.. it deploys repositories of master, because doesn't have stable_release var, and our setup role overwrite it, but it doesn't make sense, because packages are not updated (we should actually downgrade them) | 10:07 |
| sshnaidm | panda, we need to get rid off this script usage asap | 10:07 |
| jistr | jaosorior_lunch: hey sorry forgot to reconnect to ZNC when i came to work... yup i'm around | 10:08 |
| *** salmankhan has quit IRC | 10:09 | |
| *** limao has quit IRC | 10:10 | |
| panda | sshnaidm: ok, I'll put up a draft role for that, do you think it's worth trying to pass the deploy.env in some way in the meantime ? | 10:11 |
| *** funzo has joined #tripleo | 10:11 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Re-enable components tests https://review.openstack.org/443527 | 10:12 |
| *** udesale has joined #tripleo | 10:14 | |
| *** prateek__ has joined #tripleo | 10:15 | |
| *** udesale has quit IRC | 10:16 | |
| *** funzo has quit IRC | 10:16 | |
| *** udesale has joined #tripleo | 10:16 | |
| *** udesale__ has quit IRC | 10:17 | |
| *** saibarspeis has joined #tripleo | 10:18 | |
| *** prateek_ has quit IRC | 10:18 | |
| *** dsariel has joined #tripleo | 10:18 | |
| *** openstackgerrit has quit IRC | 10:18 | |
| *** tosky has joined #tripleo | 10:18 | |
| honza | sshnaidm: I have also seen a bunch of tripleo-ui jobs fail because of some multinode setting in the new oooq job. | 10:19 |
| *** akrivoka has joined #tripleo | 10:19 | |
| *** thrash|g0ne is now known as thrash | 10:19 | |
| panda | honza: links ? | 10:19 |
| honza | sshnaidm: panda: http://logs.openstack.org/53/443153/2/gate/gate-tripleo-ci-centos-7-undercloud-oooq/5a50498/logs/quickstart_install.txt.gz | 10:19 |
| honza | missing virt-customize? | 10:20 |
| panda | honza: that error should be solved now | 10:20 |
| panda | honza: we were missing a variable | 10:20 |
| honza | panda: awesome! | 10:20 |
| sshnaidm | panda, yeah, I suppose it's even enough stable_release | 10:21 |
| honza | I just triggered a recheck so we shall see. | 10:21 |
| sshnaidm | honza, yeah, recheck will help | 10:21 |
| panda | sshnaidm: ok, spending some time passing deploy.env, then working on bootstrapping role | 10:22 |
| *** dmacpher has quit IRC | 10:24 | |
| *** zoli is now known as zoli|lunch | 10:25 | |
| honza | jtomasek: florianf: unless I'm looking at this wrong, I think our tests went from about a minute to about 10s in CI (karma => jest). Not bad! | 10:26 |
| jtomasek | honza: \o/ | 10:27 |
| *** saibarspeis has quit IRC | 10:30 | |
| *** flepied has quit IRC | 10:31 | |
| *** openstackgerrit has joined #tripleo | 10:32 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 10:32 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: quickstart transition: propagate deploy.env with existing functions https://review.openstack.org/445866 | 10:34 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Populate Hiera data before PreNetworkConfig https://review.openstack.org/430215 | 10:35 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Enable DPDK on boot during the PreNetworkConfig https://review.openstack.org/437164 | 10:35 |
| *** dmacpher has joined #tripleo | 10:36 | |
| *** jtomasek has quit IRC | 10:38 | |
| *** jtomasek has joined #tripleo | 10:38 | |
| *** florianf has quit IRC | 10:39 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: keystone/containers: Add support for fernet keys https://review.openstack.org/445391 | 10:39 |
| *** udesale has quit IRC | 10:41 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] quickstart transition: propagate deploy.env with existing functions https://review.openstack.org/445866 | 10:42 |
| *** rhallisey has joined #tripleo | 10:42 | |
| *** saneax is now known as saneax-_-|AFK | 10:44 | |
| *** jtomasek has quit IRC | 10:45 | |
| *** jtomasek has joined #tripleo | 10:45 | |
| b00tcat | Hi - In THT is it possible to do a get_param for a given service VIP? | 10:46 |
| *** flepied has joined #tripleo | 10:47 | |
| *** Vijayendra has quit IRC | 10:47 | |
| b00tcat | without using the "hiera" lookup function | 10:47 |
| bandini | jaosorior_lunch: the upgrade failures with httpd restart. I have seen it very often now so opened a bz: https://bugs.launchpad.net/tripleo/+bug/1673030 | 10:47 |
| openstack | Launchpad bug 1673030 in tripleo "N-O upgrades get httpd restart failures" [High,New] | 10:47 |
| bandini | matbu, marios: ^ fyi | 10:47 |
| shardy | b00tcat: yes, you reference the EndpointMap, sec let me get an example | 10:48 |
| adarazs | panda: what is this supposed to fix? https://review.openstack.org/445866 | 10:49 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/heat-api.yaml#L110 | 10:49 |
| shardy | b00tcat: ^^ | 10:49 |
| *** karimb has quit IRC | 10:49 | |
| matbu | bandini: during which step ? | 10:49 |
| bandini | matbu: overcloud.AllNodesDeploySteps.AllNodesPostUpgradeSteps.ControllerDeployment_Step4.0 | 10:50 |
| *** florianf has joined #tripleo | 10:50 | |
| panda | adarazs: stable branches | 10:50 |
| *** jkilpatr has quit IRC | 10:50 | |
| b00tcat | shardy: and that at the same time is defined in "network/endpoints/endpoint_map.yaml" right? | 10:51 |
| jistr | jaosorior_lunch: thanks for reordering the fernet patches :) I've +A'd the 2nd one as well. | 10:51 |
| adarazs | panda: okay. if you have another patchset please mention that in the commit message for clarity. | 10:51 |
| shardy | b00tcat: yup, exactly | 10:51 |
| shardy | b00tcat: there are a few different outputs e.g host/host_nobrackets/port/protocol/uri | 10:51 |
| matbu | bandini: hm k, i haven't seen that recently but thx for the BZ :) | 10:51 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/network/endpoints/endpoint_map.yaml#L5933 | 10:51 |
| matbu | bandini: the upgrade is too quite those days :) | 10:51 |
| b00tcat | shardy: perfect, thanks! if I ever see you in person I owe you some beers | 10:51 |
| bandini | matbu: I will try to get to the bottom of it | 10:52 |
| shardy | b00tcat: hehe, no worries :) | 10:52 |
| bandini | matbu: ehehe I will join the call today and cause some trouble ;) | 10:52 |
| matbu | bandini: hehe you're always welcome :) | 10:52 |
| bandini | :) | 10:52 |
| *** chem has joined #tripleo | 10:53 | |
| *** skramaja has quit IRC | 10:56 | |
| *** skramaja has joined #tripleo | 10:56 | |
| *** morazi has quit IRC | 10:56 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 10:59 |
| sshnaidm | panda, we don't run multinode jobs for newton at all, btw | 10:59 |
| sshnaidm | panda, https://review.openstack.org/445875 | 11:00 |
| marios | thanks bandini | 11:00 |
| openstackgerrit | Merged openstack/tripleo-validations master: Use generic keystone plugins instead of version-specific https://review.openstack.org/445408 | 11:01 |
| bandini | can anyone think of a way to disable journald rate limiting in a CI job? | 11:01 |
| sshnaidm | bandini, what is it..? | 11:01 |
| *** paramite has joined #tripleo | 11:02 | |
| panda | wat ? | 11:02 |
| bandini | sshnaidm: I need to get to the bottom of https://bugs.launchpad.net/tripleo/+bug/1673030 but journald rate limiting is kicking in | 11:03 |
| openstack | Launchpad bug 1673030 in tripleo "N-O upgrades get httpd restart failures" [High,New] | 11:03 |
| bandini | and until we land https://review.rdoproject.org/r/#/c/5823/ there is little I can look at | 11:03 |
| b00tcat | shardy: at the beginning of endpoint_map.yaml it says that this file is generated automatically - can I modify it locally without fearing any change by a script? | 11:04 |
| shardy | b00tcat: you can, but it'd be better to update endpoint_data.yaml then r-run build_endpoint_map.py | 11:05 |
| shardy | re-run I mean | 11:05 |
| b00tcat | ok got it | 11:05 |
| shardy | b00tcat: we currently don't run that conversion automatically on deployment though | 11:05 |
| panda | sshnaidm: ah, because we bear the names of the old experimental oooq jobs ... | 11:06 |
| panda | bandini: what's the command to disable rate limiting ? | 11:07 |
| bandini | panda: it is a couple of settings in /etc/systemd/journald.conf | 11:07 |
| *** liverpooler has quit IRC | 11:08 | |
| bandini | and a journald restart/reload I guess | 11:08 |
| panda | bandini: what's the job you need to look at ? | 11:08 |
| sshnaidm | bandini, you need this on overcloud, right? | 11:08 |
| bandini | panda: gate-tripleo-ci-centos-7-multinode-upgrades | 11:09 |
| bandini | sshnaidm: correct | 11:09 |
| sshnaidm | bandini, then just ssh to subnode and fix journald config and then restart it | 11:09 |
| sshnaidm | bandini, after bootstrapping nodes | 11:09 |
| bandini | sshnaidm: do we have ssh interaction possibility in our CI jobs? | 11:09 |
| sshnaidm | bandini, we talk about some testing patch, not persistent solution, right? | 11:10 |
| *** athomas has quit IRC | 11:10 | |
| bandini | sshnaidm: yes just understing wth httpd seems to fail to start | 11:10 |
| sshnaidm | bandini, yeah, in multinode we can connect to subnodes and do whatever | 11:10 |
| snecklifter | Ugh, this is the first googlehit for tripleo | 11:10 |
| *** saibarspeis has joined #tripleo | 11:10 | |
| snecklifter | https://wiki.openstack.org/wiki/TripleO | 11:10 |
| bandini | sshnaidm: oh sweet. do you have more info on how that would work? | 11:11 |
| snecklifter | "tripleo is raw but usable today" | 11:11 |
| bandini | I had no idea | 11:11 |
| sshnaidm | bandini, yeah, 1 sec | 11:11 |
| bandini | thanks! | 11:11 |
| shardy | snecklifter: thanks for pointing that out, we should probably deprecate nearly all that content and provide a link to tripleo-docs | 11:12 |
| *** funzo has joined #tripleo | 11:12 | |
| snecklifter | shardy: sure, happy to take it on as a task, will try and sort today | 11:12 |
| shardy | snecklifter: Ok, I was about to offer but if you have time that's great :) | 11:13 |
| shardy | happy to be pinged to review any revisions | 11:13 |
| snecklifter | shardy: thanks | 11:13 |
| *** adarazs is now known as adarazs_errands | 11:14 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 11:15 |
| *** karimb has joined #tripleo | 11:15 | |
| *** lucasagomes is now known as lucas-pto | 11:15 | |
| *** jkilpatr has joined #tripleo | 11:16 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT MERGE: testing hack for httpd https://review.openstack.org/445880 | 11:16 |
| sshnaidm | bandini, ^^ | 11:16 |
| *** funzo has quit IRC | 11:16 | |
| sshnaidm | bandini, just depend on this patch | 11:16 |
| sshnaidm | bandini, or do the same in yours.. | 11:17 |
| bandini | sshnaidm: perfect, thanks! | 11:17 |
| *** athomas has joined #tripleo | 11:17 | |
| openstackgerrit | Merged openstack/tripleo-validations master: Typo fix: choses => chooses https://review.openstack.org/430083 | 11:18 |
| *** stendulker has quit IRC | 11:19 | |
| panda | sshnaidm: RateLimitIntervalSec | 11:19 |
| bandini | sshnaidm: since my failing patch is for stable/ocata do I need to make a stable/ocata review of yours or is leaving your on master ok? | 11:19 |
| panda | bandini: ^ | 11:19 |
| sshnaidm | panda, hmm.. on my centos it's #RateLimitInterval=30s | 11:20 |
| bandini | panda: yeah I saw, there is also another option to add, but those are details I can fix myself later as well | 11:20 |
| EmilienM | hello | 11:21 |
| sshnaidm | bandini, we fail multinode oooq jobs in ocata in this moment..\ | 11:21 |
| bandini | hi EmilienM | 11:21 |
| sshnaidm | bandini, oh, so you hit this error from when..? | 11:21 |
| *** saibarspeis has quit IRC | 11:21 | |
| sshnaidm | bandini, if from yesterday, then it's not problem | 11:22 |
| sshnaidm | EmilienM, hi | 11:22 |
| bandini | sshnaidm: I started it seeing it here: https://review.openstack.org/#/c/444928/ so first occurrence Mar 13 -> http://logs.openstack.org/28/444928/1/check/gate-tripleo-ci-centos-7-multinode-upgrades/5133416/logs/postci.txt.gz | 11:22 |
| sshnaidm | bandini, ok, so it's code problem, not CI, we don't use oooq in multinode-upgrades.. | 11:23 |
| sshnaidm | bandini, but it's impossible to run multinode for ocata atm, panda is working on this | 11:24 |
| *** jkilpatr has quit IRC | 11:24 | |
| EmilienM | sshnaidm: how is CI going? | 11:25 |
| sshnaidm | EmilienM, can you +1 please? https://review.openstack.org/#/c/445875/ | 11:25 |
| bandini | sshnaidm: ok, so if I depends-on your review will I have a disabled rate limiting in my stable/ocata review? | 11:25 |
| sshnaidm | EmilienM, if not considering stable branches problem, then pretty good | 11:25 |
| sshnaidm | bandini, yeah | 11:26 |
| *** karimb has quit IRC | 11:26 | |
| EmilienM | sshnaidm: why do you touch upgrade jobs? | 11:26 |
| bandini | sshnaidm: alright, let me give it a shot then ;) | 11:26 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 11:27 |
| sshnaidm | EmilienM, omg.. sorry, fixing | 11:27 |
| bandini | this is very useful info, thanks again sshnaidm and panda | 11:27 |
| EmilienM | sshnaidm: also, why didn't we promote? | 11:27 |
| EmilienM | it would be great to update http://tripleo.org/cistatus.html with the new jobs, like we talked yesterday | 11:28 |
| panda | we didn't ? I see all green | 11:28 |
| sshnaidm | EmilienM, didn't we? | 11:28 |
| *** karimb has joined #tripleo | 11:29 | |
| *** dtantsur|afk is now known as dtantsur | 11:29 | |
| *** yolanda has quit IRC | 11:29 | |
| openstackgerrit | Michele Baldessari proposed openstack-infra/tripleo-ci master: DONT MERGE: testing hack for httpd https://review.openstack.org/445880 | 11:29 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Add etcd image to overcloud containers https://review.openstack.org/445883 | 11:29 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement verify profiles as a custom action https://review.openstack.org/429684 | 11:29 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check ironic boot configuration as a custom action https://review.openstack.org/428225 | 11:29 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check node counts as a custom action https://review.openstack.org/429757 | 11:29 |
| EmilienM | panda, sshnaidm: I see 2 days on https://dashboards.rdoproject.org/rdo-dev | 11:30 |
| sshnaidm | EmilienM, looking.. | 11:30 |
| EmilienM | panda: I'm trying to debug why https://review.openstack.org/#/c/445702/1 doesn't pass, do you see something in logs? | 11:31 |
| EmilienM | panda: it sounds like it failed in overcloud-validate.sh | 11:32 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Add etcd image to overcloud containers https://review.openstack.org/445883 | 11:32 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates stable/ocata: WIP Upgrades: wait for galera to be settled https://review.openstack.org/444928 | 11:32 |
| EmilienM | sshnaidm, panda: could we have linkable logs? for example http://logs.openstack.org/02/445702/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/886ffba/logs/quickstart_install.txt.gz | 11:32 |
| EmilienM | we had that before | 11:32 |
| *** saneax-_-|AFK is now known as saneax | 11:32 | |
| sshnaidm | EmilienM, panda we have some problem with jenkins https://ci.centos.org/job/tripleo-dlrn-promote/ | 11:33 |
| sshnaidm | EmilienM, panda I wonder if it's related to outage we say yesterday | 11:33 |
| sshnaidm | that's why this job doesn't promote repos | 11:33 |
| panda | EmilienM: you mean with timestamps ? | 11:34 |
| EmilienM | panda: yes | 11:34 |
| *** yolanda has joined #tripleo | 11:34 | |
| EmilienM | panda: and you can click on it to have a link | 11:34 |
| sshnaidm | EmilienM, links are created in infra, they expect for exactly pattern in the start of line | 11:35 |
| EmilienM | also, where are OpenStack logs? http://logs.openstack.org/02/445702/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/886ffba/logs/subnode-2/ | 11:36 |
| panda | Maybe we can remove that file, it's equal to what we see in console anyway | 11:36 |
| sshnaidm | EmilienM, you need to look at subnode-1 | 11:36 |
| panda | EmilienM: subnode-1, subnode-2 is spurious dir | 11:36 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Remove spurious endpoint_map entry https://review.openstack.org/445890 | 11:36 |
| EmilienM | that's not obvious at all | 11:36 |
| panda | (and we should remove it ...) | 11:36 |
| sshnaidm | yeah, very confusing.. | 11:37 |
| shardy | mhenkel: Hey can you check ^^ pls | 11:37 |
| sshnaidm | panda, but iirc it's collected after job finishes, isn't it? | 11:37 |
| shardy | I think https://fedorapeople.org/~zaneb/tripleo-memory/20170314/heat_memused.png | 11:37 |
| EmilienM | sshnaidm: and I don't see where is my failure | 11:37 |
| shardy | oops | 11:37 |
| *** Vijayendra has joined #tripleo | 11:37 | |
| shardy | I think ContrailAnalyticsCollectorSandesh is a mistake? | 11:37 |
| EmilienM | shardy: nice catch | 11:37 |
| *** jkilpatr has joined #tripleo | 11:38 | |
| sshnaidm | EmilienM, http://logs.openstack.org/02/445702/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/886ffba/console.html#_2017-03-15_02_36_35_857785 | 11:38 |
| shardy | EmilienM: ref the ML discussion with zaneb I'm looking at how we can j2 render the endpoint_map.yaml | 11:38 |
| panda | sshnaidm: should be | 11:38 |
| sshnaidm | EmilienM, exactly command: http://logs.openstack.org/02/445702/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/886ffba/console.html#_2017-03-15_02_36_35_857871 | 11:38 |
| *** zoli|lunch is now known as zoli | 11:38 | |
| panda | sshnaidm: not sure if it's the outage, but jenkins is not working properly since yesterday | 11:38 |
| EmilienM | sshnaidm: the pingtest: http://logs.openstack.org/02/445702/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/886ffba/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz | 11:39 |
| sshnaidm | EmilienM, and then in log: http://logs.openstack.org/02/445702/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/886ffba/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz | 11:39 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates master: Add missing Swift services for containerized deployments https://review.openstack.org/445891 | 11:39 |
| EmilienM | sshnaidm: can we add to the list of things to do: have timestamps in logs and be able to click on it to have links? | 11:39 |
| *** dmacpher has quit IRC | 11:39 | |
| EmilienM | sshnaidm: because we're loosing this feature right now | 11:39 |
| *** dmacpher has joined #tripleo | 11:39 | |
| sshnaidm | EmilienM, but it's the same you see in console | 11:40 |
| sshnaidm | EmilienM, you can link just on console | 11:40 |
| EmilienM | sshnaidm: no | 11:40 |
| EmilienM | sshnaidm: the console doesn't show the pingtest issue | 11:40 |
| sshnaidm | EmilienM, you mean the pingtest log itself? | 11:40 |
| EmilienM | yes | 11:41 |
| EmilienM | and ALL logs | 11:41 |
| *** yolanda has quit IRC | 11:41 | |
| sshnaidm | EmilienM, yeah | 11:41 |
| sshnaidm | EmilienM, we want to do also one file with links to all relevant logs | 11:42 |
| *** gkadam has quit IRC | 11:42 | |
| sshnaidm | so jump to them directly.. | 11:42 |
| EmilienM | sshnaidm: yeah, maybe a next step | 11:43 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - test nova/policy.json https://review.openstack.org/445702 | 11:43 |
| EmilienM | sshnaidm: maybe we could have a landing page like here: http://logs.openstack.org/25/444925/1/check/gate-tempest-dsvm-multinode-full-ubuntu-xenial-nv/1bf99ed/ | 11:44 |
| EmilienM | sshnaidm: pointing to quickstart doc | 11:44 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 11:44 |
| sshnaidm | EmilienM, yeah | 11:44 |
| panda | I think the idea is to generate documentation on every build, so the logs will contain the documentation to quickstar too. | 11:45 |
| *** ccamacho1 has joined #tripleo | 11:45 | |
| EmilienM | panda: ++ | 11:45 |
| EmilienM | sshnaidm, panda: nice work otherwise, thanks :-) | 11:46 |
| panda | we need bugs, lots of them | 11:46 |
| EmilienM | and thanks for helping out | 11:46 |
| *** skramaja has quit IRC | 11:47 | |
| *** ccamacho has quit IRC | 11:47 | |
| *** skramaja_ has joined #tripleo | 11:48 | |
| sshnaidm | EmilienM, https://trello.com/c/slHYbwZp | 11:48 |
| *** yolanda has joined #tripleo | 11:48 | |
| sshnaidm | adarazs_errands, weshay ^^ | 11:48 |
| EmilienM | sshnaidm: nice :D | 11:49 |
| EmilienM | sshnaidm: and remove subnode-2 | 11:49 |
| *** gfidente has quit IRC | 11:50 | |
| sshnaidm | EmilienM, this will be more complicated.. since it's done by infra | 11:52 |
| *** morazi has joined #tripleo | 11:54 | |
| *** skramaja_ has quit IRC | 11:54 | |
| *** karimb has quit IRC | 11:55 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 11:55 |
| panda | -_- I hate me | 11:56 |
| *** skramaja_ has joined #tripleo | 11:56 | |
| *** karimb has joined #tripleo | 11:56 | |
| *** skramaja_ is now known as skramaja | 11:56 | |
| *** saibarspeis has joined #tripleo | 11:57 | |
| sshnaidm | panda, why don't you use echo_vars_to_deploy_env ? | 11:58 |
| *** oaken_chris has joined #tripleo | 11:58 | |
| *** gfidente has joined #tripleo | 11:59 | |
| *** gfidente has quit IRC | 11:59 | |
| *** gfidente has joined #tripleo | 11:59 | |
| panda | sshnaidm: because most of those variables are not defined anywhere and bash complains. I'm struggling to find the right combination of variables/position to make it stop complain about all the env vars we don't define. | 12:01 |
| *** jcoufal has joined #tripleo | 12:01 | |
| jistr | mandre, matbu, sshnaidm: hi guys, i'd be interested in helping make the baremetal->containerized CI happen | 12:01 |
| jistr | mainly i guess we'll need some CI optimizations and also implement the actual job | 12:02 |
| panda | sshnaidm: and echo_vars_to_deploy_env is still used by the old method, so I can't modify that | 12:02 |
| jistr | we had a wip attempt at making the CI job with tripleo.sh https://review.openstack.org/#/c/442614/ | 12:03 |
| jistr | but i also saw the upgrade role in OOOQ https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/overcloud-upgrade | 12:03 |
| *** mcornea has quit IRC | 12:03 | |
| jistr | matbu, sshnaidm: and i'm wondering if it might be better to try and adapt the upgrade role to support containers as well | 12:03 |
| *** psahoo has quit IRC | 12:04 | |
| *** yolanda has quit IRC | 12:05 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud master: Set OS_AUTH_TYPE on undercloud stackrc https://review.openstack.org/443776 | 12:05 |
| sshnaidm | jistr, this seems reasonable | 12:05 |
| *** ccamacho1 is now known as ccamacho|lunch | 12:06 | |
| jistr | matbu: is the upgrade role already integrated with some CI? It might be helpful for me to figure out how to best execute the upgrade via OOOQ if i saw how OOOQ is being called in such a case. | 12:08 |
| jistr | so that i could execute it on my own env | 12:08 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: remove logs from subnode https://review.openstack.org/445902 | 12:08 |
| sshnaidm | jistr, I think only downstream.. | 12:09 |
| EmilienM | jaosorior_lunch: how things are moving for keystone v3? https://review.openstack.org/#/c/445559/ | 12:09 |
| jistr | sshnaidm: that would be a fine start i think :) | 12:10 |
| sshnaidm | jistr, and downstream it runs on ovb | 12:11 |
| sshnaidm | not sure if it was used with multinode anywhere | 12:11 |
| *** rbrady-afk is now known as rbrady | 12:12 | |
| jistr | i hope that in the end that wouldn't make too much of a difference, if we pass in all the multinode specific env files too | 12:12 |
| jistr | and use all-in-one deplyment perhaps | 12:12 |
| jistr | (though i'm not too familiar with multinode...) | 12:13 |
| *** funzo has joined #tripleo | 12:13 | |
| *** yolanda has joined #tripleo | 12:14 | |
| *** yolanda has quit IRC | 12:14 | |
| *** yolanda has joined #tripleo | 12:14 | |
| EmilienM | sshnaidm, panda: do you want me to update the dashboard? | 12:15 |
| EmilienM | to have oooq jobs in cistatus | 12:15 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 12:16 |
| panda | EmilienM: trown|outtypewww said he would do it, he should be here soon | 12:17 |
| *** funzo has quit IRC | 12:17 | |
| EmilienM | ack | 12:17 |
| *** mcornea has joined #tripleo | 12:17 | |
| *** dprince has joined #tripleo | 12:18 | |
| snecklifter | shardy: trimmed it a bit but let me know if too much! https://wiki.openstack.org/wiki/TripleO | 12:19 |
| shardy | snecklifter: I think that's fine, the docs should be the primary reference, thanks! | 12:20 |
| snecklifter | shardy: np, ooo is officially no longer raw and usable, hooray | 12:20 |
| shardy | I might use https://docs.openstack.org/developer/tripleo-docs/ instead of the tripleo.org URL, although the content is the same | 12:21 |
| beagles | snecklifter: did you mean "unusable"? | 12:21 |
| shardy | beagles: the wiki was old and said TripleO was "raw but usable" | 12:21 |
| *** jaosorior_lunch is now known as jaosorior | 12:21 | |
| jaosorior | EmilienM: not so good | 12:21 |
| beagles | lol | 12:22 |
| beagles | okay | 12:22 |
| shardy | :) | 12:22 |
| *** saibarspeis has quit IRC | 12:22 | |
| EmilienM | shardy: have you seen my email about tripleo deployment guide proposal? | 12:22 |
| shardy | Now it can just say, look! we're usable, we have docs! :) | 12:22 |
| jaosorior | EmilienM: so, we could merge my patch for swiftclient, but apparently the issue goes deeper than that. With swift and glance using keystone in a very wrong way, and that's gonna require way more refactoring, which I don't really have time for : | 12:22 |
| jaosorior | :/ | 12:22 |
| shardy | EmilienM: yes, sorry been meaning to reply | 12:23 |
| EmilienM | jaosorior: refacto in which project? | 12:23 |
| shardy | I'm happy to help | 12:23 |
| EmilienM | shardy: no pressure, just to make sure the team saw it | 12:23 |
| EmilienM | shardy: it can wait next week | 12:23 |
| snecklifter | beagles: its just the deployment engineers that are raw and feel very used... | 12:23 |
| jaosorior | EmilienM: both swift and glance' | 12:23 |
| jaosorior | EmilienM: specifically swiftclient and glance_store are the once I have identified so far | 12:23 |
| *** karimb has quit IRC | 12:24 | |
| EmilienM | jaosorior: maybe we can ask for help from cschwede on the swift side | 12:24 |
| *** masco has quit IRC | 12:24 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 12:24 |
| jaosorior | EmilienM: I think the most help we need is from the glance side though | 12:24 |
| jaosorior | EmilienM: trying to work out with this commit but I'm having a hard time with it https://review.openstack.org/#/c/445802/ | 12:25 |
| EmilienM | jaosorior: can you rise it on the ML with [glance][swift] tags and maybe ask for help here? | 12:26 |
| snecklifter | shardy: updated link, ta | 12:26 |
| *** bkopilov has quit IRC | 12:27 | |
| *** rbrady has quit IRC | 12:28 | |
| *** skramaja is now known as skramaja|afk | 12:30 | |
| jaosorior | EmilienM: will keep trying a bit and then do it. | 12:32 |
| *** skramaja has joined #tripleo | 12:34 | |
| *** artom has quit IRC | 12:34 | |
| *** karimb has joined #tripleo | 12:34 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add workflow wrapping all predeployment validations https://review.openstack.org/445914 | 12:36 |
| marios | shardy: gfidente matbu i posted some comments on https://review.openstack.org/#/c/445494/3 from testing when you get a chance/fyi | 12:36 |
| *** skramaja|afk has quit IRC | 12:37 | |
| *** toure|gone is now known as toure | 12:39 | |
| gfidente | marios nice | 12:39 |
| jkilpatr | is upstream CI using OVB yet? | 12:39 |
| * jkilpatr maybe I should attend the ci squad meetings | 12:40 | |
| *** dsariel has quit IRC | 12:41 | |
| shardy | marios: ack thanks | 12:42 |
| marios | gfidente: well not really i mean it isn't clear why there was so much downtime for hte ping | 12:43 |
| shardy | jkilpatr: yes we've been using ovb for quite some time, and are also using infra multinode jobs more recently | 12:43 |
| *** karimb has quit IRC | 12:43 | |
| marios | gfidente: i mean after the batch tasks | 12:43 |
| marios | gfidente: resetting env to run again and try work out what happened. it isn't if neutron-server is down, apparently | 12:43 |
| *** shardy is now known as shardy_lunch | 12:43 | |
| jkilpatr | shardy, I was asking because I would like to push for a ~30 node gate job. which should be doable if using ovb on top of a sizable cloud. It's a lot of resources yes, but there are lot of issues that get hidden by small gate deployment sizes | 12:44 |
| marios | gfidente: actually why should i reset? i'll just re-run the ansible steps | 12:45 |
| *** pkovar has joined #tripleo | 12:45 | |
| marios | gfidente: anyway thanks for looking (i mean the env didn't fail/problem with it... i guess this time it will just be much shorter between stop and start on the batch tasks since the yum update already got the latest neutron-l3-agent | 12:45 |
| honza | jtomasek: Did I miss a discussion about the new redux-form dependency? I had some computer issues yesterday... | 12:46 |
| *** karimb has joined #tripleo | 12:46 | |
| *** jpich_ has joined #tripleo | 12:46 | |
| *** jpich has quit IRC | 12:46 | |
| *** jaosorior has quit IRC | 12:46 | |
| jtomasek | honza, just some 'why its so awesome' talk from me :) | 12:47 |
| *** jaosorior has joined #tripleo | 12:47 | |
| honza | jtomasek: haha, sounds good to me | 12:48 |
| jtomasek | honza, in general, it has great community, it has provided all the functionality I required from it for the node assignment work, I'd like it to incrementally replace all our formsy-react forms | 12:49 |
| *** ansmith has joined #tripleo | 12:49 | |
| *** rlandy has joined #tripleo | 12:49 | |
| honza | jtomasek: I have no strong opinions on this :) | 12:49 |
| *** adarazs_errands is now known as adarazs | 12:50 | |
| jtomasek | honza, cool:) | 12:50 |
| *** trown|outtypewww is now known as trown | 12:50 | |
| trown | EmilienM: ya sorry didnt get to that yesterday... doing it now | 12:51 |
| *** jpich_ is now known as jpich | 12:51 | |
| EmilienM | jaosorior: back online, sorry. we lost electricity here | 12:52 |
| EmilienM | and therefore internet | 12:52 |
| trown | EmilienM: the only job we run on other projects is the scenarioless multinode? | 12:54 |
| *** nyechiel has quit IRC | 12:54 | |
| EmilienM | trown: no, we run both scenarios & classic multinode AFIK | 12:55 |
| EmilienM | trown: https://docs.google.com/spreadsheets/d/1bLg-uEGrQXyRZ-FuR6pf1WT4XN0-6MrlfqEShI7xMxg/edit#gid=0 | 12:55 |
| *** jayg|g0n3 is now known as jayg | 12:56 | |
| gfidente | marios for ceph we have some validations | 12:56 |
| gfidente | meant to pause the execution until the cluster is in 'good' state | 12:56 |
| gfidente | after a node comes back online | 12:57 |
| gfidente | not sure if something like that would be useful for the l3 agents upgrade too | 12:57 |
| gfidente | I mean, if that helps with the timeouts | 12:57 |
| marios | gfidente: thanks will check it out | 12:57 |
| gfidente | you probably don't want two agents to be down at the same time | 12:57 |
| trown | EmilienM: ok... I guess we will just have a lot of columns for a little bit | 12:57 |
| marios | gfidente: right that is the whole point of the batch tasks i mean making l3 agent do it this way | 12:58 |
| EmilienM | trown: do you want to move these jobs today? | 12:58 |
| gfidente | marios yep but for ceph for example it takes a while after a node is back up to be 'in' the cluster | 12:58 |
| gfidente | so by relying only on the exit code you could end up with two nodes down in the same "short" period of time | 12:58 |
| marios | gfidente: right i see what you mean... but the downtime i saw yesterday was more like half hour not few mins/seconds | 12:59 |
| gfidente | ack | 12:59 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Add oooq multinode jobs to ci status page https://review.openstack.org/445937 | 12:59 |
| trown | EmilienM: lets make sure the ones we moved are working good first :) | 12:59 |
| trown | EmilienM: maybe next week if there are no snags | 12:59 |
| EmilienM | trown: indeed, reviewing now | 12:59 |
| EmilienM | trown: +2 | 13:00 |
| trown | EmilienM: lots of columns will be a bit ugly for a while, but it also will show if we have some places where we are missing coverage | 13:00 |
| EmilienM | trown: that's fine I guess and it's not super critical. | 13:00 |
| *** links has quit IRC | 13:02 | |
| *** liverpooler has joined #tripleo | 13:03 | |
| *** zoli is now known as zoli|afk | 13:03 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement verify profiles as a custom action https://review.openstack.org/429684 | 13:04 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check ironic boot configuration as a custom action https://review.openstack.org/428225 | 13:04 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check flavors as a custom action https://review.openstack.org/428178 | 13:04 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check node counts as a custom action https://review.openstack.org/429757 | 13:04 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add workflow wrapping all predeployment validations https://review.openstack.org/445914 | 13:04 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check boot images as a custom action https://review.openstack.org/428177 | 13:04 |
| gfidente | thrash ^^ sounds cool | 13:04 |
| thrash | gfidente: trying to get all that out of the client. :) | 13:04 |
| gfidente | ack | 13:04 |
| trown | thrash: nice | 13:04 |
| lhinds | trown: I have the following module: https://github.com/openstack/tripleo-puppet-elements/search?utf8=%E2%9C%93&q=auditd | 13:05 |
| lhinds | its on trunk: https://trunk.rdoproject.org/centos7-master/current/ | 13:05 |
| lhinds | ctrl-f `puppet-auditd` | 13:05 |
| *** jcoufal has quit IRC | 13:06 | |
| lhinds | I did a manual install to verify the package and this installs as expected to: /usr/share/openstack-puppet/modules/auditd | 13:06 |
| lhinds | but the module is not appearing in the master overcloud.qcow2 image | 13:06 |
| lhinds | i.e. in /etc/puppet/modules symlinked to > /usr/share/openstack-puppet/modules | 13:07 |
| lhinds | I can't work out why, and I am positive it was there before when I tested it a while ago. | 13:07 |
| *** gbarros has joined #tripleo | 13:07 | |
| trown | lhinds: ah ya, I see... you added it if using modules from source, but not to openstack-puppet-modules meta package: https://review.rdoproject.org/r/gitweb?p=openstack%2Fopenstack-puppet-modules-distgit.git;a=shortlog;h=refs%2Fheads%2Frpm-master | 13:08 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Restrict mongodb memory usage https://review.openstack.org/419090 | 13:08 |
| trown | lhinds: so it is actually a packaging thing, but really easy to fix | 13:08 |
| lhinds | oh cool | 13:08 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add workflow wrapping all predeployment validations https://review.openstack.org/445914 | 13:08 |
| trown | lhinds: you see the other commits there adding some module... just submit one of those to RDO gerrit and I will review | 13:08 |
| lhinds | thanks trown , taking a look now | 13:09 |
| trown | lhinds: `git clone https://review.rdoproject.org/r/openstack/openstack-puppet-modules-distgit` will get you a checkout to submit a review | 13:10 |
| trown | then it is just same gerrit workflow as upstream | 13:10 |
| *** pgadiya has quit IRC | 13:11 | |
| *** eck` is now known as eck`gone | 13:12 | |
| *** jmelvin has joined #tripleo | 13:12 | |
| weshay | trown, adarazs containers job is timing out, need reviews on https://review.openstack.org/#/c/445840/1/toci_instack_oooq.sh | 13:12 |
| weshay | thanks in advance | 13:13 |
| adarazs | weshay: done, though I only have +1 there. | 13:13 |
| trown | weshay: why not just put that in the containers_minimal config? | 13:14 |
| *** oaken_chris has quit IRC | 13:14 | |
| trown | seems like if it doesnt work in CI it is not going to work for a user :P | 13:14 |
| weshay | it works, it just takes too long for upstream | 13:17 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Include gate.repo in image building if exists https://review.openstack.org/445951 | 13:17 |
| lhinds | trown: I can't see if I need to checkout a branch/create a topic, or just push to master? | 13:17 |
| trown | weshay: sshnaidm ok... I am still -1 on changing behavior in CI without changing the behavior for someone trying to reproduce CI | 13:17 |
| *** ccamacho|lunch is now known as ccamacho | 13:17 | |
| *** jcoufal has joined #tripleo | 13:18 | |
| trown | lhinds: push the change on rpm-master first | 13:18 |
| trown | lhinds: does that module exist on ocata or older? | 13:18 |
| lhinds | trown: ack, thanks | 13:18 |
| sshnaidm | trown, what are we talking about...? | 13:18 |
| lhinds | just ocata and > | 13:18 |
| lhinds | trown: ^ | 13:18 |
| *** tremble has left #tripleo | 13:18 | |
| weshay | trown, run_tempest is false in the containers_min config | 13:18 |
| trown | lhinds: ok, we can backport the patch to rpm-ocata after it merges on master | 13:19 |
| lhinds | trown: thx | 13:19 |
| trown | weshay: sshnaidm why do we need to override it if it is in TRIPLEO_ROOT/tripleo-ci/scripts/quickstart/containers_minimal.yml | 13:19 |
| * trown doubts that is true | 13:19 | |
| sshnaidm | trown, config is loaded first | 13:20 |
| weshay | it's set to false in tqe/config/general_config | 13:20 |
| sshnaidm | trown, ovb-settings file overrides it | 13:20 |
| lhinds | trown: https://review.rdoproject.org/r/5828 | 13:20 |
| trown | sshnaidm: why do we have tempest config in ovb-settings? | 13:20 |
| trown | sshnaidm: that seems like the real issue | 13:20 |
| sshnaidm | trown, firstly it's old mode job | 13:20 |
| sshnaidm | trown, and where should we have it then? | 13:21 |
| weshay | I'll create a fs config for containers today | 13:21 |
| sshnaidm | trown, this patch allows containers job to pass and not to be timeouted | 13:21 |
| *** saibarspeis has joined #tripleo | 13:22 | |
| trown | sshnaidm: tempest config should not be tied to an environment like that | 13:22 |
| sshnaidm | trown, anyway everything is gonna change with ovb there | 13:22 |
| weshay | trown, the background on this.. is that the containers job was voting, then moved to -nv because of the time it took for the entire run | 13:22 |
| sshnaidm | trown, it's really not issue of current revision, config files are in progress | 13:22 |
| *** prateek__ has quit IRC | 13:22 | |
| *** tzumainn has joined #tripleo | 13:23 | |
| *** limao has joined #tripleo | 13:23 | |
| trown | sshnaidm: it is the same argument you made about your fix vs pandas fix for multinode_mode yesterday | 13:23 |
| sshnaidm | trown, sorry, it's different | 13:23 |
| trown | sshnaidm: your patch is bandaid hack, but really we should not even be tying tempest config to environment like htat | 13:24 |
| sshnaidm | trown, we didn't transit ovb jobs | 13:24 |
| chandankumar | trown: please have a look on these reviews: https://review.openstack.org/441599 https://review.openstack.org/444181 https://review.openstack.org/444277 | 13:24 |
| *** psahoo has joined #tripleo | 13:24 | |
| *** saibarspeis has quit IRC | 13:24 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the baremetal workflows to use the multiple message interface https://review.openstack.org/443053 | 13:24 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Respond to multiple messages from workflow executions https://review.openstack.org/406178 | 13:24 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements master: Adding networking-l2gw packages to overcloud-controller element https://review.openstack.org/439574 | 13:24 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Switch keystone default provider to fernet https://review.openstack.org/432420 | 13:24 |
| openstackgerrit | Merged openstack/tripleo-ui master: Remove karma, jasmine, and phantomjs dependencies https://review.openstack.org/443153 | 13:24 |
| openstackgerrit | Merged openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/445836 | 13:24 |
| sshnaidm | trown, that's right, it won't be tied when ovb is ready for transition | 13:24 |
| openstackgerrit | Merged openstack/tripleo-ui master: Add puppet to "adding a new language" docs https://review.openstack.org/440640 | 13:25 |
| *** Goneri has joined #tripleo | 13:25 | |
| trown | I should be able to run locally with config that is used upstream and get the same behavior... if you put behaviorial stuff in the environment config I cant do that | 13:25 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Mark 'undercloud deploy' as experimental https://review.openstack.org/445523 | 13:25 |
| sshnaidm | trown, ovb jobs are in progress now, what you see is not the final version of it | 13:25 |
| *** jtomasek has quit IRC | 13:25 | |
| sshnaidm | trown, I have pretty big ovb change in pipeline: https://review.openstack.org/#/c/431567/ | 13:26 |
| jistr | what about if we land the patch and reiterate on it? | 13:26 |
| trown | I just pushed it, with comment that I dont like it :P | 13:26 |
| sshnaidm | trown, weshay panda actually you can review it, maybe give it a chance to run a few days till Monday: https://review.openstack.org/#/c/431567/ | 13:27 |
| *** rbrady has joined #tripleo | 13:28 | |
| *** cdearborn has joined #tripleo | 13:28 | |
| *** jtomasek has joined #tripleo | 13:28 | |
| weshay | sshnaidm, ack | 13:28 |
| *** rhallisey has quit IRC | 13:29 | |
| panda | sshnaidm: I'll need to convert it to new format, whatever is not yet converted anyway | 13:29 |
| sshnaidm | but it does NOT include featureset and nodes config files yet | 13:29 |
| sshnaidm | panda, I don't think you need to change the playbook | 13:29 |
| sshnaidm | panda, configurations should be redefined of course.. | 13:30 |
| panda | no, the playbook can only be moved to the new position | 13:30 |
| sshnaidm | panda, so what do you need to convert? | 13:31 |
| panda | sshnaidm: ovb-settings, ovb-ha-oooq, ovb-nonha-oooq, ovb-containers-oooq it seems | 13:31 |
| sshnaidm | panda, I thought it should be done with featuresets in extras repo | 13:32 |
| panda | sshnaidm: to env specific config, featureset, nodes. | 13:32 |
| panda | yes | 13:32 |
| sshnaidm | panda, sure | 13:32 |
| sshnaidm | panda, but I want this playbook to run a little anyway | 13:33 |
| panda | actually, toci specific, env specific but not limited to toci, featureset, and nodes. | 13:33 |
| *** amoralej is now known as amoralej|lunch | 13:34 | |
| jistr | mandre, dprince, stevebaker, Slower: guys could you please review the data preservation patches when you have some bandwidth https://review.openstack.org/#/q/status:open+project:openstack/tripleo-heat-templates+branch:master+topic:containers-keep-data | 13:35 |
| sshnaidm | trown, btw, the promote jobs didn't run because of centos outage, can we do something or just to wait? https://ci.centos.org/job/tripleo-dlrn-promote/ | 13:35 |
| *** eck`gone is now known as eck` | 13:36 | |
| trown | sshnaidm: we just have to wait, but outage might be over... saw discussion about dlrn in #rdo... we can check | 13:36 |
| sshnaidm | trown, jenkins is still disabled | 13:36 |
| panda | adarazs: did you see https://bugs.launchpad.net/tripleo/+bug/1673046 ? there's also a card https://trello.com/c/slHYbwZp/627-q-to-u-add-timestamps-to-logs-to-get-linkable-lines?menu=filter&filter=label:%5BQ%20to%20U%5D | 13:36 |
| openstack | Launchpad bug 1673046 in tripleo "quickstart collected logs are missing timestamps" [High,Triaged] | 13:36 |
| trown | sshnaidm: but actual dlrn server is within that outage, we only have the mirror | 13:37 |
| adarazs | panda: yeah, I was thinking about that some time ago that it would be useful | 13:37 |
| adarazs | good that we have a card and bug for it. | 13:38 |
| *** trown is now known as trown|brb | 13:38 | |
| *** suuuper has joined #tripleo | 13:39 | |
| panda | adarazs: now it's more than just useful, it's *needed* at the highest level possible, heads will roll without it, the earth will open and swallow entire cities. | 13:40 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Upgrade webpack dependency to 2.x https://review.openstack.org/445961 | 13:40 |
| *** zoli|afk is now known as zoli | 13:40 | |
| *** aufi has quit IRC | 13:40 | |
| *** bfournie has quit IRC | 13:40 | |
| *** hewbrocca is now known as hewbrocca_afk | 13:42 | |
| panda | sshnaidm: | 13:42 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Upgrade configuration to webpack 2 https://review.openstack.org/443137 | 13:42 |
| panda | sshnaidm: http://logs.openstack.org/66/445866/9/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/0f811c3/console.html passed, using deploy.env and ocata stable branch | 13:42 |
| panda | sshnaidm: sorry, newton | 13:42 |
| panda | sshnaidm: but not sure if it's actually using newton yet. | 13:43 |
| panda | checking | 13:43 |
| EmilienM | dprince: hey! can you please review https://review.openstack.org/#/c/445700/ and tell me how it looks for containers and if I need to change something else | 13:43 |
| adarazs | panda: looks like it's a relatively easy fix, but it has to be applied at a couple of places... | 13:44 |
| adarazs | <command> | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }' | 13:44 |
| openstackgerrit | Thomas Herve proposed openstack/python-tripleoclient master: Use a Zaqar queue to get stack events https://review.openstack.org/394420 | 13:44 |
| jaosorior | EmilienM: hey, it seems to me that the current puppet-keystone manifest will cause issues with fernet tokens. If we do rotation through other means, the hieradata won't be updated, so running puppet again would overwrite the keys. | 13:45 |
| EmilienM | jaosorior: indeed | 13:45 |
| panda | sshnaidm: I think we've got it: http://logs.openstack.org/66/445866/9/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/0f811c3/logs/subnode-1/var/log/extra/rpm-list.txt.gz show packages for newton on the overcloud | 13:45 |
| EmilienM | jaosorior: I didn't specify it in the spec yet, probably worth updating it | 13:46 |
| EmilienM | jaosorior: but puppet shouldn't handle the keys anymore | 13:46 |
| panda | sshnaidm: I'll remove the WIP in the change | 13:46 |
| *** jtomasek has quit IRC | 13:46 | |
| panda | and the STABLE_BRANCH force | 13:46 |
| akrivoka | florianf: can you link me to the oooq instructions? | 13:46 |
| florianf | akrivoka: I tried the ones from the repo readme: https://github.com/openstack/tripleo-quickstart/ | 13:47 |
| akrivoka | florianf: thank you | 13:48 |
| panda | akrivoka: need help for something in particular ? | 13:48 |
| akrivoka | panda: instack-virt-setup just failed for me while trying to install a fresh environment... so I guess I should finally give oooq a try :) | 13:49 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Simplify addition of new languages https://review.openstack.org/440788 | 13:49 |
| panda | akrivoka: \o/ | 13:49 |
| *** psahoo has quit IRC | 13:49 | |
| sshnaidm | panda, great.. also need this too: https://review.openstack.org/#/c/445875/ | 13:51 |
| jaosorior | EmilienM: does puppet purge the directories when it runs? | 13:51 |
| panda | sshnaidm: yep, +1'd | 13:52 |
| sshnaidm | trown|brb, can you please review the dependency? https://review.openstack.org/#/c/442139/ | 13:53 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Add language switcher to login page https://review.openstack.org/441435 | 13:53 |
| *** trown|brb is now known as trown | 13:53 | |
| EmilienM | jaosorior: no | 13:53 |
| EmilienM | jaosorior: unless you ask for it :) | 13:53 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Highlight current i18n language in navbar https://review.openstack.org/443191 | 13:54 |
| sshnaidm | weshay, we need a tag oooq-transit-stage2 :) | 13:54 |
| openstackgerrit | Merged openstack/puppet-tripleo master: HAProxy: Refactor certificate retrieval bits https://review.openstack.org/444850 | 13:54 |
| weshay | sshnaidm, aye | 13:54 |
| *** lblanchard has joined #tripleo | 13:55 | |
| jaosorior | EmilienM: great | 13:57 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: oooq multinode: fix for stable branches https://review.openstack.org/445866 | 13:57 |
| *** udesale has joined #tripleo | 13:59 | |
| *** nyechiel has joined #tripleo | 13:59 | |
| EmilienM | panda: what about commented code? https://review.openstack.org/#/c/445866/10/scripts/common_functions.sh | 13:59 |
| *** pkovar has quit IRC | 14:00 | |
| panda | EmilienM: I love commented code :). The fact is I don't know what we're going to implement first in oooq, if the bootstrap role or the 3node jobs, if we are going to implement 3nodes first, then we'll need that commented code, and I'm sure we're going to forget it. Maybe I can just leave a TODO to remember to add the removed part in the function ? | 14:02 |
| *** oaken_chris has joined #tripleo | 14:02 | |
| *** pkovar has joined #tripleo | 14:05 | |
| *** links has joined #tripleo | 14:05 | |
| weshay | adarazs, wondering if https://review.openstack.org/#/c/430277/ should trigger the multinode jobs? | 14:05 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs master: Remove otoo CSS class https://review.openstack.org/445974 | 14:05 |
| adarazs | weshay: yes it should. I will look into it what we have to touch to trigger them | 14:06 |
| trown | adarazs: weshay I think those only run if something changes in tht or puppet-tripleo | 14:08 |
| openstackgerrit | Merged openstack/tripleo-docs master: Update installation instructions to include stable/ocata branch https://review.openstack.org/445676 | 14:09 |
| trown | adarazs: so we probably need some dependent change in puppet-tripleo for the gate check | 14:09 |
| dprince | jistr: ack, I will look at your patches today | 14:09 |
| jistr | dprince: thanks | 14:09 |
| EmilienM | panda: yeah, to be more clear in the intent | 14:10 |
| *** shardy_lunch is now known as shardy | 14:10 | |
| *** fultonj has joined #tripleo | 14:10 | |
| panda | EmilienM: ok | 14:10 |
| adarazs | trown: really? I didn't know it works like that in zuul that a dependent change can trigger a job in another project... | 14:10 |
| *** jlinkes_ is now known as jlinkes | 14:10 | |
| openstackgerrit | John Trowbridge proposed openstack/puppet-tripleo master: DNM testing scenario gates https://review.openstack.org/445977 | 14:11 |
| trown | adarazs: doing a quick check with ^ since afaik we have not actually tested those since switching | 14:12 |
| *** bkopilov has joined #tripleo | 14:12 | |
| adarazs | trown: cool. | 14:12 |
| adarazs | looks like they got triggered. good | 14:13 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: oooq multinode: fix for stable branches https://review.openstack.org/445866 | 14:13 |
| *** dprince has quit IRC | 14:14 | |
| *** funzo_ has joined #tripleo | 14:14 | |
| shardy | jkilpatr: Hey, just reading backscroll, yeah there's been discussion about having a larger periodic job, but I think with our current capacity it's very unlikely we could do such a test in the gate queue | 14:14 |
| gfidente | jistr so the reason for chown in https://review.openstack.org/#/c/441146/ is that uid might be different? | 14:14 |
| jkilpatr | shardy, what exactly is the capacity issue? you can get away with 2gb ram compute nodes although control and undercloud need to be beefy | 14:15 |
| trown | shardy: jkilpatr, maybe if it was a third party job from some other hardware pool though | 14:15 |
| *** aufi has joined #tripleo | 14:15 | |
| shardy | jkilpatr: we're running close to our current ovb cloud capacity as it is, so I think adding ~30 node gate jobs will consume too much additional resources | 14:15 |
| *** egafford has left #tripleo | 14:16 | |
| shardy | but yeah, if we find more hardware, perhaps it can be done :) | 14:16 |
| trown | ya it would really need to be a dedicated env that only runs the 30 node job | 14:16 |
| jkilpatr | trown, I have hardware and I'm building an OVB job of that scale right now | 14:16 |
| shardy | bnemec: ^^ do you have stats on our current ovb utilistion by any chance? | 14:16 |
| jkilpatr | do we have anything to take and ovb stack and make a hosts file / stackrc? | 14:16 |
| shardy | jkilpatr: ah, yeah if there's a dedicated cloud for this then I guess it's possible | 14:17 |
| skramaja | chem: if you remember, i was talking about dpdk update scripts for ovs2.6.. we are able to do this part of OS::TripleO::NodeExtraConfigPost resource. | 14:17 |
| skramaja | chem: here is the script for youre refernce - https://github.com/krsacme/tht-dpdk/blob/master/post-install-update.yaml#L75 | 14:17 |
| bnemec | shardy: It's not great, especially since we added the containers job to the check queue (which I would have nacked if I had known about it in advance) | 14:18 |
| jkilpatr | shardy, my team ends up dealing with the scale problems once they make their way downstream solving them early sounds far more ideal | 14:18 |
| bnemec | However, we might be able to add a periodic job at scale. | 14:18 |
| jkilpatr | bnemec, anything would be nice, even if it's kicked off in the dead of night on Saturday morning. | 14:18 |
| *** funzo_ has quit IRC | 14:19 | |
| weshay | ya.. a daily periodic for perf seems reasonable to me | 14:19 |
| weshay | jkilpatr, would that be sufficient? | 14:19 |
| bnemec | jkilpatr: Yeah, I think that would be doable. The big thing is that then someone needs to be very disciplined to keep track of it and report problems when they occur. | 14:19 |
| weshay | bnemec, we're building tools out now for that kind of reporting on periodic jobs | 14:19 |
| weshay | for tempest atm | 14:19 |
| weshay | but could also be used for perf | 14:20 |
| *** pradk has joined #tripleo | 14:20 | |
| jkilpatr | weshay, bnemec I'm fine with owning a 30+ node upstream job for TripleO | 14:20 |
| jkilpatr | rook, ^ | 14:20 |
| weshay | +1111 from me :) | 14:20 |
| panda | so 1141 nodes ? | 14:20 |
| bnemec | Heh | 14:20 |
| bnemec | We've got some free quota, but probably not _that_ much. | 14:21 |
| shardy | jkilpatr: yeah +100 on getting earlier coverage, I just don't think our current ovb cloud can handle it | 14:21 |
| rook | i am catching up here.. | 14:21 |
| chem | skramaja: hi, looking | 14:21 |
| jistr | gfidente: yes exactly | 14:21 |
| bnemec | shardy: We could handle running one 30 node periodic job at night. | 14:21 |
| weshay | bnemec, jkilpatr /me referring to https://review.openstack.org/#/c/420878/ | 14:21 |
| shardy | bnemec: yeah, agreed, that's why I said we'd previously discussed a scale periodic job | 14:21 |
| jistr | gfidente: we need to run the chown using the same container image which will then run the service | 14:22 |
| rook | so, weshay jkilpatr bnemec shardy this would simply be a scale play with OOO, not so much performance. | 14:22 |
| shardy | but we've not got as far as actually doing it yet | 14:22 |
| bnemec | I wouldn't even want to try running it in the experimental queue though. | 14:22 |
| jkilpatr | rook, just to make sure it deploys, babysteps | 14:22 |
| rook | we would need to define _what_ we are testing... | 14:22 |
| rook | ie - monolithic controllers, ceph?, etc | 14:22 |
| * weshay wonders why not get browbeat running in a periodic? because it's not bare metal? | 14:22 | |
| rook | weshay right. | 14:23 |
| gfidente | jistr yeah I see, I was just scared by the idea of the host not having permissions anymore | 14:23 |
| jkilpatr | weshay, we can and if you give me this job I will | 14:23 |
| *** dprince has joined #tripleo | 14:23 | |
| rook | weshay if we can have the periodic job land on our microwbrow system - sure. | 14:23 |
| bnemec | Note that anything running on rh1 is going to be pretty useless for performance testing in isolation. | 14:23 |
| jistr | gfidente: Kolla already does chown, but not recursive. Hopefully we'll ship our own entrypoints eventually and get rid of some of those init containers, but for now we need them for the BM->containers to work. | 14:23 |
| rook | bnemec: that is the fear. | 14:23 |
| jkilpatr | I'm building a downstream version of this anyways because I don't think you'll take kindly to me hogging instances for 12 hours + | 14:23 |
| bnemec | As the load on the cloud varies, so do the runtimes. By like 50%. | 14:23 |
| gfidente | jistr but still, who cares, it's going to run in the container | 14:23 |
| openstackgerrit | John Fulton proposed openstack/tripleo-specs master: Deriving TripleO Parameters https://review.openstack.org/423304 | 14:23 |
| jkilpatr | So I'll do that on my own hardware. | 14:23 |
| fultonj | skramaja: ^ | 14:23 |
| gfidente | jistr I suppose if upgrade fails we repeat until it works | 14:24 |
| bnemec | So testing that you can deploy 30 nodes is fine, testing how long it takes is probably madness. | 14:24 |
| weshay | rook, jkilpatr a daily periodic job that runs against your internal hw.. I think is possible now that we have the delorean api moving to production | 14:24 |
| rook | weshay bnemec shardy jkilpatr shall we jump on a call to avoid extra noise? | 14:24 |
| weshay | I'm in a mtg atm | 14:24 |
| jistr | gfidente: so host root will have permission, and yea we don't care about host mariadb having permission | 14:24 |
| weshay | but would like to see this move forward | 14:24 |
| rook | yeah, i have a call right now weshay | 14:25 |
| gfidente | jistr is it worth using -L too? | 14:25 |
| rook | but sometime today | 14:25 |
| shardy | rook: happy to discuss, but should we start with an openstack-dev thread? | 14:25 |
| weshay | +1 | 14:25 |
| gfidente | jistr I had to do that for ceph in the upgrade scripts because there were symlinks | 14:25 |
| rook | shardy given this is a internal resource, do we? | 14:25 |
| shardy | if someone is prepared to drive this and commit time/hardware to adding coverage, I think everyone is basically in favor :) | 14:25 |
| shardy | rook: I thought we were talking about an upstream CI job | 14:25 |
| jaosorior | EmilienM: did you get a chance to check if etcd still only supports Basic Auth? | 14:26 |
| jistr | gfidente: hm that could potentially traverse outside that directory so i think we probably don't want that | 14:26 |
| rook | shardy weshay jkilpatr this would be a u/s job? | 14:26 |
| jkilpatr | rook, u/s? | 14:26 |
| * rook might need to scroll up even more. | 14:26 | |
| rook | upstream | 14:26 |
| jkilpatr | yup | 14:26 |
| jkilpatr | on rdo cloud | 14:26 |
| weshay | ya | 14:26 |
| gfidente | jistr it goes outside the directory yes, how do symlinks behave in the container? | 14:27 |
| EmilienM | jaosorior: what do you mean by "Basic Auth" ? | 14:27 |
| jistr | gfidente: i'd rather be as conservative in our chown/chmod frenzy as possible :) | 14:27 |
| jaosorior | EmilienM: https://coreos.com/etcd/docs/latest/v2/auth_api.html | 14:27 |
| skramaja | thanks fultonj, i will look at it.. | 14:27 |
| rook | since I have no clue what you are signing up for jkilpatr maybe we should chat about it. | 14:27 |
| gfidente | jistr if they are followed I suppose we want the -L | 14:28 |
| EmilienM | jaosorior: ahh, let me check v3 | 14:28 |
| bnemec | rook: jkilpatr weshay: Just let me know if/when you want to meet. | 14:28 |
| weshay | rlandy, fyi ^ | 14:28 |
| EmilienM | jaosorior: it seems like v3 miss the doc for auth | 14:29 |
| jaosorior | EmilienM: yeah :( | 14:29 |
| jaosorior | I was noticing that | 14:29 |
| jistr | gfidente: they behave normally, i.e. they are followed, and i'm almost sure they're resolved locally within the container | 14:29 |
| jistr | gfidente: i don't think we want the -L though | 14:29 |
| gfidente | oh resolved in the container, right | 14:30 |
| jistr | gfidente: everything in /var/lib/mysql will be chowned anyway, and we don't want to traverse out | 14:30 |
| jaosorior | EmilienM: what are we currently using? v2, right? | 14:30 |
| EmilienM | jaosorior: AFIK v2 | 14:30 |
| jistr | gfidente: i'd only go for -L if really necessary. Because if for some strike of bad luck you'd have '/var/lib/mysql/root' pointing to '/', then bye bye :D | 14:31 |
| jistr | it shouldn't be there i know... but better be safe than sorry i think :) | 14:31 |
| gfidente | jistr indeed but the problem is that if we need to change ownership of data linked from /var/lib/mysql/data we'd need the -L | 14:32 |
| EmilienM | jaosorior: just fyi about releasenotes/notes/Switch-keystone's-default-token-provider-to-fernet-2542fccb5a588852.yaml | 14:32 |
| EmilienM | jaosorior: don't put ' in file names, I think it's not correct | 14:32 |
| jaosorior | EmilienM: it should work in theory :D | 14:32 |
| EmilienM | until some tooling breaks :D | 14:33 |
| jaosorior | EmilienM: anyway, will avoid it in the future | 14:33 |
| jaosorior | the tools are broken :P | 14:33 |
| EmilienM | jaosorior: it worked, I checked | 14:33 |
| EmilienM | jaosorior: but just in case, let's avoid that | 14:33 |
| jaosorior | ok | 14:33 |
| rook | bnemec wrt the RDO Cloud - this too would be nested virt -- and could have some neighbors that make performance/response times look bad.... | 14:34 |
| dprince | jistr: I think I got them all. +2 | 14:34 |
| dprince | jistr: ran 'check experimental' to see if the containers job passes | 14:34 |
| *** ckyriakidou has quit IRC | 14:34 | |
| jaosorior | EmilienM: https://github.com/coreos/etcd/blob/master/Documentation/op-guide/security.md | 14:34 |
| dprince | jistr: pending that please +A them :) | 14:34 |
| jistr | gfidente: but do we really need that? :) i mean why add danger if the need for the benefit it's adding is purely hypothetical... | 14:35 |
| gfidente | jistr agreed, if we don't need it, then it's not worth | 14:35 |
| gfidente | danger is hypothetical too | 14:35 |
| bnemec | rook: Well, the deployment wouldn't be nested. But any instances you booted on the resulting cloud would be, yes. | 14:35 |
| bnemec | And noisy neighbors are definitely a thing. | 14:35 |
| *** marst has quit IRC | 14:36 | |
| bnemec | Of course, tripleo-ci is likely to be the noisiest neighbor in RDO Cloud for the foreseeable future. | 14:36 |
| bnemec | :-) | 14:36 |
| jistr | gfidente: all danger is hypothetical :P danger = hypothetical accident waiting to happen; non-hypothetical danger == actual accident | 14:36 |
| jkilpatr | bnemec, not looking for perfect numbers, we have our own hardware for that, mostly looking to draw the attention of devs to scale problems. | 14:36 |
| *** karthiks is now known as karthiks_afk | 14:36 | |
| rook | bnemec that is my concern... | 14:36 |
| EmilienM | jaosorior: it looks quite secure, isn't? | 14:36 |
| gfidente | jistr you better hope there aren't symlinks :) | 14:37 |
| jkilpatr | A lot of bugs we report get the response "oh we've never played with that many nodes before" | 14:37 |
| jkilpatr | that's a problem. | 14:37 |
| rook | jkilpatr the issue with that is you will possibly have tons of false positives. | 14:37 |
| rook | so it will be more noise then anything | 14:37 |
| jistr | gfidente: if they point only within /var/lib/mysql, they should keep working imo | 14:37 |
| jaosorior | EmilienM: it looks nice. But now we get into the classic issue of handling certs/keys and all these sorts of things. | 14:37 |
| *** skramaja has quit IRC | 14:38 | |
| jaosorior | EmilienM: this could easily be arranged with the TLS-everywhere bits. Cause we would already have keys/certs for everything. But we don't want to enforce TLS-everywhere for all deployments | 14:38 |
| gfidente | jistr yeah so the story is that ceph creates symlinks in /var/lib/ceph/osd pointing to whichever path you provide for data | 14:38 |
| jkilpatr | rook, for perf numbers or just deployment? | 14:38 |
| jaosorior | EmilienM: another option would be to enforce having a CA. | 14:38 |
| gfidente | jistr which is why in that scenario I had to add -L (which initially I did not have) https://github.com/openstack/tripleo-heat-templates/commit/45a7fbb2104e98d9fa269da80ba1259999c1f0e7 | 14:39 |
| *** fragatina has quit IRC | 14:39 | |
| rook | jkilpatr so there are two discussions here then. 1) Scale deployments in CI. 2) Running performance in upstream CI | 14:39 |
| EmilienM | jaosorior: for etcd, I think it would make sense to have it | 14:39 |
| jistr | dprince: thanks! | 14:39 |
| EmilienM | jaosorior: though it could be optional | 14:39 |
| rook | if this discussion is about 1) sure. Then I am good with this. | 14:39 |
| rook | if it is about 2) no, I think it is useless. | 14:39 |
| *** ukalifon has joined #tripleo | 14:39 | |
| EmilienM | jaosorior: like "if rotation is enabled, then we need TLS" | 14:39 |
| *** rook is now known as rook-mtg | 14:39 | |
| *** ukalifon has quit IRC | 14:39 | |
| *** fragatina has joined #tripleo | 14:40 | |
| jaosorior | EmilienM: "if rotation is enabled, we need a CA availale" | 14:40 |
| EmilienM | jaosorior: yea | 14:40 |
| jkilpatr | rook-mtg, I agree 2 won't be super useful, but if 1 is the best we can do I'll be happy. | 14:40 |
| *** snecklifter has quit IRC | 14:40 | |
| jistr | gfidente: right in the Ceph case -L is probably necessary then | 14:41 |
| jistr | gfidente: thanks for the review :) | 14:41 |
| *** d0ugal has quit IRC | 14:42 | |
| *** links has quit IRC | 14:42 | |
| *** saibarspeis has joined #tripleo | 14:42 | |
| *** marst has joined #tripleo | 14:42 | |
| jaosorior | EmilienM: Lance mentioned they might be open to having a etcd driver in keystone. Have you poked him about that? | 14:43 |
| jaosorior | EmilienM: cause that would be really cool | 14:43 |
| *** d0ugal has joined #tripleo | 14:43 | |
| EmilienM | jaosorior: yeah, see the spec | 14:44 |
| EmilienM | jaosorior: unless you saw it in the spec | 14:44 |
| EmilienM | jaosorior: that would solve a lot of our issues | 14:44 |
| *** snecklifter has joined #tripleo | 14:44 | |
| mandre | jistr: there is one last named volume in nova_ironic | 14:44 |
| *** psahoo has joined #tripleo | 14:45 | |
| jaosorior | EmilienM: exactly | 14:48 |
| jaosorior | EmilienM: ok, lets figure out the details of how we deploy etcd, and what we need to arrange it | 14:48 |
| *** saibarspeis has quit IRC | 14:49 | |
| *** saibarspeis has joined #tripleo | 14:52 | |
| *** yamahata has joined #tripleo | 14:55 | |
| *** bfournie has joined #tripleo | 14:56 | |
| jistr | mandre: thanks! i'll check it out | 15:00 |
| *** agurenko has quit IRC | 15:00 | |
| *** jtomasek has joined #tripleo | 15:01 | |
| *** amoralej|lunch is now known as amoralej | 15:02 | |
| *** aufi has quit IRC | 15:03 | |
| EmilienM | dprince, jistr: I saw comments on https://review.openstack.org/#/c/445700/ - should I add the file in kolla_config or not? | 15:04 |
| EmilienM | I'm ok either way, just pick one solution | 15:04 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Added script to validate jinja templates https://review.openstack.org/441599 | 15:05 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Added tempest_version in validate-tempest role https://review.openstack.org/444181 | 15:06 |
| *** limao has quit IRC | 15:10 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Revert "Revert "Deploy heat APIs over httpd"" https://review.openstack.org/441776 | 15:10 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Make decision wheter to build images in CI or not https://review.openstack.org/442139 | 15:10 |
| *** limao has joined #tripleo | 15:11 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Parameterize OVB stack flavors https://review.openstack.org/445163 | 15:11 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Parameterize OVB stack flavors https://review.openstack.org/445163 | 15:11 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: New ovb playbook and settings https://review.openstack.org/431567 | 15:13 |
| *** hewbrocca_afk is now known as hewbrocca | 15:14 | |
| *** dhill_ has joined #tripleo | 15:14 | |
| *** rhallisey has joined #tripleo | 15:14 | |
| *** funzo_ has joined #tripleo | 15:15 | |
| EmilienM | dprince, jistr: thoughts? | 15:16 |
| EmilienM | I'm about to change files or not :-) | 15:16 |
| *** saibarspeis has quit IRC | 15:17 | |
| *** funzo_ has quit IRC | 15:20 | |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud stable/ocata: Align stars to fix CI https://review.openstack.org/446011 | 15:20 |
| jistr | EmilienM, dprince: whatevs :) personally i'd prefer the change coherent to what the commit message says (i.e. really make it work on containers and not rely on future features) and i think we probably should do a recheck anyway given that the container CI is red and the patch changes container things as well, BUT it won't break my arm to merge that patch if you guys think it's ok to merge :D | 15:21 |
| *** saneax is now known as saneax-_-|AFK | 15:22 | |
| *** aufi has joined #tripleo | 15:22 | |
| EmilienM | jistr: I don't have strong opinion on this one, i'll let dprince choose | 15:22 |
| sshnaidm | trown, forgot to mention, yet another perf problem - it's running virt-customize on qcow image. When it's done in tripleo.sh it takes less then running virt-customize: https://github.com/openstack-infra/tripleo-ci/blob/482ccc8ffa824afdeea4600c6da48f17a285da06/scripts/common_functions.sh#L76 | 15:24 |
| jistr | dprince: the weight of the decision is on you man :D | 15:25 |
| sshnaidm | trown, I provide it 4096M ram and 4 cpus, but still takes about 10 mins | 15:25 |
| sshnaidm | trown, and it doesn't work with initramfs images | 15:25 |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: add caching the GetParametersAction https://review.openstack.org/444220 | 15:25 |
| *** ealcaniz has quit IRC | 15:25 | |
| sshnaidm | trown, so maybe make sense to add these options to modify-image role.. | 15:26 |
| trown | sshnaidm: ya seems kind of big feature to add though | 15:28 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Don't disable satellite repo after registration https://review.openstack.org/446014 | 15:28 |
| trown | sshnaidm: can we give virt-customize more RAM or is 4G all we have? | 15:28 |
| sshnaidm | trown, maybe kind of choosing way of modifying like: "libguestfs_mode: true/false" | 15:29 |
| sshnaidm | trown, I'll try to add a bit more, but we're close. I even moved it to do before undercloud install to have more free memory | 15:29 |
| trown | sshnaidm: right that is the easy part... actually implementing libguestfs_mode=false seems large (to do in a generic way and not just copy what is in tripleo.sh) | 15:29 |
| sshnaidm | trown, ok, it could be planned and implemented later, if we agree on this task at all | 15:30 |
| trown | sshnaidm: seems useful ... especially for ironic-python-agent case | 15:31 |
| *** jlinkes has quit IRC | 15:31 | |
| adarazs | trown, weshay: FYI gate-tripleo-ci-centos-7-scenario001-multinode-oooq: SUCCESS on trown's dummy gate job, so scenario001 is working. | 15:31 |
| trown | adarazs: woot... time for a beer | 15:32 |
| trown | jk, it is 11:30am :P | 15:32 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix stackviz report generation https://review.openstack.org/444267 | 15:33 |
| sshnaidm | adarazs, trown you can track them here http://status-tripleoci.rhcloud.com/ | 15:34 |
| adarazs | trown: right, what were we thinking. we clearly need something stronger like rum or a whisky. | 15:34 |
| *** fpan has quit IRC | 15:35 | |
| *** fragatina has quit IRC | 15:35 | |
| trown | :) | 15:36 |
| EmilienM | / pradk | 15:36 |
| EmilienM | pradk: oops | 15:36 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Allow to configure policy.json for OpenStack projects https://review.openstack.org/445700 | 15:36 |
| pradk | :) | 15:37 |
| *** nyechiel has quit IRC | 15:37 | |
| *** nyechiel has joined #tripleo | 15:38 | |
| *** sanagikoki has joined #tripleo | 15:38 | |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: Stop uploading puppet artifacts to the overcloud nodes https://review.openstack.org/423190 | 15:39 |
| bandini | jaosorior: so I think I have an idea why httpd restarts fail on upgrade. I updated https://bugs.launchpad.net/tripleo/+bug/1673030 accordingly | 15:39 |
| openstack | Launchpad bug 1673030 in tripleo "N-O upgrades get httpd restart failures" [High,New] | 15:39 |
| *** dmarlin has joined #tripleo | 15:39 | |
| *** number80 has joined #tripleo | 15:40 | |
| number80 | Hi, downstream reviews needing your attention: https://review.rdoproject.org/r/#/c/5831/ | 15:40 |
| *** fpan has joined #tripleo | 15:40 | |
| *** fabbione has quit IRC | 15:41 | |
| *** limao has quit IRC | 15:42 | |
| *** fabbione has joined #tripleo | 15:42 | |
| *** limao has joined #tripleo | 15:42 | |
| jaosorior | bandini: why would heat be binding to that address? O_o | 15:45 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test ovb-oooq https://review.openstack.org/431103 | 15:45 |
| bandini | jaosorior: since it is during upgrade I guess we have some conf stuff missing ? | 15:45 |
| bandini | I *think* this is before puppet runs again | 15:45 |
| jaosorior | bandini: also, heat shouldn't even be running over httpd in ocata | 15:45 |
| bandini | jaosorior: oh | 15:45 |
| *** PhilSliderS has quit IRC | 15:45 | |
| jaosorior | bandini: that was introduced in pike | 15:46 |
| EmilienM | number80: thx. Approved. | 15:46 |
| bandini | jaosorior: hohum. so something in ci is busted? | 15:46 |
| bandini | hohum, i'll look after this call | 15:47 |
| number80 | EmilienM: thanks | 15:47 |
| jaosorior | bandini: where is that issue coming up? | 15:47 |
| bandini | jaosorior: https://review.openstack.org/#/c/444928/ the multinode-upgrades job | 15:47 |
| sshnaidm | trown, can you please re-review this? https://review.openstack.org/#/c/439198 | 15:48 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the register_or_update wrapper to use the multiple message interface https://review.openstack.org/443053 | 15:48 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Respond to multiple messages from workflow executions https://review.openstack.org/406178 | 15:48 |
| jaosorior | bandini: then yeah,s eems something is busted on CI | 15:48 |
| jaosorior | bandini: no upgrade step in stable/ocata even references heat over httpd | 15:49 |
| sshnaidm | panda, maybe this would help in case of failed_deployments log lost: https://review.openstack.org/#/c/443076/ | 15:49 |
| sshnaidm | panda, not sure it will solve, but seems logical.. | 15:49 |
| *** psahoo has quit IRC | 15:49 | |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: Stop adding FW rules for docker registry https://review.openstack.org/446024 | 15:51 |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: Rely on overcloud-prep-network to configure DNS https://review.openstack.org/446025 | 15:51 |
| *** tosky has quit IRC | 15:51 | |
| panda | sshnaidm: yes, I saw that, I'm considering it too, but I'm afraid is something more than a problem of catching deployment failure | 15:52 |
| *** PhilSliderS has joined #tripleo | 15:52 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add upgrade tasks for gnocchi container services https://review.openstack.org/445627 | 15:52 |
| sshnaidm | panda, yeah.. it's strange that we catch it in postci always | 15:52 |
| sshnaidm | panda, maybe commands differ.. | 15:52 |
| *** tosky has joined #tripleo | 15:53 | |
| *** limao has quit IRC | 15:56 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: DO NOT MERGE: Test for OVB yum update hang https://review.openstack.org/446026 | 15:56 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add bigswitch agent support https://review.openstack.org/446029 | 15:56 |
| *** mcornea has quit IRC | 15:57 | |
| dprince | EmilienM: I think kolla_config can wait for this https://review.openstack.org/#/c/445700/, But of course don't break CI | 15:57 |
| *** kjw3 has joined #tripleo | 15:58 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK https://review.openstack.org/430277 | 16:00 |
| *** artom has joined #tripleo | 16:00 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/puppet-tripleo master: Explicitly configure credentials used by ironic to access other services https://review.openstack.org/446037 | 16:00 |
| *** akrzos is now known as akrzos-spec | 16:01 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-specs master: Deriving TripleO Parameters https://review.openstack.org/423304 | 16:02 |
| *** jaosorior has quit IRC | 16:02 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test new ovb with images building https://review.openstack.org/445228 | 16:02 |
| *** jkilpatr_ has joined #tripleo | 16:03 | |
| chandankumar | trown: one more review https://review.openstack.org/#/c/444277/ | 16:04 |
| *** abehl has quit IRC | 16:04 | |
| *** trown is now known as trown|lunch | 16:05 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-docs master: Document how to configure OpenStack API policies https://review.openstack.org/446039 | 16:05 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras master: [WIP] overcloud-deploy: missing toci failure log test https://review.openstack.org/446042 | 16:06 |
| *** jkilpatr has quit IRC | 16:06 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] quickstart: test for missing failed deployment log https://review.openstack.org/446043 | 16:07 |
| bogdando | folks, PTAL https://bugs.launchpad.net/tripleo/+bug/1673136 | 16:08 |
| openstack | Launchpad bug 1673136 in tripleo "[docs] Containers deployment guide has misleading debugging-docker-puppet-py example" [Undecided,New] | 16:08 |
| mwhahaha | pradk: do you know what bug was for deployment failures from Could not find command 'panko-dbsync' | 16:10 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the register_or_update wrapper to use the multiple message interface https://review.openstack.org/443053 | 16:10 |
| pradk | mwhahaha, i did a recheck and it went away last i saw | 16:11 |
| mwhahaha | pradk: seeing it downstream so i'm wondering if it's just a package thing | 16:11 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Don't track added_files in deploy environment processing https://review.openstack.org/446045 | 16:12 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-docs master: Add Ansible tasks to container service yaml structure https://review.openstack.org/446046 | 16:12 |
| *** udesale has quit IRC | 16:12 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-specs master: Deriving TripleO Parameters https://review.openstack.org/423304 | 16:12 |
| pradk | mwhahaha, hmm possible | 16:13 |
| bandini | matbu: I might need a hand with https://bugs.launchpad.net/tripleo/+bug/1673030 since it seems more of a CI issue than anything else | 16:15 |
| openstack | Launchpad bug 1673030 in tripleo "N-O upgrades get httpd restart failures" [High,New] | 16:15 |
| *** karimb has quit IRC | 16:15 | |
| bandini | aka we should not even be installing heat under httpd in ocata. so this should not happen (TM) | 16:15 |
| *** funzo_ has joined #tripleo | 16:16 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: WIP - Add upgrade_batch_tasks to neutron-l3-agent https://review.openstack.org/445494 | 16:16 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Add lookup plugin for tripleo heat templates https://review.openstack.org/441164 | 16:16 |
| *** rlandy is now known as rlandy|afk | 16:17 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the provide workflow wrapper to use the multiple message interface https://review.openstack.org/446049 | 16:18 |
| panda | hey don't forget to review https://review.openstack.org/445866 to get oooq mutlinode to work properly with stable branches | 16:20 |
| *** funzo_ has quit IRC | 16:20 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-specs master: Enable TripleO to Deploy Ceph via Ceph Ansible https://review.openstack.org/387631 | 16:20 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Containerized undercloud - install python-heat-agents-ansible https://review.openstack.org/446052 | 16:21 |
| jistr | dprince, mandre: turns out we were missing the ansible heat-config hook for undercloud https://review.openstack.org/#/c/446052/ | 16:24 |
| jistr | wanted to post the same for doit.sh but i see stevebaker already did that :) https://github.com/dprince/undercloud_containers/commit/8edb31bd008577944894d54f0fd1160b8143e3b9 | 16:24 |
| mandre | jistr: nice catch | 16:25 |
| jistr | mandre: CI caught it :D | 16:25 |
| jistr | thanks dprince for insisting on the UC job pass, i'd have forgotten about that | 16:26 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the introspect workflow wrapper to use the multiple message interface https://review.openstack.org/446054 | 16:27 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: remove logs from subnode https://review.openstack.org/445902 | 16:27 |
| *** sshnaidm is now known as sshnaidm|nnl | 16:27 | |
| *** sshnaidm|nnl is now known as sshnaidm|bbl | 16:27 | |
| *** rbrady has quit IRC | 16:28 | |
| *** Vijayendra has quit IRC | 16:28 | |
| mandre | jistr: hmm the undercloud job is still experimental in t-h-t | 16:29 |
| *** aufi has quit IRC | 16:29 | |
| jistr | yea | 16:29 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Add lookup plugin for tripleo heat templates https://review.openstack.org/441164 | 16:29 |
| jistr | on CI too i guess, i don't see it in zuul, will request check experimental | 16:30 |
| *** Vijayendra has joined #tripleo | 16:30 | |
| *** rbrady has joined #tripleo | 16:31 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Set OS_AUTH_TYPE on undercloud stackrc https://review.openstack.org/443776 | 16:31 |
| *** hewbrocca is now known as hewbrocca_afk | 16:31 | |
| *** panda is now known as panda|bbl | 16:31 | |
| *** kbyrne has quit IRC | 16:32 | |
| mandre | EmilienM: any objection moving the undercloud-containers job to the check pipeline? | 16:33 |
| mandre | dprince: ^^ | 16:33 |
| EmilienM | is it stable? | 16:33 |
| EmilienM | does it provide logs from services? | 16:33 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the provide workflow wrapper to use the multiple message interface https://review.openstack.org/446049 | 16:34 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the introspect workflow wrapper to use the multiple message interface https://review.openstack.org/446054 | 16:34 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the register_or_update wrapper to use the multiple message interface https://review.openstack.org/443053 | 16:34 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Respond to multiple messages from workflow executions https://review.openstack.org/406178 | 16:34 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras master: [WIP] overcloud-deploy: missing toci failure log test https://review.openstack.org/446042 | 16:34 |
| * EmilienM brb 10 min | 16:34 | |
| *** dparkes has quit IRC | 16:34 | |
| *** fragatina has joined #tripleo | 16:34 | |
| mandre | EmilienM: ah! that's right, it's not pulling the logs out of the containers which make it difficult to debug | 16:34 |
| mandre | EmilienM: it's stable afaik | 16:35 |
| *** blinky_ghost has joined #tripleo | 16:36 | |
| *** kbyrne has joined #tripleo | 16:36 | |
| blinky_ghost | hi all, can anybody tell me if tripleo supports more than 3 controllers? I use newton. thanks | 16:37 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras master: [WIP] overcloud-deploy: missing toci failure log test https://review.openstack.org/446042 | 16:37 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 16:38 |
| jistr | blinky_ghost: hi, AFAIK not for the services managed by pacemaker | 16:39 |
| shardy | blinky_ghost: Hi, more than 3 controllers isn't reccomended or well tested, although from newton it's possible to independently scale services running on the controller, provided they aren't managed by pacemaker | 16:39 |
| *** ckyriakidou has joined #tripleo | 16:39 | |
| jistr | hehe :) | 16:40 |
| shardy | snap :) | 16:40 |
| lhinds | rgd'ing heat-templates in puppet/services, can you have a parameter with no default? | 16:40 |
| shardy | lhinds: yes, but that means it's mandatory, and stack create/validate will fail if it's not provided | 16:40 |
| lhinds | as in if I use default: '' <- this sets a hiera key | 16:40 |
| shardy | lhinds: ah, yeah, you can have a t-h-t default of an empty string | 16:41 |
| blinky_ghost | shardy: ok, but can I run only one controller with pacemaker? If I lose the other 2 controllers? | 16:41 |
| shardy | blinky_ghost: you can run with pacemaker enabled with either 1 or 3 controllers, and with 3 you have some HA capabilties, e.g APIs will fail over to a running controller | 16:42 |
| lhinds | shardy: its not mandatory, it has a default, its just "someclass::somevalue": "", results in Failed to validate: from the puppet manifest | 16:43 |
| *** fzdarsky_ is now known as fzdarsky | 16:43 | |
| lhinds | so if I did not have any hiera entry, the puppet manifest would quite happily apply its own default. | 16:44 |
| lhinds | I guess I could set the default, and it can be overidden | 16:44 |
| lhinds | but was curious how its wired up | 16:44 |
| *** fragatin_ has joined #tripleo | 16:44 | |
| yolanda | hi, i have a silly change pending from review https://review.openstack.org/442542 . Just adding a missing release note | 16:44 |
| shardy | lhinds: you can use a heat conditional to only set a value if something is passed as a heat parameter, e.g see | 16:45 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L149 | 16:45 |
| *** yamahata has quit IRC | 16:45 | |
| lhinds | shardy: nice, that would do it. | 16:46 |
| shardy | yolanda: spotted a small typo, otherwise lgtm | 16:46 |
| yolanda | oh, that vim! | 16:47 |
| blinky_ghost | shardy: In my lab I stopped the cluster in 2 controllers and it complains I would lose quorum. So, what happens is that haproxy, mariadb and rabbitmq go down with only controller running. | 16:47 |
| shardy | yolanda: also any idea what the rendered output will look like when you split the content like that between "features" and "security"? | 16:47 |
| *** fragatina has quit IRC | 16:47 | |
| EmilienM | mandre: I wouldn't move the job into the gate until it's a job we can debug | 16:48 |
| EmilienM | mandre: is there anyone working on it? | 16:48 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/ocata: Align hyperconverged-ceph.yaml environment and adds some validation https://review.openstack.org/439579 | 16:48 |
| shardy | I was wondering if it'd be clearer if it was just one section (I'm worried the "security" part will be rendered in a different part of the release notes) | 16:48 |
| yolanda | shardy, don't know. Is not normally used like that? | 16:48 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-heat-templates master: Add missing release note for cadf environment https://review.openstack.org/442542 | 16:48 |
| shardy | yolanda: honestly I'm not sure - I've only ever used one key/category | 16:48 |
| *** yprokule has quit IRC | 16:48 | |
| mandre | EmilienM: yes, there is https://review.openstack.org/#/c/441050/ that should get us everything we need but I can't retrigger a check experimental atm because the job is broken | 16:50 |
| EmilienM | yolanda: can you please look https://review.openstack.org/#/c/445875/ when you can? | 16:50 |
| yolanda | ++ | 16:51 |
| mandre | EmilienM: we merged a patch that broke containerized undercloud because CI was experimental and didn't see the failure :-/ | 16:51 |
| EmilienM | mandre: oops | 16:51 |
| EmilienM | mandre: ok then.. go ahead; | 16:52 |
| EmilienM | mandre: if the job pass now, then ok; but please get https://review.openstack.org/#/c/441050/ ready asap | 16:52 |
| yolanda | shardy, it looks bad, as you expected... http://docs-draft.openstack.org/42/442542/2/check/gate-tripleo-heat-templates-releasenotes/6e46ff2//releasenotes/build/html/unreleased.html | 16:52 |
| yolanda | it goes into another section | 16:52 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add upgrade tasks for aodh containers https://review.openstack.org/445600 | 16:52 |
| yolanda | shardy, so what do you prefer? i mention the cadf environment in the sentence in the security section, or i just group it on the features section? | 16:53 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the introspect workflow wrapper to use the multiple message interface https://review.openstack.org/446054 | 16:53 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the introspect manageable wrapper to use the multiple message interface https://review.openstack.org/446065 | 16:53 |
| shardy | yolanda: your choice, I'd probably just mention the environment in the features section | 16:53 |
| *** tesseract has quit IRC | 16:53 | |
| mandre | EmilienM: ack | 16:54 |
| *** paramite has quit IRC | 16:54 | |
| EmilienM | we've lost electricity and internet again, I need to relocate :-) - back in ~20 min | 16:54 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-heat-templates master: Add missing release note for cadf environment https://review.openstack.org/442542 | 16:55 |
| *** rook-mtg is now known as rook | 16:59 | |
| blinky_ghost | shardy: I think I have it working now, I set "pcs property set no-quorum-policy=ignore" | 17:02 |
| openstackgerrit | Rob Crittenden proposed openstack/python-tripleoclient master: Don't rely on umask to set appropriate permissions on overcloudrc* https://review.openstack.org/446071 | 17:03 |
| *** trown|lunch is now known as trown | 17:04 | |
| *** abishop has joined #tripleo | 17:04 | |
| *** akrzos-spec is now known as akrzos | 17:05 | |
| *** salmankhan has joined #tripleo | 17:05 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: Explicitly configure credentials used by ironic to access other services https://review.openstack.org/446074 | 17:06 |
| dprince | mandre: re. moving the job. I would love it | 17:07 |
| *** cylopez has quit IRC | 17:07 | |
| gfidente | d0ugal does https://github.com/d0ugal/mistral-ansible-actions/pull/7 look okay? | 17:07 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: DO NOT MERGE: testing ironic https://review.openstack.org/428053 | 17:07 |
| dprince | jistr: ack, thanks for pushing the ansible heat hook for tripleo.sh | 17:08 |
| d0ugal | gfidente: whoops, I seen that and then forgot about it. looking | 17:09 |
| d0ugal | gfidente: LGTM | 17:09 |
| gfidente | d0ugal I was trying to make it smarter with a try/catch | 17:10 |
| gfidente | but json.dumps works well with strings so there is no point in wrapping it there no? | 17:10 |
| *** nyechiel has quit IRC | 17:10 | |
| d0ugal | gfidente: yeah, I think it is fine. We can always add it later if needed | 17:11 |
| gfidente | d0ugal ack thanks | 17:12 |
| *** masco has joined #tripleo | 17:15 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: WIP: add timestamp to logs https://review.openstack.org/446075 | 17:16 |
| openstackgerrit | Feng Pan proposed openstack/puppet-tripleo master: Add networking-vpp ML2 mechanism driver support https://review.openstack.org/442529 | 17:16 |
| adarazs | trown, sshnaidm|bbl, panda|bbl ^ I made that timestamp change, just take a look and tell me if you think that's a good way to solve it :) | 17:16 |
| adarazs | might not even work, I didn't test it locally. | 17:16 |
| *** funzo_ has joined #tripleo | 17:16 | |
| trown | lol | 17:17 |
| fpan | hi all, could someone review https://review.openstack.org/#/c/442608/, thanks | 17:17 |
| EmilienM | mandre: feel free to propose the change and send me link | 17:18 |
| trown | adarazs: not inspiring confidence :P | 17:18 |
| adarazs | trown: well, before I spend some time to change every occurrence where we pipe logs to a file I want to make sure this is something like a decent approach. :) | 17:19 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Port the network environment validation https://review.openstack.org/341586 | 17:20 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Install python-tempestconf explicitely https://review.openstack.org/444277 | 17:20 |
| trown | adarazs: ya doing some local test of it ... you are sure that the \" get un-escaped? | 17:21 |
| *** funzo_ has quit IRC | 17:21 | |
| adarazs | trown: nope. | 17:22 |
| *** zoli is now known as zoli|gone | 17:23 | |
| adarazs | I'll check | 17:23 |
| adarazs | it doesn't... | 17:24 |
| *** yamahata has joined #tripleo | 17:24 | |
| adarazs | I'll push up another patchset. | 17:24 |
| *** florianf has quit IRC | 17:24 | |
| trown | adarazs: k, approach seems good and awk command works for me | 17:25 |
| *** oaken_chris has quit IRC | 17:25 | |
| adarazs | cool :) | 17:25 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Update the provide_manageable_nodes wrapper to use the multiple message interface https://review.openstack.org/446081 | 17:26 |
| *** dparkes has joined #tripleo | 17:26 | |
| * adarazs off for today | 17:27 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: WIP: add timestamp to logs https://review.openstack.org/446075 | 17:27 |
| *** dtantsur is now known as dtantsur|afk | 17:27 | |
| *** pcaruana has quit IRC | 17:28 | |
| *** jkilpatr has joined #tripleo | 17:29 | |
| *** rbrady is now known as rbrady-afk | 17:30 | |
| *** jkilpatr_ has quit IRC | 17:30 | |
| *** dsavineau has quit IRC | 17:31 | |
| *** jpich has quit IRC | 17:31 | |
| *** Goneri has quit IRC | 17:31 | |
| openstackgerrit | Rob Crittenden proposed openstack/instack-undercloud master: Don't rely on umask to set permissions on undercloud-passwords.conf https://review.openstack.org/446083 | 17:34 |
| *** dougbtv_ has joined #tripleo | 17:34 | |
| *** dougbtv has quit IRC | 17:34 | |
| *** dbecker has quit IRC | 17:36 | |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-docs master: Update the opstools documentation https://review.openstack.org/441092 | 17:37 |
| openstackgerrit | Merged openstack/python-tripleoclient master: undercloud deploy, delete heat containers https://review.openstack.org/440896 | 17:38 |
| *** flepied has quit IRC | 17:38 | |
| *** dsavineau has joined #tripleo | 17:40 | |
| *** trown is now known as trown|mtg | 17:43 | |
| *** dougbtv__ has joined #tripleo | 17:44 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Remove otoo CSS class https://review.openstack.org/445974 | 17:44 |
| *** jaganathan has quit IRC | 17:44 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Booting BM instance with cell_v2 https://review.openstack.org/442627 | 17:44 |
| *** Goneri has joined #tripleo | 17:45 | |
| *** dougbtv_ has quit IRC | 17:46 | |
| *** derekh has quit IRC | 17:47 | |
| *** masco has quit IRC | 17:54 | |
| sai | bnemec: if i see a stack update stuck for a long time, can i ctr+c out of it and retry or do i have to wait for it to fail before i retry? | 17:59 |
| bnemec | sai: You can ctrl-c, but you will need to manually delete the stack before retrying. | 18:00 |
| sai | bnemec: so at that point, i will have to recreate the whole stack ? cant update anymore correct? (i was trying to scale up from 103 to 115) | 18:01 |
| sai | so i will lose what i already have too | 18:01 |
| sai | ? | 18:01 |
| bnemec | sai: Oh, an update. No, then you won't want to delete. | 18:02 |
| *** rlandy|afk is now known as rlandy | 18:02 | |
| bnemec | sai: There's a heat stack-cancel-update command that you probably want to run so it goes to a failed state. | 18:02 |
| bogdando | folks, for newcomers the shortest dev path for containerized services https://docs.openstack.org/developer/tripleo-docs/containers_deployment/tips_tricks.html#debugging-docker-puppet-py is totally unusable because of the https://bugs.launchpad.net/tripleo/+bug/1673136 makes | 18:02 |
| openstack | Launchpad bug 1673136 in tripleo "[docs] Containers deployment guide has misleading debugging-docker-puppet-py example" [Undecided,New] | 18:02 |
| *** fzdarsky is now known as fzdarsky|afk | 18:02 | |
| sai | bnemec: rgr that, so ctrlc and set stack update to fail so that i dont have to wait for timeout | 18:03 |
| bnemec | sai: Although be aware that it didn't work right in some older versions of Heat. It _should_ be okay since Liberty or Mitaka though. | 18:03 |
| bnemec | sai: Right | 18:03 |
| *** skramaja has joined #tripleo | 18:03 | |
| sai | ok this is newton | 18:03 |
| sai | thanks bnemec | 18:03 |
| bnemec | sai: Yeah, if it doesn't work there please raise a bug with heat, because it is important that we have this functionality working. | 18:04 |
| sai | gotcha | 18:04 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: DPDK: Applying the RoleParameters overriding if provided for default parameter https://review.openstack.org/446093 | 18:04 |
| bnemec | And it's hard to test since we don't have negative tests in tripleo-ci. | 18:04 |
| *** marst has quit IRC | 18:05 | |
| *** marst has joined #tripleo | 18:05 | |
| *** athomas has quit IRC | 18:06 | |
| *** lmiccini has quit IRC | 18:09 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: DPDK: Applying the RoleParameters overriding if provided for default parameter https://review.openstack.org/446093 | 18:15 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 18:15 |
| skramaja | shardy: ^ | 18:15 |
| *** fzdarsky|afk has quit IRC | 18:15 | |
| skramaja | shardy: i have tried yaql to choose the right parameters, please provide your comments. | 18:15 |
| shardy | skramaja: ack, will take a look tomorrow, thanks for the update! | 18:16 |
| *** ckyriakidou has quit IRC | 18:16 | |
| skramaja | thanks. | 18:16 |
| *** bogdando has quit IRC | 18:16 | |
| *** funzo_ has joined #tripleo | 18:17 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add bigswitch agent support https://review.openstack.org/446029 | 18:18 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-specs master: Policy on Adding CI Jobs https://review.openstack.org/446102 | 18:19 |
| bnemec | EmilienM: ^ | 18:19 |
| bnemec | I would like to see us adopt that, or something like it, now that we're adding a bunch of new jobs. | 18:20 |
| rook | dsneddon question about dual stacks | 18:20 |
| rook | shardy ^ | 18:20 |
| *** tosky has quit IRC | 18:20 | |
| EmilienM | bnemec: good idea to document it, thanks. I'll review shortly | 18:20 |
| rook | I think there might be hard references to the base networks | 18:20 |
| *** skramaja has quit IRC | 18:21 | |
| shardy | rook: beyond the provider:physical_network stuff you can override via $networkNetValueSpecs? | 18:21 |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1632327 | 18:21 |
| openstack | Launchpad bug 1632327 in tripleo "Defining networks inconvenient when deploying multiple overclouds" [Medium,Triaged] - Assigned to Steven Hardy (shardy) | 18:21 |
| *** funzo_ has quit IRC | 18:22 | |
| rook | shardy: i did override them in my network env, however my controllers are using the default internal_api | 18:22 |
| rook | my compute nodes are using the override. | 18:22 |
| rook | shardy: the only way i noticed this was that i had a dup ip | 18:23 |
| EmilienM | bnemec: it sounds excellent | 18:23 |
| rook | shardy: which was alarming... however, one ip was on the interal_api, and the other was on the second internal_api network i created. | 18:24 |
| shardy | rook: Hmm, OK are you specifying static assignments for everything via *AllocationPools? | 18:24 |
| rook | No | 18:24 |
| bnemec | EmilienM: Glad to hear it. :-) | 18:24 |
| shardy | rook: Ok, thanks for the info - could you pls update the bug with notes of what you're seeing and I'll try to reproduce | 18:25 |
| EmilienM | bnemec: I gave some quick feedback but overall all good | 18:25 |
| EmilienM | bnemec: one thing, to make sure *everyone* can know about it, i would suggest an announcement on ML. | 18:25 |
| bnemec | EmilienM: That works for me. I dislike being surprised by new jobs. | 18:26 |
| shardy | rook: Ah, I wonder if https://bugs.launchpad.net/tripleo/+bug/1651541 is related | 18:26 |
| openstack | Launchpad bug 1651541 in tripleo "Custom network names prevent deployment" [High,Fix released] - Assigned to Steven Hardy (shardy) | 18:26 |
| shardy | that's not backported to newton yet and it will prevent specifying different network names | 18:27 |
| rook | yup that seems possible | 18:27 |
| rook | very much so | 18:27 |
| rook | " Don't assume default network names in net_ip*map" | 18:27 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add bigswitch agent support https://review.openstack.org/446029 | 18:27 |
| rook | right | 18:27 |
| shardy | rook: that doesn't exactly explain why we don't get an error though, when the network names collide | 18:27 |
| shardy | I'd expect the second stack to fail | 18:27 |
| *** marst has quit IRC | 18:28 | |
| rook | to avoid this ip collision, I am hoping to update the network address space and see if that solves my issue | 18:28 |
| rook | howver i hope neutron isn't smart enough to say, "i already have IPs used in this range" | 18:29 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add bigswitch agent support https://review.openstack.org/446029 | 18:29 |
| bnemec | shardy: Neutron will allow you to have multiple networks with the same name. | 18:29 |
| shardy | rook: I think with *AllocationPools it'll bypass neutron completely as we don't create any ports for the isolated networks | 18:30 |
| shardy | bnemec: aha, and overlapping ip ranges (doh!) | 18:30 |
| rook | bnemec Neutron will allow it, but I don't think OOO will | 18:30 |
| rook | however they can use the same address space | 18:31 |
| rook | which mine ur | 18:31 |
| rook | which mine are | 18:31 |
| rook | smart or dumb, I dunno yet :) | 18:31 |
| shardy | I think it's fair to say the multiple overcloud thing isn't well tested - we could use a bunch of better validation here I think | 18:31 |
| sai | bnemec: rook when a node pxes i see NBP too big for base memory, do u think this could be tripleo or the hardware? | 18:31 |
| rook | shardy ack | 18:31 |
| rook | sai i don't see how it is OOO.. did other nodes pxe just fine? | 18:32 |
| rook | does the node only have like 2GB ram? | 18:32 |
| sai | rook: yeh thats my understanding too | 18:32 |
| sai | i see this only on 730s | 18:32 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates stable/newton: Make ServiceNetMap support custom network names https://review.openstack.org/446112 | 18:33 |
| *** marst has joined #tripleo | 18:33 | |
| bnemec | sai: I have no idea what that means, so I'm not sure I can help much. | 18:33 |
| * bnemec doesn't have fancy new hardware :-) | 18:33 | |
| *** karimb has joined #tripleo | 18:33 | |
| *** shardy has quit IRC | 18:34 | |
| *** sshnaidm|bbl is now known as sshnaidm | 18:36 | |
| EmilienM | dmsimard: it would be fun to have ARA for quickstart, I don't see it: http://logs.openstack.org/00/445700/6/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq/04317d7/logs/ | 18:37 |
| EmilienM | dmsimard: I see ARA for devstack-gate | 18:37 |
| EmilienM | oh and I found this: http://logs.openstack.org/00/445700/6/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq/04317d7/console.html#_2017-03-15_16_04_46_375720 | 18:37 |
| *** jpena is now known as jpena|off | 18:39 | |
| *** natarej has quit IRC | 18:39 | |
| *** jtomasek has quit IRC | 18:39 | |
| EmilienM | dmsimard: I don't see oooq playbooks in http://logs.openstack.org/00/445700/6/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq/04317d7/logs/ara/reports/index.html | 18:40 |
| *** bnemec has quit IRC | 18:41 | |
| EmilienM | trown|mtg: I'm seeing a lot of http://logs.openstack.org/00/445700/6/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq/04317d7/console.html#_2017-03-15_16_02_08_095263 recently | 18:41 |
| *** sanagikoki has quit IRC | 18:41 | |
| *** suuuper has quit IRC | 18:44 | |
| *** bnemec has joined #tripleo | 18:44 | |
| *** trown|mtg is now known as trown | 18:46 | |
| *** dprince has quit IRC | 18:47 | |
| *** pkovar has quit IRC | 18:47 | |
| openstackgerrit | Michele Baldessari proposed openstack-infra/tripleo-ci master: WIP Make sure we upgrade to the right repos during an upgrade https://review.openstack.org/446121 | 18:47 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates stable/ocata: WIP Upgrades: wait for galera to be settled https://review.openstack.org/444928 | 18:48 |
| trown | EmilienM: hmm looks like a DLRN failure... we need to collect those build logs | 18:48 |
| *** afazekas_ is now known as afazekas | 18:50 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix stackviz report generation https://review.openstack.org/444267 | 18:50 |
| EmilienM | trown: 'recheck' ? | 18:51 |
| trown | EmilienM: I wonder if it is this http://logs.openstack.org/77/445977/1/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/ed0af89/logs/undercloud/home/jenkins/install_built_repo.log.txt.gz | 18:52 |
| EmilienM | ahh! | 18:52 |
| EmilienM | nice catch | 18:52 |
| EmilienM | we just merged something in RDO | 18:52 |
| trown | https://review.rdoproject.org/r/#/c/5831/ | 18:52 |
| *** karimb has quit IRC | 18:52 | |
| EmilienM | trown: so puppet-qdr wouldn't be found? | 18:54 |
| trown | EmilienM: I was using 445977 to do some no-op runs of the scenario jobs, and they mostly passed this morning, then all look to fail on that missing puppet-qdr | 18:54 |
| EmilienM | switching on #rdo | 18:54 |
| trown | k | 18:54 |
| trown | meant to paste https://review.openstack.org/445977 | 18:54 |
| *** marst has quit IRC | 18:54 | |
| *** blinky_ghost has quit IRC | 18:55 | |
| EmilienM | trown: nice colors :D we're almost there | 18:55 |
| *** marst has joined #tripleo | 18:55 | |
| trown | EmilienM: ya just dont check it when current run comes back... it is much less green (though I think because of ^) | 18:56 |
| EmilienM | yeah let's fix it | 18:56 |
| EmilienM | trown: maybe it's because we need a promotion? | 18:56 |
| *** rbrady-afk is now known as rbrady | 18:58 | |
| *** fragatin_ has quit IRC | 19:02 | |
| *** fragatina has joined #tripleo | 19:02 | |
| *** marrusl has joined #tripleo | 19:04 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Disable tempest in container job https://review.openstack.org/445840 | 19:04 |
| openstackgerrit | Merged openstack/instack-undercloud master: Explicitly configure credentials used by ironic to access neutron and swift https://review.openstack.org/430886 | 19:05 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Clean up oooq jobs https://review.openstack.org/445628 | 19:05 |
| openstackgerrit | Merged openstack/tripleo-common master: Make fencing action parameter optional. https://review.openstack.org/442495 | 19:05 |
| openstackgerrit | Merged openstack/tripleo-common master: Add neutron-metadata agent container image https://review.openstack.org/440543 | 19:07 |
| *** gfidente is now known as gfidente|afk | 19:08 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Role definitions https://review.openstack.org/445687 | 19:09 |
| *** dprince has joined #tripleo | 19:12 | |
| openstackgerrit | Merged openstack/tripleo-common master: Replace six.iteritems() with .items() https://review.openstack.org/408888 | 19:13 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Cleanup no longer used upgrade files https://review.openstack.org/445816 | 19:13 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart-extras master: Do not disable other repos when updating gate packages https://review.openstack.org/446129 | 19:14 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Create or update deployment plan from git repository https://review.openstack.org/417280 | 19:17 |
| sshnaidm | panda|bbl, will we have toci-wide config file for oooq? | 19:18 |
| *** funzo_ has joined #tripleo | 19:18 | |
| *** thrash is now known as thrash|biab | 19:19 | |
| *** amoralej is now known as amoralej|off | 19:20 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart-extras master: Collect DLRN build directory if it exists https://review.openstack.org/446133 | 19:21 |
| dsneddon | rook, Did shardy answer all your questions about dual stacks? | 19:21 |
| rook | dsneddon: added to the bug | 19:21 |
| *** marst has quit IRC | 19:22 | |
| *** dtrainor has quit IRC | 19:22 | |
| *** cylopez has joined #tripleo | 19:22 | |
| *** funzo_ has quit IRC | 19:23 | |
| trown | sshnaidm: panda|bbl could you look at https://review.openstack.org/#/c/446129/ and https://review.openstack.org/446133 seeing lots of failures on the multinode jobs and I think the first patch addresses some while the second patch will help diagnose the others | 19:25 |
| dsneddon | rook, Why are your networks /16? internal_api and internal_api2 have the same 172.16.0.0/16 subnet?!?! | 19:25 |
| * trown needs to catch the bus, brb | 19:25 | |
| *** trown is now known as trown|brb | 19:25 | |
| dsneddon | rook, Shouldn't that be 172.16.0.0/24 and 172.16.1.0/24, for instance? | 19:25 |
| rook | not enough IPs | 19:25 |
| rook | but yes | 19:26 |
| rook | however, i think that would of failed. | 19:26 |
| dsneddon | rook, Well, the size isn't the point, but rather that you are using the same subnet in two places. | 19:26 |
| rook | right, dsneddon agreed. | 19:26 |
| rook | they are both flat. | 19:26 |
| dsneddon | rook, that might be a red herring, it was just confusing me | 19:26 |
| rook | dsneddon: it was done this way for two reasons, if I wnated the two stacks to work together. | 19:26 |
| sshnaidm | trown|brb, which kind of failures do you see? | 19:26 |
| *** dtrainor has joined #tripleo | 19:26 | |
| rook | work/talk | 19:26 |
| rook | dsneddon: stack one is openshift | 19:27 |
| rook | stack two is Openstack | 19:27 |
| rook | however | 19:27 |
| *** deadnull has joined #tripleo | 19:27 | |
| rook | i should of done 172.16.0.0/16 with a allocation pool of 172.16.0 - 172.16.3 then the other be .4 - .6 | 19:27 |
| rook | something like that | 19:28 |
| *** b00tcat has quit IRC | 19:28 | |
| dsneddon | rook, What do you have in the ServiceNetMap for Redis network? | 19:29 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add ceilometer ipmi agent https://review.openstack.org/430447 | 19:29 |
| dsneddon | rook, You probably want to update your ServiceNetMap so that it has "internal_api2" if that's the network you want to use. | 19:30 |
| dsneddon | rook, Look at the RedisVirtualIP definition, the network name gets passed according to what's in ServiceNetMap: http://pastebin.test.redhat.com/464997 | 19:30 |
| deadnull | I am working on a tripleo deployment of newton. everything has gone smoothly until providing the nodes, they are in provisioning state "clean failed" due to not having a valid cleaning network UUID - this is not an option in undercloud.conf... any ideas what my next steps are to get it out of the clean failed state? | 19:31 |
| sshnaidm | trown|brb, I see only one delroean failure and it's because network/repo issue "[Errno 14] HTTPS Error 302" : http://logs.openstack.org/00/445700/6/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq/04317d7/logs/delorean_logs/8f/42/8f4233dda57dfbbaf0b5df1cfb70dda10745825b_dev/rpmbuild.log.txt.gz | 19:31 |
| rook | dsneddon is this documented? | 19:31 |
| dsneddon | rook, The network name gets passed here (ctlplane_vip.yaml): http://pastebin.test.redhat.com/464998 | 19:32 |
| dsneddon | rook, Well, the ServiceNetMap is documented for sure. This is a new use case, though, because you couldn't create arbitrary networks before. | 19:32 |
| rook | ahhh ok | 19:32 |
| rook | I thought I mapped the name? | 19:33 |
| EmilienM | trown|brb: have you already seen that? http://paste.openstack.org/show/LgnHPtPljtMNMYraAUyi/ | 19:35 |
| rook | dsneddon I was just shocked it used the old mapping -- just didn't expect it. | 19:37 |
| EmilienM | trown|brb: nevermind, found https://access.redhat.com/solutions/2759381 which ask to upgrade kvm | 19:37 |
| *** trown|brb is now known as trown | 19:37 | |
| rook | dsneddon the real reason I have two stacks is : https://gist.github.com/jtaleric/f970a2e89d944915d8543429950423c6#deployment-after-total-chaos | 19:37 |
| *** jmelvin has quit IRC | 19:38 | |
| rook | dsneddon: you might find https://gist.github.com/jtaleric/f970a2e89d944915d8543429950423c6#deployment-wedged interesting | 19:38 |
| rook | 51-hosts was trolling hard. | 19:38 |
| trown | sshnaidm: k maybe DLRN failure is less important one then, though I think we still want to collect those logs via collect logs since logs/delorean_logs is not created by quickstart | 19:39 |
| rook | or something was nuking what 51-hosts would put into place | 19:39 |
| dsneddon | rook, I live in plenty of fear that my Heat stack is going to get corrupted with 6 hosts, I can't even imagine rolling the dice on a 50+ node deployment. | 19:39 |
| trown | sshnaidm: other failure is on my noop test patch https://review.openstack.org/445977 (you need to look it up on zuul because not all the jobs finished) all the multinode jobs are red there | 19:40 |
| rook | dsneddon my anxiety is quite high these days | 19:40 |
| dsneddon | rook, If you ask me, recovery from bad Heat states is TripleO's Achilles heel. | 19:40 |
| rook | dsneddon oh, it is 100x better these days | 19:40 |
| rook | While it was a PITA, I am happy the workarounds i put into place worked | 19:41 |
| trown | sshnaidm: and it is because we are disabling all the repos when installing the gated change which will break any time we add a new dependency (until we promote a new image with said dependency) | 19:41 |
| rook | it isn't all rainbows and unicorns but it is way better than before. | 19:41 |
| sshnaidm | trown, what does it mean - not created by quickstart?? | 19:41 |
| dsneddon | rook, True enough, I used to have to rebuild my undercloud after nearly every deployment in the bad old days. | 19:41 |
| rook | lol yup | 19:41 |
| trown | sshnaidm: ignore the collect logs patch for now :P, it is much lower priority now that I realize those logs are present | 19:41 |
| dsneddon | rook, BTW, you are paving new ground, so if you take some notes it will help me and others write documentation for this stuff. | 19:42 |
| sshnaidm | trown, I don't follow, we collect delorean logs with quickstart, why do we need collect repo? | 19:42 |
| trown | sshnaidm: ok, cool I will abandon that patch... please look at https://review.openstack.org/446129 though | 19:43 |
| sshnaidm | trown, I really don't think it's an issue with disabling repo, it was working so for years, I assume it should be ok today too.. | 19:43 |
| rook | dsneddon: that gist i Sent you is everything I am trying to capture. | 19:43 |
| dsneddon | rook, I mean, obviously you are taking notes, but I mean make some notations with an eye toward what is *not* documented yet. | 19:43 |
| rook | right | 19:43 |
| rook | dsneddon exactly | 19:43 |
| sshnaidm | trown, can you show what exactly error do you see and logs? | 19:43 |
| trown | sshnaidm: it only breaks if a new dependency is added | 19:43 |
| sshnaidm | trown, for example? | 19:43 |
| sshnaidm | trown, dependency in rpm you mean? | 19:44 |
| *** morazi has quit IRC | 19:44 | |
| trown | sshnaidm: ya... we added puppet-qdr dep to puppet-tripleo and it breaks | 19:44 |
| trown | sshnaidm: http://logs.openstack.org/77/445977/1/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/ed0af89/logs/undercloud/home/jenkins/install_built_repo.log.txt.gz | 19:45 |
| trown | sshnaidm: puppet-qdr is available in the delorean-current repo, but we disable it | 19:45 |
| sshnaidm | trown, I see, I thought you talk about dependency in gerrit.. | 19:45 |
| trown | sshnaidm: ya it is a very corner case we would not have caught only sporadically running jobs | 19:46 |
| sshnaidm | trown, yeah, it looks like a problem.. | 19:46 |
| trown | sshnaidm: we rarely add new rpm deps | 19:46 |
| trown | sshnaidm: and it will only affect patches which have puppet-tripleo in gate repo (in this case) | 19:46 |
| sshnaidm | trown, ok, let's do it so then, but with caution.. because I didn't test it (and nobody I think) | 19:46 |
| *** morazi has joined #tripleo | 19:46 | |
| trown | sshnaidm: we will have the same behavior because of yum priorities | 19:47 |
| sshnaidm | trown, I very hope :) | 19:47 |
| trown | and DLRN nvr should be very nearly always higher for our built package too | 19:47 |
| sshnaidm | trown, I mean it *should* work, yes, but there is always something (c) | 19:47 |
| trown | there is a very rare corner case where we build a package at same time as some actual DLRN server but just slightly faster where we would get the wrong package, but in that case priorities would take over anyways | 19:48 |
| sshnaidm | trown, yeah, I see | 19:49 |
| trown | not even sure above would happen as we have already "pinned" current before we do the package build I think so our package would be strictly newer version | 19:49 |
| trown | hmm... we dont put an actual hash in our delorean repo files, so the corner case does exist, but it is taken care of via priorities anyways | 19:50 |
| *** thrash|biab is now known as thrash | 19:51 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: add featureset config for a containerised overcloud https://review.openstack.org/446151 | 19:52 |
| *** fultonj has quit IRC | 19:56 | |
| *** dougbtv__ is now known as dougbtv | 19:56 | |
| openstackgerrit | Rob Crittenden proposed openstack/instack-undercloud master: Don't rely on umask to set permissions on undercloud-passwords.conf https://review.openstack.org/446083 | 19:57 |
| *** rbrady is now known as rbrady-errand | 19:58 | |
| *** fultonj has joined #tripleo | 19:58 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: scenario001/pingtest: enable Gnocchi resource again https://review.openstack.org/442795 | 19:58 |
| weshay | Slower, did that CI work for you yesterday | 19:59 |
| weshay | Slower, I just got a clean run | 20:00 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Enable quickstart to work with clean base OS images. https://review.openstack.org/442764 | 20:02 |
| *** salmankhan has quit IRC | 20:08 | |
| *** jmelvin has joined #tripleo | 20:09 | |
| *** dsneddo__ has joined #tripleo | 20:15 | |
| *** funzo_ has joined #tripleo | 20:19 | |
| *** marst has joined #tripleo | 20:19 | |
| *** rwsu has joined #tripleo | 20:21 | |
| *** funzo_ has quit IRC | 20:24 | |
| *** marst_ has joined #tripleo | 20:25 | |
| *** trown is now known as trown|outtypewww | 20:26 | |
| *** marst has quit IRC | 20:28 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Standardize example role definitions https://review.openstack.org/445687 | 20:29 |
| *** fultonj has quit IRC | 20:29 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DONT MERGE: testing build role https://review.openstack.org/446168 | 20:31 |
| sshnaidm | trown|outtypewww, ^^ just to be sure we didn't break role | 20:32 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add bigswitch agent support https://review.openstack.org/446029 | 20:34 |
| *** salmankhan has joined #tripleo | 20:35 | |
| *** marst_ has quit IRC | 20:36 | |
| *** dprince has quit IRC | 20:38 | |
| *** stream10 has joined #tripleo | 20:40 | |
| *** saneax-_-|AFK has quit IRC | 20:41 | |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-common master: Enhance plan creation and update with plan-environment https://review.openstack.org/414169 | 20:43 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: oooq multinode: fix for stable branches https://review.openstack.org/445866 | 20:43 |
| *** saneax-_-|AFK has joined #tripleo | 20:44 | |
| EmilienM | sshnaidm: see my comment on https://review.openstack.org/#/c/437260/ (not urgent at all) | 20:47 |
| EmilienM | weshay: featureset really rocks | 20:48 |
| EmilienM | weshay: the concept I mean, super cool | 20:48 |
| *** ansmith has quit IRC | 20:49 | |
| *** fzdarsky has joined #tripleo | 20:49 | |
| rbowen | I wonder if one of you TripleO wizards could take a look at https://ask.openstack.org/en/question/82790/openstack-baremetal-introspection-internal-server-error/ and tell me if this is a known problem, and if so, fixed, and, if not, what a reasonable answer to this post might be. | 20:50 |
| rbowen | Thanks. | 20:50 |
| weshay | EmilienM, woot..glad you like it | 20:53 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras master: [WIP] overcloud-deploy: missing toci failure log test https://review.openstack.org/446042 | 20:53 |
| *** akrivoka has quit IRC | 20:53 | |
| panda|bbl | uuhh, something urgent to review ? | 20:54 |
| *** morazi has quit IRC | 20:55 | |
| dsneddon | rbowen, We don't use sqlite for Ironic Inspector any more, so that Ask Openstack question is pretty obsolete | 20:55 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Standardize example role definitions https://review.openstack.org/445687 | 20:55 |
| rbowen | dsneddon: Ok, thank you. | 20:55 |
| *** panda|bbl is now known as panda | 20:55 | |
| *** rwsu has quit IRC | 20:59 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Standardize example role definitions https://review.openstack.org/445687 | 21:03 |
| pradk | mwhahaha, so looking at your roles patch ^^ .. do you mind if i push a telemetry role to this patch? | 21:06 |
| mwhahaha | pradk: that would be fine, that's the point of this reorg :) | 21:06 |
| *** stream10 has quit IRC | 21:06 | |
| pradk | cool, lemme push | 21:06 |
| weshay | sshnaidm, you're not still working are you? | 21:07 |
| *** lblanchard has quit IRC | 21:07 | |
| sshnaidm | weshay, it couldn't be called work.. :D | 21:07 |
| weshay | sshnaidm, ur crazy | 21:07 |
| weshay | :) | 21:07 |
| sshnaidm | just waiting jobs to finish | 21:07 |
| *** morazi has joined #tripleo | 21:09 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Standardize example role definitions https://review.openstack.org/445687 | 21:11 |
| pradk | mwhahaha, ^^ so for telemetry role i included dependencies such as keystone, mongo and redis in the yaml so its deployable as a standalone role | 21:13 |
| mwhahaha | pradk: sounds good | 21:14 |
| pradk | cool | 21:14 |
| dmsimard | EmilienM: sorry I've been stuck in snow or meetings | 21:16 |
| dmsimard | you said you had issues with ara ? | 21:16 |
| dmsimard | last I know it worked in oooq | 21:16 |
| dmsimard | we made sure of that before merging, so maybe something happened | 21:16 |
| EmilienM | dmsimard: did you see my links? I didn't find the ara for oooq but only for devstack-gate | 21:16 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add bigswitch agent support https://review.openstack.org/446029 | 21:17 |
| *** jcoufal has quit IRC | 21:17 | |
| * dmsimard looks | 21:17 | |
| dmsimard | EmilienM: hm... maybe the devstack-gate one is overwriting the oooq one ? | 21:18 |
| dmsimard | checking if it's the same path | 21:18 |
| dmsimard | no, not the same path -- it looks like the oooq report is generated at /opt/stack/logs/ara, is that recovered somewhere ? | 21:19 |
| dmsimard | the job ends right after that, at least in the console you linked, so I'm not sure | 21:20 |
| dmsimard | http://logs.openstack.org/00/445700/6/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq/04317d7/console.html#_2017-03-15_16_05_01_921310 | 21:20 |
| *** salmankhan has quit IRC | 21:25 | |
| *** jtomasek has joined #tripleo | 21:28 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: fix featureset001/2 network-isolation settings for libvirt https://review.openstack.org/446181 | 21:30 |
| *** abishop has quit IRC | 21:30 | |
| *** salmankhan has joined #tripleo | 21:30 | |
| *** flepied has joined #tripleo | 21:30 | |
| *** Goneri has quit IRC | 21:35 | |
| *** eck` is now known as eck`gone | 21:35 | |
| *** dparkes has quit IRC | 21:36 | |
| *** jkilpatr has quit IRC | 21:37 | |
| openstackgerrit | Chris Jones proposed openstack/tripleo-common stable/ocata: Make fencing action parameter optional. https://review.openstack.org/446184 | 21:37 |
| *** fzdarsky has quit IRC | 21:39 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: add missing skip case for extra environment for composable scenario https://review.openstack.org/445508 | 21:40 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: [overcloud-deploy] Update deprecated references to Heat https://review.openstack.org/409178 | 21:40 |
| *** dsariel has joined #tripleo | 21:42 | |
| EmilienM | fpan: https://bugs.launchpad.net/tripleo/+bug/1673266 | 21:43 |
| openstack | Launchpad bug 1673266 in tripleo "default etcd deployments share the same cluster token" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 21:43 |
| EmilienM | fpan: this is really bad. | 21:44 |
| *** pmannidi has joined #tripleo | 21:52 | |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-docs master: Update the opstools documentation https://review.openstack.org/441092 | 21:54 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Generate EtcdInitialClusterToken https://review.openstack.org/446194 | 21:54 |
| *** jkilpatr has joined #tripleo | 21:57 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: etcd: secure EtcdInitialClusterToken parameter https://review.openstack.org/446195 | 21:58 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add bigswitch agent support https://review.openstack.org/446029 | 22:01 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras master: overcloud-deploy: unset -e flag in deploy script https://review.openstack.org/446042 | 22:02 |
| *** jmelvin has quit IRC | 22:02 | |
| *** morazi has quit IRC | 22:04 | |
| panda | can someone suggest a set of overcloud deploy arguments that make the deploy fail very fast ? | 22:04 |
| panda | EmilienM: ^ | 22:05 |
| panda | but it has to go to status CREATE_FAILED | 22:05 |
| bkero | panda: --fake-option? :) | 22:05 |
| panda | bkero: that prevents the deployment to even start | 22:05 |
| *** ccamacho has quit IRC | 22:06 | |
| panda | I need it to start and fail with CREATE_FAILED almost immediately | 22:06 |
| bkero | oh | 22:06 |
| *** cwolferh has joined #tripleo | 22:06 | |
| matbu | panda: you can use big flavor for the controller | 22:07 |
| matbu | a big* | 22:07 |
| panda | matbu: nice --flavor big ? :) | 22:07 |
| matbu | you will hit no valid host was found very fast | 22:07 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: derive overcloud_public_ip https://review.openstack.org/441569 | 22:07 |
| panda | matbu: what if I give --control-scale 9000 ? | 22:08 |
| panda | 9001 even better | 22:08 |
| matbu | panda: never tried as much but failed also :) | 22:09 |
| bkero | Can computers count that high? | 22:09 |
| matbu | panda: but im not sure if it will start the heat deployment | 22:09 |
| matbu | panda: i think tripleoclient check the hypervisor start before deploy | 22:10 |
| matbu | s/start/stats/ | 22:10 |
| panda | :/ I need to test it on multinode | 22:10 |
| panda | not sure the flavor will affect that | 22:10 |
| bkero | matbu: that check is disabled for multinode | 22:11 |
| bkero | since nova will never have the nodes available | 22:11 |
| bkero | panda: maybe --control-scale=0? | 22:11 |
| panda | I can't believe we are here talking about a way | 22:12 |
| panda | to break things, and we can't find a solution | 22:12 |
| panda | after all we've done in these months to break things | 22:12 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add bigswitch agent support https://review.openstack.org/446029 | 22:12 |
| matbu | panda: no there is too much solutions :) | 22:13 |
| panda | bkero: I'll try | 22:13 |
| bkero | You want it to fail instantly. I can give you a lot of ways to make it fail after a little bit, or hang indefinitely | 22:13 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] quickstart: test for missing failed deployment log https://review.openstack.org/446043 | 22:14 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: define non_root_group explicitly https://review.openstack.org/439133 | 22:14 |
| panda | let's see hot it goes | 22:14 |
| panda | anyway, that should really be it with https://review.openstack.org/446042, that's why we didn't have failed_deployments logs | 22:15 |
| panda | I just need a proof | 22:16 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements master: Add qpid-dispatch-router to overcloud-controller element https://review.openstack.org/437942 | 22:18 |
| openstackgerrit | Merged openstack/tripleo-incubator master: make doc job of jenkins executing successfully https://review.openstack.org/444580 | 22:19 |
| *** funzo has joined #tripleo | 22:20 | |
| *** kjw3 has quit IRC | 22:25 | |
| *** funzo has quit IRC | 22:25 | |
| *** rwsu has joined #tripleo | 22:30 | |
| *** bfournie has quit IRC | 22:31 | |
| EmilienM | panda: you around? trying to get the infra patch merged... | 22:31 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add ceilometer ipmi agent https://review.openstack.org/430447 | 22:32 |
| panda | EmilienM: need any help ? | 22:34 |
| *** rbrady-errand is now known as rbrady | 22:34 | |
| *** Goneri has joined #tripleo | 22:35 | |
| EmilienM | panda: you see my ping on #openstack-infra I guess | 22:35 |
| EmilienM | panda: clarkb ask for logging | 22:35 |
| *** cdearborn has quit IRC | 22:36 | |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Create or update deployment plan from git repository https://review.openstack.org/417280 | 22:37 |
| *** ckyriakidou has joined #tripleo | 22:38 | |
| *** Goneri has quit IRC | 22:40 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: add caching the GetParametersAction https://review.openstack.org/444220 | 22:48 |
| *** ansmith has joined #tripleo | 22:48 | |
| *** dsneddo__ has quit IRC | 22:54 | |
| *** jtomasek has quit IRC | 22:56 | |
| *** kjw3 has joined #tripleo | 23:00 | |
| *** pradk has quit IRC | 23:02 | |
| *** dmarlin has quit IRC | 23:08 | |
| *** ckyriakidou has quit IRC | 23:11 | |
| *** rbowen has quit IRC | 23:16 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: Deploy NTP with puppet-tripleo profile https://review.openstack.org/443334 | 23:18 |
| *** rwsu has quit IRC | 23:21 | |
| *** funzo has joined #tripleo | 23:21 | |
| *** salmankhan has quit IRC | 23:23 | |
| *** salmankhan has joined #tripleo | 23:24 | |
| *** funzo has quit IRC | 23:26 | |
| panda | bkero: lol http://logs.openstack.org/43/446043/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/2e6dc66/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-03-15_22_50_02 | 23:27 |
| panda | bkero: with --control-scale 0 it creted the stack successfully in ~10 seconds | 23:27 |
| bkero | nice | 23:27 |
| panda | a very fast deployment of nothing | 23:27 |
| panda | and no failure ... | 23:28 |
| panda | bah I'll go to bed | 23:28 |
| bkero | panda: you could try 9000 while you sleep | 23:28 |
| panda | good idea | 23:29 |
| *** panda is now known as panda|Zz | 23:30 | |
| panda|Zz | ... 10 ... | 23:30 |
| panda|Zz | ... 58 ... | 23:30 |
| rook | dsneddon: around? | 23:46 |
| rook | or anyone really... | 23:46 |
| rook | .Properties.NetworkHostnameMap: Collection length exceeds 1000 elements | 23:46 |
| dsneddon | rook, Yeah... | 23:46 |
| rook | ^ i am getting a error there | 23:46 |
| rook | 2017-03-15 23:38:38Z [overcloud-openstack]: UPDATE_FAILED StackValidationFailed: resources.ComputeIpListMap: Property error: ComputeIpListMap.Properties.NetworkHostnameMap: Collection length exceeds 1000 elements | 23:47 |
| dsneddon | rook, Wow, never seen that one before. You're good at finding new errors. | 23:47 |
| rook | i am trying to find where in the code that even exists. | 23:47 |
| dsneddon | rook, I'm guessing you are hitting some sort of Heat limit. | 23:47 |
| dsneddon | rook, Although I'm not aware of any such limit. | 23:47 |
| bnemec | Yeah, we generate multiple hostnames per node, so you probably hit 1000 way before 1000 nodes. | 23:47 |
| rook | right bnemec | 23:48 |
| rook | well ~300 | 23:48 |
| rook | bnemec: where would I see this / tune it? | 23:49 |
| bnemec | rook: I agree with dsneddon that this sounds like a Heat limitation, so if it's tunable it would be in heat.conf. | 23:49 |
| bnemec | If it's not tunable then I think it's a bug. | 23:49 |
| rook | max_resources_per_stack = -1 | 23:50 |
| rook | the default is 1000 | 23:50 |
| rook | but we hvae it set to -1 | 23:50 |
| bnemec | Yeah, I don't think this is that. | 23:50 |
| bnemec | You'd have hit that limit long ago. | 23:50 |
| rook | #max_events_per_stack = 1000 ? | 23:50 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo master: Include ceph::profile::client from rgw.pp https://review.openstack.org/446231 | 23:50 |
| rook | hm | 23:50 |
| rook | it seems like something related to validations does it not? | 23:51 |
| rook | StackValidation ? | 23:51 |
| bnemec | rook: Yeah, it sounds like Heat limits their internal data structures to 1000 entries. | 23:52 |
| *** dmacpher has quit IRC | 23:52 | |
| bnemec | I suspect you're going to need a heat person to tell you if/how to get past this though. | 23:52 |
| openstackgerrit | Merged openstack/diskimage-builder master: Use sphinx warning-is-error https://review.openstack.org/445254 | 23:57 |
| rook | bnemec: ugh | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
