| *** rlandy has quit IRC | 00:04 | |
| *** gfidente has quit IRC | 00:04 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 00:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 00:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 00:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| *** milan has joined #tripleo | 00:12 | |
| *** _milan_ has quit IRC | 00:12 | |
| *** lblanchard has joined #tripleo | 00:31 | |
| *** limao has joined #tripleo | 00:45 | |
| *** cshastri has joined #tripleo | 00:52 | |
| *** homeski has joined #tripleo | 00:56 | |
| *** dixiaoli has joined #tripleo | 00:57 | |
| *** dixiaoli has quit IRC | 00:57 | |
| *** dixiaoli has joined #tripleo | 00:58 | |
| *** dixiaoli has quit IRC | 00:58 | |
| *** dixiaoli has joined #tripleo | 00:58 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Delete docker-centos-tripleoupstream.yaml https://review.openstack.org/487613 | 01:00 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Containerize virtlogd https://review.openstack.org/469116 | 01:02 |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 01:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 01:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 01:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| *** _milan_ has joined #tripleo | 01:15 | |
| *** milan has quit IRC | 01:16 | |
| *** lblanchard has quit IRC | 01:22 | |
| *** shreshtha_ has quit IRC | 01:24 | |
| *** jmelvin has joined #tripleo | 01:39 | |
| pabelanger | EmilienM: weshay: mwhahaha: just spend 20mins looking at tripleo logs | 01:41 |
| pabelanger | ultinode-oooq/4e30192/logs/undercloud/home/jenkins/undercloud.conf.txt.gz | 01:41 |
| pabelanger | err | 01:41 |
| pabelanger | http://logs.openstack.org/37/493937/3/gate/gate-tripleo-ci-centos-7-scenario002-multinode-oooq/4e30192/logs/undercloud/home/jenkins/undercloud.conf.txt.gz | 01:42 |
| pabelanger | undercloud_nameservers = 8.8.8.8 | 01:42 |
| pabelanger | that is wrong | 01:42 |
| pabelanger | we need to stop defaulting to google DNS | 01:42 |
| *** itlinux has joined #tripleo | 01:42 | |
| pabelanger | and use unbound service with is 127.0.0.1 | 01:42 |
| *** itlinux has quit IRC | 01:43 | |
| pabelanger | http://logs.openstack.org/37/493937/3/gate/gate-tripleo-ci-centos-7-scenario002-multinode-oooq/4e30192/logs/undercloud/etc/resolv.conf.save.gz is correct | 01:43 |
| pabelanger | http://logs.openstack.org/37/493937/3/gate/gate-tripleo-ci-centos-7-scenario002-multinode-oooq/4e30192/logs/undercloud/etc/resolv.conf.txt.gz is wrong | 01:43 |
| *** itlinux has joined #tripleo | 01:43 | |
| pabelanger | so that explains why we are getting DNS errors | 01:44 |
| pabelanger | http://logs.openstack.org/83/493383/2/gate/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-puppet/2777163/logs/undercloud/home/jenkins/failed_deployment_list.log.txt.gz | 01:44 |
| pabelanger | bug 1711262 | 01:50 |
| openstack | bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Undecided,New] https://launchpad.net/bugs/1711262 | 01:50 |
| *** ramishra has joined #tripleo | 01:59 | |
| *** michapma has quit IRC | 02:06 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 02:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 02:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 02:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| *** dixiaoli has quit IRC | 02:11 | |
| *** dixiaoli has joined #tripleo | 02:12 | |
| *** itlinux has quit IRC | 02:15 | |
| *** itlinux has joined #tripleo | 02:15 | |
| *** homeski has quit IRC | 02:16 | |
| *** itlinux has quit IRC | 02:17 | |
| *** mkovacik__ has joined #tripleo | 02:18 | |
| *** _milan_ has quit IRC | 02:18 | |
| *** itlinux has joined #tripleo | 02:19 | |
| *** tzumainn has quit IRC | 02:23 | |
| mwhahaha | pabelanger: good spot. | 02:30 |
| mwhahaha | pabelanger: I'll fix it in the morning if no one else does | 02:30 |
| *** jmelvin has quit IRC | 02:42 | |
| EmilienM | pabelanger: ok I'll work on that | 02:51 |
| EmilienM | with mwhahaha | 02:51 |
| EmilienM | and yes, good spot, thanks | 02:51 |
| *** dmacpher has joined #tripleo | 02:51 | |
| *** artom has quit IRC | 02:56 | |
| EmilienM | stevebaker: if you want to help on testing containers, it seems like tempest doesn't success in tests | 02:58 |
| EmilienM | http://logs.openstack.org/84/494284/2/check/gate-tripleo-ci-centos-7-containers-multinode/c7c5f63/logs/tempest.html.gz | 02:58 |
| EmilienM | I haven't debugged yet | 02:58 |
| EmilienM | I need to get dinner first | 02:58 |
| *** eck` is now known as eck`gone | 03:05 | |
| *** homeski has joined #tripleo | 03:07 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 03:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 03:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 03:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| *** mdnadeem has joined #tripleo | 03:19 | |
| *** michapma has joined #tripleo | 03:19 | |
| *** psahoo has joined #tripleo | 03:19 | |
| *** milan has joined #tripleo | 03:21 | |
| *** mkovacik__ has quit IRC | 03:22 | |
| *** homeski has quit IRC | 03:38 | |
| *** yamahata has quit IRC | 03:39 | |
| *** ykarel|afk has joined #tripleo | 03:51 | |
| *** ykarel|afk is now known as ykarel | 03:53 | |
| *** ramishra has quit IRC | 03:54 | |
| *** shreshtha_ has joined #tripleo | 03:55 | |
| *** ramishra has joined #tripleo | 03:56 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: overcloud_containers.yaml.j2 map images to services https://review.openstack.org/448328 | 03:56 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Deprecate --pull-source for container prepare command https://review.openstack.org/494366 | 03:58 |
| *** gkadam has joined #tripleo | 03:58 | |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Filter container images by deployed services https://review.openstack.org/494367 | 03:59 |
| *** gkadam_ has joined #tripleo | 04:01 | |
| *** links has joined #tripleo | 04:01 | |
| *** gkadam has quit IRC | 04:03 | |
| *** fpan has quit IRC | 04:04 | |
| *** gkadam_ is now known as gkadam | 04:04 | |
| *** fpan has joined #tripleo | 04:04 | |
| weshay | jebus | 04:07 |
| *** homeski has joined #tripleo | 04:08 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 04:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 04:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 04:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| *** homeski has quit IRC | 04:14 | |
| *** udesale has joined #tripleo | 04:15 | |
| *** limao has quit IRC | 04:21 | |
| *** limao has joined #tripleo | 04:22 | |
| *** _milan_ has joined #tripleo | 04:23 | |
| *** milan has quit IRC | 04:24 | |
| *** limao has quit IRC | 04:26 | |
| *** yamahata has joined #tripleo | 04:26 | |
| *** homeski has joined #tripleo | 04:27 | |
| *** anshul has joined #tripleo | 04:34 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Set default OSD pool size to 1 in scenario 001/004 containers https://review.openstack.org/494176 | 04:37 |
| *** sri_ has quit IRC | 04:42 | |
| *** openstack has quit IRC | 04:42 | |
| *** openstack has joined #tripleo | 04:44 | |
| *** jrist has joined #tripleo | 04:44 | |
| *** honza has joined #tripleo | 04:44 | |
| *** milan has joined #tripleo | 04:44 | |
| *** rook has joined #tripleo | 04:44 | |
| *** myoung has joined #tripleo | 04:44 | |
| *** anshul has joined #tripleo | 04:44 | |
| *** hewbrocca_afk has joined #tripleo | 04:44 | |
| *** lhinds has joined #tripleo | 04:44 | |
| *** spredzy has joined #tripleo | 04:44 | |
| *** pliu has joined #tripleo | 04:44 | |
| *** jrist has quit IRC | 04:44 | |
| *** jrist has joined #tripleo | 04:44 | |
| *** EmilienM has joined #tripleo | 04:44 | |
| *** vpickard_ has joined #tripleo | 04:45 | |
| *** eck`gone has joined #tripleo | 04:45 | |
| *** akrzos has joined #tripleo | 04:45 | |
| *** rook is now known as Guest88899 | 04:45 | |
| *** oanson has joined #tripleo | 04:45 | |
| *** mhenkel has joined #tripleo | 04:45 | |
| *** honza is now known as Guest28838 | 04:45 | |
| *** leseb has joined #tripleo | 04:45 | |
| *** SlickNik has joined #tripleo | 04:45 | |
| *** dobson has joined #tripleo | 04:45 | |
| *** sai has joined #tripleo | 04:45 | |
| *** mandre has joined #tripleo | 04:46 | |
| *** dmsimard|off has joined #tripleo | 04:46 | |
| *** EmilienM has quit IRC | 04:46 | |
| *** EmilienM has joined #tripleo | 04:46 | |
| *** dixiaoli has joined #tripleo | 04:46 | |
| *** fpan has joined #tripleo | 04:46 | |
| *** zoli has joined #tripleo | 04:46 | |
| *** faceman has joined #tripleo | 04:46 | |
| *** rasca has joined #tripleo | 04:46 | |
| *** ipsecguy has joined #tripleo | 04:46 | |
| *** leifmadsen has joined #tripleo | 04:46 | |
| *** weshay has joined #tripleo | 04:46 | |
| *** amoralej has joined #tripleo | 04:46 | |
| *** arxcruz has joined #tripleo | 04:46 | |
| *** gbarros has joined #tripleo | 04:46 | |
| *** number80 has joined #tripleo | 04:46 | |
| *** openstackstatus has joined #tripleo | 04:46 | |
| *** dtantsur has joined #tripleo | 04:47 | |
| *** bandini has joined #tripleo | 04:47 | |
| *** kbyrne has joined #tripleo | 04:47 | |
| *** ChanServ sets mode: +v openstackstatus | 04:47 | |
| *** leifmadsen has quit IRC | 04:47 | |
| *** leifmadsen has joined #tripleo | 04:47 | |
| *** gchamoul has joined #tripleo | 04:47 | |
| *** zzzeek has joined #tripleo | 04:47 | |
| *** mdbooth has joined #tripleo | 04:47 | |
| *** markmc has joined #tripleo | 04:47 | |
| *** kambiz has joined #tripleo | 04:47 | |
| *** vkhanna has joined #tripleo | 04:48 | |
| *** numans has joined #tripleo | 04:48 | |
| *** mburned_out has joined #tripleo | 04:48 | |
| *** radez has joined #tripleo | 04:48 | |
| *** thrash has joined #tripleo | 04:48 | |
| *** percevalbot has joined #tripleo | 04:48 | |
| *** thrash has quit IRC | 04:48 | |
| *** thrash has joined #tripleo | 04:48 | |
| *** lucasagomes has joined #tripleo | 04:49 | |
| *** shadower has joined #tripleo | 04:49 | |
| *** jistr has joined #tripleo | 04:49 | |
| *** PhilSliderS has joined #tripleo | 04:49 | |
| *** ianw has joined #tripleo | 04:49 | |
| *** jpena|off has joined #tripleo | 04:49 | |
| *** larsks has joined #tripleo | 04:50 | |
| *** jaosorior has joined #tripleo | 04:50 | |
| *** andreaf has joined #tripleo | 04:50 | |
| *** panda has joined #tripleo | 04:50 | |
| *** mrunge has joined #tripleo | 04:50 | |
| *** bcafarel has joined #tripleo | 04:50 | |
| *** funzo has joined #tripleo | 04:50 | |
| *** trown has joined #tripleo | 04:50 | |
| *** dmanchad has joined #tripleo | 04:50 | |
| *** karthiks has joined #tripleo | 04:51 | |
| *** bugzy has joined #tripleo | 04:51 | |
| *** jschlueter|znc has joined #tripleo | 04:51 | |
| *** jidar has joined #tripleo | 04:51 | |
| *** jtomasek has joined #tripleo | 04:51 | |
| *** ramishra has joined #tripleo | 04:51 | |
| *** bnemec has joined #tripleo | 04:51 | |
| *** gbarros has quit IRC | 04:51 | |
| *** cshastri has joined #tripleo | 04:52 | |
| *** dsavineau has joined #tripleo | 04:52 | |
| *** migi has joined #tripleo | 04:52 | |
| *** michapma has joined #tripleo | 04:52 | |
| *** limao has joined #tripleo | 04:54 | |
| *** rodrigods has joined #tripleo | 04:54 | |
| *** cinerama has joined #tripleo | 04:54 | |
| *** dalvarez has joined #tripleo | 04:54 | |
| *** limao has quit IRC | 04:57 | |
| *** limao has joined #tripleo | 04:57 | |
| *** mdnadeem has quit IRC | 05:04 | |
| *** anshul has quit IRC | 05:06 | |
| *** tristanC has joined #tripleo | 05:09 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 05:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 05:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 05:10 |
| *** stendulker has joined #tripleo | 05:10 | |
| *** itlinux has quit IRC | 05:11 | |
| *** dpawar has joined #tripleo | 05:12 | |
| *** limao_ has joined #tripleo | 05:13 | |
| *** limao has quit IRC | 05:14 | |
| *** fabbione has joined #tripleo | 05:14 | |
| *** yprokule has joined #tripleo | 05:16 | |
| *** iranzo has joined #tripleo | 05:20 | |
| *** marios has joined #tripleo | 05:20 | |
| *** openstackgerrit has joined #tripleo | 05:24 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/container: run Barbican non-containerized https://review.openstack.org/493734 | 05:24 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo master: Move barbican's database creation to mysql profile https://review.openstack.org/493953 | 05:24 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/multinode: do not run containerized Zaqar https://review.openstack.org/494005 | 05:25 |
| *** pdeore has joined #tripleo | 05:25 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common stable/ocata: Prompt to clear breakpoints when using deployed-server https://review.openstack.org/491115 | 05:26 |
| *** milan has quit IRC | 05:27 | |
| *** milan has joined #tripleo | 05:28 | |
| *** itlinux has joined #tripleo | 05:31 | |
| *** jaosorior has quit IRC | 05:32 | |
| *** jaosorior has joined #tripleo | 05:33 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo master: Release Pike rc1 - 7.3.0 https://review.openstack.org/494378 | 05:33 |
| *** agurenko has joined #tripleo | 05:34 | |
| *** udesale has joined #tripleo | 05:34 | |
| *** itlinux has quit IRC | 05:41 | |
| *** udesale__ has joined #tripleo | 05:43 | |
| *** pdeore_ has joined #tripleo | 05:43 | |
| *** dmacpher has joined #tripleo | 05:44 | |
| *** pdeore has quit IRC | 05:45 | |
| *** udesale has quit IRC | 05:45 | |
| *** mdnadeem has joined #tripleo | 05:46 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable listening on TLS for the internal network for horizon https://review.openstack.org/489596 | 05:48 |
| *** dmsimard|off has quit IRC | 05:48 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make swift's endpoint type configurable for gnocchi storage https://review.openstack.org/494103 | 05:49 |
| *** dmsimard has joined #tripleo | 05:49 | |
| *** dmsimard is now known as dmsimard|off | 05:49 | |
| *** Guest28838 is now known as honza | 05:50 | |
| *** rcernin has joined #tripleo | 05:57 | |
| *** ramishra has quit IRC | 06:01 | |
| *** nyechiel has joined #tripleo | 06:03 | |
| *** janki has joined #tripleo | 06:03 | |
| *** ramishra has joined #tripleo | 06:03 | |
| *** florianf has joined #tripleo | 06:07 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 06:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 06:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 06:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable listening on TLS for the internal network for horizon https://review.openstack.org/489596 | 06:10 |
| *** skramaja has joined #tripleo | 06:13 | |
| *** pgadiya has joined #tripleo | 06:16 | |
| *** jfrancoa has joined #tripleo | 06:19 | |
| *** brault has joined #tripleo | 06:20 | |
| *** pgadiya has quit IRC | 06:22 | |
| *** dmacpher has quit IRC | 06:28 | |
| *** milan has quit IRC | 06:30 | |
| *** masco has joined #tripleo | 06:30 | |
| *** pdeore_ has quit IRC | 06:32 | |
| *** paramite has joined #tripleo | 06:33 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Introduced extra_tempest_config flag for tempest.conf https://review.openstack.org/494394 | 06:34 |
| *** milan has joined #tripleo | 06:34 | |
| *** brault has quit IRC | 06:36 | |
| *** brault has joined #tripleo | 06:37 | |
| *** milan has quit IRC | 06:38 | |
| *** pdeore_ has joined #tripleo | 06:38 | |
| *** udesale has joined #tripleo | 06:40 | |
| *** mdnadeem has quit IRC | 06:40 | |
| *** pdeore_ is now known as pdeore | 06:40 | |
| *** jlinkes has joined #tripleo | 06:41 | |
| *** mdnadeem has joined #tripleo | 06:41 | |
| *** udesale__ has quit IRC | 06:42 | |
| *** pcaruana has joined #tripleo | 06:43 | |
| dpawar | mwhahaha: any tentative date for merge of branching request https://review.rdoproject.org/r/#/c/8155/ ? | 06:43 |
| *** tesseract has joined #tripleo | 06:47 | |
| *** anshul has joined #tripleo | 06:49 | |
| *** mcornea has joined #tripleo | 06:54 | |
| *** limao_ has quit IRC | 06:56 | |
| *** limao has joined #tripleo | 06:56 | |
| *** mbu has joined #tripleo | 06:57 | |
| *** aufi has joined #tripleo | 06:58 | |
| *** cylopez has joined #tripleo | 06:59 | |
| *** limao has quit IRC | 07:00 | |
| *** limao has joined #tripleo | 07:01 | |
| *** shardy has joined #tripleo | 07:03 | |
| *** michapma has quit IRC | 07:04 | |
| *** yprokule_ has joined #tripleo | 07:06 | |
| *** yprokule has quit IRC | 07:09 | |
| *** yprokule_ is now known as yprokule | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 07:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 07:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 07:10 |
| *** ykarel is now known as ykarel|afk | 07:10 | |
| *** hewbrocca_afk is now known as hewbrocca | 07:11 | |
| *** ebarrera has joined #tripleo | 07:11 | |
| *** ccamacho has joined #tripleo | 07:12 | |
| *** jpich has joined #tripleo | 07:13 | |
| *** jbadiapa_ has joined #tripleo | 07:17 | |
| *** apetrich has quit IRC | 07:18 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Remove duplicate Iscsid service in resource registry https://review.openstack.org/492480 | 07:21 |
| *** jpena|off is now known as jpena | 07:24 | |
| *** mdnadeem has quit IRC | 07:24 | |
| *** gkadam has joined #tripleo | 07:26 | |
| jaosorior | shardy: I'm getting errors running the pingtest, and I see the following issue in the compute logs. http://paste.openstack.org/show/618635/ any idea what that means? | 07:29 |
| jaosorior | marios: ^^ | 07:31 |
| jaosorior | or mandre ^^ (since it's in a containerized environment) | 07:31 |
| shardy | jaosorior: sorry not seen that one before, looks like the volume driver can't find a configured disk? | 07:34 |
| marios | o/ not seen that before jaosorior | 07:34 |
| *** jprovazn has joined #tripleo | 07:34 | |
| *** shardy is now known as shardy_afk | 07:34 | |
| mandre | same here jaosorior, first time I see this error | 07:35 |
| *** agurenko has quit IRC | 07:36 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Avoid overcloud validate timeout on stack failure https://review.openstack.org/493383 | 07:36 |
| jaosorior | where is that disk configured? | 07:36 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/newton: Fix selinux unit tests https://review.openstack.org/493619 | 07:36 |
| *** mdnadeem has joined #tripleo | 07:37 | |
| marios | jfrancoa: about that error you mentioned , I found the 'original' (I mean subnode 2 logs), might be worth digging here too http://logs.openstack.org/99/491399/9/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/49da6f3/logs/subnode-2/var/log/messages.txt.gz#_Aug_16_14_17_05 in case you didn't see it yet | 07:39 |
| *** mrch has joined #tripleo | 07:39 | |
| marios | jfrancoa: i mean the "ERROR! no action detected in task" thing | 07:40 |
| jfrancoa | marios: yes, thanks for checking it! I will dig in a little bit more. Let's see if I manage to find why it is failing | 07:40 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-quickstart master: Add featureset33 for scenario009 https://review.openstack.org/494418 | 07:41 |
| *** sshnaidm|afk has joined #tripleo | 07:41 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:41 | |
| *** mdnadeem has quit IRC | 07:47 | |
| *** dpawar has quit IRC | 07:47 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Switch scenario004 to run Tempest https://review.openstack.org/491113 | 07:52 |
| *** mbu is now known as matbu | 07:53 | |
| *** honza has quit IRC | 07:54 | |
| *** honza has joined #tripleo | 07:54 | |
| *** honza is now known as Guest11829 | 07:55 | |
| *** zzzeek has quit IRC | 07:56 | |
| *** agurenko has joined #tripleo | 07:58 | |
| *** tesseract has quit IRC | 07:58 | |
| *** zzzeek has joined #tripleo | 07:59 | |
| *** athomas has quit IRC | 07:59 | |
| jaosorior | mandre: oh right. I think I found the issue. It's because I'm not running a containerized compute but tried to run a containerized iscsid. | 08:00 |
| jaosorior | marios: ^^ | 08:01 |
| *** yprokule_ has joined #tripleo | 08:02 | |
| *** mdnadeem has joined #tripleo | 08:04 | |
| *** derekh has joined #tripleo | 08:04 | |
| *** yprokule has quit IRC | 08:04 | |
| *** yprokule_ is now known as yprokule | 08:04 | |
| *** athomas has joined #tripleo | 08:06 | |
| *** oidgar has joined #tripleo | 08:06 | |
| mandre | jaosorior: ack, that could very well explain it, yeah | 08:06 |
| mandre | jaosorior: but it seems like a bug to me because in theory the services should be able to run independent on BM or in containers | 08:07 |
| mandre | bbl | 08:07 |
| jaosorior | mandre: yep, it does seem to me like a bug. | 08:08 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 08:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 08:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 08:10 |
| *** dpawar has joined #tripleo | 08:15 | |
| *** egonzalez has joined #tripleo | 08:16 | |
| *** openstackgerrit has quit IRC | 08:17 | |
| *** gfidente has joined #tripleo | 08:18 | |
| *** gfidente has quit IRC | 08:18 | |
| *** gfidente has joined #tripleo | 08:18 | |
| *** shardy_afk is now known as shardy | 08:20 | |
| *** openstackgerrit has joined #tripleo | 08:21 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Move creating fake image to oooq extras https://review.openstack.org/494235 | 08:21 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Remove iscsid from TLS everywhere docker environment https://review.openstack.org/494429 | 08:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: TLS everywhere/docker: add nova services to environment https://review.openstack.org/494430 | 08:23 |
| jaosorior | shardy, marios, mandre: The patch removing iscsid fixed the issue ^^ | 08:23 |
| jaosorior | * removing containerized iscsid | 08:24 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Move creating fake image to oooq extras https://review.openstack.org/494233 | 08:24 |
| openstackgerrit | Jan Provaznik proposed openstack/tripleo-heat-templates master: [test] Let mds create manila key and fs https://review.openstack.org/494431 | 08:25 |
| jprovazn | gfidente: I wonder if this one passes 001 CI job ^ | 08:26 |
| jaosorior | marios: and I just tested the containerized nova services with TLS. Thus I added them to the environment file https://review.openstack.org/494430 . compute and libvirt are missing still. | 08:26 |
| marios | jaosorior: ack +2d the iscsi one waiting for ci on the compute services? | 08:26 |
| jaosorior | marios: well, it's not used anywhere yet. | 08:26 |
| marios | jaosorior: ah ok so not exercised by any job | 08:26 |
| jaosorior | marios: that file right now is more to enable people to test the containerized TLS services. | 08:26 |
| openstackgerrit | zenghui.shi proposed openstack/os-net-config master: Add NIC Mapping Reporting Feature https://review.openstack.org/383516 | 08:28 |
| marios | jaosorior: ack | 08:28 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Introduced extra_tempest_config flag for tempest.conf https://review.openstack.org/494394 | 08:29 |
| *** apetrich has joined #tripleo | 08:31 | |
| gfidente | morning jprovazn | 08:32 |
| openstackgerrit | Martin Kopec proposed openstack/tripleo-quickstart-extras master: Allow removing of options from tempest conf https://review.openstack.org/477079 | 08:32 |
| gfidente | marios in I am just copy/pasting the existing parameter definition | 08:32 |
| gfidente | from all-nodes-config.yaml | 08:32 |
| gfidente | so the same parameter is consumed by both | 08:32 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Add support for Dell EMC Unity Cinder backend https://review.openstack.org/487599 | 08:36 |
| marios | gfidente: ah ok didn't realise thanks | 08:38 |
| oidgar | hi everybody, after couple of days that I'm trying to deploy overcloud without any success I found that there's a problem/bug with nova-scheduler regarding table schema (the environment is brand new. undercloud was installed from scratch) | 08:38 |
| *** dpawar has quit IRC | 08:38 | |
| gfidente | marios though the pep8 error is real, fixing itnow | 08:38 |
| oidgar | the error messages in nova-scheduler.log: http://paste.openstack.org/show/618640/ | 08:38 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Swith to the appropriate ceph-ansible playbook on upgrade https://review.openstack.org/494336 | 08:39 |
| shardy | oidgar: interesting, any idea why we're not seeing the issue in CI? | 08:39 |
| oidgar | anyone else encountered this problem? the deployment is failed at step 4.0 because of compute timeout | 08:39 |
| oidgar | shardy, not a clue | 08:39 |
| oidgar | shardy: I think I was installing a brand new undercloud for about 6 or 7 times | 08:39 |
| oidgar | shardy: I'm using tripleo-ci for the installation | 08:40 |
| oidgar | shardy: but up to last week it worked perfectly | 08:40 |
| *** gvrangan_odl has joined #tripleo | 08:41 | |
| shardy | owalsh: Hey don't suppose the issue pasted by oidgar looks familiar to you by any chance? | 08:41 |
| oidgar | this is a real blocker for me as I can't work at all, if someone knows what can cause it and how to solve it I'll be grateful :) | 08:46 |
| shardy | oidgar: have you tried manually syncing the nova DB? | 08:47 |
| oidgar | have to drop off for a while, will be back later. thanks in advance :) | 08:47 |
| openstackgerrit | Derek Higgins proposed openstack/instack-undercloud master: Provide LOCAL_IP_WRAPPED as a instack env variable https://review.openstack.org/494440 | 08:47 |
| openstackgerrit | Derek Higgins proposed openstack/instack-undercloud master: Parse DSN strings with regex https://review.openstack.org/494441 | 08:47 |
| openstackgerrit | Derek Higgins proposed openstack/instack-undercloud master: Wrap IPv6 addresses in square brackets https://review.openstack.org/494442 | 08:47 |
| openstackgerrit | Derek Higgins proposed openstack/instack-undercloud master: Create a IPv6 ctlplane subnet if using IPv6 https://review.openstack.org/494443 | 08:47 |
| oidgar | shardy: no, I'll talk with you when I come back to understand what to do if it's ok with you | 08:47 |
| shardy | oidgar: ack | 08:47 |
| oidgar | shardy: thanks shardy! | 08:47 |
| *** hanish has joined #tripleo | 08:49 | |
| *** dpawar has joined #tripleo | 08:53 | |
| *** lyarwood has joined #tripleo | 08:55 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Add support for Dell EMC Unity Manila Backend https://review.openstack.org/491078 | 08:56 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart master: Remove hardcoded overcloud_release value https://review.openstack.org/480461 | 08:58 |
| *** salmankhan has joined #tripleo | 08:59 | |
| *** aditya_r has joined #tripleo | 09:00 | |
| *** apetrich has quit IRC | 09:01 | |
| *** apetrich has joined #tripleo | 09:02 | |
| *** Guest11829 is now known as honza | 09:09 | |
| *** remix_tj has joined #tripleo | 09:10 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 09:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 09:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 09:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 09:10 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/494127 | 09:10 |
| *** fzdarsky has joined #tripleo | 09:10 | |
| *** chem has joined #tripleo | 09:13 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo master: Configure cache when setting up docker_registry https://review.openstack.org/494451 | 09:16 |
| *** afazekas is now known as afazekas|sick | 09:19 | |
| *** zshi has joined #tripleo | 09:19 | |
| *** zshi has quit IRC | 09:19 | |
| *** zshi has joined #tripleo | 09:20 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-quickstart-extras master: Fix conditional in undercloud-deploy role https://review.openstack.org/472609 | 09:21 |
| *** aditya_r has quit IRC | 09:22 | |
| matbu | marios: ccamacho is it possible to put the +A for https://review.openstack.org/#/c/479212 ? | 09:22 |
| *** tosky has joined #tripleo | 09:24 | |
| marios | matbu: sec on phone will look in sec | 09:24 |
| *** limao has quit IRC | 09:25 | |
| apetrich | matbu, I didn't know you could do conditionals like that. Nice! | 09:27 |
| *** social has joined #tripleo | 09:27 | |
| *** zoli is now known as __mlisik__ | 09:27 | |
| matbu | apetrich: :) hey man | 09:28 |
| *** __mlisik__ is now known as zoli | 09:28 | |
| marios | matbu: ugh ci ran >1 month ago there :) | 09:31 |
| *** gkadam is now known as gkadam-afk | 09:32 | |
| shardy | marios: hey, is there an up to date list of outstanding patches for minor update? | 09:32 |
| shardy | I was looking for the one adding the noop environment, related to https://review.openstack.org/#/c/486180/ | 09:33 |
| marios | shardy: i think yprokule may have one (list of patches you are applying for update) | 09:33 |
| shardy | https://etherpad.openstack.org/p/pike-update has a list but it doesn't look current | 09:33 |
| shardy | marios: cool, I wanted to chat about the workflow where we noop the deploy steps | 09:33 |
| marios | shardy: ah we didn't land one for the noop | 09:33 |
| *** spredzy has quit IRC | 09:33 | |
| marios | shardy: i mean we were adding it locally to override | 09:33 |
| shardy | that sounds reasonable, but we'll need to think about how to restore the steps after the update | 09:33 |
| chem | shardy: at the bottom of the etherpad you have the upstream list | 09:33 |
| shardy | or things like config changes won't work after the update | 09:33 |
| chem | shardy: tested on tuesday for upstream | 09:34 |
| shardy | chem: ah, didn't scroll down enough , thanks :) | 09:34 |
| marios | shardy: yeah so, we *just* had a chat with social and yprokule . the idea is we need 2 things still clientwise... one is some kind of --minor-update-init (similar to --update-plan-only) which will noop and run the stack update | 09:34 |
| yprokule | marios: I'm using the same one as shardy | 09:34 |
| shardy | marios: I was originally thinking we'd rework the overcloud update command | 09:34 |
| marios | shardy: and then another for the playbook invocation | 09:34 |
| shardy | and that could pass in either a flag or a special environment | 09:35 |
| shardy | then the overcloud deploy command would restore it | 09:35 |
| shardy | that could do the stack update, the config download, then prompt for each role to apply the ansible playbook | 09:35 |
| matbu | marios: shardy actually the update command could it all (i mean config dowload and noop | 09:35 |
| matbu | ) | 09:36 |
| shardy | matbu: yeah, exactly | 09:36 |
| matbu | i think i started something like that before my ptos | 09:36 |
| social | the update command should not eat new env :) | 09:36 |
| shardy | Cool, probably not the highest priority right now, but something to think about when the basic workflow is proven | 09:36 |
| social | you whould have to provide update command with environment files | 09:37 |
| social | as we do expect them to change | 09:37 |
| matbu | shardy: yep sure | 09:37 |
| marios | shardy: ok... social was going to look at implementing something like --minor-update-init today so this is good time to have this discussino :) | 09:37 |
| shardy | social: no the update command doesn't currently support changing the configuration, the -e args etc don't work AFAIK? | 09:37 |
| shardy | which makes this easier, as we just do one "special" stack update prior to the ansible things? | 09:38 |
| social | shardy: yes it does not, thats why we have update-plan-only | 09:38 |
| *** dixiaoli has quit IRC | 09:38 | |
| shardy | social: yeah true, we'd need to fix it so we could reference the update environment, or switch to using a flag | 09:38 |
| social | shardy: idea was that if you just want to run yum update you don't need to update-plan-only but if you are delivering fixes from tripleo you do update-plan-only (usually you need fixes from tripleo to restart things correctly as our packages often get broken) | 09:39 |
| shardy | social: yup, I guess this workflow is just different now we have to consider containers vs just yum | 09:39 |
| marios | shardy: so it sounds reasonable and I think in the end it is the same (except the running the playbooks too part)... either run openstack update stack ... or openstack overcloud deploy --minor-update-init ... to set the OS::Heat::None on PostDeploymentSteps and run the stack update | 09:39 |
| shardy | containers make updates easier! Oh wait ;) | 09:40 |
| social | shardy: we most likely will still end up delivering hacks/workarounds for the update_tasks that we'll need before running update | 09:40 |
| social | shardy: idea of having noop option inside the overcloud deploy seems fine to me | 09:40 |
| shardy | marios: ack, yeah I don't have a strong opinion on the interface, I just thought the docs/operator impact would be less if we maintained overcloud update to hide the workflow changes | 09:40 |
| social | question is how long would that run? | 09:41 |
| shardy | social: if we disable the PostDeploySteps the stack update should be pretty fast but it'll still take a few minutes I expect | 09:42 |
| matbu | i guess around 10 / 15 minutes | 09:42 |
| matbu | depending on hardware | 09:42 |
| *** spredzy has joined #tripleo | 09:42 | |
| *** salmankhan1 has joined #tripleo | 09:43 | |
| yprokule | matbu: depending on deployment size ? | 09:43 |
| social | ;.; 10~15 minutes is 9~14 minutes regression | 09:43 |
| *** salmankhan has quit IRC | 09:44 | |
| *** salmankhan1 is now known as salmankhan | 09:44 | |
| social | but in overall picture it's not bad and it can be ran before the downtime | 09:45 |
| social | I mean maintenance window | 09:45 |
| shardy | Yeah compared with the time to do the rolling update of every node it should be a small overhead | 09:45 |
| shardy | not ideal, but hopefully workable | 09:45 |
| marios | shardy: social took ~20 mins on my slow beaker box last week to do stack update with noop | 09:45 |
| shardy | marios: I can do a full overcloud deploy in less time than that locally ;) | 09:46 |
| shardy | but yeah I guess it's going to vary | 09:46 |
| marios | shardy: heh as i said *slow* beaker box :) | 09:46 |
| shardy | hehe :) | 09:46 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Add check on download_overcloud_templates_rpm to trigger custom-tht https://review.openstack.org/479212 | 09:46 |
| *** afazekas|sick is now known as afazekas | 09:47 | |
| social | shardy: any idea how to sneak the oop in without uploading it to swift? | 09:48 |
| shardy | social: I think we have to update the entire plan in swift anyway? E.g if t-h-t changed and new update_tasks exist etc | 09:50 |
| social | shardy: yes, that has to happen I was just thinking to not to put noop there, but on the other hand scale and deploy will always reupload the environment and remove the noop. yes? | 09:51 |
| shardy | social: yeah we'll just have to check that running another deploy restores the non-noop resource_registry mapping, which I think it should because we include the base overcloud-resource-registry-puppet by default | 09:53 |
| shardy | in future this will be easier when we just maintain a list of environments in the plan vs merging it all in tripleoclient | 09:53 |
| * shardy needs to rebase his patches that do that | 09:54 | |
| *** masco is now known as masco_lunch | 09:55 | |
| chandankumar | sshnaidm: please have a look https://review.openstack.org/493030 | 09:59 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable listening on TLS for the internal network for horizon https://review.openstack.org/489596 | 10:00 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Add -n/--networks-data option https://review.openstack.org/493933 | 10:02 |
| *** dsariel has quit IRC | 10:04 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 10:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 10:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 10:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 10:10 |
| *** florianf has quit IRC | 10:12 | |
| *** bogdando has joined #tripleo | 10:14 | |
| *** ykarel|afk is now known as ykarel | 10:15 | |
| *** udesale has quit IRC | 10:18 | |
| owalsh | oidgar, shardy: yea, looks like the db sync timeout | 10:19 |
| owalsh | oidgar: you hit this a while back too IIRC. Is this master, and have you increased the timeout? | 10:20 |
| *** florianf has joined #tripleo | 10:25 | |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy OpenShift using OOO on the overcloud https://review.openstack.org/494470 | 10:25 |
| *** saneax_-_ has joined #tripleo | 10:28 | |
| lyarwood | marios: re https://bugs.launchpad.net/tripleo/+bug/1708115 are you still looking for people to test this series? I finally have a free env to run through this if you are. | 10:29 |
| openstack | Launchpad bug 1708115 in tripleo "Ensure non-controller are usable after upgrade and before converge." [Critical,Triaged] | 10:29 |
| * lyarwood isn't sure from the etherpad what the current status is tbh | 10:30 | |
| marios | lyarwood: yeah testing the things on comment #8 would be good thanks | 10:30 |
| marios | lyarwood: which etherpad are you looking at | 10:31 |
| lyarwood | marios: cool, do you have a hack/command/script for applying all of these scripts btw? Just so I don't reinvent the wheel here. | 10:31 |
| lyarwood | marios: https://etherpad.openstack.org/p/tripleo-pike-updates-upgrades at the top | 10:31 |
| lyarwood | s/scripts/reviews/g | 10:32 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Make env variables for multinode into ansible vars https://review.openstack.org/483930 | 10:32 |
| marios | lyarwood: ah ack yeah no i don't have one already but we can use that .. there was one more patch that was needed on the client i'll add there | 10:32 |
| openstackgerrit | Merged openstack/instack-undercloud master: Add PATCH to list of allowed methods for Ironic https://review.openstack.org/494228 | 10:32 |
| lyarwood | marios: cool thanks | 10:32 |
| oidgar | owalsh: the db sync timeout I hit before was for low spec environments. in here I didn't use any environment files | 10:35 |
| owalsh | oidgar: looks like a db sync didn't happen.. which release is it? | 10:35 |
| oidgar | owalsh: latest/master | 10:36 |
| owalsh | oidgar: with or without containers? | 10:36 |
| oidgar | owalsh: using tripleo-ci to install the undercloud, then running "openstack overcloud deploy --templates" | 10:36 |
| oidgar | owalsh: without containers. simplest one | 10:36 |
| oidgar | owalsh: I had problems deploying overcloud with containers so I tried the basic deployment just to make sure it works... which didn't | 10:37 |
| oidgar | owalsh: another thing, shardy suggested to run the db sync manually. it failed on the controller | 10:37 |
| owalsh | oidgar: anything output when it failed or in /var/log/nova/nova-manage.log? | 10:38 |
| oidgar | owalsh: the result of "nova-manage db sync": http://paste.openstack.org/show/618653/ | 10:39 |
| oidgar | owalsh: it is the stdout from the command. looking at nova-manage.log now | 10:40 |
| oidgar | owalsh: the logs says basically the same, just the traceback is shorter | 10:41 |
| marios | lyarwood: should do it | 10:41 |
| oidgar | owalsh: "Table 'instances' already exists" looks like the main issue IMO | 10:41 |
| owalsh | oidgar: suggest a dbsync was killed | 10:41 |
| oidgar | owalsh: what do you mean? in a middle of overcloud deployment while heat stack is running? | 10:42 |
| lyarwood | marios: ack cool thanks | 10:42 |
| owalsh | oidgar: yes, do you have puppet logs? | 10:42 |
| *** bogdando has quit IRC | 10:43 | |
| oidgar | owalsh: yes, from /var/log/messages but they don't say anything special. really short | 10:43 |
| *** ansiwen has joined #tripleo | 10:43 | |
| oidgar | owalsh: anything special you suggest to search for in the log? | 10:43 |
| *** jkilpatr has quit IRC | 10:44 | |
| owalsh | oidgar: anything releated to the nova dbsyncs | 10:44 |
| *** akrivoka has joined #tripleo | 10:44 | |
| shardy | oidgar: how much memory does the overcloud node have? | 10:47 |
| *** apetrich_ has joined #tripleo | 10:47 | |
| *** dsariel has joined #tripleo | 10:47 | |
| shardy | could the sync have gotten killed by oom or something? | 10:47 |
| *** apetrich has quit IRC | 10:47 | |
| owalsh | shardy: shouldn't the deploy fail if this happened? | 10:48 |
| shardy | owalsh: well it did fail, it won't necessarily say you ran out of memory tho | 10:48 |
| shardy | that should be in the logs of course | 10:48 |
| *** bogdando has joined #tripleo | 10:48 | |
| oidgar | shardy: 12GB for controller and 8GB for compute | 10:48 |
| *** ramishra has quit IRC | 10:48 | |
| shardy | Ok I guess not oom then | 10:49 |
| oidgar | owalsh,shardy: looking now at the logs to find anything related to nova dbsync | 10:49 |
| owalsh | oidgar: check /var/log/nova/nova-manage.log too. It shows the DB migrations e.g http://logs.openstack.org/92/494292/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/b37b2a4/logs/overcloud-controller-0/var/log/nova/nova-manage.log.txt.gz | 10:49 |
| *** ramishra has joined #tripleo | 10:50 | |
| oidgar | owalsh: what is the easiest way to upload the log file to some public url so I'll be able to share it? (don't want to copy paste from cli... too much time :) ) | 10:53 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Keep floating ip reachability during pacemaker migration. https://review.openstack.org/474967 | 10:53 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Keep floating ip reachability during pacemaker migration. https://review.openstack.org/474967 | 10:53 |
| *** agurenko has quit IRC | 10:53 | |
| shardy | oidgar: you can install fpaste from epel, then you can do cat foo | fpaste | 10:55 |
| *** agurenko has joined #tripleo | 10:55 | |
| oidgar | shardy: cool! | 10:55 |
| shardy | oidgar: there is a size limit so you probably want to use head/tail on large logfiles | 10:55 |
| oidgar | shardy: ack | 10:55 |
| *** pkovar has joined #tripleo | 10:56 | |
| social | shardy: marios: https://paste.fedoraproject.org/paste/LpPqLnco62rgC9z7V5qOhw/ | 10:57 |
| social | those are atm "significant times" with the noop | 10:57 |
| oidgar | owalsh,shardy: nova-manage.log: https://paste.fedoraproject.org/paste/jmbYKTf0nDWmqUWrLpf2Bw | 10:57 |
| marios | cool thanks social so it 'sounds' like ~ 20 mins on that box too | 10:58 |
| social | yeah I just wonder what we can cut out of it | 10:58 |
| *** cshastri has quit IRC | 10:59 | |
| *** akrivoka has quit IRC | 11:00 | |
| *** shardy is now known as shardy_lunch | 11:01 | |
| social | marios: shardy_lunch: but this might be a big issue https://paste.fedoraproject.org/paste/BRelexdbyk4~t6dKpLPX6A/ - it ran stuff before we do any yum modifications | 11:03 |
| social | I mean it would be changing stuff while it's not supposed to touch the nodes | 11:04 |
| oidgar | owalsh: also found in /var/log/messages: http://paste.openstack.org/show/618657/ | 11:04 |
| owalsh | oidgar: yea, about to say the migration was interrupted here: 2017-08-16 14:45:49.902 49780 INFO migrate.versioning.api [req-ca6dfc8a-be0b-4782-8d5b-7154004eb143 - - - - -] 215 -> 216... | 11:05 |
| owalsh | oidgar: what kind of storage is this on? | 11:06 |
| oidgar | owalsh: guess ceph...? :) I'm working on RDO cloud | 11:08 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 11:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 11:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 11:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 11:10 |
| owalsh | oidgar: ah, known issues with terrible I/O on RDO cloud | 11:10 |
| oidgar | owalsh: yay :) I don't have resources for installing tripleO except this... kind of locked | 11:11 |
| oidgar | owalsh: any insights/suggestions...? | 11:11 |
| owalsh | oidgar: bump the timeouts and cross fingers :-) | 11:11 |
| oidgar | owalsh: lol, maybe I should use the low spec env file | 11:12 |
| *** apetrich_ has quit IRC | 11:12 | |
| oidgar | owalsh: do you think we can get more details from the current deployment or should I redeploy again and delete the current overcloud? | 11:13 |
| *** stendulker has quit IRC | 11:14 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Separate config_volume for ringbuilder https://review.openstack.org/494008 | 11:14 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Use Docker registry by default https://review.openstack.org/494483 | 11:14 |
| *** ramishra has quit IRC | 11:14 | |
| *** ramishra has joined #tripleo | 11:15 | |
| *** akrivoka has joined #tripleo | 11:16 | |
| *** ramishra has quit IRC | 11:17 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Adds PostUpgradeConfigStepsDeployment to drive post config ansible https://review.openstack.org/493878 | 11:17 |
| *** ramishra has joined #tripleo | 11:18 | |
| owalsh | oidgar: don't think we need any more details. Redploy with increased timeouts might work | 11:18 |
| owalsh | oidgar: or I think there are no dbsync timeout when using containers... but you may hit other issues if the storage is very slow | 11:20 |
| owalsh | oidgar: would be interesting to know how the containerized deployment handles this FWIW | 11:20 |
| *** ramishra has quit IRC | 11:21 | |
| *** ramishra has joined #tripleo | 11:21 | |
| *** jschlueter|znc is now known as jschlueter | 11:23 | |
| *** jkilpatr has joined #tripleo | 11:23 | |
| bogdando | EmilienM: hi. wrt upgrade jobs update failures, http://logs.openstack.org/00/461000/37/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/1beac0e/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-17_01_03_09 fails cuz of the "Repository 'delorean-current': Error parsing config: Error parsing" error it seems. Elastic-recheck shows 280 hits for it, all failures | 11:27 |
| *** hewbrocca is now known as hewbrocca_afk | 11:27 | |
| bogdando | EmilienM, pabelanger: ^^ shall we add e-r query for it? | 11:27 |
| *** masco_lunch is now known as masco | 11:27 | |
| bogdando | but all hits are for that only job type | 11:28 |
| oidgar | owalsh: I've deployed environments with containers many times without any issues. I thought it will be better to try the baremetal one but what you're saying is different. | 11:29 |
| bogdando | EmilienM: there is also glance api error "'Connection aborted.', BadStatusLine("''",)" looks odd | 11:29 |
| oidgar | owalsh: I'll try to redploy again first baremetal version with low spec env to see if there's any change | 11:30 |
| oidgar | owalsh: and will update once finished | 11:30 |
| *** lucasagomes is now known as lucas-hungry | 11:30 | |
| owalsh | oidgar: ack, if containers works ok on this env then I expect it will work with the low-mem environment | 11:31 |
| oidgar | owlash: meanwhile, many thanks for the assistance! | 11:32 |
| bogdando | EmilienM: but the latter can be ignored, according to the e-r (~2000 hits, 80% success rates) | 11:32 |
| owalsh | oidgar: np. TBH I think the dbsync timeouts are a bad idea... I've never seen it solve a problem but it causes lots... but I've not convinced everyone yet :-) | 11:35 |
| *** jkilpatr has quit IRC | 11:37 | |
| *** jkilpatr has joined #tripleo | 11:37 | |
| *** Guest88899 is now known as rook | 11:39 | |
| *** atoth has joined #tripleo | 11:40 | |
| *** abishop has joined #tripleo | 11:40 | |
| *** bogdando_ has joined #tripleo | 11:51 | |
| *** rhallisey has joined #tripleo | 11:51 | |
| *** bogdando_ has quit IRC | 11:51 | |
| *** bogdando has quit IRC | 11:52 | |
| *** artom has joined #tripleo | 11:53 | |
| *** pchavva has joined #tripleo | 11:55 | |
| jaosorior | mandre: are we setting any docker labels to the containers at the moment? | 11:55 |
| *** shardy_lunch is now known as shardy | 11:57 | |
| *** pradk has joined #tripleo | 11:58 | |
| *** bogdando has joined #tripleo | 11:59 | |
| *** pdeore_ has joined #tripleo | 11:59 | |
| jaosorior | shardy: tripleo runs containers using paunch, right? | 12:01 |
| *** pdeore has quit IRC | 12:02 | |
| *** cdearborn has joined #tripleo | 12:03 | |
| owalsh | jaosorior: can check in CI e.g http://logs.openstack.org/92/494292/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq/cde8acf/logs/overcloud-controller-0/var/log/extra/docker/containers/nova_api/docker_info.log.txt.gz | 12:04 |
| owalsh | jaosorior: and yea, paunch | 12:04 |
| jaosorior | owalsh: nice, nice | 12:06 |
| jaosorior | owalsh: any hints on where the command paunch command is executed? Would like to pass extra labels for the containers | 12:06 |
| *** masco has quit IRC | 12:07 | |
| ff | bogdando: hey there | 12:07 |
| bogdando | ff hey | 12:07 |
| ff | whoops O.o | 12:07 |
| *** ff is now known as flaper87 | 12:07 | |
| flaper87 | bogdando: ok, now it's better | 12:07 |
| flaper87 | :D | 12:07 |
| bogdando | flaper87: yeah | 12:07 |
| flaper87 | bogdando: have a sec to talk about kubespray? got a query for you | 12:07 |
| *** flaper87 is now known as Guest38917 | 12:08 | |
| owalsh | jaosorior: I think you can just set labels in config_data https://github.com/openstack/paunch/blob/master/paunch/builder/compose1.py#L20 | 12:08 |
| *** marios has quit IRC | 12:08 | |
| bogdando | Guest38917: sure go on | 12:08 |
| Guest38917 | freaking freenode is driving me crazy today | 12:08 |
| *** marios has joined #tripleo | 12:08 | |
| Guest38917 | bogdando: so, I'm running kubespray with `--skip-flags docker`, which uses the already installed docker | 12:08 |
| Guest38917 | that works almost ok except, in CentOS, the service file uses MountFlags=slave | 12:09 |
| bogdando | Guest38917: do you mean skip-tags? | 12:09 |
| Guest38917 | bogdando: crap, yeah, tags | 12:09 |
| Guest38917 | bogdando: so, the question is, what do you think about having 2 different tags for docker? docker and docker-systemd ? | 12:09 |
| bogdando | Guest38917: sigh. tags/skip-tags was never tested, just an experimental thing. If it works as you expect, you're lucky | 12:09 |
| shardy | jaosorior: see here https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml#L52 | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 12:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| Guest38917 | that way we can have kubespray configure docker's service file but not installing it | 12:10 |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 12:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 12:10 |
| *** dpawar has quit IRC | 12:10 | |
| Guest38917 | bogdando: it works fine 'cause I'm just skipping the entire role | 12:10 |
| *** jlabarre has joined #tripleo | 12:10 | |
| bogdando | Guest38917: yes, we can propose a patch for a new tag, why not | 12:10 |
| jaosorior | owalsh: seems to me that it's expecting the labels fromt he command line arguments, and not from config_data | 12:11 |
| Guest38917 | bogdando: cool, lemme do that | 12:11 |
| owalsh | jaosorior: yea, just about to say that | 12:11 |
| Guest38917 | now I need to get my nick back | 12:11 |
| *** Guest38917 has quit IRC | 12:11 | |
| *** ff has joined #tripleo | 12:11 | |
| owalsh | jaosorior: https://github.com/openstack/heat-agents/blob/master/heat-config-docker-cmd/install.d/hook-docker-cmd.py | 12:12 |
| jaosorior | owalsh: what's that? thoughtt we were calling it explicitly, as shardy pointed, here https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml#L52 | 12:13 |
| shardy | yeah we're not using the docker-cmd hook since we switched to ansible driving paunch | 12:14 |
| owalsh | shardy, jaosorior: ah, ignore me then :-) | 12:14 |
| *** jtomasek_ has joined #tripleo | 12:14 | |
| shardy | owalsh: paunch was derived from the same code, so they should work similarly | 12:14 |
| openstackgerrit | Karthik S proposed openstack/tripleo-heat-templates master: NetworkDeploymentActions shall be made role specific https://review.openstack.org/490474 | 12:15 |
| jaosorior | shardy, owalsh: So for TLS everywhere, certmonger used to restart or reload the specific service that's using the cert after the cert is gotten (this is useful, for instance, for a certificate renewal). And now I'm looking for a way to do this but on containers. | 12:15 |
| *** jtomasek has quit IRC | 12:16 | |
| jaosorior | Else, the certificate will get automatically renewed, but the service will be serving the old certificate, and connections will start to fail (due to the expired certificates) | 12:16 |
| shardy | jaosorior: you can change anything in the environment that's passed to the container then it'll get restarted | 12:16 |
| *** morazi has joined #tripleo | 12:17 | |
| shardy | jaosorior: e.g in docker-puppet we calculate a hash of the directory of /etc files generated by puppet | 12:17 |
| jaosorior | shardy: how? | 12:17 |
| bogdando | shardy, jaosorior: by the next stack update. right? | 12:17 |
| shardy | so we restart if the hash changes | 12:17 |
| *** morazi has quit IRC | 12:17 | |
| shardy | bogdando: yes, or the next run of paunch | 12:17 |
| jaosorior | shardy: problem is, the certificates are not in the /etc files generated by puppet. That's in the host. | 12:17 |
| shardy | jaosorior: paunch compares the container with the json, if the json changes, the container is restarted | 12:17 |
| *** morazi has joined #tripleo | 12:18 | |
| jaosorior | shardy: and the path to the certificate remains the same, it's just the content that changes. | 12:18 |
| shardy | jaosorior: yeah but you could perhaps use a similar approach e.g calculate a hash of the contents? | 12:18 |
| shardy | jaosorior: when is it generated, e.g is it via a stack update? | 12:19 |
| *** marios has quit IRC | 12:19 | |
| bogdando | amoralej: hi. I'm trying to locate the "Package Review" bz for the ansible-pacemaker package to follow that for my task, could you please help to locate it? | 12:20 |
| jaosorior | shardy: Well, on overcloud deploy the certificates are requested (and gotten if all goes well). If there's a stack update, that doesn't necessarily trigger a certificate resubmit. If the cert is fine, it will do nothing. | 12:20 |
| bogdando | jpena: ^^ hi | 12:20 |
| *** marios has joined #tripleo | 12:20 | |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-puppet.py#L362 | 12:20 |
| *** dprince has joined #tripleo | 12:20 | |
| shardy | jaosorior: that's how we add the config hash - I'm wondering if we could add a TRIPLEO_CERT_HASH or something | 12:20 |
| jaosorior | shardy: certmonger runs all the time in the host, and keeps track of the certificates, and if it encounters that one needs a re-submission (cause it will expire) it will request one again by itself. | 12:20 |
| jpena | bogdando: give me a sec, I'll find it | 12:21 |
| bogdando | amoralej, jpena: https://review.rdoproject.org/r/#/c/8424/2 should be absolutely the same as the ansible-pacemaker story, it seems | 12:21 |
| jaosorior | shardy: is paunch running as a daemon and detects changes on the fly? | 12:21 |
| shardy | jaosorior: Ok so it's not done via a stack update | 12:21 |
| bogdando | jpena: thanks! | 12:21 |
| shardy | jaosorior: No | 12:21 |
| *** liverpooler has joined #tripleo | 12:21 | |
| *** eck`gone is now known as eck` | 12:22 | |
| jpena | bogdando: https://bugzilla.redhat.com/show_bug.cgi?id=1406728 | 12:22 |
| openstack | bugzilla.redhat.com bug 1406728 in Package Review "ansible-pacemaker - Ansible library for tripleo composable upgrade" [Unspecified,Closed: errata] - Assigned to jpena | 12:22 |
| bogdando | jpena: thanks a bunch! | 12:22 |
| jaosorior | shardy: so I had the idea to add some labels to the templates for services that should be restarted together (services running over httpd for instance), and use that to filter using "docker ps". This way I could just restart the certificates once it's needed. | 12:23 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui master: Make validation groups labels clickable https://review.openstack.org/494513 | 12:24 |
| shardy | jaosorior: yeah, or changing a hash in the json passed to paunch, then re-running paunch would basically do the same I think | 12:24 |
| jaosorior | shardy: that could work too. I'm not entirely sure how that json is generated though. | 12:25 |
| *** dciabrin_ has joined #tripleo | 12:25 | |
| *** dciabrin has quit IRC | 12:25 | |
| shardy | jaosorior: it's generated by ansible in t-h-t | 12:25 |
| *** jprovazn has quit IRC | 12:25 | |
| *** dciabrin_ is now known as dciabrin | 12:25 | |
| *** jprovazn has joined #tripleo | 12:25 | |
| shardy | jaosorior: I'm thinking something like certmonger writes the certs and we calculate the checksum in the ansible workflow or something | 12:26 |
| *** tzumainn has joined #tripleo | 12:26 | |
| jaosorior | dciabrin: are HA containers also started by paunch? | 12:26 |
| *** sshnaidm is now known as sshnaidm|afk | 12:26 | |
| shardy | jaosorior: no they're managed by pacemaker | 12:26 |
| dciabrin | jaosorior, shardy was faster than me :) | 12:26 |
| jaosorior | shardy, dciabrin: Would it be REALLY problematic to do a "docker restart" on a container ran by pacemaker? | 12:26 |
| dciabrin | jaosorior, short answer: yes | 12:27 |
| shardy | lol :) | 12:27 |
| jaosorior | long answer? | 12:27 |
| dciabrin | jaosorior, for haproxy? | 12:27 |
| jaosorior | for example | 12:27 |
| dciabrin | jaosorior, so for haproxy we may have a means of reloading config on sighup | 12:27 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common master: Fix destination path when pushing puppet-elements https://review.openstack.org/494517 | 12:27 |
| jaosorior | dciabrin: right, though we will need something similar for mysql and rabbitmq | 12:28 |
| dciabrin | for the other services, if you kill a container under pacemaker control, you never know when the next pacemaker probe will be, so it might compete/race against docker restart and cause troubles | 12:28 |
| *** limao has joined #tripleo | 12:28 | |
| owalsh | shardy: kindof related to restarting containers... how/do we control start order when a node is rebooted? | 12:28 |
| *** bfournie has quit IRC | 12:28 | |
| *** bfournie has joined #tripleo | 12:29 | |
| *** limao has quit IRC | 12:29 | |
| *** ff is now known as flaper87282809 | 12:29 | |
| *** flaper87282809 is now known as fl7134a92per87 | 12:29 | |
| *** fl7134a92per87 is now known as ff | 12:29 | |
| jaosorior | dciabrin, shardy: OK, so there's two issues: 1. restarting non-pacemaker containers with certmonger. 2. restarting pacemaker containers with certmonger | 12:29 |
| jaosorior | first lets figure out #1 | 12:30 |
| *** sbrzozow has joined #tripleo | 12:30 | |
| owalsh | shardy: do we let them all start concurrently and fail/resart until all deps are up? | 12:30 |
| shardy | owalsh: yeah currently we don't - I expected services when configured should tolerate coming up in any order, but if that proves incorrect we'll have to not start them on boot and re-run the ansible steps every reboot | 12:30 |
| bogdando | dciabrin, jaosorior, shardy: it seems like we have an architecture challenge for run-time notifications of the services in containers | 12:30 |
| dciabrin | jaosorior, agreed, #2 is more involved, and we must guarantee availability | 12:31 |
| *** jcoufal has joined #tripleo | 12:31 | |
| shardy | owalsh: some more eyes/testing on that would be very helpful | 12:31 |
| bogdando | logrotate should do that, monitoring/helth checks should do that, certs updates as well, name it | 12:31 |
| jaosorior | shardy: certmonger writes the certificate in the first puppet step. you're thinking about adding an ansible task in-between that that will write the hash? | 12:31 |
| owalsh | shardy: looks like we might have a problem https://bugzilla.redhat.com/show_bug.cgi?id=1473111#c10 | 12:31 |
| openstack | bugzilla.redhat.com bug 1473111 in openstack-tripleo-heat-templates "openstack-nova: After rebooting compute: state of nova-compute service is down" [High,Assigned] - Assigned to imain | 12:31 |
| shardy | jaosorior: yes, or make docker-puppet do it like it already does for the config | 12:31 |
| bogdando | how come we have no a use case for it from the earlier releases? basically this applies to non containerized services as well | 12:32 |
| owalsh | shardy: trying to reproduce it now | 12:32 |
| oidgar | owalsh: I wonder why dbsync takes so much time anyway, isn't it mainly creates and updates table schemas (and probably indexes)? there isn't any data in the db at deployment time... | 12:33 |
| jaosorior | bogdando: what do you mean? | 12:33 |
| *** bfournie has quit IRC | 12:33 | |
| jaosorior | bogdando: aah, well, certs are monitored with certmonger. Thing is, we need to tell certmonger how to restart or reload a service once the certificate has been renewed (which is something that certmonger does) | 12:34 |
| mcornea | shardy: hey o/ I noticed that some new deprecated_param were added in roles_data.yaml and I was wondering how should we handle them in case of upgrading a composable roles environment? Do they need to be added manually to the custom roles_data.yaml? | 12:34 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: [WIP] Build a list of playbooks out of CephAnsibleUpgradePlaybook https://review.openstack.org/494521 | 12:34 |
| jaosorior | bogdando: and it's already working with non-containerized services. telling it how to restart containers is the problem right now. Since some containers share the certificate (like those whose services run over httpd) | 12:34 |
| owalsh | oidgar: it's normally doesn't on an SSD... 30s IIRC, but the first nova migration is big and not very efficient AFAIK | 12:35 |
| jaosorior | shardy: What runs first? docker-puppet or regular puppet? | 12:36 |
| *** rlandy has joined #tripleo | 12:36 | |
| *** jpena is now known as jpena|lunch | 12:36 | |
| owalsh | oidgar: creating tables may not be that fast on the DB side either | 12:36 |
| bogdando | jaosorior: how does it work for non containerized services? | 12:37 |
| oidgar | owalsh: not very efficient can explain it. alter empty table shouldn't take so long. maybe MySql is not efficient also... maybe one day I'll understand what happens there :) | 12:38 |
| bogdando | jaosorior: I mean how those get restarted on certs updates in <Pike? | 12:38 |
| owalsh | oidgar: there were some effort to optimized the nova migrations but it's not very high priority | 12:38 |
| *** aditya_r has joined #tripleo | 12:39 | |
| shardy | jaosorior: check deploy-steps-tasks.yaml, regular puppet runs first | 12:39 |
| jaosorior | bogdando: certmonger has a postsave command option that you pass. I merely give "systemctl reload httpd" for the certificates used by httpd. | 12:39 |
| jaosorior | bogdando: systemctl reload haproxy, for haproxy, and so on. | 12:40 |
| jaosorior | bogdando: https://github.com/openstack/puppet-tripleo/blob/master/manifests/certmonger/httpd.pp#L59-L72 | 12:41 |
| bogdando | jaosorior: well, I think the trick like this https://review.openstack.org/#/c/490048/7/templates/logrotate/containers_logrotate.conf.erb@9 may help to reload for certs changes as well | 12:41 |
| jaosorior | bogdando: note that we don't support TLS everywhere <Pike | 12:41 |
| *** fultonj has joined #tripleo | 12:43 | |
| *** lucas-hungry is now known as lucasagomes | 12:44 | |
| jaosorior | bogdando: I don't fully understand what you're suggesting. Can you ellaborate? | 12:44 |
| bogdando | jaosorior: a custom postsave command based on lsof outputs | 12:44 |
| bogdando | jaosorior: f.e. if old certificates will get unlinked, but remaining opened by services, the command I linked would work | 12:45 |
| bogdando | just like the logfiles | 12:45 |
| bogdando | here we should think of rotated logs == renewed cert files | 12:46 |
| *** lblanchard has joined #tripleo | 12:46 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart master: Set undercloud_docker_registry_mirror for all container jobs https://review.openstack.org/494525 | 12:47 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy kubernetes using TripleO on the overcloud https://review.openstack.org/471759 | 12:47 |
| ff | bogdando: hacked it there ^ | 12:47 |
| jaosorior | bogdando: that sounds like it could work | 12:47 |
| *** catintheroof has joined #tripleo | 12:49 | |
| bogdando | jaosorior: and automagically works for pacemaker | 12:49 |
| bogdando | unless services support sighup ofc | 12:49 |
| bogdando | w/o a silent death or so | 12:50 |
| bogdando | but even if stopped on reload, they will come back the next monitoring event | 12:50 |
| *** catintheroof has quit IRC | 12:50 | |
| *** catintheroof has joined #tripleo | 12:50 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:51 | |
| jaosorior | dciabrin: what do you think of that? ^^ | 12:51 |
| * dciabrin reads backlog | 12:51 | |
| *** eck` is now known as eck`gone | 12:52 | |
| pabelanger | bogdando: no, that issue should be fixed. Are you still seeing jobs fail because of it? The fix landed 12hours ago | 12:53 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Add Neutron SR-IOV agent container https://review.openstack.org/469066 | 12:53 |
| bogdando | pabelanger: can't answer that, no data but that e-r check gives | 12:53 |
| bogdando | and I was looking for the specifc job log EmilienM mentioned openstack-dev | 12:54 |
| pabelanger | bogdando: http://status.openstack.org/elastic-recheck/#1674681 shows the outage from yesterday | 12:55 |
| pabelanger | haven't see any new issues after fix was merged | 12:55 |
| pabelanger | see https://review.openstack.org/494265/ | 12:55 |
| sshnaidm | pabelanger, hi, I have a few question about dns bug.. | 12:56 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: Add options to use local DLRN and CentOS mirrors https://review.openstack.org/494262 | 12:56 |
| sshnaidm | pabelanger, firstly, what dns do you use in infra? | 12:56 |
| *** eck`gone is now known as eck` | 12:57 | |
| sshnaidm | pabelanger, do you use 8.8.8.8 too or something else? | 12:57 |
| mwhahaha | sshnaidm: all the nodes have caching locally, so you should use 127.0.0.1 | 12:57 |
| *** gvrangan_odl has quit IRC | 12:57 | |
| mwhahaha | sshnaidm: that's the unbound thing that's running on the nodes | 12:57 |
| sshnaidm | mwhahaha, does it mean they are built with cache already on them..? | 12:57 |
| mwhahaha | sshnaidm: yes | 12:57 |
| dciabrin | bogdando, jaosorior I see several concerns here, but baring technicalities I think we still have 1) can we guarantee that certs are updated _before_ they expire, otherwise service gets disrupted, and 2) as soon as we have a means to do 1) how would we implement a rolling reload, to avoid service disruption. i.e "I don't want to rebootstrap galera everytime certificates expire on all controllers." | 12:58 |
| pabelanger | right, we setup a forward-zone in unbound on all images: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/finalise.d/89-unbound#n29 | 12:58 |
| sshnaidm | mwhahaha, and in case it doesn't have the address, how does it resolve? what is forwarder? | 12:58 |
| sshnaidm | pabelanger, ^^ | 12:58 |
| *** jmelvin has joined #tripleo | 12:58 | |
| mwhahaha | sshnaidm: the nodepool static nameservers, see previous comment by pabelanger | 12:58 |
| pabelanger | it will always have an address | 12:59 |
| openstackgerrit | Feodor Tersin proposed openstack/os-net-config master: This patch adds initial support for the Contrai vRouter interface https://review.openstack.org/492492 | 12:59 |
| pabelanger | today we use both nodepool elements and glean to ensure DNS is properly configured on nodes | 12:59 |
| sshnaidm | pabelanger, you can't cache the whole internet though, which nodepool nameservers do you use in infra? | 12:59 |
| mwhahaha | sshnaidm: it doesn't matter | 12:59 |
| mwhahaha | sshnaidm: it's the local nameservers | 12:59 |
| mwhahaha | sshnaidm: this is not our concern, basically we need to configure our stuff to use the local systems | 13:00 |
| pabelanger | sshnaidm: any DNS request made, will cache into unbound. If unbound doesn't know the DNS, it will forward to google or opendns | 13:00 |
| mwhahaha | it's a continuation of the whole mirror stuff. we need to stop using global things | 13:00 |
| pabelanger | the issue here, is every request is hitting google, and likely getting rate limited | 13:00 |
| mwhahaha | it's a ci config, we just need to specify 127.0.0.1 | 13:00 |
| bogdando | pabelanger: ack, thanks | 13:01 |
| sshnaidm | mwhahaha, I try to understand where and how the unbound came up | 13:02 |
| *** dhill_ has joined #tripleo | 13:02 | |
| pabelanger | not sure I understand? Are you wanting to know more about how we manage unbound services? | 13:03 |
| sshnaidm | pabelanger, how *we* manage unbound services :) | 13:04 |
| pabelanger | sshnaidm: well, you don't need to manage unbound because openstack-infra does that today. The images you get for centos-7 have that already configured: see http://nb03.openstack.org/dib.centos-7.log | 13:05 |
| *** bfournie has joined #tripleo | 13:05 | |
| pabelanger | there is still a part that I am not sure, but anything launched on the overcloud, out side of nodepool nodes should also not be using google DNS | 13:07 |
| pabelanger | I am unsure how that DNS is configured ATM | 13:07 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates master: Deploy Mistral with Keystone v3 options (authtoken) https://review.openstack.org/461040 | 13:07 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: Use local mirrors for multinode during DLRN build https://review.openstack.org/494537 | 13:08 |
| sshnaidm | pabelanger, just curious, do you use unbound in devstack too? | 13:08 |
| pabelanger | sshnaidm: yes, all jobs use it today | 13:09 |
| sshnaidm | pabelanger, and also for my curiosity, why not dnsmasq that we have already installed, but some weird server? | 13:09 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 13:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 13:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 13:10 |
| sshnaidm | pabelanger, because dnsmasq is actually dns caching server too.. | 13:12 |
| openstackgerrit | Attila Darazs proposed openstack/puppet-tripleo master: GATE TEST: do not merge https://review.openstack.org/494538 | 13:12 |
| pabelanger | sshnaidm: I think for recursive queries, unbound supports that | 13:13 |
| pabelanger | also, so far unbound has just worked across all distros | 13:13 |
| sshnaidm | pabelanger, dnsmasq too | 13:14 |
| *** udesale has joined #tripleo | 13:16 | |
| pabelanger | sure, but I think it was because of recursive queries. | 13:16 |
| *** pdeore_ has quit IRC | 13:16 | |
| pabelanger | But info should be in our commit messages also | 13:16 |
| pabelanger | I am sure personal preference also came into play at some point | 13:17 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/puppet-unbound/commit/?id=e5e832f5c0f985f546fb2e695ef239421b60c822 | 13:17 |
| *** mdnadeem has quit IRC | 13:18 | |
| *** ansmith has joined #tripleo | 13:19 | |
| *** michapma has joined #tripleo | 13:20 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Switch to overcloudrc.v3 for running tempest https://review.openstack.org/493030 | 13:20 |
| *** gbarros has joined #tripleo | 13:20 | |
| oidgar | shardy,owalsh: deployment worked with low-memory-usage.yaml :) | 13:21 |
| oidgar | shardy, owalsh: in addition, I ran iostat and the highest write latency was 750ms... which is really bad... | 13:22 |
| owalsh | oidgar: :-) 750ms ouch... | 13:23 |
| shardy | oidgar: good news, and ouch! :) | 13:23 |
| *** hewbrocca_afk is now known as hewbrocca | 13:23 | |
| oidgar | shardy,owalsh: but I think there's another thing which can cause this slowlyness, I'm using OVB to create the baremetal nodes (compute & controller) | 13:23 |
| *** skramaja has quit IRC | 13:24 | |
| EmilienM | hello | 13:24 |
| oidgar | shardy,owalsh: as I see, it does not create any volume but relies on the image directly. correct me if I'm wrong but in this situation the virtual disk resides on the hypervisor's local drive | 13:24 |
| oidgar | shardy,owalsh: so my overcloud might not reside on ceph storage... but I'm not sure. | 13:25 |
| *** bogdando has quit IRC | 13:25 | |
| oidgar | shardy,owalsh: btw, iostat result in case you are interested: http://paste.openstack.org/show/618682/ | 13:26 |
| *** pdeore has joined #tripleo | 13:27 | |
| *** jprovazn has quit IRC | 13:28 | |
| *** jprovazn has joined #tripleo | 13:28 | |
| *** bswartz has joined #tripleo | 13:29 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Introduced extra_tempest_config flag for tempest.conf https://review.openstack.org/494394 | 13:29 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Set undercloud nameserver to localhost 127.0.0.1 https://review.openstack.org/494545 | 13:30 |
| openstackgerrit | Martin Kopec proposed openstack/tripleo-quickstart-extras master: Allow removing of options from tempest conf https://review.openstack.org/477079 | 13:31 |
| bnemec | oidgar: Are you on RDO cloud by any chance? | 13:32 |
| oidgar | bnemec: yup | 13:33 |
| bnemec | oidgar: Yeah, extremely slow ephemeral storage is a known problem there. | 13:33 |
| chandankumar | sshnaidm: EmilienM need your views on hti spatch https://review.openstack.org/#/c/491189/ | 13:34 |
| bnemec | oidgar: I'm hoping to push a change to OVB today that will allow boot-from-volume, which seems to help a lot. | 13:34 |
| oidgar | bnemec: wow that's sounds great! | 13:34 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui master: Make validation groups labels clickable https://review.openstack.org/494513 | 13:34 |
| oidgar | bnemec: Is there anything I can help with regarding OVB? are you one of the developers of this? | 13:35 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Provide sample environment for composable roles https://review.openstack.org/487226 | 13:35 |
| bnemec | oidgar: I'm the primary developer on OVB. The patch is working for me locally, but I need to test a lot of use cases to make sure I haven't broken anything before I push it live. | 13:36 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Provide sample environment for composable roles https://review.openstack.org/487226 | 13:37 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Provide sample environment for composable roles https://review.openstack.org/487226 | 13:38 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Add logrotate-crond configuration https://review.openstack.org/490048 | 13:38 |
| sshnaidm | chandankumar, we have it here: https://github.com/openstack/tripleo-quickstart-extras/blob/317b6409344e749fe3b797022160271e9454ef81/roles/validate-tempest/tasks/tempest-status.yml#L2-L5 | 13:39 |
| chandankumar | sshnaidm: it is based on tempest_result.rc which comes from https://github.com/openstack/tripleo-quickstart-extras/blob/317b6409344e749fe3b797022160271e9454ef81/roles/validate-tempest/tasks/run-tempest.yml#L5 | 13:40 |
| *** amoralej is now known as amoralej|lunch | 13:40 | |
| sshnaidm | bnemec, hi, do you plan to put OVB to openstack repos? | 13:40 |
| pabelanger | okay, so we should get 11 merges incoming | 13:40 |
| pabelanger | and gate pipeline should be back under 6 hrs | 13:41 |
| pabelanger | down from 19h4mins | 13:41 |
| sshnaidm | chandankumar, yep, so what is problem with this? | 13:41 |
| adarazs | woo, results. :) | 13:41 |
| chandankumar | sshnaidm: we wanted to know tempest status it means tempest tests run is successfull, i was not able to understand how it is used to determine ststaus https://github.com/openstack/tripleo-quickstart-extras/blob/317b6409344e749fe3b797022160271e9454ef81/roles/validate-tempest/tasks/run-tempest.yml#L5 | 13:41 |
| dalvarez | guys noob question here, i've written some patches for puppet-tripleo, puppet-neutron and tripleo-heat-templates. With a script I clone puppet-neutron and puppet-tripleo repos, apply my patches and copy the code to the overcloud image at /usr/share | 13:42 |
| dalvarez | When i deploy the oc, i'm getting this error: | 13:42 |
| dalvarez | Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Rabbitmq]: has no parameter named 'ipv6' at /etc/puppet/modules/tripleo/manifests/profile/base/rabbitmq.pp:114:7 on node overcloud-controller-0.localdomain | 13:42 |
| *** jpena|lunch is now known as jpena | 13:42 | |
| dalvarez | what could I be missing? what could I do to debug it? | 13:42 |
| dalvarez | thanks a lot :) | 13:43 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: upload containers to undercloud in upgrade scenario https://review.openstack.org/493972 | 13:43 |
| sshnaidm | chandankumar, when testr was running it was exiting with failure status code if some of tests failed | 13:43 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: README: Fix CI coverage layout https://review.openstack.org/494279 | 13:43 |
| adarazs | sshnaidm, pabelanger: hmm, so I'm trying to grab /etc/yum.repos.d/CentOS-Base.repo from the undercloud slave when I build DLRN packages to get the local mirrors, and I just noticed all the relevant CentOS repos are enabled=0, any idea why? | 13:43 |
| adarazs | sshnaidm, pabelanger: e.g. http://logs.openstack.org/38/494538/1/check/gate-tripleo-ci-centos-7-undercloud-oooq/88253c0/logs/delorean_logs/19/d5/19d5c61889bfb70744c0a83fc744a95a48823ca9_dev/rpmbuild.log.txt.gz | 13:43 |
| sshnaidm | chandankumar, it was last command that was executed in run_tempest.sh, so its code was the last | 13:43 |
| jprovazn | vkmc: hi | 13:44 |
| openstackgerrit | Merged openstack/puppet-tripleo master: HAProxy: Set listen options for internal services too https://review.openstack.org/491832 | 13:44 |
| pabelanger | adarazs: I think that is a bug in quickstart, there is likley a centos-base.repo too | 13:44 |
| sshnaidm | adarazs, yep, it was recently patch that left all repos in place, but disabled them | 13:44 |
| pabelanger | adarazs: note the case | 13:44 |
| sshnaidm | adarazs, you need the repos with lowcase | 13:44 |
| fultonj | jistr: would you mind reviewing https://review.openstack.org/#/c/479426 ? | 13:44 |
| adarazs | sshnaidm, pabelanger: hmm, thanks :) | 13:45 |
| sshnaidm | pabelanger, it's not an issue with quickstart | 13:45 |
| vkmc | jprovazn, o/ | 13:45 |
| jprovazn | vkmc: some good news, after moving around backend config from pacemaker manifest to base, manila-share container deploys fine (even with initialized driver) | 13:46 |
| pabelanger | sshnaidm: issue no, but optimization sure: http://logs.openstack.org/59/493259/10/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e8e97e5/logs/dstat-csv_log.txt.gz don't think you need duplicate .repo files with 1 enable and the other disabled | 13:46 |
| sshnaidm | adarazs, pabelanger https://github.com/openstack/tripleo-quickstart/commit/84e0168e49fce2f3d79559866ebea8f7050eef51 | 13:46 |
| jprovazn | vkmc: how is your overcloud doing? | 13:47 |
| michapma | trown: is collect-logs only run on the undercloud? I don't see anything for /var/log on subnode-2 in a multinode job | 13:47 |
| vkmc | jprovazn, \o/ | 13:47 |
| chandankumar | sshnaidm: os bascally last command is tempest test run (ostestr) or tempest cleanup from run_tempest.sh script if it fails their exit status will be returned , if inderstand correctly | 13:47 |
| sshnaidm | pabelanger, hmm.. what does this dstat show? | 13:47 |
| vkmc | jprovazn, got the same feedback from bandini and dciabrin :) | 13:47 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Create separate resource for HAProxy horizon endpoint https://review.openstack.org/491437 | 13:47 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Enable TLS in the internal network for horizon https://review.openstack.org/489593 | 13:47 |
| vkmc | jprovazn, thanks for all the help | 13:47 |
| pabelanger | sshnaidm: sorry, wrong link: http://logs.openstack.org/38/494538/1/check/gate-tripleo-ci-centos-7-undercloud-oooq/88253c0/logs/undercloud/etc/yum.repos.d/ | 13:47 |
| trown | michapma: it should run on every node in the inventory | 13:47 |
| vkmc | jprovazn, in my env... I cannot deploy the overcloud... it's failing in phase 1 | 13:48 |
| sshnaidm | chandankumar, no sure I understood.. | 13:48 |
| jprovazn | vkmc: great, I read michele's reply - merging your patch and then iterate on fixes makes sense | 13:48 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable listening on TLS for the internal network for horizon https://review.openstack.org/489596 | 13:48 |
| jprovazn | vkmc: what error are you getting? | 13:48 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart master: Use <release> for docker namespace in rdoproject.org https://review.openstack.org/494555 | 13:48 |
| sshnaidm | pabelanger, I'm not sure why it was done so, maybe mwhahaha can explain | 13:49 |
| bandini | vkmc: o/ do ping if you're stuck on unrelated issues. chances are we hit those recently as well | 13:49 |
| *** b3nt_pin is now known as beagles | 13:49 | |
| vkmc | jprovazn, bandini, http://paste.openstack.org/ | 13:49 |
| vkmc | http://paste.openstack.org/show/618686/ ^ | 13:50 |
| sshnaidm | chandankumar, testr was last command that ran in run_tempest.sh, so if run_tempest.sh has status code != 0, it meant that tempest failed. That's like it worked | 13:50 |
| chandankumar | sshnaidm: last command would be https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/run-tempest.sh.j2#L31 or https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/run-tempest.sh.j2#L40 and if any tests fails their exist status determine the tempest status that i wanted to say what i understand from above discussion | 13:50 |
| chandankumar | on https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/tasks/run-tempest.yml#L6 | 13:50 |
| chandankumar | sshnaidm: but there is longer testr command, we are using wrappers | 13:51 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: Add options to use local DLRN and CentOS mirrors https://review.openstack.org/494262 | 13:51 |
| sshnaidm | chandankumar, any command you run - ostestr, whatever | 13:51 |
| *** pdeore has quit IRC | 13:51 | |
| jprovazn | vkmc: it looks like there are not enough available hosts, what is "ironic node-list" output? | 13:51 |
| sshnaidm | chandankumar, but I see last command now is 'cleanup --dry-run' - what is it for? | 13:51 |
| vkmc | jprovazn, yeah, I'm not entirely sure if I should kill the hosts and restart | 13:51 |
| vkmc | or how should I proceed | 13:52 |
| bandini | vkmc: so I usually spend like 5 minutes to try and see if I can beat ironic/nova into submission after which I typically reprovision the undercloud :/ | 13:52 |
| bandini | unless I have loads of time at hand that's the fastest route for me | 13:52 |
| bandini | not ideal though | 13:52 |
| chandankumar | sshnaidm: if tests fails, tempest cleanup --dry-run stores the list of resources created or leftover due to faild tests which will be deleted by suing tempest cleanup | 13:53 |
| jprovazn | vkmc: so there are not existing running vms, right? | 13:53 |
| vkmc | bandini, I see... I'm not sure how I reached that state | 13:53 |
| vkmc | jprovazn, http://paste.openstack.org/show/618688/ | 13:53 |
| michapma | trown: it should be under here, right? http://logs.openstack.org/05/486905/23/experimental/gate-tripleo-ci-centos-7-scenario008-multinode-oooq-nv/e98301a/logs/subnode-2/ | 13:54 |
| sshnaidm | chandankumar, so if tests fail, how does it run with "-eux"..? | 13:54 |
| openstackgerrit | Attila Darazs proposed openstack/puppet-tripleo master: GATE TEST: do not merge https://review.openstack.org/494538 | 13:54 |
| michapma | trown: oh there was an error in collect logs, never mind I'll figure it out | 13:54 |
| *** itlinux has joined #tripleo | 13:55 | |
| jprovazn | vkmc: interesting, nodes look fine, not sure why these are not exposed to nova | 13:55 |
| * jprovazn relocates | 13:56 | |
| *** jprovazn has quit IRC | 13:56 | |
| vkmc | jprovazn, yeah... it's kinda odd | 13:56 |
| *** ramishra has quit IRC | 13:56 | |
| jistr | fultonj: it's merging | 13:56 |
| *** ykarel has quit IRC | 13:56 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 13:56 |
| fultonj | jistr: thanks | 13:57 |
| sshnaidm | chandankumar, it doesn't seem to run: http://logs.openstack.org/85/494085/2/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha-tempest-oooq/cd33137/logs/undercloud/home/jenkins/tempest_output.log.txt.gz | 13:57 |
| mwhahaha | adarazs: the repos are quickstart-* | 13:57 |
| *** ramishra has joined #tripleo | 13:57 | |
| mwhahaha | adarazs: we disable the existing ones on the images because they don't reference the mirrors and quickstart isn't updating the existing files so it creates new | 13:57 |
| *** jlabarre has quit IRC | 13:57 | |
| mwhahaha | adarazs: there was a change yesterday to rename them to quickstart-* | 13:58 |
| *** oidgar has quit IRC | 13:58 | |
| gfidente | jistr good news is https://review.openstack.org/#/c/479288/ | 13:58 |
| *** MVenesio has joined #tripleo | 13:58 | |
| adarazs | mwhahaha: I still see just lowercase centos-*: http://logs.openstack.org/38/494538/1/check/gate-tripleo-ci-centos-7-undercloud-oooq/88253c0/logs/undercloud/etc/yum.repos.d/ | 13:58 |
| gfidente | jistr so we can add mds to scenario004 and get that tested too, thanks! | 13:58 |
| adarazs | mwhahaha: which change alters them? | 13:58 |
| mwhahaha | adarazs: it might still be in the gate | 13:58 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 13:59 |
| mwhahaha | adarazs: was supposed to be https://review.openstack.org/#/c/494056/ gues sit failed and needs a recheck | 13:59 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 14:00 |
| *** jlabarre has joined #tripleo | 14:00 | |
| EmilienM | chandankumar: ok | 14:01 |
| mwhahaha | adarazs: so until that patch lands, the lower case centos-* ones are the ones that are actually enabled CI | 14:01 |
| adarazs | mwhahaha: thanks for the heads up, I will include it in the depends-on chain :) | 14:01 |
| *** mrch has quit IRC | 14:02 | |
| EmilienM | chandankumar: isn't it something you can solve with arxcruz ? | 14:02 |
| EmilienM | chandankumar: I don't understand the last comment yet | 14:03 |
| * arxcruz reading | 14:03 | |
| jistr | vkmc: i had a very similar issue some time ago, in my case it was some problem with libvirt on the host | 14:04 |
| jistr | if i run on undercloud | 14:04 |
| jistr | (undercloud) [stack@undercloud ~]$ sudo vbmc show compute_0 | 14:04 |
| jistr | it prints among other things | 14:05 |
| pabelanger | EmilienM: weshay: mwhahaha: who do you recommend to change http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/build-test-packages/tasks/dlrn-build.yml#n27 to use http://mirror.ord.rax.openstack.org:8080/rdo/rdoinfo/ now | 14:05 |
| jistr | libvirt_uri | qemu+ssh://stack@192.168.23.1/session?socket=/run/user/1003/libvirt/libvirt-sock&keyfile=/root/.ssh/id_rsa_virt_power&no_verify=1&no_tty=1 | 14:05 |
| pabelanger | EmilienM: weshay: mwhahaha: so we remove the dependency of github.com | 14:05 |
| jistr | vkmc: the problem i had was that /run/user/1003/libvirt didn't exist at all on the bare metal host, so there was no libvirt socket to connect to | 14:05 |
| EmilienM | pabelanger: I think adarazs and amoralej|lunch are on it | 14:05 |
| jistr | vkmc: i had to reinstall... | 14:05 |
| vkmc | jistr, let me check that | 14:06 |
| weshay | pabelanger, adarazs | 14:06 |
| adarazs | yeah. | 14:06 |
| pabelanger | EmilienM: weshay: adarazs: okay thanks, do you want a new bug or just update bug 1710678 | 14:07 |
| openstack | bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] https://launchpad.net/bugs/1710678 - Assigned to Attila Darazs (adarazs) | 14:07 |
| EmilienM | pabelanger: update ^ I think | 14:07 |
| vkmc | jistr, in my case... (undercloud) [stack@undercloud ~]$ sudo vbmc show compute-0 | No domain with matching name compute-0 was found | 14:07 |
| vkmc | which it seems fine since I did openstack stack delete overcloud | 14:08 |
| jistr | vkmc: i think there is underscore instead of hyphen. (`sudo vbmc list` can show the right name) | 14:08 |
| vkmc | so those domains shouldn't exist | 14:08 |
| weshay | adarazs, would you prefer 1 or 2 lumps.. er.. bugs | 14:08 |
| vkmc | jistr, you are right | 14:08 |
| vkmc | http://paste.openstack.org/show/618691/ | 14:09 |
| vkmc | I thought those should be the ironic node | 14:09 |
| vkmc | s | 14:09 |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs https://review.openstack.org/494043 | 14:09 |
| adarazs | weshay, pabelanger: wait, do we have some upstream mirrors instead of these URLs? https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/defaults/main.yml#L5-L6 | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 14:10 |
| arxcruz | EmilienM: chandankumar sshnaidm I'll keep my -1, as I already discuss several times, it's working as it is, as always was, I don't see any reasons to change it. I'm always angry to have to scroll down a lot the tempest_output.log because of the ostestr -l that list all the tests for no reason, I don't walt to scroll the failures also. Plus, stackviz and tempest.html have the same information in a more | 14:10 |
| adarazs | weshay: I can change these within the same bug. | 14:10 |
| arxcruz | fashion and easy way | 14:10 |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 14:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 14:10 |
| jistr | yea i think they don't get deleted in ironic and vbmc when overcloud is deleted. They are only removed in Nova i think. | 14:10 |
| weshay | adarazs, the openstack git | 14:10 |
| weshay | adarazs, already in the bug :) | 14:10 |
| pabelanger | adarazs: weshay: no, git mirror. Just the folder on http://trunk.rdoproject.org. | 14:11 |
| sshnaidm | arxcruz, chandankumar but please fix the bug with "dry cleanup" that doesn't run if tempest failed (as it has "-eux") | 14:11 |
| pabelanger | adarazs: weshay: long term, RDO should be producing pip packages on their post pipeline and storing them some place, ideally trunk.rdoproject.org, then you could pip install from the remove URL. For now, I'd like to stop git cloning that info | 14:12 |
| *** apetrich_ has joined #tripleo | 14:12 | |
| arxcruz | sshnaidm: as far as I can say, it's working properly, if there are tests failing, the job will return as failed, as expected | 14:12 |
| *** trown is now known as trown|brb | 14:12 | |
| owalsh | vkmc: what does ironic node-list show? | 14:13 |
| sshnaidm | arxcruz, yeah, but I talk about this line: https://github.com/openstack/tripleo-quickstart-extras/blob/b0b7f95435675446a9639eb4f69b596e4e628555/roles/validate-tempest/templates/run-tempest.sh.j2#L40 | 14:13 |
| sshnaidm | arxcruz, what is this line for? | 14:13 |
| weshay | in 15min.. tripleo-ci-squad mtg fyi https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:14 |
| adarazs | ah, yeah, forgot to ping people here because of the changes :) | 14:14 |
| adarazs | weshay: thanks | 14:14 |
| weshay | np | 14:15 |
| arxcruz | sshnaidm: I was against the patch that added this because we don't re-run tempest if it fails, except if we do it manually and want to test things, and even so, I would like to see what was left behind in order to understand where it's failing, but... | 14:15 |
| jistr | owalsh: http://paste.openstack.org/show/618688/ http://paste.openstack.org/show/618686/ | 14:15 |
| adarazs | EmilienM, trown|brb, sshnaidm, panda, rlandy, arxcruz: CI squad meeting in 15 minutes | 14:15 |
| *** hanish has quit IRC | 14:15 | |
| arxcruz | adarazs: yes sir! :) | 14:16 |
| jistr | i'm thinking if it may be https://bugs.launchpad.net/tripleo-quickstart/+bug/1650238 again | 14:16 |
| openstack | Launchpad bug 1650238 in tripleo-quickstart "Systemd freezing execution in VMs on manually running virsh list" [Low,Expired] | 14:16 |
| *** lblanchard has quit IRC | 14:16 | |
| sshnaidm | arxcruz, but anyway it won't run now if tests failed.. is it as designed? | 14:16 |
| jistr | i don't recall the error message i got exactly, but it wasn't very telling IIRC | 14:16 |
| *** trown|brb is now known as trown | 14:17 | |
| EmilienM | adarazs: ack | 14:18 |
| jistr | so in case /run/user/1000/libvirt/libvirt-sock vanished from the host, it probably needs a full OOOQ redeploy | 14:18 |
| adarazs | pabelanger: I'm not sure how to change the URL to clone it from a local server. http://mirror.ord.rax.openstack.org:8080/rdo/rdoinfo/ is not a git repo so "{{ lookup('env', 'NODEPOOL_RDO_PROXY') }}/rdoinfo" won't work. | 14:19 |
| vkmc | owalsh, http://paste.openstack.org/show/618688/ | 14:19 |
| pabelanger | adarazs: right, it might be easier if that was a tarball over extract directly. Otherwise, you'll have to recursive wget / curl that folder | 14:20 |
| *** ykarel has joined #tripleo | 14:20 | |
| adarazs | can't we mirror/clone the whole repo instead of just the data? | 14:21 |
| pabelanger | adarazs: not from github.com, it is unrealible | 14:22 |
| pabelanger | adarazs: same for DLRN, we shouldn't be git clone DLRN from github, but pip install DLRN from pypi | 14:22 |
| adarazs | hrm, not sure if we have it anywhere else. | 14:22 |
| pabelanger | we do have pypi mirrors in place | 14:22 |
| bnemec | sshnaidm: Sorry, was in a meeting. I've added formalizing OVB as a tracking item for Queens, so hopefully it will get done then. | 14:22 |
| sshnaidm | bnemec, great news, thanks | 14:23 |
| pabelanger | https://pypi.python.org/pypi/DLRN for DLRN, we jobs can use that | 14:23 |
| adarazs | pabelanger: okay, so this might be another bug actually and directed to the rdo infra people. | 14:23 |
| pabelanger | rdoinfo, isn't released ATM. So we'll need a master tarball or something | 14:23 |
| *** jlabarre has quit IRC | 14:23 | |
| owalsh | vkmc: hmm, that looks ok yea? | 14:23 |
| adarazs | hm, okay, I can change the role to use pip at least for DLRN | 14:24 |
| pabelanger | Ya, that once is easier. I'll try and get rdoinfo to be a tarball | 14:24 |
| adarazs | pabelanger: okay. | 14:24 |
| *** amoralej|lunch is now known as amoralej | 14:25 | |
| pabelanger | adarazs: what bits in rdoinfo does DLRN need? Just the yaml files or also the python bits | 14:26 |
| *** jlabarre has joined #tripleo | 14:26 | |
| owalsh | vkmc: does openstack baremetal list --unassociated --no-maintenance also return both nodes? | 14:27 |
| adarazs | pabelanger: I don't know, it's a parameter for the DLRN command so it might use whatever from it. I'm not that familiar with DLRN. | 14:29 |
| adarazs | pabelanger: maybe amoralej can tell. | 14:29 |
| amoralej | adarazs, pabelanger we used rdoinfo from rdopkg | 14:31 |
| amoralej | we need the full content as it provides methods to parse it | 14:31 |
| EmilienM | chandankumar: if you can, we're having the tripleo ci squad meeting | 14:32 |
| mrunge | EmilienM: I'd be interested in that as well | 14:32 |
| mrunge | where is it? | 14:32 |
| EmilienM | mrunge: https://bluejeans.com/u/whayutin/ | 14:34 |
| mrunge | thanks | 14:34 |
| mrunge | EmilienM: ^^ | 14:34 |
| *** cdearborn has quit IRC | 14:35 | |
| *** cdearborn has joined #tripleo | 14:36 | |
| vkmc | owalsh, same output | 14:37 |
| vkmc | looks ok | 14:38 |
| owalsh | vkmc: then I'm confused. AFAICT that's what mistral runs to check the node count - https://github.com/openstack/tripleo-common/blob/4e7554b7f9e65faa8a818e44bb8ac2cb46023529/workbooks/validations.yaml#L580 | 14:39 |
| *** janki has quit IRC | 14:41 | |
| vkmc | owalsh, I'm too yes... | 14:42 |
| pabelanger | amoralej: amoralej: okay, thanks. So we need to either have RDO release that to pypi, which we mirror, or create a python-tarball on trunk.project.org, which we also mirror. | 14:42 |
| *** itlinux has quit IRC | 14:43 | |
| amoralej | pabelanger, the plan is to separate data from python code, i think jruzicka will work on that | 14:44 |
| amoralej | then it'll be easy to push python module to pypi | 14:44 |
| *** jbadiapa_ has quit IRC | 14:45 | |
| owalsh | vkmc: had you just deleted the stack? maybe there is a slight delay in the nodes going back to available | 14:45 |
| *** itlinux has joined #tripleo | 14:46 | |
| pabelanger | amoralej: I am unsure, but if you could just fetch the yaml from one place then pip install other, that seems like a good plan | 14:46 |
| amoralej | pabelanger, yes, that's the plan | 14:46 |
| amoralej | but that needs some work | 14:46 |
| amoralej | so far we need to pull it together | 14:46 |
| pabelanger | amoralej: okay. So I'll push #rdo to atleast create a tarball of rdoinfo and see if they can publish that in to trunk.rdoproject.org | 14:47 |
| vkmc | owalsh, no no... I've deleted the stack yesterday | 14:50 |
| vkmc | owalsh, and I keep hitting that | 14:50 |
| owalsh | vkmc: err... I'm out of ideas | 14:51 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add Collector service to roles_data https://review.openstack.org/494589 | 14:52 |
| *** ramishra has quit IRC | 14:52 | |
| *** ramishra has joined #tripleo | 14:52 | |
| *** ykarel has quit IRC | 14:53 | |
| EmilienM | mwhahaha, shardy : hey - just talked with release folks - if we tag RC1 this week, we *have* to branch stable/pike - thing we don't want I think, since we still have a bunch of things to land IMHO | 14:54 |
| EmilienM | deadline for RC is R+O Aug 28 - Sep 01 | 14:54 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: Add options to use local DLRN and CentOS mirrors https://review.openstack.org/494262 | 14:54 |
| EmilienM | I think we should postpone RC1 to R+0 | 14:54 |
| EmilienM | so we can continue to improve CI and land stuffs we really need | 14:54 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Add converge step to the container upgrade https://review.openstack.org/494590 | 14:55 |
| mwhahaha | EmilienM: k | 14:55 |
| EmilienM | as well as improving testing coverage | 14:55 |
| EmilienM | and eventually find bugs / fix them | 14:55 |
| EmilienM | shardy: does it work for you as well? | 14:55 |
| owalsh | vkmc: I think it must be releated to mistral though, so maybe restart the mistral services | 14:55 |
| d0ugal | owalsh, vkmc - what's up? | 14:55 |
| openstackgerrit | Jiri Stransky proposed openstack/puppet-tripleo master: Allow configuring multiple insecure registries https://review.openstack.org/492612 | 14:55 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates master: Gate test - do not merged https://review.openstack.org/494593 | 14:56 |
| *** gbarros has quit IRC | 14:57 | |
| owalsh | d0ugal: vkmc is hitting http://paste.openstack.org/show/618686/ | 14:57 |
| shardy | EmilienM: +1, particularly given the gate issues I think deferring is a good idea | 14:58 |
| owalsh | d0ugal: but CLI shows 2 nodes available and no associated | 14:58 |
| vkmc | d0ugal, o/ | 14:58 |
| EmilienM | shardy, mwhahaha: ack. Update sent on ML. | 14:58 |
| d0ugal | owalsh, vkmc - that error seems to come from here: https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/validations.py#L457-L461 | 14:59 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add Ceilometer API and Collector service to roles_data https://review.openstack.org/494589 | 14:59 |
| *** jprovazn has joined #tripleo | 15:00 | |
| owalsh | d0ugal: yea, which get's the counts from https://github.com/openstack/tripleo-common/blob/4e7554b7f9e65faa8a818e44bb8ac2cb46023529/workbooks/validations.yaml#L580 | 15:00 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Provide sample environment for composable roles https://review.openstack.org/487226 | 15:00 |
| *** udesale has quit IRC | 15:00 | |
| d0ugal | owalsh: right | 15:00 |
| *** gbarros has joined #tripleo | 15:00 | |
| *** paramite has quit IRC | 15:00 | |
| owalsh | d0ugal: but when vkmc runs openstack baremetal list --unassociated --no-maintenance it reports 2 nodes | 15:01 |
| d0ugal | vkmc: What does this give you? mistral run-action ironic.node_list '{"provision_state":"available", "associated": false, "maintenace": false}' | 15:01 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Write user parameters environment to swift https://review.openstack.org/450264 | 15:01 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Write rhel registration parameters env to swift https://review.openstack.org/450708 | 15:01 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Write breakpoint cleanup env to swift https://review.openstack.org/450709 | 15:01 |
| d0ugal | vkmc: but with maintenance typed correctly :) | 15:02 |
| d0ugal | brb | 15:02 |
| *** dsariel has quit IRC | 15:03 | |
| *** ff is now known as flaper87 | 15:04 | |
| *** flaper87 has quit IRC | 15:04 | |
| *** flaper87 has joined #tripleo | 15:04 | |
| shardy | matbu: Hey were you planning to push another revision of https://review.openstack.org/#/c/485732/ or should I do it? | 15:04 |
| *** aditya_r has quit IRC | 15:04 | |
| vkmc | d0ugal, http://paste.openstack.org/show/618693/ | 15:04 |
| *** dsariel has joined #tripleo | 15:05 | |
| d0ugal | vkmc: interesting, so there seems to be two nodes there. | 15:06 |
| vkmc | yeah... | 15:06 |
| *** oidgar has joined #tripleo | 15:06 | |
| d0ugal | vkmc: I think the issue is in the statistics variable | 15:08 |
| matbu | shardy: hm i can do it if you are busy on other stuff | 15:08 |
| vkmc | d0ugal, not entirely sure how I reached this state though | 15:09 |
| d0ugal | vkmc: mistral run-action nova.hypervisors_statistics | 15:09 |
| vkmc | d0ugal, this was a clean unercloud | 15:09 |
| shardy | matbu: ack, sure if you have time that would be great, thanks! | 15:09 |
| vkmc | undercloud* | 15:09 |
| d0ugal | vkmc: Good question! | 15:09 |
| matbu | shardy: but thx for the reminder, i totally miss it, i was on CI debuging today | 15:09 |
| matbu | shardy: yep im adding it on my todo list :) | 15:09 |
| vkmc | d0ugal, I used docker registry to deploy the overcloud... I hit DNS issues | 15:09 |
| vkmc | so I delete the overcloud, created a local registry | 15:09 |
| vkmc | redeploy | 15:10 |
| vkmc | and hit that | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 15:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 15:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 15:10 |
| vkmc | d0ugal, http://paste.openstack.org/show/618695/ | 15:10 |
| d0ugal | vkmc: so I think "count": 0 is the issue | 15:11 |
| d0ugal | vkmc: but I don't know much about this code | 15:11 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Refactor setup_docker_host.sh as host_prep_tasks https://review.openstack.org/469163 | 15:11 |
| vkmc | d0ugal, shall I report a bug? | 15:12 |
| vkmc | I don't have much info on how to reproduce | 15:12 |
| *** aufi has quit IRC | 15:12 | |
| d0ugal | vkmc: Yeah, that might be best. Maybe thrash will make more sense of this, I think he wrote or ported this code to Mistral | 15:12 |
| * thrash scrolls back | 15:13 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo master: Configure cache when setting up docker_registry https://review.openstack.org/494451 | 15:14 |
| owalsh | vkmc: should have been done when providioning but could you try running sudo nova-manage cell_v2 discover_hosts --verbose | 15:14 |
| jistr | vkmc: btw could you please check if /run/user/1000/libvirt/libvirt-sock exists on the host? I'm quite interested if this is related to the libvirt session dissapearing sometimes, might be good to prove/disprove that. | 15:15 |
| vkmc | owalsh, http://paste.openstack.org/show/618696/ | 15:15 |
| *** yprokule has quit IRC | 15:15 | |
| vkmc | jistr, so... I deployed the undercloud with tripleo-quickstart... I should check if that exist on the undercloud, right? or in the virthost? | 15:16 |
| sshnaidm | EmilienM, about downloading qemu-img after a repo setup, I think it's ready for review/merge: https://review.openstack.org/#/c/494233/ https://review.openstack.org/#/c/494235/ | 15:16 |
| jistr | vkmc: virthost | 15:16 |
| *** dsariel has quit IRC | 15:16 | |
| *** ccamacho has quit IRC | 15:16 | |
| EmilienM | sshnaidm: ok thx | 15:17 |
| vkmc | jistr, it does exist! | 15:17 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Remove duplicate parameter_defaults section https://review.openstack.org/494603 | 15:17 |
| jistr | vkmc: ok that part is good then. Thanks for checking :) | 15:18 |
| marios | EmilienM: bnemec o/ folks wdyt about https://review.openstack.org/#/c/474578/ - i think bnemec was ready to +2 but wondering about ffe. I thought it is covered "because upgrades" . though not workflow per se, but validations. there is some risk, but mitigated by the fact that even when validation fails, it won't fail the upgrade (and they run at the end of the upgrade) | 15:19 |
| owalsh | vkmc: hrmm, discover hosts showed that one node wasn't mapped. What is the count in nova hypervisor-stats now? | 15:19 |
| vkmc | jistr, it's good because we are not hitting the same... it's bad because we are now hitting some unknown bug | 15:19 |
| jistr | yea :) | 15:19 |
| vkmc | :/ | 15:19 |
| vkmc | owalsh, all 0 | 15:20 |
| vkmc | owalsh, http://paste.openstack.org/show/618698/ | 15:20 |
| *** itlinux has quit IRC | 15:21 | |
| thrash | vkmc: what does openstack server list show? | 15:21 |
| *** jlinkes has quit IRC | 15:21 | |
| thrash | if it shows anything, that's bad. | 15:21 |
| vkmc | thrash, anything | 15:21 |
| thrash | vkmc: without a stack, you shouldn't have anything listed. | 15:22 |
| vkmc | thrash, then it's ok... I don't have any nova instance running | 15:22 |
| *** bfournie has quit IRC | 15:22 | |
| thrash | ok.. | 15:23 |
| thrash | strange why nova thinks it doesn't have anything available... | 15:23 |
| vkmc | yeah... hmm | 15:23 |
| owalsh | vkmc: anything suspicious in /var/log/nova/nova-compute.log? | 15:24 |
| vkmc | owalsh, checking | 15:24 |
| vkmc | owalsh, no errors... just debug info | 15:25 |
| vkmc | let's see if I can pick something with journalctl | 15:25 |
| vkmc | no... | 15:25 |
| *** thrash is now known as thrash|biab | 15:26 | |
| *** jpena is now known as jpena|off | 15:26 | |
| bnemec | marios: Hmm, good point about it being upgrade-related. | 15:27 |
| bnemec | I guess I didn't think about that because it's not something to make upgrades work, but to verify what happened. | 15:27 |
| bnemec | I'd be okay merging it under the upgrades umbrella though. | 15:27 |
| *** jlabarre has quit IRC | 15:29 | |
| owalsh | vkmc: hmm, could share the logs somewhere (no idea how)? | 15:30 |
| *** jfrancoa has quit IRC | 15:30 | |
| *** Slower has joined #tripleo | 15:30 | |
| *** jlabarre has joined #tripleo | 15:31 | |
| vkmc | owalsh, I'll check out other nova, mistral and ironic logs | 15:31 |
| *** rcernin has quit IRC | 15:32 | |
| owalsh | vkmc: ok. This is a recent quickstart deployment of master yea? I've not hit anything similar and I've deleted/redployed the stack a lot | 15:33 |
| vkmc | owalsh, yes | 15:33 |
| *** bfournie has joined #tripleo | 15:33 | |
| *** iranzo has quit IRC | 15:35 | |
| *** gbarros has quit IRC | 15:35 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Remove printing the introspection status https://review.openstack.org/494249 | 15:37 |
| chandankumar | EmilienM: sorry not able to make it to any Bj call this week due to poor network at Home town | 15:38 |
| jrist | EmilienM: we are almost ready with bp-websocket-logging, fyi | 15:38 |
| marios | bnemec: ack thanks putting it out there for discussion and its still early in US day /me off in a bit. not urgent, but would be nice to have, and i think it is not too dangerous, since it doesn't explode when validation fails | 15:39 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 15:40 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 15:40 |
| *** gbarros has joined #tripleo | 15:40 | |
| *** MVenesio has quit IRC | 15:41 | |
| pabelanger | weshay: EmilienM: mwhahaha: things are starting to click in gate pipeline. Looks like another wave of merges | 15:43 |
| weshay | pabelanger, thanks again for your help | 15:43 |
| *** ccamacho has joined #tripleo | 15:44 | |
| *** ccamacho has quit IRC | 15:44 | |
| *** dtrainor has joined #tripleo | 15:45 | |
| fultonj | jistr: would you please review https://review.openstack.org/#/c/479288 ? | 15:46 |
| jistr | looking | 15:47 |
| fultonj | thanks | 15:47 |
| fultonj | i think it's finally ready :) | 15:47 |
| *** dansmith has joined #tripleo | 15:47 | |
| dansmith | owalsh: vkmc ohai | 15:47 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Enable TLS configuration for containerized HAProxy https://review.openstack.org/491602 | 15:48 |
| *** vpickard_ is now known as vpickard | 15:49 | |
| owalsh | dansmith: ola | 15:49 |
| gfidente | jistr fultonj it's blocked by the dependency | 15:49 |
| jistr | yea i see | 15:49 |
| gfidente | they don't really depend on each other | 15:49 |
| gfidente | EmilienM ^^ | 15:49 |
| jistr | i'll just put +2 for now | 15:49 |
| gfidente | but I guess point is to make it pass pingtest too | 15:49 |
| gfidente | which it does, because of the depends-on | 15:50 |
| gfidente | EmilienM if you rebase https://review.openstack.org/#/c/490129/ we try to merge https://review.openstack.org/#/c/479288 too | 15:50 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Render IP map and host maps according to network_data.yaml https://review.openstack.org/493984 | 15:51 |
| fultonj | thanks jistr | 15:52 |
| *** apetrich_ has quit IRC | 15:52 | |
| *** eglynn has joined #tripleo | 15:53 | |
| owalsh | dansmith: quick TL;DR vkmc is hitting http://paste.openstack.org/show/618686/ when deploying an overcloud | 15:54 |
| owalsh | dansmith: ironic nodes appear to be ok, 2 avaialable, but nova doesn't know about them http://paste.openstack.org/show/618698/ | 15:54 |
| dansmith | owalsh: and only one nova-compute I assume? | 15:54 |
| EmilienM | chandankumar: no worries | 15:55 |
| EmilienM | jrist: cool | 15:55 |
| owalsh | dansmith: on undercloud yes, there is only one nova-compute service | 15:55 |
| *** udesale has joined #tripleo | 15:55 | |
| EmilienM | pabelanger: thx for the help again | 15:55 |
| EmilienM | gfidente: looking | 15:55 |
| dansmith | owalsh: is that compute properly mapped into the cell and such? | 15:56 |
| melwitt | dansmith: it looks like she ran discover_hosts already (there's a paste in the scrollback) | 15:56 |
| dansmith | the way hypervisor stats works is that it iterates cells not mappings, so really the mappingness doesn't matter | 15:56 |
| dansmith | melwitt: I don't have scrollback here since I just joined | 15:56 |
| melwitt | dansmith: k, this one http://paste.openstack.org/show/618696/ | 15:57 |
| pabelanger | 3 more incoming merges | 15:57 |
| pabelanger | a good run but telnet://15.184.68.88:19885 is going to break it | 15:57 |
| owalsh | dansmith: http://paste.openstack.org/show/618696/ | 15:58 |
| dansmith | owalsh: vkmc: is there evidence in the compute logs that it's seeing the ironic nodes? | 15:58 |
| *** udesale has quit IRC | 15:58 | |
| pabelanger | 493953 will fail, and will look why | 15:58 |
| *** udesale has joined #tripleo | 15:58 | |
| EmilienM | gfidente: let me remove -Depends On on https://review.openstack.org/#/c/479288 so we can land it ok? | 15:58 |
| EmilienM | gfidente: now we know it works :D | 15:58 |
| gfidente | EmilienM also good yes | 15:59 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 15:59 |
| gfidente | I thought you wanted both in | 15:59 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 15:59 |
| EmilienM | gfidente: that's fine | 15:59 |
| EmilienM | gfidente: approved | 15:59 |
| EmilienM | gfidente: hopefullt it merges today | 15:59 |
| EmilienM | gfidente: meantime, I'm rebasing quickstart things and make them land as well | 15:59 |
| *** ebarrera has quit IRC | 16:00 | |
| EmilienM | gfidente, fultonj, jistr: good work to make it working :) | 16:00 |
| EmilienM | now upgrade :P | 16:00 |
| *** trown is now known as trown|lunch | 16:00 | |
| fultonj | thanks EmilienM! | 16:00 |
| owalsh | dansmith: bah, got it "Auto-disabled due to 10 build failures" | 16:00 |
| gfidente | EmilienM yeah WIP | 16:00 |
| dansmith | owalsh: that shouldn't affect hypervisor-stats that I know of | 16:00 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Create a whitelist for /etc configs https://review.openstack.org/493973 | 16:00 |
| dansmith | oh, no, it should actually | 16:01 |
| dansmith | buried deep in the db layer is a filter for disabled==false, which is back from the days where one compute service was one hypervisor | 16:01 |
| dansmith | so yeah, that should explain it | 16:01 |
| *** ebarrera has joined #tripleo | 16:02 | |
| *** shardy has quit IRC | 16:02 | |
| owalsh | dansmith: yes, reenabled the service and count is now 2 | 16:02 |
| dansmith | sweet | 16:02 |
| *** jmelvin has quit IRC | 16:03 | |
| *** egonzalez has quit IRC | 16:04 | |
| *** aditya_r has joined #tripleo | 16:04 | |
| vkmc | owalsh, reestarting nova did the trick? | 16:04 |
| owalsh | vkmc: re-enabling the nova-compute service: http://paste.openstack.org/show/618703/ | 16:05 |
| EmilienM | gfidente, fultonj : how is scenario004 deployment going with ceph-ansible? last time I checked it didn't work well with glance | 16:05 |
| owalsh | dansmith: thanks | 16:05 |
| gfidente | EmilienM so plans are to merge mds and rgw | 16:05 |
| gfidente | because scenario004 uses those | 16:05 |
| gfidente | and then convert that one too | 16:05 |
| dansmith | owalsh: all I did was say "what's in the logs" but .. sure :D | 16:05 |
| gfidente | mds got merged earlier today, rgw probably needs a change in ceph-ansible so it might take a bit longer | 16:06 |
| gfidente | we can't deploy ceph services some with ansible some with puppet, so we can only migrate scenario004 when mds and rgw are both available via ansible | 16:06 |
| *** marios has quit IRC | 16:06 | |
| adarazs | weshay: so as far as I can tell from the console logs of the job related to https://bugs.launchpad.net/tripleo/+bug/1710678 it should be working. | 16:07 |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 16:07 |
| vkmc | owalsh, dansmith, thanks | 16:07 |
| *** lvdombrkr has joined #tripleo | 16:07 | |
| adarazs | pabelanger: ^ | 16:07 |
| vkmc | owalsh, any idea why I could have reached to that? | 16:07 |
| pabelanger | EmilienM: like to pull the trigger on sshnaidm namesever patch: https://review.openstack.org/494545/ | 16:07 |
| adarazs | pabelanger, weshay: I will tackle the DLRN cloning and maybe the rdoinfo tomorrow. | 16:07 |
| EmilienM | pabelanger: done | 16:08 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Enable TLS for containerized haproxy https://review.openstack.org/489900 | 16:08 |
| *** cylopez has quit IRC | 16:08 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Remove extra keystone admin haproxy listen and allow TLS https://review.openstack.org/493937 | 16:08 |
| *** rcernin has joined #tripleo | 16:08 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Enable TLS for containerized MySQL https://review.openstack.org/493561 | 16:08 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Tag the ha containers with 'pcmklatest' at deploy time https://review.openstack.org/491705 | 16:08 |
| adarazs | pabelanger, weshay: I updated the bug with all the related changes. | 16:08 |
| pabelanger | adarazs: weshay: Thanks, I'll look at logs shortly | 16:09 |
| adarazs | they should be probably ready to go if https://review.openstack.org/494538 passes. | 16:09 |
| * adarazs has to log off for today. | 16:09 | |
| owalsh | dansmith: in the output for discover hosts it added a cell mapping yea http://paste.openstack.org/show/618696/ | 16:09 |
| EmilienM | sshnaidm: http://logs.openstack.org/33/494233/3/check/gate-tripleo-ci-centos-7-containers-multinode/100517e/logs/undercloud/home/jenkins/overcloud_prep_images.log.txt.gz#_2017-08-17_09_21_20 | 16:10 |
| dansmith | owalsh: yes, although the way stats works that shouldn't have mattered | 16:10 |
| EmilienM | sshnaidm: are we sure we pull it from the mirror? | 16:10 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 16:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 16:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 16:10 |
| owalsh | dansmith: builds would fail until that mapping existed though, correct? | 16:10 |
| dansmith | owalsh: but what it tells us is that it is at least reporting to its own cell db | 16:10 |
| dansmith | owalsh: correct | 16:10 |
| *** hewbrocca is now known as hewbrocca_afk | 16:10 | |
| dansmith | owalsh: you think that's whyfor the auto disableification? | 16:10 |
| EmilienM | sshnaidm: I guess yes | 16:10 |
| owalsh | dansmith: yea | 16:10 |
| dansmith | owalsh: makes sense | 16:11 |
| sshnaidm | EmilienM, I think so.. maybe it's pulled as dependency of something, need to check | 16:11 |
| EmilienM | sshnaidm: I checked, it's good. | 16:11 |
| sshnaidm | ok | 16:11 |
| EmilienM | sshnaidm: we setup mirrors BEFORE creating fake image | 16:11 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on 2 scenarios container jobs https://review.openstack.org/490129 | 16:13 |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates master: Set notification_format to 'unversioned' https://review.openstack.org/494628 | 16:13 |
| owalsh | vkmc: did you say that the first deploy ran but failed? | 16:13 |
| vkmc | owalsh, yeah, I hit some issues with keytone | 16:13 |
| sshnaidm | EmilienM, it's installed there: http://logs.openstack.org/33/494233/3/check/gate-tripleo-ci-centos-7-containers-multinode/100517e/logs/undercloud/home/jenkins/install_packages.sh.log.txt.gz#_2017-08-17_08_51_20 | 16:13 |
| vkmc | s/keytone/keystone/g | 16:13 |
| sshnaidm | EmilienM, so we're good | 16:13 |
| EmilienM | sshnaidm: perfecto | 16:14 |
| *** jpich has quit IRC | 16:14 | |
| owalsh | dansmith: then I'm not sure why happened here... initiall all was well and it looks like the mapping was deleted | 16:14 |
| *** gvrangan_odl has joined #tripleo | 16:14 | |
| *** itlinux has joined #tripleo | 16:15 | |
| dansmith | owalsh: meaning you think the mapping was in place initially, then deleted, then re-added in the pastebin you shows? | 16:16 |
| dansmith | *showed | 16:16 |
| owalsh | dansmith: yea, otherwise the first time vkmc tried to deploy it would have failed at the same place but it got much further | 16:16 |
| dansmith | owalsh: okay, well, I don't have the full context of the story I guess, but.. mappings are never deleted that I knwo of | 16:17 |
| owalsh | vkmc: did you do anything with the ironic nodes? E.g delete and re add them? | 16:17 |
| openstackgerrit | Andy Smith proposed openstack/tripleo-heat-templates master: WIP OpenStack containerized qpid-dispatch-router service https://review.openstack.org/479049 | 16:17 |
| vkmc | owalsh, no, just overcloud delete | 16:18 |
| *** achadha has joined #tripleo | 16:19 | |
| *** gbarros has quit IRC | 16:19 | |
| *** itlinux has quit IRC | 16:19 | |
| *** ramishra has quit IRC | 16:20 | |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs https://review.openstack.org/494043 | 16:20 |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs https://review.openstack.org/494043 | 16:21 |
| *** gbarros has joined #tripleo | 16:21 | |
| *** ramishra has joined #tripleo | 16:22 | |
| *** achadha has quit IRC | 16:23 | |
| *** dsariel has joined #tripleo | 16:23 | |
| *** agurenko has quit IRC | 16:24 | |
| *** itlinux has joined #tripleo | 16:25 | |
| owalsh | dansmith: not related to cells | 16:25 |
| owalsh | vkmc: not enough free space on the undercloud VM, ironic needs to create some large tmp files | 16:27 |
| openstackgerrit | John Fulton proposed openstack/tripleo-docs master: Update Pike storage documentation for ceph-ansible https://review.openstack.org/487155 | 16:28 |
| *** aditya_r has quit IRC | 16:28 | |
| owalsh | vkmc: could you redeploy the undercloud with a 100GB disk. docker images take quite a lot of space | 16:29 |
| *** udesale__ has joined #tripleo | 16:29 | |
| *** udesale has quit IRC | 16:29 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-docs master: Update Pike storage documentation for ceph-ansible https://review.openstack.org/487155 | 16:30 |
| *** udesale__ has quit IRC | 16:30 | |
| *** udesale has joined #tripleo | 16:30 | |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs https://review.openstack.org/494043 | 16:33 |
| openstackgerrit | John Fulton proposed openstack/tripleo-docs master: Update Pike storage documentation for ceph-ansible https://review.openstack.org/487155 | 16:33 |
| owalsh | d0ugal, jistr, vkmc: so... it turns out the problem was that the undercloud didn't have much disk space remaining so ironic builds failed, and nova now auto disables computes that fail 10 consecutive builds | 16:34 |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs https://review.openstack.org/494043 | 16:34 |
| *** pchavva has quit IRC | 16:34 | |
| owalsh | so not a bug, as such, but maybe something we could catch in validation? | 16:34 |
| *** itlinux has quit IRC | 16:34 | |
| owalsh | ^H^H^H by catch I mean report the actual issue, validation did catch this I guess | 16:35 |
| *** brault has quit IRC | 16:36 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Mount ceph config on gnocchi statsd https://review.openstack.org/494639 | 16:36 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Keep floating ip reachability during pacemaker migration. https://review.openstack.org/474967 | 16:38 |
| *** eglynn has quit IRC | 16:38 | |
| openstackgerrit | Andy Smith proposed openstack/tripleo-common master: Add Qdrouterd to the overcloud containers https://review.openstack.org/491889 | 16:39 |
| *** anshul has quit IRC | 16:40 | |
| *** jtomasek_ has quit IRC | 16:41 | |
| *** sshnaidm is now known as sshnaidm|off | 16:44 | |
| *** homeski has joined #tripleo | 16:47 | |
| *** udesale has quit IRC | 16:50 | |
| *** achadha has joined #tripleo | 16:51 | |
| *** tellesnobrega has joined #tripleo | 16:52 | |
| d0ugal | owalsh: woah, nice debugging :) a validation for that would be good! | 16:52 |
| *** derekh has quit IRC | 16:52 | |
| d0ugal | open a feature-bug? | 16:52 |
| *** itlinux has joined #tripleo | 16:53 | |
| *** jmelvin has joined #tripleo | 16:53 | |
| jistr | also i wonder if we should think about bumping the default UC specs | 16:53 |
| jistr | though mine has 30 GB used 20 GB free | 16:53 |
| jistr | (and i do have container images there) | 16:54 |
| *** lucasagomes is now known as lucas-afk | 16:54 | |
| *** achadha has quit IRC | 16:55 | |
| *** ebarrera has quit IRC | 16:55 | |
| pabelanger | gate reset, as expected. looking at 493953 logs | 16:56 |
| *** oidgar has quit IRC | 16:57 | |
| jistr | it could be a matter of garbage collection maybe. I can imagine if the UC gets reused a few times, including downloading fresh images from upstream, it could fill up the disk quite easily. I think the old ones don't get deleted automatically. | 16:57 |
| pabelanger | http://logs.openstack.org/53/493953/3/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/27ca6d9/logs/devstack-gate-setup-workspace-new.txt | 16:57 |
| pabelanger | failed to clone from zm07 | 16:57 |
| pabelanger | since infracloud, this is likely a network issue | 16:58 |
| *** achadha has joined #tripleo | 17:01 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario007 to run Tempest https://review.openstack.org/494293 | 17:03 |
| lvdombrkr | guys, anybode bofore deplot overcloud with compute and controller in one node? | 17:03 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Use sample environments for OVB configuration https://review.openstack.org/494643 | 17:04 |
| *** brault has joined #tripleo | 17:04 | |
| *** achadha has quit IRC | 17:06 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: Set the undercloud_docker_registry_mirror for upstream jobs https://review.openstack.org/494644 | 17:07 |
| *** achadha has joined #tripleo | 17:07 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Use sample environments for OVB configuration https://review.openstack.org/494643 | 17:08 |
| *** salmankhan has quit IRC | 17:08 | |
| *** slagle has joined #tripleo | 17:09 | |
| lvdombrkr | guys, anybode bofore deplot overcloud with compute and controller in one node? | 17:09 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 17:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 17:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 17:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario003 to run Tempest https://review.openstack.org/494290 | 17:10 |
| slagle | lvdombrkr: yes, we do in ci | 17:11 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario007 to run Tempest https://review.openstack.org/494293 | 17:12 |
| slagle | lvdombrkr: you can see an example list of services for an all-in-one at ci/environments/multinode.yaml | 17:12 |
| slagle | in tripleo-heat-templates | 17:12 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: moving docker registry config to environment https://review.openstack.org/494647 | 17:13 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 17:13 |
| EmilienM | mcornea: hey have you tried to spawn + ssh vms on containerized overclouds? | 17:15 |
| lvdombrkr | slagle: i try it wich this config (http://paste.openstack.org/show/618524) but get beck error that there is required 2 nodes | 17:15 |
| EmilienM | I'm trying with tempest right now and it fails: http://logs.openstack.org/84/494284/2/check/gate-tripleo-ci-centos-7-containers-multinode/c7c5f63/logs/tempest.html.gz | 17:15 |
| lvdombrkr | slagle: i cant undrstood where is my mistake | 17:15 |
| EmilienM | weshay: ^ very close of running tempest on containers-multinode FYI | 17:15 |
| *** brault has quit IRC | 17:15 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 17:16 |
| slagle | lvdombrkr: you may need to pass --compute-scale 0 to the deploy command | 17:16 |
| weshay | EmilienM, nice brotha | 17:17 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Add config/nodes definitions for OVB https://review.openstack.org/466367 | 17:17 |
| mcornea | EmilienM: yes, but only after applying workarounds for a https://bugzilla.redhat.com/show_bug.cgi?id=1459592 and https://bugzilla.redhat.com/show_bug.cgi?id=1464182 | 17:18 |
| openstack | bugzilla.redhat.com bug 1459592 in libvirt "error : Unable to move /dev/log mount to /var/run/libvirt/qemu/instance-00000002.log: No such file or directory" [Unspecified,Post] - Assigned to mprivozn | 17:18 |
| openstack | bugzilla.redhat.com bug 1464182 in openstack-nova "openstack-nova: unable to launch an instance: InternalError: Unable to get host UUID: /etc/machine-id is empty" [High,New] - Assigned to m.andre | 17:18 |
| EmilienM | mcornea: weird, sounds like the instance was created on my side, it just can't be ssh'ed | 17:19 |
| EmilienM | the instance was created: http://logs.openstack.org/84/494284/2/check/gate-tripleo-ci-centos-7-containers-multinode/c7c5f63/logs/subnode-2/var/log/libvirt/qemu/instance-00000001.log.txt.gz | 17:19 |
| EmilienM | (I don't run any workaround FYI) | 17:19 |
| *** trown|lunch is now known as trown | 17:20 | |
| mcornea | EmilienM: yes, looks that the instances was created but it's not reacheable | 17:20 |
| *** jmelvin has quit IRC | 17:20 | |
| EmilienM | mcornea: error is here http://logs.openstack.org/84/494284/2/check/gate-tripleo-ci-centos-7-containers-multinode/c7c5f63/logs/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-08-16_22_58_51 | 17:20 |
| EmilienM | yeah | 17:20 |
| EmilienM | i'll debug neutron | 17:20 |
| mcornea | EmilienM: is this a clean deployment or an upgraded one? | 17:21 |
| EmilienM | http://logs.openstack.org/84/494284/2/check/gate-tripleo-ci-centos-7-containers-multinode/c7c5f63/logs/subnode-2/var/log/containers/neutron/neutron-metadata-agent.log.txt.gz#_2017-08-16_22_53_45_592 | 17:21 |
| EmilienM | mcornea: clean | 17:21 |
| EmilienM | sounds like neutron metadata agent is crying | 17:21 |
| EmilienM | dprince, jistr: did you get to the point where you can spawn + ssh a vm with containerized overcloud? | 17:21 |
| dprince | EmilienM: I've done it before | 17:22 |
| EmilienM | with containerized neutron? | 17:22 |
| dprince | EmilienM: but this was a few weeks back | 17:22 |
| dprince | EmilienM: yes, containerized neutron | 17:22 |
| EmilienM | dprince: we can't ssh a VM now | 17:22 |
| dprince | EmilienM: okay, upgrades job? | 17:23 |
| EmilienM | dprince: no | 17:23 |
| EmilienM | gate-tripleo-ci-centos-7-containers-multinode | 17:23 |
| EmilienM | fresh install | 17:23 |
| *** pkovar has quit IRC | 17:24 | |
| *** tosky has quit IRC | 17:27 | |
| *** MVenesio has joined #tripleo | 17:28 | |
| *** achadha has quit IRC | 17:32 | |
| *** achadha has joined #tripleo | 17:33 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add Ceilometer API and Collector service to roles_data https://review.openstack.org/494589 | 17:33 |
| lvdombrkr | slagle: now after : openstack-tripleo-heat-templates]$ openstack overcloud deploy --templates one.yaml --compute-scale 0 | 17:36 |
| lvdombrkr | i get [Errno 20] Not a directory: '/usr/share/openstack-tripleo-heat-templates/one.yaml' | 17:36 |
| atoth | beagles, EmilienM, I need to understand something I think, do ensure_packages calls actually install the packages if they are not there or just check for their inclusion? | 17:37 |
| *** achadha has quit IRC | 17:38 | |
| pabelanger | EmilienM: mwhahaha: weshay: adarazs: I've +1 https://review.openstack.org/494262 but with comments. We still have other repos that need to hit our mirrors, so we can either fix this or stack on another patch. | 17:38 |
| *** thrash|biab is now known as thrash | 17:39 | |
| *** achadha has joined #tripleo | 17:39 | |
| atoth | beagles, EmilienM, because I was just reading some posts referencing norpm as that is what is referenced in my log files, which seems to be the reason for the packages not being installed | 17:39 |
| weshay | looking | 17:39 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add certmonger user profile to all overcloud roles https://review.openstack.org/494653 | 17:40 |
| weshay | pabelanger, ah.. like delorean-pike-testing and the two others | 17:42 |
| pabelanger | weshay: yup, we need to use the reverse proxy cache for those too | 17:43 |
| slagle | lvdombrkr: --templates takes a directory, or no argument at all | 17:43 |
| slagle | lvdombrkr: try --help | 17:43 |
| slagle | lvdombrkr: i'm guessing you may be missing a -e before one.yaml | 17:43 |
| pabelanger | weshay: so, landing that will make an impact and allow us to iterate on the next patch | 17:43 |
| lvdombrkr | slagle: yea, you are right, but now i got :Not enough nodes - available: 0, requested: 1 | 17:44 |
| weshay | ya.. just getting familiar w/ the related patches atm | 17:44 |
| *** bkopilov has quit IRC | 17:44 | |
| slagle | lvdombrkr: then you have 0 nodes availalble, you won't be able to deploy anything. | 17:45 |
| EmilienM | dprince: I reported it https://bugs.launchpad.net/tripleo/+bug/1711425 - any help is welcome, please | 17:45 |
| openstack | Launchpad bug 1711425 in tripleo "Impossible to ssh a VM when TripleO is containerized" [Critical,Triaged] | 17:45 |
| slagle | lvdombrkr: did you walk though all the documented steps of registering nodes, etc? | 17:45 |
| lvdombrkr | slagle: yes, shore under openstack baremetal node list i see one node available | 17:46 |
| weshay | pabelanger, so https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L69-L72 | 17:47 |
| weshay | needs to change to NODEPOOL_MIRROR_HOST? | 17:47 |
| *** brault has joined #tripleo | 17:48 | |
| lvdombrkr | slagle: http://paste.openstack.org/raw/618711/ | 17:48 |
| *** jmelvin has joined #tripleo | 17:48 | |
| slagle | lvdombrkr: check the "Provisioning State" and "Maintenance". does those show "available" and False? | 17:48 |
| lvdombrkr | slagle:http://paste.openstack.org/raw/618711/ | 17:48 |
| slagle | lvdombrkr: it's in maintenance mode | 17:48 |
| slagle | which means that, for some reason Ironic was unable to contact the node | 17:49 |
| slagle | probably while checking the power credentials | 17:49 |
| slagle | check the Ironic logs for the possible error | 17:49 |
| slagle | how did you set the environment up? | 17:49 |
| lvdombrkr | slagle: what you excatly mean? | 17:50 |
| slagle | lvdombrkr: check the ironic logs under /var/log/ironic to see why it put the node into maintenance mode | 17:50 |
| homeski | set term_charset utf-8 | 17:50 |
| *** homeski has quit IRC | 17:51 | |
| *** brault has quit IRC | 17:52 | |
| *** apetrich has joined #tripleo | 17:52 | |
| *** nyechiel has quit IRC | 17:53 | |
| *** itlinux has quit IRC | 17:53 | |
| lvdombrkr | slagle: During sync_power_state, max retries exceeded for node ce96b035-77f1-4469-a7f9-b78228e0c6de, node state None does not match expected state 'power off'. Updating DB state to 'None' Switching node to maintenance mode. Error: IPMI call failed: power status. | 17:54 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Add TLS for nova metadata service https://review.openstack.org/494657 | 17:55 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for nova-metadata https://review.openstack.org/494658 | 17:55 |
| *** homeski has joined #tripleo | 17:55 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Generalize OVB settings to fit feature set pattern https://review.openstack.org/494659 | 17:55 |
| slagle | lvdombrkr: ironic can't use the provided ipmi connection info and credentials to get the power state for the node | 17:58 |
| *** itlinux has joined #tripleo | 17:58 | |
| slagle | lvdombrkr: that has to work before ironic will let you deploy anything to the node | 17:59 |
| lvdombrkr | slagle: but if provisioning state is : available thats means it worked before? | 17:59 |
| lvdombrkr | or im wrong? | 18:00 |
| *** brault has joined #tripleo | 18:00 | |
| slagle | lvdombrkr: that's not what it means | 18:00 |
| *** dprince has quit IRC | 18:00 | |
| lvdombrkr | slagle: i belive before that available means ready for deployment | 18:01 |
| *** jkilpatr has quit IRC | 18:02 | |
| slagle | when maintenance mode is True, you can't deploy to the node | 18:03 |
| *** jkilpatr has joined #tripleo | 18:03 | |
| slagle | even though it says "available" | 18:03 |
| slagle | the ironic state machine is confusing, i admit | 18:04 |
| *** brault has quit IRC | 18:04 | |
| lvdombrkr | can i move it to managable back and try introspection again>? | 18:05 |
| *** jcoufal has quit IRC | 18:05 | |
| slagle | if ironic can't sync the power state, that needs to be fixed first | 18:06 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Generalize OVB settings to fit feature set pattern https://review.openstack.org/494659 | 18:08 |
| lvdombrkr | slagle: mhmm, its misconfiguration in bios or BMC? | 18:09 |
| *** anshul has joined #tripleo | 18:09 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 18:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711425 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 18:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 18:10 |
| openstack | Launchpad bug 1711425 in tripleo "Impossible to ssh a VM when TripleO is containerized" [Critical,Triaged] | 18:10 |
| EmilienM | owalsh: do we have ongoing effort to containerized nova-api-metadata? | 18:10 |
| slagle | lvdombrkr: i really couldn't say tbh. check what the creds and IPMI address are in ironic and if those match the expected values | 18:11 |
| openstackgerrit | John Fulton proposed openstack/tripleo-docs master: Update Pike storage documentation for ceph-ansible https://review.openstack.org/487155 | 18:12 |
| *** gregwork has quit IRC | 18:13 | |
| lvdombrkr | slagle: but if node is registred that means that the creditianals ar correct | 18:14 |
| lvdombrkr | ? | 18:14 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for nova-metadata https://review.openstack.org/494658 | 18:15 |
| *** jprovazn has quit IRC | 18:15 | |
| *** dprince has joined #tripleo | 18:16 | |
| slagle | lvdombrkr: nope. that's not the case at all | 18:16 |
| lvdombrkr | slagle: i can ipmtool node with same creditionals as in instackenv.json | 18:17 |
| *** cdearborn has quit IRC | 18:17 | |
| lvdombrkr | i thing its enought? | 18:17 |
| *** jcoufal has joined #tripleo | 18:17 | |
| EmilienM | mcornea: how could you run your tests if nova-api-metadata isn't containerized ? | 18:25 |
| EmilienM | mcornea: did you deploy it on the host in the classic way? | 18:25 |
| mcornea | EmilienM: yes, I believe so - the neutron services are running on the host, are not containerized | 18:26 |
| EmilienM | ok that's why, I'll do the same | 18:26 |
| EmilienM | thx | 18:26 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 18:28 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Workaround for RHEL registration as "localhost" https://review.openstack.org/494669 | 18:28 |
| jaosorior | EmilienM, mcornea: Which neutron services are running uncontainerized? neutron-metadata? | 18:29 |
| mcornea | jaosorior: EmilienM in downstream all neutron related services: http://paste.openstack.org/show/618714/ | 18:30 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: ci: do not try to containerized nova-api-metadata https://review.openstack.org/494671 | 18:31 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs010: enable tempest https://review.openstack.org/494284 | 18:32 |
| EmilienM | jaosorior, mcornea : ^ doing this workaround for now. | 18:32 |
| slagle | lvdombrkr: then check in the ironic logs why it failed | 18:34 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 18:35 |
| * beagles playing catchup | 18:36 | |
| *** florianf has quit IRC | 18:37 | |
| * beagles also discovers that overcloud nodes running containers do not appear to like being rebooted | 18:37 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 18:40 |
| *** jlabarre has quit IRC | 18:41 | |
| *** jlabarre has joined #tripleo | 18:42 | |
| lvdombrkr | slagle: node get into maintance mode just today in the morning | 18:44 |
| lvdombrkr | but yesterday it was okay | 18:44 |
| lvdombrkr | (i see that in logs) | 18:45 |
| slagle | lvdombrkr: does it say why? | 18:45 |
| slagle | mhenkel: fyi, https://review.openstack.org/#/c/494669/ | 18:45 |
| slagle | mhenkel: maybe give it a try and let us know if it helps at all if you have the time | 18:45 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Run nova-metadata container from nova-api template https://review.openstack.org/494673 | 18:46 |
| jaosorior | EmilienM: what about that? ^^ | 18:46 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: WIP create default settings for environment types https://review.openstack.org/494674 | 18:46 |
| lvdombrkr | slagle: thats all what i see : node state None does not match expected state 'power off'. Updating DB state to 'None' Switching node to maintenance mode. Error: IPMI call failed: power status. | 18:46 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Enable TLS configuration for containerized RabbitMQ https://review.openstack.org/491604 | 18:46 |
| openstackgerrit | Merged openstack/tripleo-common stable/ocata: Prompt to clear breakpoints when using deployed-server https://review.openstack.org/491115 | 18:46 |
| *** pchavva has joined #tripleo | 18:47 | |
| slagle | lvdombrkr: try setting debug=True in ironic.conf, and restarting all ironic services, then move node out of maintenance mode, see what happens | 18:47 |
| lvdombrkr | slagle: any quick quide how to move node from maintanence mode? | 18:49 |
| slagle | lvdombrkr: you can do it via the ironic cli, but i don't have the command right off | 18:50 |
| slagle | check the help, or ironic docs | 18:50 |
| EmilienM | jaosorior: it doesn't fit with micro services architecture, right? | 18:51 |
| EmilienM | jaosorior: services should be separated IMHO | 18:51 |
| *** catintheroof has quit IRC | 18:54 | |
| *** lyarwood has quit IRC | 18:55 | |
| *** nyechiel has joined #tripleo | 18:56 | |
| jaosorior | EmilienM: it doesn't. But it sure makes it way easier than depending on the puppet-generated configuration that it needs from nova-api | 18:57 |
| jaosorior | EmilienM: they're not fully independent unfortunately | 18:58 |
| EmilienM | jaosorior: I talked with nova guys and they told me they are | 18:58 |
| jaosorior | EmilienM: so one of two. Either we start referencing the puppet-generated configurations from nova-api, from nova-metadata. Or we do it in the same template as I did there. | 18:58 |
| EmilienM | ok | 18:58 |
| *** gfidente is now known as gfidente|afk | 18:58 | |
| jaosorior | EmilienM: then I'm probably wrong and life is good :) | 18:58 |
| jaosorior | EmilienM: maybe I have outdated info | 18:59 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: WIP create default settings for environment types https://review.openstack.org/494674 | 18:59 |
| jaosorior | EmilienM: alright so lets do this, I won't abandon that patch just yet. If all else fails, lets try out that one. | 18:59 |
| jaosorior | as a last resort | 18:59 |
| EmilienM | jaosorior: ok | 19:00 |
| EmilienM | but yeah, api servis should be separated as an end result | 19:00 |
| *** amoralej is now known as amoralej|off | 19:01 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: WIP create default settings for environment types https://review.openstack.org/494674 | 19:02 |
| EmilienM | bnemec, mwhahaha : I saw you approved https://review.openstack.org/#/c/474578/ | 19:03 |
| *** gfidente|afk has quit IRC | 19:03 | |
| EmilienM | it's not something we need for upgrades, it's a features that improves upgrades | 19:03 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 19:03 |
| EmilienM | but I guess that's fine. I just don't want to increase the risk of regression at this stage | 19:03 |
| *** mcornea has quit IRC | 19:03 | |
| EmilienM | if something breaks I'll revert it though. | 19:04 |
| *** itlinux has quit IRC | 19:04 | |
| mwhahaha | EmilienM: eh we can hold off if you think it's better to | 19:05 |
| *** itlinux has joined #tripleo | 19:05 | |
| EmilienM | mwhahaha: it's fine | 19:05 |
| *** nyechiel has quit IRC | 19:06 | |
| mwhahaha | EmilienM: on second thought it's probably better not to add this in right now given everything. I'll remove my +A | 19:06 |
| weshay | EmilienM, how do you like your words cooked? | 19:06 |
| EmilienM | mwhahaha: thanks | 19:07 |
| EmilienM | weshay: hot | 19:07 |
| *** gbarros has quit IRC | 19:07 | |
| weshay | EmilienM, ok.. get ready to eat'em up :) http://paste.openstack.org/show/618715/ | 19:07 |
| EmilienM | weshay: like I said, you didn't deploy nova-api-metadata in a container | 19:08 |
| EmilienM | weshay: run a ps on the controller | 19:08 |
| EmilienM | and show me the nice process running | 19:08 |
| EmilienM | weshay: and btw where is the tempest test that try ssh to vm? I don't see it. | 19:09 |
| weshay | nova-api is running in a container on the controller | 19:09 |
| fultonj | EmilienM: regarding https://review.openstack.org/#/c/479288 , i just added a comment on why it didn't pass there. Do we want to add the depends on to the oooq change that was removed? | 19:09 |
| weshay | :)) | 19:09 |
| EmilienM | weshay: I'm talking about nova-api-metadata | 19:09 |
| weshay | k.. /me eats my words cold | 19:09 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Remove docker references from stable/ocata scenarios. https://review.openstack.org/489874 | 19:09 |
| weshay | and looks | 19:09 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Mount NFS volume to docker container. https://review.openstack.org/490839 | 19:10 |
| fultonj | i'd add it but want to know if it was removed for a reason; i'll add it back, just let me know. thanks | 19:10 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 19:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711425 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 19:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 19:10 |
| openstack | Launchpad bug 1711425 in tripleo "Impossible to ssh a VM when TripleO is containerized" [Critical,Triaged] | 19:10 |
| *** salmankhan has joined #tripleo | 19:10 | |
| EmilienM | weshay: pastbin me a ps on the controller host, please | 19:10 |
| weshay | EmilienM, it's in the paste no? | 19:11 |
| EmilienM | weshay: also show me the total list of tests you ran with tempest. I don't care about mistral now | 19:11 |
| EmilienM | weshay: not docker ps | 19:11 |
| EmilienM | weshay: "ps" | 19:11 |
| weshay | k | 19:11 |
| weshay | lolz | 19:11 |
| weshay | don't get EmilienM fired up.. ever | 19:11 |
| weshay | :) | 19:11 |
| *** itlinux has quit IRC | 19:12 | |
| *** pchavva has quit IRC | 19:12 | |
| EmilienM | I'm just bringing up nobody tested to boot and ssh a VM (which is a very basic thing) with containerized services :D | 19:12 |
| EmilienM | and we're about to release | 19:12 |
| slagle | yea, but ... containers | 19:13 |
| slagle | you don't need VM's | 19:13 |
| *** liverpooler has quit IRC | 19:13 | |
| mwhahaha | ಠ_ಠ| 19:14 |
| EmilienM | fultonj: ok, we can wait or add depends-on; up to you | 19:14 |
| fultonj | if you'll vote again, then i'll depend-on to get it in sooner :) | 19:15 |
| fultonj | if you won't vote again, i'll wait :) | 19:15 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Generalize OVB settings to fit feature set pattern https://review.openstack.org/494659 | 19:15 |
| fultonj | whatever you prefer | 19:16 |
| weshay | chunk.io isn't working :( | 19:17 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 19:18 |
| EmilienM | fultonj: I'll vote as soon as you ask me | 19:18 |
| fultonj | thanks EmilienM | 19:18 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 19:20 |
| fultonj | thanks | 19:21 |
| *** gkadam-afk has quit IRC | 19:35 | |
| EmilienM | cool fultonj | 19:36 |
| EmilienM | good work | 19:36 |
| fultonj | i think the docs are ready, i know you wanted those landing same time https://review.openstack.org/#/c/487155/ | 19:37 |
| *** gbarros has joined #tripleo | 19:37 | |
| fultonj | though dsuntur was revinewing and you have enough to review :) | 19:38 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Add converge step to the container upgrade https://review.openstack.org/494590 | 19:41 |
| *** eck` is now known as eck`gone | 19:41 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Add converge step to the container upgrade https://review.openstack.org/494590 | 19:41 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 19:41 |
| *** athomas has quit IRC | 19:42 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 19:43 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 19:43 |
| EmilienM | pabelanger: when you have 2s https://review.openstack.org/#/c/494596 | 19:44 |
| *** trown is now known as trown|brb | 19:46 | |
| *** brault has joined #tripleo | 19:47 | |
| *** nyechiel has joined #tripleo | 19:47 | |
| *** itlinux has joined #tripleo | 19:47 | |
| *** anshul has quit IRC | 19:51 | |
| *** gvrangan_odl has quit IRC | 19:52 | |
| *** beagles has left #tripleo | 19:53 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 19:54 |
| *** trown|brb is now known as trown | 19:59 | |
| *** akrivoka has quit IRC | 20:00 | |
| hamzy | does anyone know what I might be doing wrong. newton undercloud, master overcloud, openstack overcloud deploy fails: http://paste.openstack.org/show/618719/ | 20:01 |
| mwhahaha | hamzy: i think you need profile:control not control_profile:1 | 20:04 |
| mwhahaha | not sure that's probably a question for the ironic guys but i've only ever seen profile:<flavor> | 20:05 |
| *** MVenesio has quit IRC | 20:07 | |
| *** stee_3_ has joined #tripleo | 20:08 | |
| *** jkilpatr has quit IRC | 20:09 | |
| *** bugzy_ has joined #tripleo | 20:09 | |
| dprince | EmilienM: Nova metadata was finished. Your comment to me seems to be a bit of an overstatement | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 20:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711425 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 20:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 20:10 |
| openstack | Launchpad bug 1711425 in tripleo "Impossible to ssh a VM when TripleO is containerized" [Critical,Triaged] | 20:10 |
| dprince | EmilienM: Revert f24d5d4c0237d2703cf2744aa6db65865401e94e is potentially an option here to fix the CI regression. 3 days old | 20:10 |
| dprince | jaosorior: ^^^^ | 20:10 |
| EmilienM | in a mtg | 20:11 |
| owalsh | EmilienM: ack from earlier... looking into it. I thought we had this from the initial services that dprince implemented but it seems we don't | 20:12 |
| *** bugzy has quit IRC | 20:12 | |
| dprince | owalsh: it ran via Apache :) | 20:12 |
| dprince | owalsh: sorry eventlet | 20:12 |
| dprince | owalsh: but now we switched back to Apache and it got left behind again | 20:12 |
| *** stee_3 has quit IRC | 20:12 | |
| dprince | owalsh: I commented here https://bugs.launchpad.net/tripleo/+bug/171 | 20:13 |
| openstack | Launchpad bug 36187 in debian-cd (Ubuntu) "duplicate for #171 Should set casper-udeb/snapshot/backing-file to the path to the cloop image" [Medium,Fix released] - Assigned to Colin Watson (cjwatson) | 20:13 |
| dprince | owalsh: https://bugs.launchpad.net/tripleo/+bug/1711425 | 20:13 |
| openstack | Launchpad bug 1711425 in tripleo "Impossible to ssh a VM when TripleO is containerized" [Critical,Triaged] | 20:13 |
| *** eck`gone is now known as eck` | 20:15 | |
| *** rcernin has quit IRC | 20:16 | |
| *** trozet has joined #tripleo | 20:22 | |
| *** sbrzozow has quit IRC | 20:23 | |
| owalsh | dprince: ack, thanks. I can look into it | 20:25 |
| *** nyechiel has quit IRC | 20:25 | |
| EmilienM | dprince: nice spot | 20:25 |
| EmilienM | though I'm not sure it worked before | 20:25 |
| EmilienM | since it wasn't tested in CI... | 20:25 |
| dprince | EmilienM: I said it worked for me. It did. | 20:26 |
| dprince | EmilienM: please don't say it wasn't implemented. It was | 20:26 |
| dprince | EmilienM: as for CI regressions... it could certainly be related to something else | 20:26 |
| dprince | EmilienM: but the feature was working at some point | 20:26 |
| weshay | EmilienM, is that what you want me to try? ssh into an instance? | 20:26 |
| dprince | EmilienM: and we need CI on it absolutely | 20:26 |
| *** eck` is now known as eck`gone | 20:27 | |
| dprince | owalsh: ack, that would be great. I can help too if you need it | 20:27 |
| dprince | owalsh: late for you. Hit me on IRC tomorrow morning if you want | 20:28 |
| EmilienM | dprince: ok fair enough, I believe you ;-) | 20:28 |
| EmilienM | dprince: I'm just saying, I didn't see it before so it's hard to know | 20:29 |
| EmilienM | dprince: anyway, do you think jaosorior's patch is the way to go? | 20:29 |
| EmilienM | I think we should split it and manage the httpd service in the nova-metadata docker service | 20:29 |
| *** eck`gone is now known as eck` | 20:30 | |
| *** jkilpatr has joined #tripleo | 20:30 | |
| dprince | EmilienM: funny thing is I did all these w/ Httpd initially :) | 20:30 |
| dprince | EmilienM: and due to Nova API's httpd breakage we switched it back to eventlet | 20:30 |
| dprince | EmilienM: and now we are switching back to HTTP again | 20:30 |
| dprince | EmilienM: the intitial patches took over a month to land so only the Gerrit history would have this I think | 20:31 |
| dprince | owalsh: ^^^ | 20:31 |
| dprince | and even then it may have gotten lost in the rebase'ing | 20:31 |
| hamzy | mwhahaha, yay, thanks! http://tripleo-docs.readthedocs.io/en/latest/advanced_deployment/profile_matching.html#manual-profile-tagging could be worded better | 20:31 |
| mwhahaha | hamzy: yea i've noticed when we have multiple ways of doing things it's hard to follow in the docs | 20:32 |
| dprince | EmilienM: back to your question, are you suggesting we revert his patch? | 20:32 |
| mwhahaha | hamzy: i've run into this with some of the container docs as well | 20:32 |
| dprince | EmilienM: I dunno, I could probably knock it out quickly w/ owalsh. Reverting Juan's patch for a few days doesn't seem to be a big deal though if you want it working today again | 20:32 |
| dprince | EmilienM: giving a few days to implement it seems reasonable | 20:34 |
| *** nyechiel has joined #tripleo | 20:34 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-docs master: Update Pike storage documentation for ceph-ansible https://review.openstack.org/487155 | 20:36 |
| *** ansmith has quit IRC | 20:39 | |
| *** nyechiel has quit IRC | 20:39 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Check for stack failure earlier in undercloud deploy https://review.openstack.org/487509 | 20:40 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Fix rpms being installed via DeployArtifactURLs https://review.openstack.org/493866 | 20:40 |
| owalsh | dprince: I think we just need to run the eventlet api too, it just runs the metadata api when we are using http | 20:40 |
| *** stevebaker has joined #tripleo | 20:41 | |
| *** ebarrera has joined #tripleo | 20:41 | |
| pabelanger | EmilienM: mwhahaha: 2 more merges! | 20:41 |
| EmilienM | dprince: yeah it's funny we do back and forth :( | 20:41 |
| EmilienM | pabelanger: yes that's cool | 20:41 |
| dprince | owalsh: cool, the slight rub here is that code probably belongs in the nova-metadata.yaml rather than in nova-api.yaml | 20:41 |
| dprince | owalsh: so a bit more work to wire it in... but shouldn't be too bad I think | 20:42 |
| owalsh | dprince: yea, figured that but for a quick test I can just add it back in nova-api | 20:42 |
| EmilienM | dprince: please do not block https://review.openstack.org/#/c/494671/ - so I can enable pingtest or tempest on container jobs | 20:42 |
| stevebaker | morning | 20:42 |
| EmilienM | dprince: and we can iterate later on enabling nova-metadata | 20:42 |
| EmilienM | stevebaker: hey | 20:42 |
| EmilienM | dprince: right now container jobs have zero tests | 20:43 |
| EmilienM | and this one is our last blocker for now | 20:43 |
| dprince | EmilienM: not blocking it at all. Just wanted the commit message to reflect what caused the regression | 20:43 |
| EmilienM | ah ok I see | 20:43 |
| dprince | EmilienM: "was finished" got broken bit "git ID" | 20:43 |
| EmilienM | I'll update once the patch in depends-on finish to run the job | 20:44 |
| EmilienM | I want to see how that works | 20:44 |
| dprince | EmilienM: ack, I'll +2 then | 20:44 |
| EmilienM | ok | 20:44 |
| EmilienM | dprince: the work we have been doing, running tempest instead of pingtest, was actually useful: pingtest doesn't test SSH to the VM while tempest does. | 20:49 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario004 to run Tempest https://review.openstack.org/491113 | 20:49 |
| EmilienM | we like it or not, it brings useful feedback and doesn't take much more time, we're carefuly watching this | 20:50 |
| *** ebarrera has quit IRC | 20:50 | |
| dprince | EmilienM: "much more time" how much? | 20:50 |
| EmilienM | dprince: I can show you number, give me 5 min | 20:51 |
| *** dsariel has quit IRC | 20:52 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario002 to run Tempest https://review.openstack.org/491102 | 20:52 |
| EmilienM | scenario001 is a good example because we run both | 20:52 |
| EmilienM | execution of tempest tests take 4 min 40 on the scenario001: http://logs.openstack.org/39/494639/1/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/0600f2b/logs/ara_oooq/file/33dceb23-1afd-4241-98a6-66205cdf3588/#line-2 | 20:53 |
| EmilienM | and let me check what pingtest takes... | 20:54 |
| dprince | EmilienM: you are comparing very different things | 20:54 |
| *** gbarros has quit IRC | 20:54 | |
| EmilienM | http://logs.openstack.org/88/479288/21/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/41d6ac8/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz | 20:54 |
| EmilienM | it takes 3 min to run pingtest | 20:54 |
| dprince | EmilienM: 4 minutes isn't bad though, but as we've discussed exhaustively the focus of Tempest and pingtest is quite different | 20:55 |
| dprince | EmilienM: and, we could make pingtest do the ssh if we want it too :) | 20:55 |
| EmilienM | running tempest has a lot of benefit to have advanced scenarios like autoscaling in telemetry that comes for free | 20:55 |
| dprince | EmilienM: pingtest is like deep cross service testing, that is meant to be as fast as possible | 20:56 |
| EmilienM | tempest has a lot of integration tests that come for free and we'll use it in the tripleo scenarios | 20:56 |
| *** gbarros has joined #tripleo | 20:56 | |
| dprince | EmilienM: "free" is nice. but it using anything is an investment | 20:56 |
| EmilienM | I know you want to keep pingtest but it has been proven to not being enough | 20:56 |
| EmilienM | we're deploying some services that can't be tested by pingtest | 20:56 |
| dprince | EmilienM: tools can evolve. All I'm saying | 20:57 |
| EmilienM | for example, ec2api | 20:57 |
| dprince | EmilienM: my concern would be when "free" accidentally gets us a runtime of +15 minutes | 20:57 |
| EmilienM | dprince: we don't have resources to have someone working on pingtest | 20:57 |
| dprince | EmilienM: and we didn't notice it | 20:57 |
| EmilienM | everyone is busy afiu | 20:57 |
| EmilienM | tempest comes for free | 20:57 |
| EmilienM | it's not 15m | 20:57 |
| EmilienM | we verified and the overlap is very minimal for now | 20:57 |
| EmilienM | and we'll be careful in what we run | 20:57 |
| dprince | EmilienM: that runtime isn't guaranteed though. COuld go up or down at any time | 20:58 |
| EmilienM | we discussed about that during the tripleo ci squad meeting this morning | 20:58 |
| dprince | EmilienM: my point is, sometimes having a trusted suite that changes less often is beneficial too | 20:58 |
| EmilienM | we can keep pingtest on ovb if you like | 20:58 |
| EmilienM | but for the scenarios, I see tempest an excellent fit | 20:58 |
| dprince | EmilienM: "free" isn't always free is all I'm saying. There is an invenstment in any tool. If we drop pingtest we need to invest in getting parity tests in Tempest that run just as fast. | 20:59 |
| dprince | EmilienM: otherwise, we've lost something | 20:59 |
| dprince | EmilienM: you think free is nice because you see tool X has some features we don't yet. That is nice | 21:00 |
| dprince | EmilienM: but it isn't everything | 21:00 |
| EmilienM | I've been running one single scenario that cover the same thing as ping test (boot from volume + vm connectivity) (actually tempest also cover SSH access) and we only lost 1 min40 | 21:01 |
| EmilienM | I'm pretty sure we can survive 1min40 and that will enable more testing that come from upstream projects | 21:01 |
| EmilienM | the key thing of tempest is now we're able to tell what doesn't work | 21:01 |
| EmilienM | which test doesn't work and notify the right teams with that | 21:02 |
| dprince | EmilienM: lots of subtle' thiings in booting an instance. How is it booted. Does it use a volume. Etc. | 21:02 |
| EmilienM | that's the thing we're working on https://trello.com/c/9a28fWKc/258-provide-a-way-to-notify-teams-when-a-specific-job-fails-in-a-specific-project | 21:02 |
| dprince | EmilienM: those things control what services get tested... and just leaving it to some other core team to adjust those when they want seems risky to me | 21:02 |
| EmilienM | dprince: yes it boot from volume like I said | 21:02 |
| *** trown is now known as trown|outtypewww | 21:02 | |
| dprince | EmilienM: the investment here is, watching it and making sure those tests stay as we think they are | 21:02 |
| dprince | EmilienM: are you signing up to do that? | 21:03 |
| EmilienM | dprince: we have a bunch of people who want to be involved in CI but don't know where to start. Having tempest in the game will help them to understand how do we test. Tempest is popular and easy to use. And it doesn't require Heat as a dependency. | 21:04 |
| EmilienM | again, we're trying here | 21:04 |
| EmilienM | we're not saying we'll do that for life, but we're trying to do better | 21:04 |
| EmilienM | any help is welcome... | 21:04 |
| *** tonyb has joined #tripleo | 21:05 | |
| dprince | EmilienM: I'm trying to help | 21:05 |
| dprince | EmilienM: :) | 21:05 |
| dprince | EmilienM: my concern is would Tempest gate on us | 21:05 |
| dprince | EmilienM: do we have resources for that | 21:06 |
| EmilienM | yes | 21:06 |
| EmilienM | I talked with them already | 21:06 |
| EmilienM | and yes | 21:06 |
| dprince | EmilienM: and does it actually prevent complexity like you say it does | 21:06 |
| EmilienM | we already have puppet jobs in their gates | 21:06 |
| dprince | EmilienM: for me it is more complext | 21:06 |
| EmilienM | since more than a year | 21:06 |
| dprince | EmilienM: but I know heat | 21:06 |
| weshay | ya.. we need it to start getting tripleo jobs running across other projects | 21:06 |
| weshay | ping test doesn't do the trick | 21:06 |
| EmilienM | it doesn't | 21:06 |
| EmilienM | and only a very small subset of people here thinks pingtest is better | 21:07 |
| weshay | and we desperately need more projects to run non-voting ooo jobs | 21:07 |
| EmilienM | we have been dealing with promotion pipelines for a long time now | 21:07 |
| weshay | I like the ping test.. but it's not great at everything | 21:07 |
| EmilienM | we know that pingtest is good but not enough. | 21:07 |
| dprince | EmilienM: that very small subset is likely the older TripleO core's | 21:07 |
| abishop | assistance required to triage https://bugs.launchpad.net/tripleo/+bug/1711462 | 21:08 |
| openstack | Launchpad bug 1711462 in tripleo "HCI derived parameters workflow not using NovaVcpuPinSet" [Undecided,New] - Assigned to Alan Bishop (alan-bishop) | 21:08 |
| abishop | it's medium importance, and I'd like it targeted for pike-rc1 | 21:08 |
| abishop | I assigned to myself and am already working on the fix | 21:08 |
| dprince | EmilienM: so, just like quickstart.... we are dividing the group here I think again | 21:08 |
| EmilienM | people who are driving decisions are these who are making the change | 21:08 |
| dprince | EmilienM: complexity... goes up. CI running 2 suites doesn't seem better. If you must choose... choose 1 | 21:08 |
| EmilienM | as far I can tell not everyone here works in CI | 21:08 |
| dprince | EmilienM: you can't make everyone happy | 21:08 |
| EmilienM | so if you don't like it, you join CI squad, participate and make the change | 21:09 |
| dprince | EmilienM: at the root of all this I feel like that is what is happening to an extent. | 21:09 |
| dprince | at the cost of complexity | 21:09 |
| EmilienM | but i'm tired to hear quickstart vs X or vs Y | 21:09 |
| EmilienM | abishop: do you need a fix in pike rc1? | 21:09 |
| dprince | As I'm tire to hear Tempest vs. Pingtest | 21:09 |
| abishop | yes | 21:09 |
| EmilienM | abishop: triaged | 21:09 |
| dprince | but you run both now... so I guess that says something | 21:10 |
| EmilienM | abishop: I'll add you to tripleo group in launchpad so next time you can do it yourself | 21:10 |
| abishop | EmilienM: thx | 21:10 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 21:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711425 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 21:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 21:10 |
| openstack | Launchpad bug 1711425 in tripleo "Impossible to ssh a VM when TripleO is containerized" [Critical,Triaged] | 21:10 |
| *** gbarros has quit IRC | 21:10 | |
| abishop | EmilienM: thx, even better :) | 21:10 |
| EmilienM | done | 21:10 |
| EmilienM | dprince: we run pingtest everywhere now, nothing changed | 21:11 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Generalize OVB settings to fit feature set pattern https://review.openstack.org/494659 | 21:12 |
| *** bugzy_ is now known as bugzy | 21:13 | |
| weshay | dprince, EmilienM I'm ssh'd into a cirros instance on a container deployment. | 21:14 |
| EmilienM | weshay: without an SSH key ;-) | 21:14 |
| EmilienM | you're still trying to cheat me :D | 21:14 |
| weshay | yes indeeed | 21:15 |
| EmilienM | dude you won't got me so easily | 21:15 |
| *** dprince has quit IRC | 21:19 | |
| openstackgerrit | Alan Bishop proposed openstack/tripleo-common master: Use NovaVcpuPinSet when deriving HCI parameters https://review.openstack.org/489239 | 21:21 |
| *** nyechiel has joined #tripleo | 21:21 | |
| openstackgerrit | OpenStack Release Bot proposed openstack/os-apply-config stable/pike: Update .gitreview for stable/pike https://review.openstack.org/494708 | 21:21 |
| openstackgerrit | OpenStack Release Bot proposed openstack/os-apply-config stable/pike: Update UPPER_CONSTRAINTS_FILE for stable/pike https://review.openstack.org/494709 | 21:21 |
| openstackgerrit | OpenStack Release Bot proposed openstack/os-collect-config stable/pike: Update .gitreview for stable/pike https://review.openstack.org/494710 | 21:21 |
| openstackgerrit | OpenStack Release Bot proposed openstack/os-collect-config stable/pike: Update UPPER_CONSTRAINTS_FILE for stable/pike https://review.openstack.org/494711 | 21:21 |
| openstackgerrit | OpenStack Release Bot proposed openstack/os-net-config stable/pike: Update .gitreview for stable/pike https://review.openstack.org/494712 | 21:21 |
| openstackgerrit | OpenStack Release Bot proposed openstack/os-refresh-config stable/pike: Update .gitreview for stable/pike https://review.openstack.org/494713 | 21:21 |
| openstackgerrit | OpenStack Release Bot proposed openstack/os-refresh-config stable/pike: Update UPPER_CONSTRAINTS_FILE for stable/pike https://review.openstack.org/494714 | 21:21 |
| openstackgerrit | OpenStack Release Bot proposed openstack/paunch stable/pike: Update .gitreview for stable/pike https://review.openstack.org/494715 | 21:22 |
| openstackgerrit | OpenStack Release Bot proposed openstack/paunch master: Update reno for stable/pike https://review.openstack.org/494716 | 21:22 |
| EmilienM | stevebaker: we released paunch and created stable/pike branch ^ | 21:22 |
| EmilienM | bfournie: ^ same for os-net-config | 21:23 |
| *** tosky has joined #tripleo | 21:24 | |
| *** slagle has quit IRC | 21:28 | |
| *** gbarros has joined #tripleo | 21:29 | |
| *** abishop has quit IRC | 21:31 | |
| *** jcoufal has quit IRC | 21:32 | |
| *** nyechiel has quit IRC | 21:32 | |
| *** ansmith has joined #tripleo | 21:34 | |
| pradk | can someone +a this https://review.openstack.org/#/c/494639/ | 21:37 |
| EmilienM | pradk: what if ceph isn't used as backend? | 21:37 |
| EmilienM | the mount will fail, isn't? | 21:38 |
| *** salmankhan has quit IRC | 21:38 | |
| *** oanson has quit IRC | 21:42 | |
| *** oanson has joined #tripleo | 21:44 | |
| EmilienM | larsks: hey, I'm testing upgrades from ocata to pike on scenario001 and seeing an issue with collectd, have you seen http://paste.openstack.org/show/618726/ already? | 21:45 |
| EmilienM | http://logs.openstack.org/00/461000/41/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container-upgrades-nv/62ff380/logs/subnode-2/var/log/messages.txt.gz#_Aug_17_21_12_53 | 21:46 |
| stevebaker | EmilienM: ok, thanks | 21:47 |
| EmilienM | larsks: nevermind, it's a quickstart thing, we're investigating | 21:53 |
| *** jmelvin has quit IRC | 21:53 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Revert "Make containerized nova-api run with httpd" https://review.openstack.org/494723 | 22:00 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: enable enable_opstools_repo where opstools are deployed https://review.openstack.org/494724 | 22:01 |
| EmilienM | mwhahaha: ^ | 22:01 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 22:02 |
| *** bfournie has quit IRC | 22:04 | |
| *** ecerquei_ has joined #tripleo | 22:07 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711425 | 22:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 22:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 22:10 |
| openstack | Launchpad bug 1711425 in tripleo "Impossible to ssh a VM when TripleO is containerized" [Critical,Triaged] | 22:10 |
| *** jlabarre has quit IRC | 22:11 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Restore nova metadata api docker service https://review.openstack.org/494727 | 22:13 |
| *** brault_ has joined #tripleo | 22:18 | |
| *** brault has quit IRC | 22:19 | |
| *** gbarros has quit IRC | 22:23 | |
| *** itlinux has quit IRC | 22:23 | |
| *** itlinux has joined #tripleo | 22:25 | |
| *** brault has joined #tripleo | 22:30 | |
| *** thrash is now known as thrash|g0ne | 22:31 | |
| *** brault_ has quit IRC | 22:32 | |
| *** openstackgerrit has quit IRC | 22:33 | |
| pradk | EmilienM, well we do the same for other containers like metricd and its working fine in swift case .. i think all thats doing really is trying to copy the files if available? .. and you were fine with https://review.openstack.org/#/c/482500/ as you +2 evidently ;) | 22:35 |
| *** itlinux has quit IRC | 22:35 | |
| *** bfournie has joined #tripleo | 22:36 | |
| *** bfournie has quit IRC | 22:37 | |
| owalsh | EmilienM: ssh works for me with https://review.openstack.org/494727 | 22:43 |
| *** kbyrne has quit IRC | 22:52 | |
| *** openstackgerrit has joined #tripleo | 22:54 | |
| openstackgerrit | Merged openstack/tripleo-common master: overcloud_containers.yaml.j2 map images to services https://review.openstack.org/448328 | 22:54 |
| *** kbyrne has joined #tripleo | 22:55 | |
| *** morazi has quit IRC | 23:07 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Restore nova metadata api docker service https://review.openstack.org/494727 | 23:08 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 23:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711262 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1711425 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 23:10 |
| openstack | Launchpad bug 1711262 in tripleo "tripleo job are overwriting nameserver with 8.8.8.8" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 23:10 |
| openstack | Launchpad bug 1711425 in tripleo "Impossible to ssh a VM when TripleO is containerized" [Critical,In progress] - Assigned to Oliver Walsh (owalsh) | 23:10 |
| *** achadha has quit IRC | 23:11 | |
| *** achadha has joined #tripleo | 23:11 | |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Filter container images by deployed services https://review.openstack.org/494367 | 23:25 |
| *** ecerquei_ has quit IRC | 23:27 | |
| *** tosky has quit IRC | 23:29 | |
| *** numans has quit IRC | 23:30 | |
| *** michapma has quit IRC | 23:31 | |
| *** achadha has quit IRC | 23:33 | |
| *** achadha has joined #tripleo | 23:33 | |
| *** achadha has quit IRC | 23:35 | |
| *** achadha has joined #tripleo | 23:35 | |
| *** achadha has quit IRC | 23:39 | |
| *** michapma has joined #tripleo | 23:45 | |
| EmilienM | owalsh: where can I see results? | 23:46 |
| EmilienM | pradk: are we sure it's not failing when swift is used? | 23:46 |
| owalsh | EmilienM: local env, but jaosorior propsed basically the same change earlier and you -1 it :-) | 23:47 |
| EmilienM | and dprince is right we need to split | 23:47 |
| owalsh | EmilienM: yup, spitting it now | 23:47 |
| EmilienM | well, we need to split services | 23:47 |
| *** ebarrera has joined #tripleo | 23:47 | |
| EmilienM | which is why we want containers, iiuc | 23:47 |
| EmilienM | owalsh: thanks for the work | 23:48 |
| owalsh | EmilienM: np | 23:48 |
| EmilienM | owalsh: I'm happy if it works, so we can move forward | 23:48 |
| EmilienM | owalsh: if you don't mind, in the meantime I'll deploy it in non-containerized in the CI | 23:49 |
| EmilienM | owalsh: so we can make progress | 23:49 |
| EmilienM | owalsh: ok I give up my workaround and adds your patch in depends-on | 23:56 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs010: enable tempest https://review.openstack.org/494284 | 23:58 |
| EmilienM | owalsh: if you upload a new PS, please recheck ^ | 23:58 |
| * EmilienM out | 23:58 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-common master: Add NovaMetadataApi to nova-api image params https://review.openstack.org/494750 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!