Wednesday, 2017-11-08

*** tosky has quit IRC00:03
*** salmankhan has quit IRC00:04
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172740600:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925300:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958600:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011100:10
openstackLaunchpad bug 1727406 in tripleo "Zaqar subscriptions failed to report deployment error" [Critical,Triaged] - Assigned to Thomas Herve (therve)00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047700:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)00:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)00:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]00:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)00:10
*** mary_grace has quit IRC00:11
*** mary_grace1 has joined #tripleo00:12
*** hewbrocca is now known as hewbrocca_afk00:16
*** gbarros has joined #tripleo00:20
*** zshi has joined #tripleo00:31
*** gbarros has quit IRC00:40
*** bobh has joined #tripleo00:41
*** gbarros has joined #tripleo00:44
openstackgerritGiulio Fidente proposed openstack/tripleo-common master: Fail ceph-ansible workflow if there is stale puppet-ceph hieradata  https://review.openstack.org/51147400:46
openstackgerritGiulio Fidente proposed openstack/tripleo-common master: Fail ceph-ansible workflow if there is stale puppet-ceph hieradata  https://review.openstack.org/51147400:46
*** zshi has quit IRC00:53
*** zshi has joined #tripleo00:53
*** dhill_ has quit IRC00:54
*** cschwede has quit IRC00:54
*** cschwede has joined #tripleo00:54
*** dhill_ has joined #tripleo00:54
*** rhallisey has quit IRC01:01
openstackgerritGiulio Fidente proposed openstack/tripleo-docs master: Update upgrade documentation with the Ceph related steps  https://review.openstack.org/51163501:05
*** gfidente has quit IRC01:05
*** ecerquei has joined #tripleo01:09
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172740601:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925301:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958601:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011101:10
openstackLaunchpad bug 1727406 in tripleo "Zaqar subscriptions failed to report deployment error" [Critical,Triaged] - Assigned to Thomas Herve (therve)01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047701:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)01:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)01:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]01:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)01:10
*** bobh has quit IRC01:16
*** cshastri has joined #tripleo01:17
*** bobh has joined #tripleo01:21
*** gbarros has quit IRC01:21
*** fragatina has quit IRC01:24
*** fragatina has joined #tripleo01:25
*** fragatina has quit IRC01:25
*** bobh has quit IRC01:32
*** d0ugal has quit IRC01:37
*** itlinux has joined #tripleo01:37
*** itlinux has quit IRC01:37
*** gkadam has joined #tripleo01:49
*** achadha has joined #tripleo01:55
*** achadha has quit IRC02:00
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Remove scenario001-multinode.yaml  https://review.openstack.org/51842302:00
*** mary_grace1 has quit IRC02:03
*** mary_grace has joined #tripleo02:06
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172740602:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925302:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958602:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011102:10
openstackLaunchpad bug 1727406 in tripleo "Zaqar subscriptions failed to report deployment error" [Critical,Triaged] - Assigned to Thomas Herve (therve)02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047702:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)02:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)02:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]02:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)02:10
*** jlabarre has quit IRC02:14
*** newmember has joined #tripleo02:15
*** jkilpatr has quit IRC02:21
*** shreshtha has quit IRC02:22
*** dhill_ has quit IRC02:26
*** dhill_ has joined #tripleo02:30
*** atoth has quit IRC02:31
*** mary_grace has quit IRC02:37
*** zshi has quit IRC02:38
*** zshi has joined #tripleo02:38
*** zshi has quit IRC02:46
*** gkadam has quit IRC02:55
*** zshi has joined #tripleo03:05
*** gvrangan has joined #tripleo03:07
*** moshele has joined #tripleo03:08
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172740603:10
openstackLaunchpad bug 1727406 in tripleo "Zaqar subscriptions failed to report deployment error" [Critical,Triaged] - Assigned to Thomas Herve (therve)03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925303:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958603:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011103:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047703:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)03:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)03:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]03:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)03:10
*** mdnadeem has joined #tripleo03:44
*** stendulker has joined #tripleo03:49
*** links has joined #tripleo03:51
*** rlandy has quit IRC03:51
*** fragatina has joined #tripleo03:51
*** fragatina has quit IRC03:53
openstackgerritRicardo Noriega proposed openstack/tripleo-heat-templates master: Fixing typos in the Bagpipe driver environment file  https://review.openstack.org/51843803:55
*** fragatina has joined #tripleo03:55
*** gbarros has joined #tripleo03:55
*** pdeore has joined #tripleo03:56
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates master: OVN: Provide the option to define NeutronBridgeMappings as a role parameter  https://review.openstack.org/51844004:01
*** gbarros has quit IRC04:01
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates master: OVN: Provide the option to define NeutronBridgeMappings as a role parameter  https://review.openstack.org/51844004:03
*** psachin has joined #tripleo04:05
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172740604:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925304:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958604:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011104:10
openstackLaunchpad bug 1727406 in tripleo "Zaqar subscriptions failed to report deployment error" [Critical,Triaged] - Assigned to Thomas Herve (therve)04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047704:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)04:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)04:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]04:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)04:10
*** moshele has quit IRC04:10
*** pdeore has quit IRC04:11
*** udesale has joined #tripleo04:12
*** aditya_r has joined #tripleo04:14
*** shreshtha has joined #tripleo04:15
*** dbecker has quit IRC04:21
*** ykarel|away has joined #tripleo04:29
*** anilvenkata has quit IRC04:34
*** anilvenkata has joined #tripleo04:34
*** gvrangan has quit IRC04:37
*** dbecker has joined #tripleo04:38
*** itlinux has joined #tripleo04:58
*** itlinux has quit IRC04:58
*** gvrangan has joined #tripleo05:02
*** gbarros has joined #tripleo05:05
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Add tags to baremetal cron removal tasks  https://review.openstack.org/51708205:07
*** ratailor has joined #tripleo05:08
openstackgerritDirk Mueller proposed openstack/diskimage-builder master: zypper-minimal: Set default locale env to C.UTF-8  https://review.openstack.org/51844705:09
openstackgerritMerged openstack/python-tripleoclient master: Refactor undercloud deploy to use Ansible.  https://review.openstack.org/51596705:09
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172740605:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925305:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958605:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011105:10
openstackLaunchpad bug 1727406 in tripleo "Zaqar subscriptions failed to report deployment error" [Critical,Triaged] - Assigned to Thomas Herve (therve)05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047705:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)05:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)05:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]05:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)05:10
*** jaganathan has joined #tripleo05:11
*** cylopez has joined #tripleo05:13
openstackgerritMerged openstack/tripleo-heat-templates master: Add docker templates to configure Ironic inspector  https://review.openstack.org/45782205:15
openstackgerritMerged openstack/tripleo-heat-templates master: Add condition to os-net-config run during upgrade.  https://review.openstack.org/51790505:15
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Enable Cinder as a backend for Glance  https://review.openstack.org/51610005:15
openstackgerritMerged openstack/python-tripleoclient master: Tripleoclient unified queue name.  https://review.openstack.org/50990205:15
*** ykarel|away is now known as ykarel05:17
*** cylopez has quit IRC05:17
*** aditya_r has quit IRC05:19
*** skramaja has joined #tripleo05:20
*** janki has joined #tripleo05:21
*** gvrangan has quit IRC05:26
*** cylopez has joined #tripleo05:35
*** aditya_r has joined #tripleo05:40
*** ckanade has joined #tripleo05:51
ckanadeAfter running ironic inspector local_gb uses smallest device. Is there any way to specify the device to ironic to use for local_gb05:52
Tenguhello ckanade, you should ask on #openstack-ironic, you'll get better answer as it's related to that project :)05:53
jaosoriorgood morning!05:54
Tenguhello jaosorior !05:54
*** ckanade has left #tripleo05:55
Tengujaosorior: your masked password has apparently been merged tonight in master :). Now we can go further with the other one.05:55
jaosoriorexcellent05:56
jaosoriorwell, gotta add tests for this one https://review.openstack.org/#/c/517894/05:56
Tengumight be good, indeed.05:57
Tengutests are life in such a huge code thing ;)05:58
Tengubtw, i'll try to add some tests to certodia as well. that will be interesting.05:58
Tengubut first: finish the ACME client.05:58
openstackgerritEmilien Macchi proposed openstack/instack-undercloud master: CI test - never merge  https://review.openstack.org/51845306:00
openstackgerritDan Prince proposed openstack/tripleo-heat-templates master: Add Docker service to the undercloud roles  https://review.openstack.org/51845406:02
*** abregman has joined #tripleo06:02
*** ansmith has quit IRC06:08
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172740606:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925306:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958606:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011106:10
openstackLaunchpad bug 1727406 in tripleo "Zaqar subscriptions failed to report deployment error" [Critical,Triaged] - Assigned to Thomas Herve (therve)06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047706:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)06:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)06:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]06:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)06:10
*** tzumainn has quit IRC06:10
*** ansmith has joined #tripleo06:14
openstackgerritRajesh Tailor proposed openstack/puppet-tripleo master: Add TLS for ec2api metadata service  https://review.openstack.org/51177206:17
lvdombrkrhello folks, who tried add lecencrypt certeficate to overcloud?06:17
*** moshele has joined #tripleo06:17
Tengume06:20
Tengulvdombrkr: doing a script for that right now - should be working by next week.06:20
Tengulvdombrkr: there are "some" issues with LE on an HA deploy (i.e. 3 controllers) due to the load-balancing. my script will overcome those issues.06:21
*** marios has joined #tripleo06:21
*** gbarros has quit IRC06:21
*** suuuper has joined #tripleo06:22
*** shreshtha has quit IRC06:22
*** shreshtha has joined #tripleo06:24
*** janki has quit IRC06:26
lvdombrkrTengu: i tried to do that manually right now (without HA), and when tried to do that get always error on deployment: http://paste.openstack.org/raw/625758/ , deploy commands what i use : http://paste.openstack.org/raw/625795/06:27
*** janki has joined #tripleo06:27
*** janki has quit IRC06:28
Tengulvdombrkr: hmm, I wouldn't do that with tripleo/heat.06:28
*** janki has joined #tripleo06:28
lvdombrkrTengu: mhm, but how?06:28
Tengulvdombrkr: basically, you'll need to use the webroot validation, unless you're using some certbot-friendly DNS provider like route53 (aws).06:29
Tengulvdombrkr: the path for the validation token is in /var/www (the documentroot for "horizon" httpd vhost)06:30
Tenguthat said, there might be a small chicken-and-egg issue with let's encrypt: haproxy does the TLS-endpoint, thus needs a TLS certificate to be started, but it can't have one because let's encrypt requires one to validate the domain. but let's encrypt can't do that because haproxy is down…06:31
*** oidgar has joined #tripleo06:32
Tengulvdombrkr: so the script I'm creating will take care of that kind of issue, and even checks whether haproxy is listening on ports 80/443, and chose if it must start a standalone server if nothing listens on http/https.06:32
jaosoriorlvdombrkr: what release are you trying out?06:33
Tengulvdombrkr: btw, you pushed the private key in the paste…06:33
lvdombrkrTengu: thanks! im using Ocata06:34
lvdombrkrjaosorior: ^06:34
jaosoriorlvdombrkr: can you try the same but disabling telemetry?06:34
lvdombrkrjaosorior: sure, give me some quick clues how to disable it06:35
lvdombrkrTengu: so if i use webroot i dont need to update overcloud after it?06:35
Tengulvdombrkr: indeed. but I strongly advice you to recreate the whole certificate: you just leaked its private key, so anyone can impersonate dev-openstack.makonix.com06:36
lvdombrkrTengu: yes sore, i will do that06:37
*** fragatin_ has joined #tripleo06:37
*** fragatina has quit IRC06:39
*** agurenko has joined #tripleo06:40
jaosoriorlvdombrkr: same command, but use the environment environments/disable-telemetry.yaml also06:40
*** dsariel has joined #tripleo06:41
*** brault has joined #tripleo06:47
*** aufi has joined #tripleo06:50
*** anshul has joined #tripleo06:52
lvdombrkrjaosorior, Tengu: before to try what excatly i need to put in inject-trust-anchor.yaml in lecencrypy case?06:53
*** yprokule has joined #tripleo06:53
Tengulvdombrkr: hmm nothing I think.06:54
TenguLet's encrypt is trusted on centos, debian and other linux.06:54
jaosoriorlvdombrkr: well, if your letsencrypt certificate is publicly trusted, as Tengu mentioned, you don't need to put anything.06:54
jaosoriorright06:54
lvdombrkrjaosorior: thanks i re-create certeficate and on the way to re-deploy06:55
*** aditya_r has quit IRC06:56
*** aditya_r has joined #tripleo06:57
Tenguhmm, more over, the trustanchor would be for internal TLS validation, in your case you're using let's encrypt for the public endpoint, lvdombrkr, right?06:57
lvdombrkrTengu: yes, right06:58
lvdombrkrTengu: in certbot webroot case which folder i need to link as a webroot ? /var/www/?06:58
*** chem has quit IRC06:59
Tengulvdombrkr: yup, /var/www - you can get that value from the /etc/httpd/conf.d/10-horizon_vhost.conf file, look for the "DocumentRoot" entry.06:59
Tengulvdombrkr: ah, certodia (my script) will allow to share the very same certificat between multiple nodes using a secure storage backend (not managed by certodia, and currently "only" custodia will be supported - a configuration example will be provided)07:00
*** jfrancoa has joined #tripleo07:02
*** janki has left #tripleo07:04
*** chem has joined #tripleo07:04
*** janki has joined #tripleo07:04
*** moshele has quit IRC07:04
*** aditya_r has quit IRC07:04
*** iranzo has joined #tripleo07:05
*** aditya_r has joined #tripleo07:08
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925307:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958607:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011107:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047707:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)07:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)07:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]07:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)07:10
lvdombrkrjaosorior: i need to destroy cloud and deploy new, or i can update existing with new certeficates?07:13
lvdombrkrTengu: uhh, i think it will be very useful07:13
jaosoriorlvdombrkr: depends, is it containerized and HA?07:14
jaosorioroh wait07:14
jaosoriorit was ocata07:14
jaosorioryeah, you could update07:14
lvdombrkrjaosorior: ok, thanks07:15
*** spectr has joined #tripleo07:15
*** aditya_r has quit IRC07:15
Tengulvdombrkr: pretty sure it will be indeed :).07:16
Tengulvdombrkr: and it will embed its own ACME client - meaning you won't need to add EPEL in order to get certbot command on your centos :).07:17
Tengu(and no subprocess for that part either, pretty clean)07:17
lvdombrkrTengu: yes, i just some sec ago added EPEL ))07:21
lvdombrkrnot realated issue with this07:21
Tengulvdombrkr: beware! adding epel without any filter will break your system07:22
lvdombrkrbut yesterday when tried update overcloud get several times error like : http://paste.openstack.org/raw/625796/07:22
lvdombrkri didnt try scale overcloud or something like just07:22
lvdombrkrtrying change some tls configuration07:23
Tenguit apparently contains some openstack-related lib that aren't the same version, and a `yum update' will replace them, and, in my case, I had a broken system after that (unable to update due to version conflict).07:23
Tengulvdombrkr: are all the hosts in an active state in ironic?07:23
lvdombrkrTengu:yes both hosts are active07:25
Tenguno maintenance?07:25
Tenguand you're sure you didn't change the numbers for the amount of hosts?07:26
*** ykarel is now known as ykarel|lunch07:26
lvdombrkrTengu: no maintenence, and im pretty sure that i keep same amount of hosts07:28
Tengumight be worth checking the latest statement.07:30
*** agurenko has quit IRC07:30
*** dparkes has joined #tripleo07:31
lvdombrkrTengu: when you update yum, you will not enable only update existing deployment, or also not able to create now one?07:32
Tengulvdombrkr: I don't understand that question07:33
lvdombrkrTengu, jaosorior : adding environments/disable-telemetry.yaml resolves issue, there no errors this time07:34
Tenguah. interesting.07:34
jaosoriorlvdombrkr: can you file a bug? Seems to be a bug in ceilometer.07:34
jaosoriorwe resolved the same thing in pike, so it seems that some patches will need to be backported.07:34
*** oidgar has quit IRC07:35
Tenguah, that's why I didn't see that issue in pike :D07:35
*** aditya_r has joined #tripleo07:35
lvdombrkrTengu: you mentioned that you updated yum and after that broke your system, as i understood..07:37
lvdombrkryou do that on undercloud or overcloud?07:37
*** jprovazn has joined #tripleo07:37
Tengulvdombrkr: ah, I enbled EPEL without any filter on the packages on the overcloud, and it was a bad idea. wait, I can provide the filtered content so that you can avoid any issue.07:38
Tengulvdombrkr: add this to your EPEL source:         includepkgs=certbot* python2-mock python2-future python2-dialog python-ndg_httpsclient python-parsedatetime python-psutil python-zope-component python-zope-event python2-acme python2-certbot python2-configargparse python2-pyrfc333907:38
Tenguhmm. darn. acme lib is also in epel.07:39
lvdombrkrjaosorior: sure i can, i will do that in a while07:39
Tenguanyway, it will be far better.07:39
*** abregman is now known as abregman|mtg07:40
lvdombrkrTengu: ooh, thanks, i will keep in mind07:40
Tenguyou're welcome ;).07:40
TenguI had some issues with certbot, as you may expect.07:40
Tenguthat's one of the reasons I want to make the certodia script.07:40
Tenguadding the capacity to manage haproxy backends on the fly, upload/fetch certificate from a secure backend, all of that is needed if you want let's encrypt on a multi-controller infra.07:41
*** agurenko has joined #tripleo07:48
*** dpawar has joined #tripleo07:49
*** achadha has joined #tripleo07:56
jaosoriorsshnaidm: hey, was there any update about the patches you had to test public TLS again?07:57
jaosoriorsshnaidm: it's a real bummer that it's not being tested in CI anymore :/07:57
lvdombrkrTengu: some weeks ago you wrote me something like " i just begin understood how tripleO works exactly etc.." i see you make a huge progress ))))08:00
Tengulvdombrkr: oh well… being on that little baby for over 2 months help a lot ;). More over, the certodia thing isn't really "openstack-only",it could be used in many other cases.08:02
*** achadha has quit IRC08:02
*** ebarrera has joined #tripleo08:04
*** ccamacho has joined #tripleo08:09
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925308:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958608:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011108:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047708:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)08:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)08:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]08:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)08:10
openstackgerritMerged openstack/tripleo-heat-templates master: Add containerized barbican environment  https://review.openstack.org/51734908:13
openstackgerritMerged openstack/tripleo-heat-templates master: Add validation task in docker services [Ceilometer]  https://review.openstack.org/50779608:13
openstackgerritMerged openstack/python-tripleoclient stable/pike: Add missing puppet configuration to undercloud deploy  https://review.openstack.org/51761308:13
openstackgerritMerged openstack/tripleo-heat-templates master: Env files for ODL deployments  https://review.openstack.org/51250108:13
openstackgerritMerged openstack/tripleo-heat-templates master: Allow containerized undercloud deploy with SELinux  https://review.openstack.org/51366908:13
openstackgerritMerged openstack/tripleo-heat-templates master: Only mount selinux sysfs in nova_libvirt container  https://review.openstack.org/51596608:13
openstackgerritMerged openstack/tripleo-quickstart-extras master: Simplify network_args  https://review.openstack.org/49968008:13
openstackgerritMerged openstack/python-tripleoclient master: Allow the user to run ansible update on all nodes  https://review.openstack.org/50971808:13
*** tesseract has joined #tripleo08:15
*** pgadiya has joined #tripleo08:19
*** ykarel|lunch is now known as ykarel08:21
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Fix static-inventory option for minor update  https://review.openstack.org/51497508:22
jaosoriormandre: you around?08:23
*** dsariel has quit IRC08:23
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Fix static-inventory option for minor update  https://review.openstack.org/51497508:24
mandrejaosorior: yep, for about 20 min08:24
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Fix static-inventory option for minor update  https://review.openstack.org/51497508:25
jaosoriormandre: got any suggestions for this https://review.openstack.org/#/c/518019/1 ?08:25
mandrejaosorior: what's in the default config for rsyslog?08:28
jaosoriormandre: nothing we can use, the default config is just a placeholder08:28
jaosoriormandre: what I want is a small sidecar image that will have that static configuration[08:28
mandrejaosorior: so rsyslog does nothing by default?08:28
jaosoriormandre: no. that would not be a good idea :)08:29
jaosoriormandre: but yeah, adding that configuration to be done by puppet is overkill08:29
jaosoriorwhich is why I tried to add it to the image08:29
jaosoriorcause we're gonna have two types of rsyslog containers08:29
jaosoriorone fullfledged server that forwards the logs to a centralized logging system08:30
mandrejaosorior: right... but it's not one size fits all, we're not the only consumers of kolla images08:30
jaosoriorand another small one that runs besides services to just output stuff to stdout08:30
jaosoriorwhich has a configuration that never changes08:30
mandreif you want to keep the config in the image, you'll have to convince eduardo it's useful for more than us08:30
jaosoriorok08:31
jaosoriorI'll try08:31
*** spectr has quit IRC08:33
*** spectr has joined #tripleo08:34
mandrejaosorior: UDP port 514 is standard for syslog, so you don't really have reason to change08:34
mandrejaosorior: but what creates the /sockets/log file?08:34
mandreI mean, what sends data to it?08:35
*** bogdando has joined #tripleo08:36
*** oidgar has joined #tripleo08:39
*** gkadam has joined #tripleo08:39
*** ffiore has joined #tripleo08:42
jaosoriormandre: so, it's a sidecar container, so by itself, nothing. something needs to mount a volume with the /sockets/ directory, and log to that.08:42
jaosoriormandre: which is the next bit. Make haproxy and swift to use /sockets/log instead of /dev/log08:43
*** jpena|off is now known as jpena08:45
*** adac has joined #tripleo08:46
*** jpich has joined #tripleo08:50
*** ffiore has quit IRC08:51
lvdombrkrTengu:  after i added epel repos to overcloud and update deployment i cant retrieve instances,volumes nothing... it can be really releated with epel addition?08:51
*** d0ugal has joined #tripleo08:52
Tengulvdombrkr: hm, don't know. Might be, especially if you didn't add the filter I gave here earlier.08:52
Tenguyou might want to check the status of the different services on the overcloud.08:52
Tengua `systemctl status' will show you if you have to worry, and `systemctl' will list you all the known services loaded or not, with their status (running, dead, and so on).08:53
openstackgerritOr Idgar proposed openstack/tripleo-common master: Add ansible playbook for configuring octavia  https://review.openstack.org/48745008:58
*** mcornea has joined #tripleo08:58
openstackgerritRadomir Dopieralski proposed openstack/tripleo-heat-templates master: Fix rights to local_settings.d for dockerized Horizon  https://review.openstack.org/51838408:59
*** amoralej|off is now known as amoralej09:00
openstackgerritRadomir Dopieralski proposed openstack/tripleo-heat-templates master: Fix rights to local_settings.d for dockerized Horizon  https://review.openstack.org/51838409:00
*** florianf has joined #tripleo09:01
*** Guest99448 has joined #tripleo09:03
*** ffiore has joined #tripleo09:04
lvdombrkrTengu, jaosorior: interesting if i tried deploy letsencrypt certificate to new overcloud ( not update existing) i get this error: http://paste.openstack.org/raw/625802/09:04
jaosoriorlvdombrkr: that's cause you didn't set the SSLKey, as you can see in the error message you posted.09:05
jaosoriorit's a required parameter if you enable TLS09:05
jaosoriorbrb lunch09:06
lvdombrkrjaosorior: damn.. thanks09:09
Tengulvdombrkr: btw, there's also an entry for the chaincert instead of passing it as the certificate. Won't really change, but at least it will be more readable. also, you might want to put the SSLcert/key/chain in some dedicated env file so that, next time you copy things, they won't leak (especially the private key - the others are public anyway)09:09
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925309:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958609:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011109:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047709:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)09:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]09:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)09:10
*** Guest99448 has quit IRC09:11
Tengulvdombrkr: the chain part being: SSLIntermediateCertificate (in the heat env file)09:13
*** lucas-afk is now known as lucasagomes09:14
owalshjaosorior: hey, so https://bugs.launchpad.net/tripleo/+bug/1730650... do we need that environment file to build docker images?09:17
openstackLaunchpad bug 1730650 in tripleo "cannot use enable-internal-tls.yaml due to /usr/share/openstack-tripleo-heat-templates/extraconfig/nova_metadata/krb-service-principals.yaml doesn't exist" [Critical,Triaged]09:17
*** mary_grace has joined #tripleo09:18
owalshjaosorior: runs fine without it AFAICT, but am I missing something?09:18
*** aditya_ra has joined #tripleo09:19
*** moshele has joined #tripleo09:22
*** aditya_r has quit IRC09:22
lvdombrkrTengu: sorry, im a liitle bit confused about SSLIntermediateCertificate, so except what i already added: sslkey  and sslcerteficate i need to add something else to deploy command?09:25
*** dpawar has quit IRC09:26
*** dpawar has joined #tripleo09:26
Tengulvdombrkr: not "need", it's optional, and allows to get a better understanding about what file is what. you can keep the things like you have done until now.09:26
*** tosky has joined #tripleo09:27
*** myoung has quit IRC09:29
*** abregman|mtg is now known as abregman09:30
*** aditya_ra has quit IRC09:30
*** mary_grace has quit IRC09:31
*** aditya_r has joined #tripleo09:31
*** moshele has quit IRC09:32
*** myoung has joined #tripleo09:38
*** sshnaidm has quit IRC09:39
*** derekh has joined #tripleo09:41
*** ffiore has quit IRC09:43
*** colonwq has quit IRC09:43
*** sdake has quit IRC09:43
*** NobodyCam has quit IRC09:43
*** bryan_att has quit IRC09:43
*** gregwork has quit IRC09:43
*** Vijayendra has quit IRC09:43
*** jidar has quit IRC09:43
*** jbadiapa has quit IRC09:43
*** lvdombrkr has quit IRC09:43
*** aputtur has quit IRC09:43
*** amoralej has quit IRC09:43
*** vkhanna has quit IRC09:43
*** radez has quit IRC09:43
*** akrzos has quit IRC09:43
*** saneax has quit IRC09:43
*** tonyb has quit IRC09:43
*** gchamoul has quit IRC09:43
*** zul has quit IRC09:43
jaosoriorowalsh: no, you don't need that file.09:44
*** gfidente[m] has quit IRC09:47
owalshjaosorior: ack, guess we need to do something about it in case we add jinja2 templating to an environment file that we do need, but not a critical bug09:47
*** dsariel has joined #tripleo09:48
*** ffiore has joined #tripleo09:49
*** colonwq has joined #tripleo09:49
*** sdake has joined #tripleo09:49
*** gregwork has joined #tripleo09:49
*** NobodyCam has joined #tripleo09:49
*** bryan_att has joined #tripleo09:49
*** Vijayendra has joined #tripleo09:49
*** jidar has joined #tripleo09:49
*** jbadiapa has joined #tripleo09:49
*** lvdombrkr has joined #tripleo09:49
*** aputtur has joined #tripleo09:49
*** amoralej has joined #tripleo09:49
*** vkhanna has joined #tripleo09:49
*** radez has joined #tripleo09:49
*** akrzos has joined #tripleo09:49
*** saneax has joined #tripleo09:49
*** tonyb has joined #tripleo09:49
*** gchamoul has joined #tripleo09:49
*** zul has joined #tripleo09:49
*** percevalbot has quit IRC09:49
*** moshele has joined #tripleo09:49
*** moshele has quit IRC09:49
*** moshele has joined #tripleo09:49
*** mhenkel has quit IRC09:49
*** dtantsur|afk is now known as dtantsur09:51
*** percevalbot has joined #tripleo09:51
*** sshnaidm has joined #tripleo09:51
*** mhenkel has joined #tripleo09:53
*** salmankhan has joined #tripleo09:53
openstackgerritDougal Matthews proposed openstack/tripleo-common-tempest-plugin master: Simplify the tox.ini  https://review.openstack.org/51850509:57
openstackgerritDougal Matthews proposed openstack/tripleo-common-tempest-plugin master: Update the .gitignore  https://review.openstack.org/51850609:57
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates master: OVN: Provide the option to define NeutronBridgeMappings as a role parameter  https://review.openstack.org/51844009:57
*** gfidente[m] has joined #tripleo09:59
*** dougbtv has joined #tripleo09:59
*** dougbtv_ has quit IRC10:00
*** dougbtv_ has joined #tripleo10:01
*** dougbtv has quit IRC10:04
*** hanish has joined #tripleo10:06
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add timestamper_cmd for tripleo-validations  https://review.openstack.org/49001310:08
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add pre-deployment negative tests for validations  https://review.openstack.org/48849510:08
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add post-deployment negative tests for validations  https://review.openstack.org/50401410:08
openstackgerritDougal Matthews proposed openstack/tripleo-common-tempest-plugin master: Fix gating on tripleo-common-tempest-plugin  https://review.openstack.org/51850510:09
openstackgerritDougal Matthews proposed openstack/tripleo-common-tempest-plugin master: Update the .gitignore  https://review.openstack.org/51850610:09
*** rdopiera has joined #tripleo10:09
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925310:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958610:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011110:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173047710:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)10:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]10:10
openstackLaunchpad bug 1730477 in tripleo "legacy-puppet-syntax-3 job missing on instack-undercloud stable/newton branch" [Critical,In progress] - Assigned to Attila Darazs (adarazs)10:10
rdopieraHello gentle folk, I know you are very busy, but can I divert your attention for a moment to this patch? It has two +2s already and it just needs a little gentle push to be merged, and then we can backport it to Pike... https://review.openstack.org/#/c/514421/10:10
openstackgerritDougal Matthews proposed openstack/tripleo-common-tempest-plugin master: Fix gating on tripleo-common-tempest-plugin  https://review.openstack.org/51802510:10
openstackgerritDougal Matthews proposed openstack/tripleo-common-tempest-plugin master: Update the .gitignore  https://review.openstack.org/51850610:10
rdopieraWhen it merges, we will be able to test Horizon's plugins.10:11
openstackgerritDougal Matthews proposed openstack/tripleo-common-tempest-plugin master: Update the .gitignore  https://review.openstack.org/51850610:13
*** cshastri has quit IRC10:13
d0ugalrdopiera: looks okay to me (but I'm no t-h-t core). Are the CI failures anything to worry about?10:14
d0ugalI assume that is why it doesn't have a +W10:14
rdopierad0ugal: to be honest I have no idea, as I don't understand the test job output10:15
d0ugalah10:15
*** aditya_r has quit IRC10:15
hanishHi i am working on add a postconfig for compute node, but my yaml could not invoke the puppet manifest10:16
hanishCan you please help me on this10:16
rdopierad0ugal: the error seems to be: 2017-10-26 07:23:50.890348 | primary | sort: write failed: standard output: Broken pipe10:17
rdopierad0ugal: but I have no idea what that signifies10:17
*** moshele has quit IRC10:17
d0ugalrdopiera: I'm not sure either. Sorry :)10:19
rdopieraI will try a recheck...10:19
*** mrunge has quit IRC10:19
therverdopiera, The real error is usually before that10:20
*** mrunge has joined #tripleo10:22
rdopieratherve: like the cp: cannot stat ‘/home/zuul/workspace/logs/undercloud/home/zuul/tempest’: No such file or directory ?10:24
therverdopiera, No before again10:24
owalshrdopiera: looks like timeouts http://logs.openstack.org/21/514421/13/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq/b3d082d/job-output.txt.gz#_2017-10-26_07_26_09_92050110:25
rdopierathen maybe the recheck will help10:27
rdopieraowalsh: thank you10:30
lvdombrkrTengu: did i need to customize endpoints-public-ips.yaml file, when deploying with tls?10:30
lvdombrkrTengu: right now i keep it by default10:31
Tengulvdombrkr: you can keep it as is, it takes variables from within tripleo10:31
openstackgerritOr Idgar proposed openstack/tripleo-common master: Add ansible playbook for configuring octavia  https://review.openstack.org/48745010:32
lvdombrkrTengu: my problem i that when i enable tls with letcencrypt certeficate, i cannot retreive any information in dashboard but are services are running10:33
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Extract all errors from tripleo deployment  https://review.openstack.org/51640210:49
sshnaidmpanda|ruck, fixed ^^10:49
lvdombrkrTengu: maybe some ideas where start investigate?10:49
openstackgerritMerged openstack/tripleo-heat-templates master: Tell HAProxy container template that the public cert was autogenerated  https://review.openstack.org/51761510:52
*** adac has quit IRC10:52
*** adac has joined #tripleo10:52
*** udesale has quit IRC10:53
*** ykarel is now known as ykarel|afk10:56
*** dpawar has quit IRC10:57
*** shreshtha has quit IRC10:57
Tengulvdombrkr: I don't really understand the issue. you know services are running, but you can't access horizon, right? is haproxy running?11:01
*** mary_grace has joined #tripleo11:01
*** mary_grace has quit IRC11:01
*** dpawar has joined #tripleo11:01
*** stendulker has quit IRC11:03
jaosoriorlvdombrkr: depends, if you're using an FQDN instead of an IP address for the certificate, I suggest you use endpoints-public-dns.yaml instead.11:04
jaosoriorand make sure that the entry is avilable via your DNS server or the /etc/hosts entries.11:04
lvdombrkrTengu: after i deploying tls and getting into dashboard, i can retrieve any information, see links, https://pasteboard.co/GSGqGI2.png https://pasteboard.co/GSGqNoj.png11:04
jaosoriorlvdombrkr: anything in the logs?11:05
sshnaidmpanda|ruck, adarazs|rover FYI: https://bugs.launchpad.net/tripleo/+bug/173093111:07
openstackLaunchpad bug 1730931 in tripleo "Github.com is used in TripleO CI for cloning distgit repos" [Critical,Triaged]11:07
*** iranzo1 has joined #tripleo11:07
*** pchavva has joined #tripleo11:08
adarazs|roversshnaidm: hmm, I thought we solved this a while back. but maybe only for the non-distgit repos.11:08
*** Goneri has joined #tripleo11:08
Tenguo_O github.com unresolvable?! duh…11:08
sshnaidmadarazs|rover, well, I didn't know we need to clone distgit repos too11:08
adarazs|roversshnaidm: to build images with DLRN, for sure.11:09
sshnaidmadarazs|rover, packages you mean?11:09
adarazs|roversshnaidm: yes, I mean packages :)11:09
sshnaidmadarazs|rover, yeah, but not sure - where we need to clone them from? is it hosted in rdoproject too?11:10
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925311:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958611:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011111:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)11:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)11:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]11:10
*** iranzo1 is now known as iranzo1|AFK11:10
*** iranzo1|AFK is now known as iranzo111:10
adarazs|roversshnaidm: probably. the rdo infra team probably knows.11:10
lvdombrkrjaosorior: i think that i should use endpoints-public-dns.yaml as you mentioned11:10
*** dtantsur is now known as dtantsur|bbl11:11
jaosoriorlvdombrkr: sounds about right11:11
*** ramishra has quit IRC11:12
*** hanish has quit IRC11:12
*** pchavva has quit IRC11:12
jaosoriorTengu: do you happen to have links to the pastes I sent yesterday with examples on the JSON logging?11:13
Tengujaosorior: of course. checking my logs, 2s.11:14
Tengujaosorior: a request: https://paste.fedoraproject.org/paste/9bo47C~09in2vXF1k02CfA   and a stacktrace: https://paste.fedoraproject.org/paste/RA6AsS3SHPe2bGyfF6aggg11:15
*** thrash|g0ne is now known as thrash11:15
Tengujaosorior: ah, regarding the "multi-line as a json LIST": there's a small issue doing so: a list is not sorted.11:15
Tengujaosorior: this means the list output can be different than the input.11:15
Tengu~> a stacktrace content being scrambled is bad :/11:15
jaosoriorTengu: thanks11:16
jaosoriorTengu: dammit, that sure is11:16
jaosoriorTengu: lets open a bug report and fix that11:16
jaosoriorTengu: wait, isn't the list supposed to be ordered?11:17
Tenguin json? not 100% sure.11:17
*** ramishra has joined #tripleo11:17
*** panda|ruck is now known as panda|ruck|bbl11:17
jaosoriorTengu: maybe not in JSON, but in python at least.11:17
Tengujaosorior: unorderd.11:17
sshnaidmadarazs|rover, panda|ruck seems like my patch for devmode solved only one problem, but it still fails on ipmi errors. It was resolved by rlandy before in https://review.openstack.org/#/c/486595 but seems like was broken in last patches, but should be resolved by https://review.openstack.org/#/c/512798/11:18
Tengumeaning the process "I take my JSON and push it to an ELK" might scramble the pseudo-order.11:18
jaosoriorok11:18
jaosoriorthat's good to know11:18
Tenguwould be better to ensure this doesn't happen and stringify the list.11:18
jaosoriorgotta test another patch that makes that configurable.11:18
sshnaidmadarazs|rover, panda|ruck|bbl so if you can merge https://review.openstack.org/#/c/512798/, would be great11:18
adarazs|roversshnaidm: looking.11:18
jaosoriorTengu: I'11:18
Tengujaosorior: maybe that should be an option embedded in the "elk_friendly_json"11:18
jaosoriorright11:19
jaosoriorTengu: I'll initially propose not having that as a list by deafult11:19
jaosoriorif that doesn't fly, then I'll make it configurable11:19
Tengufine for me11:19
Tengujaosorior: need any help on that, do you want me to open the launchpad issue or anything?11:19
rdopieraThe recheck on the https://review.openstack.org/#/c/518384/ worked, can anybody give this patch a push so it can merge?11:20
jaosoriorTengu: uhm... maybe, if it's something that you have observed already (the list not being ordered) then we should definitely have a bug.11:20
Tenguwell, not observed, but it's a fact in JSON :)11:21
*** gfidente has joined #tripleo11:21
*** gfidente has quit IRC11:21
*** gfidente has joined #tripleo11:21
TenguI'll open it.11:22
Tenguoslo.log I guess?11:22
jaosorioryes11:22
adarazs|roversshnaidm: the tqe gate failed on it, but I think it's only because it's not based on https://review.openstack.org/51833611:23
sshnaidmadarazs|rover, yeah, same error11:24
*** jaganathan has quit IRC11:25
Tengujaosorior: https://bugs.launchpad.net/oslo.log/+bug/173093611:26
openstackLaunchpad bug 1730936 in oslo.log "JSON list are unordered" [Undecided,New]11:26
chemdciabrin: he, do you need help on this one https://bugs.launchpad.net/tripleo/+bug/1724636 (wrong permission of mysql at step3)11:28
openstackLaunchpad bug 1724636 in tripleo "[ocata to pike] upgrade failing at step3 during dbsync" [Critical,Triaged]11:28
*** rhallisey has joined #tripleo11:29
*** ramishra_ has joined #tripleo11:29
*** ramishra has quit IRC11:29
dciabrinchem, hmm yeah I would definitely benefit from having direct access to an env, I have the feeling this is a job misconfiguration rather than a bug in the code11:32
*** rhallisey has quit IRC11:33
jaosoriorTengu: awesome, thanks!11:33
*** rhallisey has joined #tripleo11:33
dciabrinchem will probably ping you this afternoon, but for a minor update issue11:33
Tenguwell, that's nothing, jaosorior - the real job is the correction ;).11:33
chemdciabrin: I'm going to trigger a new job as, maybe (as in idwid) this can help https://review.openstack.org/#/c/517909/11:33
*** dciabrin is now known as dciabrin|away11:34
*** iranzo1 is now known as iranzo1|AFK11:35
*** rhallisey has quit IRC11:36
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Fix wrong permission on database during mysql_init tasks.  https://review.openstack.org/51851311:36
*** skramaja has quit IRC11:38
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Upgrade CI test on stable/pike  https://review.openstack.org/50062511:41
*** dsariel has quit IRC11:42
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates master: OVN: Provide the option to define NeutronBridgeMappings as a role parameter  https://review.openstack.org/51844011:43
*** salmankhan has quit IRC11:44
*** abregman is now known as abregman|mtg11:44
*** adarazs|rover is now known as adarazs11:45
*** adarazs is now known as adarazs|rover11:45
*** pchavva has joined #tripleo11:45
*** abregman|mtg has quit IRC11:49
*** salmankhan has joined #tripleo11:50
d0ugalAnyone got a moment to review a patch that fixes the gate on a new tripleo repo? https://review.openstack.org/#/c/518025/11:55
*** ykarel|afk is now known as ykarel11:56
thrashd0ugal: looking11:57
*** dsariel has joined #tripleo11:57
gfidentemarios your comment in https://review.openstack.org/51147411:57
gfidentesounds a good idea to me but I am not sure how to put it in place11:57
gfidentewere you thinking about an upgrade task which does the same check?11:58
*** shreshtha has joined #tripleo12:01
lvdombrkrjaosorior: mhmm adding tls-endpoints-public-dns.yaml not helped12:02
mariosgfidente: sure, so it could definitely be a step0 validation task, and that is pretty early before anything major happens. i was even thinking you could add a bash one-liner here https://github.com/openstack/tripleo-heat-templates/blob/86a8ad9d897083e00bf0b7d94fe72e46fea4bcaa/environments/major-upgrade-composable-steps-docker.yaml#L7-L1112:02
mariosgfidente: that happens even before we run upgrade_tasks12:02
jaosoriorlvdombrkr: ok, what does it say in the logs? the UI messages are not very informative :/12:03
gfidentemarios bash sounds okay to me12:03
gfidenteon which nodes is that executed? all overcloud nodes?12:03
mariosgfidente: yeah, really the step0 validation sounds more 'proper' but that env file really is a requirement if you are running upgrade, so it is also 'proper' if you see what i mean. is just bash :)12:04
lvdombrkrjaosorior: with which logs is good start investigate?12:04
gfidentemarios I think bash runs too early though, the nodes have pre-existing and valid hieradata at that stage?12:04
gfidentewe need to refresh hieradata for the check to be 'effective'12:04
*** abregman|mtg has joined #tripleo12:05
jaosoriorlvdombrkr: nova API to start with. Since it couldn't retrieve instances.12:05
gfidenteso step0 is probably the way to go? has os-collect-config dumped new hieradata at step0 already?12:05
*** rhallisey has joined #tripleo12:05
*** ansmith has quit IRC12:05
*** lucasagomes is now known as lucas-hungry12:05
*** raildo has joined #tripleo12:05
openstackgerritMerged openstack/tripleo-common-tempest-plugin master: Fix gating on tripleo-common-tempest-plugin  https://review.openstack.org/51802512:06
*** dsariel has quit IRC12:06
mariosgfidente: ah right yeah you'd have to check the sequencing for the new hiera but this is where the init stuff happens (per node yeah) https://github.com/openstack/tripleo-heat-templates/blob/86a8ad9d897083e00bf0b7d94fe72e46fea4bcaa/puppet/role.role.j2.yaml#L509-L519 and so it happens on each stack update, which is why we unset it here12:06
marioshttps://github.com/openstack/tripleo-heat-templates/blob/86a8ad9d897083e00bf0b7d94fe72e46fea4bcaa/environments/major-upgrade-converge-docker.yaml#L9-L1012:07
*** amoralej is now known as amoralej|lunch12:07
*** ffiore has quit IRC12:07
mariosgfidente: so yeah step0 should be fine for the new hiera so that sounds like the right thing to do. it is after all a validation?12:07
*** ramishra_ has quit IRC12:07
gfidentemarios yep12:07
gfidenteI'll see if I can put up something12:08
gfidentethough the change for tripleo-common is also valid imho12:08
gfidenteI'd merge both12:08
mariosgfidente: yeah that is why i said " also" :)12:08
*** dsariel has joined #tripleo12:08
gfidentemarios okay, thanks!12:08
mariosgfidente: because it is in particular the upgrade case that would suffer more from a tripleo-common only check12:08
gfidenteyep I get it12:08
mariosgfidente: so in this case, lets also guard by adding this step012:08
gfidenteack12:08
mariosgfidente: so if you see this diagram12:08
mariosgfidente: sec i have some slides12:09
*** ffiore has joined #tripleo12:09
mariosgfidente: thanks :)12:09
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925312:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958612:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011112:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)12:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)12:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]12:10
lvdombrkrjaosorior: there is no information in nova-api log, or neutron server.log about any requests or errors12:11
Tengudarn… what's the command already in order to ensure nova-compute service is running on a node?12:11
jaosorior  lvdombrkr I'm not very acquainted with debugging horizon.12:11
Tengulvdombrkr: might be in horizon (openstack-dashboard) logs directly, or even in keystone.12:12
jaosoriorTengu: you should do openstack hypervisor list to list them all, then hypervisor show?12:12
Tenguah12:12
Tenguhmmm.12:13
Tenguno,there's another one.12:13
jaosoriornova service-list ?12:13
jaosoriorI think the equivalent would be openstack compute service list12:13
*** ramishra has joined #tripleo12:14
Tenguthat's it!12:14
Tenguthanks12:14
Tenguhmm but that's not the issue I have. ook.12:14
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Remove workaround for LP#1701239 bug  https://review.openstack.org/51124212:14
lvdombrkrTengu, jaosorior: mhm, seems something useful http://paste.openstack.org/raw/625826/12:15
*** ffiore has quit IRC12:15
Tengugrmbl. what might raise a "No valid host was found. No valid host found for resize (HTTP 400) (Request-ID: req-06e936f4-5141-46b0-9c61-0f0e4c36ec64)" when we want to resize an instance, when there are only two "hosts", and both of them have plenty of space, resources and so on?12:17
*** atoth has joined #tripleo12:17
Tengulvdombrkr: apparently haproxy isn't working as expected.12:17
Tengulvdombrkr: ah, no. did you actually activate internal-TLS ?12:17
Tenguseems to be the case, seeing the URI schem. Unable to establish connection to https://overcloud.localdomain <- httpS12:18
Tengudid you partially activate that without fully enabling internal-TLS support?12:18
lvdombrkrTengu: yes i activate only public tls12:20
Tengulvdombrkr: apparently, something partially activated internal TLS as well.12:20
jaosoriorlvdombrkr: that seems to be the issue. you need to set CloudName to point to the domain that you want to use12:20
*** pgadiya has quit IRC12:21
Tengujaosorior: ah, didn't think about that one. good catch!12:21
lvdombrkrjaosorior, Tengu:  additional .yaml to deploy commend with something like :parameter_defaults:12:23
lvdombrkr    CloudName: domain12:23
lvdombrkr?12:23
Tenguyup12:23
Tenguor in the environment file you already have.12:24
Tenguno need to create 10'000 env file.12:24
tdasilvamarios: hello! do you have a minute to take a look at this patch https://review.openstack.org/#/c/517209 ?12:25
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Fix /etc/openstack-dashboard/ permissions for access to *policy.json  https://review.openstack.org/51510512:26
lvdombrkrTengu, jaosorior: folks, its my third overcloud update in last 1.5 hours, i cross fingers ))12:26
jaosoriorlvdombrkr: good luck!12:26
*** catintheroof has joined #tripleo12:27
Tengulvdombrkr: well, I had to deploy over 50 times the overcloud before I could even dream of an update XD.12:28
mariostdasilva: o/ ack lgtm12:28
tdasilvamarios: thanks! just need a +A now :)12:29
mariostdasilva: yah was just poking at the failed container job12:29
mariostdasilva: and looks unrelated but12:31
*** abregman|mtg has quit IRC12:31
mariostdasilva: i think we need to hold on merging for a bit yet http://lists.openstack.org/pipermail/openstack-dev/2017-November/124294.html12:31
*** abregman has joined #tripleo12:32
*** jkilpatr has joined #tripleo12:32
*** shreshtha has quit IRC12:32
tdasilvamarios: ah! thanks for the heads up12:32
*** panda|ruck|bbl is now known as panda|ruck12:33
*** ffiore has joined #tripleo12:33
*** catintheroof has quit IRC12:33
mariostdasilva: np12:34
sri_beagles: Hi, quick question is there any support for "networking-ovs-dpdk" in tripleo ?12:35
*** d0ugal has quit IRC12:36
beaglessri_, there is support for DPDK in tripleo12:36
beaglessri_, what do you mean by "networking-ovs-dpdk"?12:37
sri_beagles: cool, I mean by default we are not enableing dpdk in tripleo right ?12:38
*** ratailor has quit IRC12:38
beaglessri_, that's right12:38
sri_beagles: ok, we can enable dpdk by jest using -e environments/neutron-ovs-dpdk.yam !!, do i need pass any extra configuration ?12:40
*** catintheroof has joined #tripleo12:40
*** raildo has quit IRC12:40
*** brault has quit IRC12:40
*** raildo has joined #tripleo12:41
*** brault has joined #tripleo12:41
beaglessri_, https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ovs_dpdk_config.html12:41
sri_beagles: thanks :)12:42
*** catintheroof has quit IRC12:43
openstackgerritMerged openstack/python-tripleoclient stable/pike: Deprecate --pull-source for container prepare command  https://review.openstack.org/51804012:45
openstackgerritMerged openstack/python-tripleoclient stable/pike: Rename options to the prepare command for clarity  https://review.openstack.org/51804112:45
openstackgerritMerged openstack/python-tripleoclient master: Support to drive undercloud deploy via undercloud.conf  https://review.openstack.org/51135012:45
openstackgerritMerged openstack/puppet-tripleo stable/pike: Add new MySQL server option to mysql_bundle  https://review.openstack.org/51792012:45
*** agurenko has quit IRC12:45
*** udesale has joined #tripleo12:46
lvdombrkrTengu: no luck unfortunately http://paste.openstack.org/raw/625831/ ...yesterday i imagine that adding certificate will be easy game ))12:46
*** d0ugal has joined #tripleo12:47
lvdombrkrmaybe i need to destroy and deploy from zero12:48
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add role to setup any centos vm to look like nodepool vm  https://review.openstack.org/51741112:48
*** mary_grace has joined #tripleo12:50
*** mary_grace has quit IRC12:50
*** shreshtha has joined #tripleo12:52
openstackgerritDamien Ciabrini proposed openstack/puppet-tripleo stable/pike: Galera: add support for encrypted SST  https://review.openstack.org/51852112:53
*** dciabrin|away is now known as dciabrin12:54
*** jpena is now known as jpena|lunch12:55
*** links has quit IRC12:56
*** jlabarre has joined #tripleo12:57
*** rlandy has joined #tripleo12:59
*** shreshtha has quit IRC13:01
*** shreshtha has joined #tripleo13:01
*** ffiore has quit IRC13:01
*** pradk has quit IRC13:01
*** brault has quit IRC13:03
*** rfolco has quit IRC13:03
*** lucas-hungry is now known as lucasagomes13:03
*** sri__ has joined #tripleo13:04
*** sri_ has quit IRC13:05
*** trown|outtypewww is now known as trown13:06
*** rfolco has joined #tripleo13:08
*** liverpooler has quit IRC13:08
*** abishop has joined #tripleo13:09
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925313:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958613:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011113:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)13:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)13:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]13:10
*** brault has joined #tripleo13:11
*** moshele has joined #tripleo13:11
dciabrinweshay, hey I'm trying to understand why major upgrade fails (https://bugs.launchpad.net/tripleo/+bug/1724636), I'd very much appreciate to have direct access to an env. can you help?13:17
openstackLaunchpad bug 1724636 in tripleo "[ocata to pike] upgrade failing at step3 during dbsync" [Critical,Triaged]13:17
dciabrinchem bandini  ^^13:17
*** aditya_r has joined #tripleo13:19
chemdciabrin: have you seen my last comment on this one, basically I'm waiting for a new run on https://review.openstack.org/#/c/500625/13:19
bandinidciabrin: I think we can check via https://review.openstack.org/#/c/500625/ if we have fresh results?13:19
bandiniwhat chem says :)13:19
*** moshele has quit IRC13:19
chembandini: dciabrin do you think that https://review.openstack.org/#/c/518513/ could be in any way related or did I just exposed my complete lack of understanding ?13:20
dciabrinchem bandini oh excellent. let's wait for the run then13:20
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Clone kubespray to location accessible by the user  https://review.openstack.org/51837613:21
bandinichem: it looks sane to me. I am surprised things worked without that. dciabrin, ideas?13:21
dciabrinbandini, I'm trying to figure out in which case this would work and in which case it wouldn't13:22
jaosoriormandre: could you check this out https://review.openstack.org/#/c/517076/ ?13:23
openstackgerritLuke Hinds proposed openstack/tripleo-heat-templates master: Implements AIDE Intrusion Detection System  https://review.openstack.org/48302813:24
lvdombrkrTengu, jaosorior: folks thanks in assist , i deployed tls, adding CLOUDNAME helped after re-creating overcloud13:26
jaosoriornice :)13:26
*** aditya_r has quit IRC13:26
openstackgerritLuke Hinds proposed openstack/puppet-tripleo master: Implements AIDE Intrusion Detection System  https://review.openstack.org/48302513:27
dciabrinbandini, chem I don't think rw is needed, the puppet code will contact the mariadb server which is running in its own container ("mysql") which should have rw access to it13:27
*** ansmith has joined #tripleo13:28
honzaflorianf: hey, would you mind taking a look at this patch?  https://review.openstack.org/#/c/516674/13:28
*** agurenko has joined #tripleo13:29
fultonjwould anyone mind reviewing https://review.openstack.org/#/c/516224 ? thanks13:29
florianfhonza: on it13:29
honzaflorianf: sweet, thanks man13:29
chemdciabrin: well rw was needed for the related bug I think, just wondering if this would help that issue as well13:29
*** tzumainn has joined #tripleo13:29
*** tvignaud has quit IRC13:29
chemdciabrin: it seems that myslq_init wasn't doing anything useful13:29
*** amoralej|lunch is now known as amoralej13:29
openstackgerritLuke Hinds proposed openstack/tripleo-heat-templates master: Implements AIDE Intrusion Detection System  https://review.openstack.org/48302813:29
TenguI think I just hit a nice bug in nova.13:30
Tenguthat one will hurt.13:30
*** hewbrocca_afk is now known as hewbrocca13:31
*** ffiore has joined #tripleo13:32
openstackgerritJustin Kilpatrick proposed openstack/tripleo-quickstart-extras master: Enable OVN deployment  https://review.openstack.org/50372813:33
florianfhonza: hilarious comment on RolesStep.js :-)13:34
*** cylopez has quit IRC13:35
*** aputtur_ has joined #tripleo13:36
honzaflorianf: oh yeah?13:36
*** ansmith_ has joined #tripleo13:36
jaosoriormandre: thanks13:36
*** moshele has joined #tripleo13:36
*** udesale has quit IRC13:36
florianfhonza: the general critique of the English language. ;-)13:37
honzaflorianf: about capitals?  remind you of some other language?13:37
honzaflorianf: haha, yeah13:37
mandrejaosorior : np13:37
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates master: Add CephAnsiblePlaybookVerbosity parameter to overcloud-ceph-ansible  https://review.openstack.org/51838313:37
*** aditya_r has joined #tripleo13:37
*** dsariel has quit IRC13:41
*** liverpooler has joined #tripleo13:44
*** zshi has quit IRC13:44
Tenguuhu, even better. the issue is not related to nova, but to oslo.rootwrap, in fact. It fails in a nasty corner case.13:46
*** tvignaud has joined #tripleo13:47
dciabrinchem: wow that's odd that we didn't spotted that ro flag earlier... fun fact it's already rw in the pacemaker profile13:47
chemdciabrin: sad fact is that we can't share more between the two13:48
chemdciabrin: but +1 on pacemaker :)13:48
dciabrinchem agreed, it's far from ideal :(13:48
*** pchavva has quit IRC13:49
weshaymwhahaha, is the queue really at 8hr or did you slice and dice it?13:50
mwhahahai didn't touch it13:50
openstackgerritMerged openstack/puppet-pacemaker master: Allow multiple stonith declarations in one manifest  https://review.openstack.org/51425213:50
mwhahahabut we did drop non-containerized jobs13:50
mwhahahaso13:50
mwhahahamaybe that helped13:50
jankimandre, hey, if a service inside container dies, does tripleo restart it?13:51
openstackgerritJustin Kilpatrick proposed openstack/tripleo-quickstart-extras master: Enable OVN deployment  https://review.openstack.org/50372813:51
mwhahahaweshay: https://review.openstack.org/#/c/518413/413:51
mwhahahanot sure the impact on coverage yet so i'm going ot take a look this morning13:51
mandrejanki: yes, all containers have the 'restart: always' option set, docker will restart them if they die13:51
jankimandre, so container in itself is restarted or the service inside the same container is restarted?13:52
*** ykarel is now known as ykarel|away13:52
openstackgerritMerged openstack/puppet-pacemaker master: Use resource collectors for ordering in stonith manifests  https://review.openstack.org/51426613:53
*** oidgar has quit IRC13:53
mandrejanki: the container restart13:53
*** lblanchard has joined #tripleo13:54
*** marrusl has quit IRC13:54
jankimandre, got it. and if I manually delete a container post deployment, can I bring it back up again? so redeployment of overcloud is the only option?13:54
mandrejanki: if you stop a container, you can restart it, if you delete it, then it's gone13:55
jankimandre, ack. thanks13:55
mandrejanki: but you can recreate it with paunch13:56
jankimandre, how?13:56
openstackgerritRadomir Dopieralski proposed openstack/tripleo-heat-templates stable/pike: Fix rights to local_settings.d for dockerized Horizon  https://review.openstack.org/51853313:56
openstackgerritMerged openstack/tripleo-validations master: Fix typos in pacemaker module documentation  https://review.openstack.org/51833013:56
mandrejanki: https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/tips_tricks.html#debugging-with-paunch13:57
*** jmelvin has joined #tripleo13:57
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements  https://review.openstack.org/51163313:59
jankimandre, cool..also I want to test few puppet changes and I tried DeployArtifacts but didnot seem to work with containers. I need to change some haproxy manifests. How I do go around it?13:59
*** pradk has joined #tripleo13:59
beaglesbandini, bounce a question off of you? if I have a situation where it looks like a service is being added to haproxy.cfg without a pacemaker deployment and it is missing when it is a pacemaker deployment - where should I look for where "breakage"?13:59
*** jpena|lunch is now known as jpena13:59
jankimandre, would chaging in /var/lib/config-data/puppet-generated/haproxy work? (I am not sure if there actually is such folder)14:00
*** mary_grace has joined #tripleo14:01
*** mary_grace has quit IRC14:01
amoralejjfrancoa, i'm using https://review.openstack.org/#/c/518367 to run some experimental jobs i need to test14:03
beaglesbandini, might be that there is no correlation - I need to look deeper but thought I'd bounce this off of you anyways14:03
amoralejyou can ignore results14:03
bandinibeagles: sure, sec14:03
openstackgerritAndy Smith proposed openstack/puppet-tripleo master: WIP: Updates to separate messaging backends alternative  https://review.openstack.org/51068414:04
*** mdnadeem has quit IRC14:04
bandinibeagles: is this pike/queens? both pacemaker haproxy profiles include ::tripleo::profile::base::haproxy which ends up including tripleo::haproxy14:05
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Add new SELinux validation check  https://review.openstack.org/51121114:06
beaglesbandini, pike as it happens I think14:06
bandinibeagles: my hunch is that if you are using containers that some haproxy init container is not being called in HA (for whatever reason)14:06
*** ykarel|away has quit IRC14:06
*** toure_biab is now known as toure14:06
beaglesbandini, that's what I thought from the code .. oh could be. I'll look14:07
weshaymwhahaha, that patch helped that much?14:07
weshaywow14:07
mandrejanki: are you saying that uploading the new puppet modules and then re-deploying doesn't pick up your puppet changes?14:08
weshaypanda|ruck, adarazs|rover, mwhahaha how do things look in your humble opinions? move ci back to green?14:08
*** marrusl has joined #tripleo14:08
jankimandre, ya. I followed shardy's blog and tried. http://hardysteven.blogspot.in/2016/08/tripleo-deploy-artifacts-and-puppet.html14:08
*** spectr has quit IRC14:09
*** spectr has joined #tripleo14:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432814:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925314:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958614:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011114:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]14:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)14:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)14:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]14:10
adarazs|roverweshay, panda|ruck, mwhahaha: the gate looks good to me, but I'm not sure if this is only because of the submit-ban. we have quite decent gate-job pass rate today. 94.5% up from 91.8% of yesterday.14:10
mandrejanki: odd, there is nothing special to do for containers... check that /etc/puppet/modules has your changes on the deployed nodes14:11
jankimandre, will do. ALso I am hit with https://bugzilla.redhat.com/show_bug.cgi?id=1496203. I didnt find any upstream bug for this? There is a workaround and a patch in review that you are aware of?14:12
openstackbugzilla.redhat.com bug 1496203 in instack-undercloud "tripleo quickstart fails with error lvs: lvm lvs --help: Invalid units specification" [Unspecified,New] - Assigned to jslagle14:12
panda|ruckI think it is for the submit ban, and that proves that we have to move in two directions: fix the gates, and limit the queue length.14:13
panda|ruckadarazs|rover: ^14:13
*** moshele has quit IRC14:13
mandreadarazs|rover, panda|ruck: I bet this has to do with the puppet-zaqar fix that merged in master and stable/pike14:16
*** lblanchard has quit IRC14:17
mandrehttps://review.openstack.org/#/c/517889/14:17
panda|ruckmandre: how did this help the queue ?14:18
mandrepanda|ruck: first, failing jobs wouldn't be sitting there for hours, and 2. this greatly reduced the number of failed jobs, and thus the queue being reset14:20
jfrancoaamoralej: ok, no problem. thanks for the heads up14:21
owalshmwhahaha: hey, I see https://review.openstack.org/506595 just got +W. It depends on https://review.openstack.org/513867, could you take a look?14:21
owalshmandre: would you mind reviewing the latest patchset in https://review.openstack.org/513867?14:22
*** eck`gone is now known as eck`14:22
mandreowalsh: looks good, thanks for making the changes14:24
panda|ruckmandre: fair enough, we'll see what happens when the ban is lifted14:25
mandrepanda|ruck: from what I saw the ban wasn't very well respected ;-)14:28
*** panda|ruck is now known as panda|ruck|brb14:29
mandreI guess a lot of devs didn't know about the ban14:29
*** cdearborn has joined #tripleo14:29
*** etingof has quit IRC14:29
*** udesale has joined #tripleo14:31
*** hewbrocca is now known as hewbrocca_afk14:32
*** udesale has quit IRC14:34
*** moshele has joined #tripleo14:34
*** udesale has joined #tripleo14:34
*** hjensas has quit IRC14:35
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/pike: Refactor cellv2 host discovery logic to avoid races  https://review.openstack.org/51854614:35
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/pike: Update default cell_v2 cell when it already exists  https://review.openstack.org/51338314:36
*** chem has quit IRC14:37
*** ukalifon has joined #tripleo14:37
*** ykarel|away has joined #tripleo14:38
*** abregman is now known as abregman|mtg14:38
*** ykarel|away is now known as ykarel|afk14:38
*** chlong has joined #tripleo14:38
openstackgerritOliver Walsh proposed openstack/puppet-tripleo stable/pike: Unset MountFlags in docker.service systemd directives  https://review.openstack.org/51854814:40
*** chem has joined #tripleo14:40
*** lblanchard has joined #tripleo14:42
*** pdeore has joined #tripleo14:42
jfrancoamarios: these seem to be the jobs they plan to remove from CI upstream https://review.openstack.org/#/c/518405/ and leave only the upgrade jobs in RDO cloud14:43
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: nova-placement: switch auth_uri to keystone versionless endpoint  https://review.openstack.org/51474114:43
*** shreshtha_ has joined #tripleo14:45
jaosoriorowalsh: can you check this out https://review.openstack.org/#/c/517927/ ?14:45
*** shreshtha has quit IRC14:45
jaosoriorweshay, panda|ruck|brb, sshnaidm, adarazs|rover: Would really cool if we could test TLS everywhere as part of regular CI when all the pieces are there.14:46
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/pike: Set bind mount propegatation to shared for /var/lib/nova.  https://review.openstack.org/51855414:47
weshayjaosorior, ya.. I would really like to spend a few minutes w/ you and hrybacki to understand the gaps there14:47
owalshjaosorior: LGTM14:47
sshnaidmweshay, jaosorior, it's possible right now with network isolation only. Working without net iso will require an effort, but not too big..14:48
sshnaidmweshay, please include me in this discussion14:48
adarazs|roverweshay, panda|ruck|brb: we need this change for it merged to unblock it: https://review.openstack.org/51833614:48
weshaysshnaidm, yes agree :)14:48
jaosoriorsshnaidm: net-iso only is the way that I've always tested it :D14:49
sshnaidmjaosorior, yeah, but it's only 2 ovb jobs that use it :)14:50
jaosoriorwell, lets hook it to one of them14:50
*** udesale has quit IRC14:50
jaosoriorweshay, sshnaidm: All we need is to take into use the code that bnemec introduced to get an extra node from OVB; we already have code in quickstart that installs FreeIPA, so it would be a matter of using that.14:51
jaosoriorweshay: surely hrybacki and I could help out.14:51
sshnaidmjaosorior, it's used in legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset02414:51
* hrybacki listens14:51
sshnaidmjaosorior, hrybacki is bnemecs work ready?14:52
weshayjaosorior, k cool. I'll reach out to hrybacki later today to get a card on the tripleo-ci board for sshnaidm to review14:52
jaosoriorsshnaidm: well, we could try there, in that case it would now be 1ctlr_1comp_1ceph_1extra14:52
jaosoriorweshay: that would be awesome14:52
*** moshele has quit IRC14:52
jaosoriorlets get this rolling :D14:52
hrybackisshnaidm: he landed the last patch that was blocking us (AFAIK)14:52
sshnaidmjaosorior, hrybacki when bnemec has it ready, we are free to go14:53
openstackgerritMerged openstack/tripleo-ui master: Improve node count indicator in roles step  https://review.openstack.org/51667414:53
jaosoriorsshnaidm, weshay: Also, it would be great if we would start testing public TLS again in upstream CI. IIRC sshnaidm had some test patches already up for doing that.14:53
*** etingof has joined #tripleo14:53
sshnaidmjaosorior, that's the job I posted above, it has overcloud ssl14:53
jaosoriorsshnaidm: is it doing SSL? I didn't know :O14:54
weshaymatbu, with regards to doing more on rdo-cloud. Can you please add this bug to be discussed w/ the upgrade squad https://bugs.launchpad.net/tripleo/+bug/172432814:54
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]14:54
sshnaidmjaosorior, yep :)14:54
toskyunrelated topic: in order to enable Manila Tempest tests in the TripleO CI, we need to prepare the environment a bit, and I'm not sure where to put the code14:54
jaosoriorsshnaidm: awesome! thanks!14:54
toskysee the last comment in https://review.openstack.org/#/c/509554/14:54
jaosoriorsshnaidm: but yeah, getting TLS everywhere to be tested upstream would be awesome14:54
toskyor better, the last two comments14:54
jaosoriorsshnaidm: TLS everywhere != public TLS14:54
weshaymatbu, enabling rdo-cloud to upgrade will unblock work moving to rdo-cloud14:54
openstackgerritHonza Pokorny proposed openstack/tripleo-ui stable/pike: Improve node count indicator in roles step  https://review.openstack.org/51855514:55
*** pradk has quit IRC14:55
*** ykarel|afk is now known as ykarel14:55
toskydoes anyone know what is the cleanest way to add some post-deployment steps in some scenarios?14:55
honzaflorianf: jrist: ^^^ please & thank you14:55
beaglesbandini, Keepalived_vrrp[17922]: /usr/bin/systemctl status haproxy.service exited with status 314:55
beaglesNov  8 14:55:04 localhost Keepalived_vrrp[17922]: /usr/bin/systemctl status haproxy.service exited with status 314:55
beagles spamming /var/log/messages. Mean anything to you?14:55
bandinibeagles: is that on the undercloud or the overcloud?14:56
beaglesbandini, overcloud14:56
bandinibeagles: then this is a non-ha deployment correct?14:56
beaglesbandini, correct14:56
bandiniah ok14:56
bandinijust making sure :)14:56
sshnaidmjaosorior, sorry, it's legacy-tripleo-ci-centos-7-ovb-ha-oooq : http://logs.openstack.org/59/510059/5/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq/1833716/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2017-11-06_12_27_0714:56
bandinibeagles: no never seen (but I never ever deploy nonha envs)14:57
beaglesbandini, I realized just after I fired that off that was probably the case :)14:57
bandinisorry :)14:57
jaosoriorsshnaidm: dumb question: What does the legacy- prefix mean?14:58
beaglesbandini, :)14:58
florianfhonza: AFAIK only one +2 needed for backports, right?14:58
honzaflorianf: correct14:58
sshnaidmjaosorior, it means jobs run in non-native mode after moving to zuulv3. Actually it means nothing :)14:58
honzaflorianf: if the backport is proposed by a core...14:58
florianfhonza: which it clearly is14:58
honza:)14:58
* honza is special14:59
* beagles just realizes that this is keepalive asking of haproxy is running via systemctl (which it is not) so I'd expect it to complain somewhat14:59
florianfhonza: done14:59
florianf:-)14:59
jaosoriorsshnaidm: lol ok14:59
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260715:00
*** etingof has quit IRC15:00
dciabrinchem, are you familiar with TRIPLEO_CONFIG_HASH and minor updates?15:02
chemdciabrin: very, they are coming to dinner tonight15:03
*** pdeore has quit IRC15:06
*** rfolco has quit IRC15:09
chemdciabrin: hum ... you wondering for the volume configuration, right ?15:09
dciabrinchem, correct15:10
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432815:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925315:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958615:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011115:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)15:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)15:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]15:10
*** janki has quit IRC15:10
chemdciabrin: I looked at the common_* which is used I didn't spot anything relevant15:11
*** d0ugal has quit IRC15:11
dciabrinchem, essentially i'd like to understand when the tag is set and when it's updated15:12
dciabrinchem, seems like docker-puppet.py sets it in the first place?15:12
*** dtantsur|bbl is now known as dtantsur15:13
*** etingof has joined #tripleo15:13
chemdciabrin: I'm a bit lost at what you're looking for/at, but yeah docker-puppet.py seems to be the place15:14
*** mdnadeem has joined #tripleo15:14
dtantsurhi folks! https://review.openstack.org/#/c/517356/ is an upgrade-critical backport, could you please consider it?15:15
*** panda|ruck|brb is now known as panda|ruck15:15
chemdciabrin: do you know if depends-on get applied legacy-tripleo-ci-centos-7-containers-multinode-upgrades jobs (or the person I should query ?)15:17
chemdciabrin: seems like they are, sorry for the noise15:18
dciabrinchem I have a low pass filter anyway , that's fine :)15:19
chemdciabrin: nice, I'm the rumble under 1000hz15:20
*** mary_grace has joined #tripleo15:20
*** mary_grace has quit IRC15:20
*** bobh has joined #tripleo15:20
*** mcornea has quit IRC15:22
dciabrinchem, haha!15:22
*** rfolco has joined #tripleo15:22
*** psachin has quit IRC15:23
mwhahahalooks like telemetry tempest failed in the gate15:24
mwhahahahttp://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/e05c657/job-output.txt.gz#_2017-11-08_14_57_00_10494215:24
*** d0ugal has joined #tripleo15:25
*** d0ugal has quit IRC15:25
*** d0ugal has joined #tripleo15:25
matbuweshay: sorry i was on a mtg, the LP ^ looks pretty new, stupid question but its for ocata promotion ? there is no upgrade jobs in the rdo promotion ?15:26
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Switch scenario004-containers to use ceph-ansible  https://review.openstack.org/51621815:27
openstackgerritMerged openstack/tripleo-heat-templates master: Set become:false for undercloud plays  https://review.openstack.org/51832315:27
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add various functionality to nodepool role  https://review.openstack.org/51856015:31
sshnaidmtrown, ^^15:31
*** mcornea has joined #tripleo15:34
*** dsariel has joined #tripleo15:34
*** udesale has joined #tripleo15:35
weshaymatbu, that bug was opened by the RDO-Cloud admins, they are in process of upgrading from newton to ocata15:35
trownsshnaidm: I am reorganizing the etc-nodepool bits to work with ovb, but I will make sure to rebase your patch correctly :)15:36
matbuweshay: ha ok15:36
sshnaidmtrown, sure15:36
sshnaidmrlandy, trown do we need to generate "subnodes" hosts file after ovb role as well? for nodepool-setup15:37
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Rename Undercloud->External deployment  https://review.openstack.org/51856115:37
weshaymatbu, by ensuring RDO-Cloud admins are unblocked, we will be helping ourselves bring up more upgrade jobs in rdo-cloud15:37
openstackgerritLiz Blanchard proposed openstack/tripleo-ui master: Remove trash icons on delete modals  https://review.openstack.org/51856215:38
*** dsariel_ has joined #tripleo15:38
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Temporarily disable fluentd from scenario001-multinode-containers  https://review.openstack.org/51749715:38
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Add --detailed-exitcodes when running puppet via ansible  https://review.openstack.org/51702215:38
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Upgrade CI test on stable/pike  https://review.openstack.org/50062515:38
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements  https://review.openstack.org/51163315:38
trownsshnaidm: I dont think so, I was going to actually split tasks we need only for multinode into their own list, and just dynamically include that task list only for multinode15:39
*** dpawar has quit IRC15:39
chemdciabrin: so inclusion of the rw patch doesn't do the trick.  I've rebased https://review.openstack.org/#/c/500625/21 to include latest patch but I think the error will still be there.15:39
rlandyoops - right - now we are split answering sshnaidm  question15:39
chembandini: ^15:39
chembandini: don't remember why you were included in this upgrade failing cinder db manage issue btw.15:40
*** mdnadeem has quit IRC15:40
*** ykarel is now known as ykarel|afk15:41
*** dsariel has quit IRC15:41
*** dsariel__ has joined #tripleo15:43
*** udesale has quit IRC15:43
jaosoriorsshnaidm, mwhahaha: what's the "featureset" that defines legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container ?15:46
*** dsariel_ has quit IRC15:46
weshayjaosorior, in the job log search for the variable jobtype15:47
openstackgerritLuke Hinds proposed openstack/tripleo-heat-templates master: Implements AIDE Intrusion Detection System  https://review.openstack.org/48302815:47
mwhahahajaosorior: i think it's 1615:47
*** dsariel_ has joined #tripleo15:48
jaosoriorweshay: and where are the scenarios defined?15:49
weshayjaosorior, https://docs.openstack.org/tripleo-quickstart/latest/feature-configuration.html15:49
*** mary_grace has joined #tripleo15:50
sshnaidmjaosorior, scenarios are in tht repo..15:50
*** dsariel__ has quit IRC15:51
*** mary_grace has quit IRC15:52
*** pradk has joined #tripleo15:52
*** dsariel__ has joined #tripleo15:53
jaosoriorsshnaidm, weshay thanks15:53
*** mary_grace has joined #tripleo15:53
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Add tags to plays  https://review.openstack.org/51856615:53
*** mary_grace has quit IRC15:54
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Add tags to plays  https://review.openstack.org/51856615:55
*** dsariel_ has quit IRC15:56
*** ykarel|afk has quit IRC15:57
*** spectr has quit IRC15:57
*** dsariel_ has joined #tripleo15:58
*** itlinux has joined #tripleo15:58
*** Goneri has quit IRC16:00
jaosoriorweshay, sshnaidm do you know how can I find the tempest logs to figure out why a commit failed? For instance, this http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/e05c657/job-output.txt.gz#_2017-11-08_14_57_00_10494216:01
*** dsariel__ has quit IRC16:01
*** Goneri has joined #tripleo16:01
*** suuuper has quit IRC16:02
weshayhttp://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/e05c657/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-08_14_56_1816:02
weshayjaosorior, ^16:02
sshnaidmjaosorior, http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/e05c657/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-08_14_52_5016:02
sshnaidmoops :)16:02
*** mary_grace has joined #tripleo16:03
sshnaidmweshay, is faster16:03
*** dsariel__ has joined #tripleo16:03
weshaysshnaidm, only at cut and paste :)16:03
jaosoriorssh timeout, funky16:05
jaosoriorweshay, sshnaidm: thanks!16:05
dciabrinchem: available for a quick chat on bluejeans with bandini?16:05
dciabrinchem (hint: it's a trap)16:05
bandinidciabrin: how can he possibly say no to such a sweet offer!16:06
jaosoriorlol16:06
dciabrinbandini, he may have his low pass filter engaged16:06
*** dsariel_ has quit IRC16:06
bandiniahaha16:07
*** ramishra has quit IRC16:07
chemdciabrin: I got to setup that filter ...16:07
*** ramishra has joined #tripleo16:07
openstackgerritIan Main proposed openstack/python-tripleoclient master: Mount a tmpfs filesystem for heat tmpfiles  https://review.openstack.org/50855816:08
*** dsariel_ has joined #tripleo16:08
chemdciabrin: where is that trap unfolding so I can jump right into it with big guns and shoot everything ?16:08
dciabrinchem, pm'ed you16:09
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432816:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925316:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958616:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011116:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)16:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)16:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]16:10
*** dsariel__ has quit IRC16:11
*** dsariel__ has joined #tripleo16:13
chemdciabrin: bandini https://github.com/redhat-openstack/tripleo-upgrade16:16
*** dsariel_ has quit IRC16:17
openstackgerritJan Provaznik proposed openstack/tripleo-heat-templates master: Add support for ceph-nfs manila backend  https://review.openstack.org/47124516:17
weshaymwhahaha, just fyi.. please don't kick this out of the queue https://review.openstack.org/#/c/517428/16:17
* mwhahaha gets his boots on16:17
*** ukalifon has quit IRC16:17
*** dsariel_ has joined #tripleo16:18
mwhahahaweshay: i think it might be ok, i think https://review.openstack.org/#/c/517038/ is the patch to make our lives easier16:18
mwhahahawhich is queued before that patch16:18
mwhahahawe keep failing on that cinder stuff so hopefully 038 fixes that16:19
* mwhahaha is keeping an eye on it16:19
weshayrock16:19
mwhahahaunfortunately there are 8 patches before that one :D16:19
mwhahahaso it may be 20 hours before it merges16:19
weshaylol16:20
*** ebarrera has quit IRC16:20
weshaymwhahaha, I think all managers should have to watch the zuul queue for a week16:20
amoralejmwhahaha, what's the status of merging reviews in tripleo repos, do you think we may get https://review.openstack.org/#/c/518037/ today?16:21
mwhahahaand try and figure out what went wrong16:21
*** dsariel__ has quit IRC16:21
*** achadha has joined #tripleo16:22
mwhahahaamoralej: probably not, but we'll see. i'll keep it open and aprove it if the queue stops having issues16:22
jristhonza: done16:22
amoralejmwhahaha, ok, thx16:22
*** achadha_ has joined #tripleo16:22
*** dsariel__ has joined #tripleo16:23
mwhahahaamoralej: what's the prioirity of that patch? it doesn't have a bug associated with it16:23
mwhahahaamoralej: is it preventing promotion or something16:23
amoralejit's blocking pushing ovs-2.816:23
amoralejwhich is needed for networking-ovn team16:24
mwhahahaamoralej: we should have a bug for that  so we can track it16:24
mwhahahaor some sort of bp reference16:24
chemdciabrin: bandini http://logs.openstack.org/25/500625/21/check/legacy-tripleo-ci-centos-7-containers-multinode-upgrades/d7a1ff4/logs/subnode-2/var/log/cinder/cinder-manage.log.txt.gz#_2017-11-08_13_49_58_85216:24
amoralejmwhahaha, you mean ovn bp or some tripleo one?16:24
mwhahahaamoralej: some reference to something, the problem is context on that patch16:25
mwhahahaamoralej: right now if you didn't mention it, i wouldn't know it's a thing we need16:25
*** adac has quit IRC16:25
*** ramishra has quit IRC16:25
mwhahahaamoralej: we don't need to add it to that patch now as a respin of ci would be a waste, but in the future it helps if we have bugs or bps associated so we know why we need this change16:26
amoralejmwhahaha, i can create a LP bug and add it in a comment if it helps16:26
*** achadha has quit IRC16:26
mwhahahaamoralej: yes please do that so we can have some tracking of this. thanks16:27
*** dsariel_ has quit IRC16:27
*** ramishra has joined #tripleo16:27
*** dsariel_ has joined #tripleo16:28
*** abregman|mtg has quit IRC16:28
*** gfidente has quit IRC16:28
*** aditya_r has quit IRC16:29
*** gfidente has joined #tripleo16:29
*** gfidente has quit IRC16:29
*** gfidente has joined #tripleo16:29
openstackgerritFrançois Charlier proposed openstack/tripleo-quickstart-extras master: validate-tempest: better workers default  https://review.openstack.org/51857516:29
*** dsariel__ has quit IRC16:31
*** newmember has quit IRC16:32
*** dsariel__ has joined #tripleo16:33
*** newmember has joined #tripleo16:33
*** aufi has quit IRC16:33
*** mary_grace has quit IRC16:34
*** marios has quit IRC16:34
bandinichem: https://review.openstack.org/#/c/517576/16:35
*** aditya_r has joined #tripleo16:35
*** dsariel_ has quit IRC16:36
*** dsariel_ has joined #tripleo16:38
*** gvrangan_ has joined #tripleo16:41
*** jkilpatr has quit IRC16:41
*** aditya_r has quit IRC16:41
*** dsariel__ has quit IRC16:42
*** iranzo has quit IRC16:42
*** dsariel__ has joined #tripleo16:43
*** moshele has joined #tripleo16:44
*** moshele has quit IRC16:46
*** akrivoka has joined #tripleo16:46
*** dsariel_ has quit IRC16:46
*** openstackgerrit has quit IRC16:48
*** dsariel_ has joined #tripleo16:48
*** moshele has joined #tripleo16:48
*** anshul has quit IRC16:48
*** openstackgerrit has joined #tripleo16:51
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Make sure /var/lib/mysql rights are setup correctly.  https://review.openstack.org/51857816:51
*** artom has joined #tripleo16:51
*** dsariel__ has quit IRC16:51
*** dsariel__ has joined #tripleo16:53
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Make sure /var/lib/mysql rights are setup correctly.  https://review.openstack.org/51857916:53
*** jkilpatr has joined #tripleo16:54
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Upgrade CI test on stable/pike  https://review.openstack.org/50062516:56
*** tesseract has quit IRC16:56
*** dsariel_ has quit IRC16:56
*** moshele has quit IRC16:57
*** dsariel_ has joined #tripleo16:58
*** dparkes has quit IRC16:58
*** aditya_r has joined #tripleo17:00
*** dsariel__ has quit IRC17:01
*** jfrancoa has quit IRC17:02
*** dsariel__ has joined #tripleo17:03
*** jkilpatr has quit IRC17:05
openstackgerritIhar Hrachyshka proposed openstack/tripleo-heat-templates master: Revert "Addition of Nuage as mechanism driver for ML2"  https://review.openstack.org/51858217:06
openstackgerritIhar Hrachyshka proposed openstack/tripleo-heat-templates master: Revert "Addition of Nuage as mechanism driver for ML2"  https://review.openstack.org/51858217:06
*** dsariel_ has quit IRC17:07
*** dsariel_ has joined #tripleo17:08
*** itlinux has quit IRC17:09
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432817:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925317:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958617:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011117:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]17:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)17:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)17:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]17:10
*** yprokule has quit IRC17:11
*** salmankhan has quit IRC17:12
openstackgerritMerged openstack/tripleo-common master: Healthcheck for nova_migration_target container  https://review.openstack.org/51829817:12
*** salmankhan has joined #tripleo17:12
*** dsariel__ has quit IRC17:12
*** dsariel__ has joined #tripleo17:13
honzajrist: florianf: thanks17:14
jristnp17:14
*** ffiore has quit IRC17:16
*** itlinux has joined #tripleo17:17
*** dsariel_ has quit IRC17:17
*** jkilpatr has joined #tripleo17:18
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add various functionality to nodepool role  https://review.openstack.org/51856017:19
*** dsariel__ has quit IRC17:19
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Add missing host_prep_tasks for log dirs creation  https://review.openstack.org/51858317:20
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Write readme.txt into old log directories  https://review.openstack.org/51858417:20
openstackgerritMerged openstack/tripleo-ui stable/pike: Improve node count indicator in roles step  https://review.openstack.org/51855517:21
jristwoot17:22
*** fragatin_ has quit IRC17:23
*** lucasagomes is now known as lucas-afk17:24
*** rfolco is now known as rfolco_brb17:24
*** panda|ruck is now known as panda|ruck|bbl17:26
*** achadha_ has quit IRC17:26
*** achadha has joined #tripleo17:27
*** jpich has quit IRC17:30
*** ccamacho has quit IRC17:31
*** adac has joined #tripleo17:32
*** openstackgerrit has quit IRC17:32
* mwhahaha crosses fingers so the patches start merging17:34
*** aditya_r has quit IRC17:36
*** openstackgerrit has joined #tripleo17:38
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: DNM, update the undercloud install for containers  https://review.openstack.org/51744417:38
bnemecmwhahaha: So are we good to start approving things again, or should we wait and see how it goes for a while?17:39
mwhahahaI would like to wait17:39
bnemecAck17:40
mwhahahai'm still waiting for the cinder patch to merge17:40
mwhahahaso until that merges we should hold our breath :D17:40
*** gkadam has quit IRC17:40
mandrejaosorior: you may be interested in https://review.openstack.org/#/c/518589/17:40
openstackgerritIhar Hrachyshka proposed openstack/tripleo-heat-templates master: Change default ml2 firewall driver back to iptables_hybrid  https://review.openstack.org/51859017:43
openstackgerritMerged openstack/tripleo-heat-templates master: Fix rights to local_settings.d for dockerized Horizon  https://review.openstack.org/51838417:44
jschlueterhmm I wonder if we can quiet down the copy logs section of CI jobs in the gates?17:44
*** etingof has quit IRC17:45
bnemecjschlueter: +117:45
jschlueterhalf of the job-output.txt file is entries from TASK [Copy files from /home/zuul/workspace/ on node]17:45
*** fragatina has joined #tripleo17:45
openstackgerritIhar Hrachyshka proposed openstack/tripleo-heat-templates master: Change default ml2 firewall driver back to iptables_hybrid  https://review.openstack.org/51859017:46
abishophey, I notice several THTs backported to stable/ocata still say "heat_template_version: pike"17:47
abishopI guess that's not an issue?17:47
abishop% grep -rl "heat_template_version: pike"17:48
*** itlinux has quit IRC17:49
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add various functionality to nodepool role  https://review.openstack.org/51856017:52
*** mcornea has quit IRC17:53
*** mary_grace has joined #tripleo17:53
*** rfolco_brb has quit IRC17:57
*** dtantsur is now known as dtantsur|afk17:59
mwhahahawell17:59
mwhahahathe patch i was hoping to fix the problems, didn't17:59
mwhahahaas it failed17:59
*** chem has quit IRC17:59
mwhahaha‎(ノಥ益ಥ)ノ oǝʃdıɹʇ18:00
mwhahahahttp://logs.openstack.org/38/517038/4/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/f6ed2aa/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2017-11-08_17_48_0918:00
mwhahahaabishop, mandre: https://review.openstack.org/#/c/517038/ failed with a deploy timeout18:01
abishopmwhahaha: yeah, looking at it now18:02
*** derekh has quit IRC18:03
*** amoralej is now known as amoralej|off18:03
*** dsneddon has quit IRC18:03
*** itlinux has joined #tripleo18:04
*** jkilpatr has quit IRC18:05
*** dsneddon has joined #tripleo18:05
*** rmascena has joined #tripleo18:05
openstackgerritMerged openstack/tripleo-quickstart-extras master: Add steps needed when using OVB with a testenv broker  https://review.openstack.org/51279818:05
openstackgerritMerged openstack/tripleo-heat-templates master: Do not set cluster in maintenance mode during split stack upgrade  https://review.openstack.org/51365418:05
openstackgerritMerged openstack/tripleo-common master: Add missing IronicInspector service  https://review.openstack.org/51475918:06
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Add all services to container scenarios  https://review.openstack.org/51230518:06
openstackgerritMerged openstack/tripleo-heat-templates master: Explicitly set healthcheck command.  https://review.openstack.org/50284318:06
openstackgerritMerged openstack/tripleo-heat-templates master: Add option for Neutron containers to log to stdout/stderr  https://review.openstack.org/51046418:06
*** raildo has quit IRC18:07
abishopmwhahaha: meaningful? http://logs.openstack.org/38/517038/4/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/f6ed2aa/logs/subnode-2/var/log/messages.txt.gz#_Nov__8_16_37_5018:08
mwhahahaabishop: i don't think so18:09
*** rfolco_brb has joined #tripleo18:10
mwhahahasince these are public vms we get a bunch of extraneous traffic18:10
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432818:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925318:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958618:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011118:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]18:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)18:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)18:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]18:10
mwhahahaunless we're supposed to be serving something on port 854518:10
mwhahahabut 8545 looks to be a port for ethereum18:11
abishopmwhahaha: don't know, just seemed suspicious given that's the timeframe when the timeout occurred18:11
mwhahahathere was a nother bug for just timeouts, doesn't appear to be an alert bug though18:11
mwhahahalet me see if i can dig that one up18:11
abishopmwhahaha: I'll keep poking, but this stuff is outside my wheelhouse18:12
openstackgerritPradeep Kilambi proposed openstack/python-tripleoclient master: Add support to generate a sample undercloud.conf  https://review.openstack.org/51859518:12
*** jpena is now known as jpena|off18:12
*** shreshtha_ has quit IRC18:17
*** jkilpatr has joined #tripleo18:18
*** trown is now known as trown|lunch18:19
mwhahahasimilar https://bugs.launchpad.net/tripleo/+bug/165126718:19
openstackLaunchpad bug 1651267 in tripleo "Timeouts in multinode jobs" [High,Triaged]18:19
*** moshele has joined #tripleo18:22
abishopmwhahaha: fwiw this test passed last run: http://logs.openstack.org/38/517038/4/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1f3fcbe/job-output.txt.gz#_2017-11-05_18_41_56_79703518:22
mwhahahayea i know18:22
mwhahahait means we have multiple issues still outstanding18:22
mwhahahawithout a possible fix :/18:22
*** mary_grace has quit IRC18:23
openstackgerritMarius Cornea proposed openstack/tripleo-heat-templates stable/pike: Do not set cluster in maintenance mode during split stack upgrade  https://review.openstack.org/51859718:23
mwhahahai'm going ot look further at it. we should recheck that patch for now because we known we need it18:23
*** mary_grace has joined #tripleo18:26
*** bogdando has quit IRC18:32
*** newmember has quit IRC18:44
*** dhill_ has quit IRC18:53
*** Goneri has quit IRC18:56
*** mary_grace has quit IRC18:59
*** jkilpatr has quit IRC19:00
*** dhill_ has joined #tripleo19:00
*** mary_grace has joined #tripleo19:00
*** gfidente is now known as gfidente|afk19:02
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: Fix node count formatting on deployment page  https://review.openstack.org/51860719:06
*** anilvenkata has quit IRC19:07
*** agurenko has quit IRC19:08
*** anilvenkata has joined #tripleo19:08
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432819:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925319:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958619:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011119:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]19:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)19:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)19:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]19:10
*** jkilpatr has joined #tripleo19:12
*** Usman has joined #tripleo19:15
*** trown|lunch is now known as trown19:17
weshayah.. openstack-health is back online19:19
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Fix devmode by right order of playbooks  https://review.openstack.org/51833619:19
*** jprovazn has quit IRC19:19
openstackgerritJustin Kilpatrick proposed openstack/tripleo-quickstart-extras master: Enable OVN deployment  https://review.openstack.org/50372819:23
*** rwsu has quit IRC19:39
*** chlong has quit IRC19:39
*** achadha_ has joined #tripleo19:42
*** achadha has quit IRC19:43
*** achadha_ has quit IRC19:46
*** moshele has quit IRC19:47
*** dparkes has joined #tripleo19:51
mwhahahatherve, zaneb: FYI I just created https://bugs.launchpad.net/tripleo/+bug/1731032 It seems that it's just hanging in heat but I have no idea what to start looking at. So if you have a chance please take a look at it tomorrow when you're in19:53
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]19:53
*** abregman has joined #tripleo19:54
*** chlong has joined #tripleo19:55
*** ffiore has joined #tripleo19:57
*** salmankhan has quit IRC20:00
*** ffiore has quit IRC20:01
*** mary_grace has quit IRC20:02
*** mary_grace has joined #tripleo20:07
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432820:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925320:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958620:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011120:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103220:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)20:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)20:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]20:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]20:10
openstackgerritKeith Schincke proposed openstack/puppet-tripleo master: Set ACLs on ceph client keyrings  https://review.openstack.org/51734720:10
*** atoth has quit IRC20:11
*** mary_grace has quit IRC20:11
*** mary_grace has joined #tripleo20:12
openstackgerritKeith Schincke proposed openstack/tripleo-heat-templates master: Set restrictive file permissions on Ceph keyrings for non-containerized deployment  https://review.openstack.org/50902020:13
*** marrusl has quit IRC20:13
*** marrusl has joined #tripleo20:13
openstackgerritIhar Hrachyshka proposed openstack/tripleo-heat-templates master: Change default ml2 firewall driver back to iptables_hybrid  https://review.openstack.org/51859020:13
*** itlinux has quit IRC20:15
*** ramishra has quit IRC20:17
openstackgerritIhar Hrachyshka proposed openstack/tripleo-heat-templates master: Change default ml2 firewall driver back to iptables_hybrid  https://review.openstack.org/51859020:18
*** anilvenkata has quit IRC20:19
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Fix devmode by right order of playbooks  https://review.openstack.org/51833620:20
*** anilvenkata has joined #tripleo20:21
*** chlong has quit IRC20:22
*** eck` is now known as eck`gone20:23
*** kbyrne has quit IRC20:24
*** kbyrne has joined #tripleo20:26
*** itlinux has joined #tripleo20:26
openstackgerritMerged openstack/tripleo-heat-templates master: Migrates storage-environment.yaml to ceph-ansible  https://review.openstack.org/51256920:28
openstackgerritMerged openstack/puppet-tripleo stable/pike: Galera: add support for encrypted SST  https://review.openstack.org/51852120:28
*** gvrangan_ has quit IRC20:28
*** mary_grace has quit IRC20:29
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: DNM, update the undercloud install for containers  https://review.openstack.org/51744420:29
*** kbyrne has quit IRC20:31
*** kbyrne has joined #tripleo20:32
*** jkilpatr_ has joined #tripleo20:32
*** jkilpatr has quit IRC20:35
*** chlong has joined #tripleo20:36
openstackgerritMerged openstack/tripleo-quickstart-extras master: create subunit test results for the undercloud sanity check  https://review.openstack.org/51765720:38
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: DNM, update undercloud install options for containers  https://review.openstack.org/51744520:38
openstackgerritMerged openstack/tripleo-heat-templates master: Add support for multiple Cinder RBD backends  https://review.openstack.org/50671520:38
*** etingof has joined #tripleo20:38
*** rfolco_brb has quit IRC20:42
openstackgerritMerged openstack/tripleo-quickstart-extras master: tempest: conditional --provider:physical_network  https://review.openstack.org/51431320:43
openstackgerritMerged openstack/puppet-tripleo master: Unset MountFlags in docker.service systemd directives  https://review.openstack.org/51817920:43
openstackgerritMerged openstack/tripleo-heat-templates master: ovn: Remove setting of DockerNeutronApiImage param  https://review.openstack.org/51090020:43
*** newmember has joined #tripleo20:50
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: DNM, update undercloud install options for containers  https://review.openstack.org/51744520:51
*** liverpooler has quit IRC20:51
*** rmascena has quit IRC20:52
openstackgerritMerged openstack/tripleo-quickstart master: Get DLRN hash before setting up repos and images  https://review.openstack.org/51285520:53
*** achadha has joined #tripleo20:55
*** paramite has quit IRC20:55
*** achadha_ has joined #tripleo20:56
*** eck`gone is now known as eck`20:56
openstackgerritMerged openstack/tripleo-puppet-elements master: Remove ceilometer-collector from overcloud image  https://review.openstack.org/51231520:57
*** rwsu has joined #tripleo20:57
*** iranzo has joined #tripleo20:58
*** iranzo has quit IRC20:58
*** iranzo has joined #tripleo20:58
*** achadha has quit IRC20:59
*** toure is now known as toure_biab21:04
openstackgerritMerged openstack-infra/tripleo-ci master: Use rdoproject-based settings for containers and dlrn_hash for images  https://review.openstack.org/51466821:07
openstackgerritMerged openstack/tripleo-heat-templates master: Set bind mount propegatation to shared for /var/lib/nova.  https://review.openstack.org/51818021:07
*** iranzo has quit IRC21:08
*** jtomasek has joined #tripleo21:09
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432821:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925321:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958621:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011121:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103221:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)21:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)21:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]21:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]21:10
*** newmember has quit IRC21:10
*** rhallisey has quit IRC21:11
*** rhallisey has joined #tripleo21:12
*** aputtur_ has quit IRC21:16
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: Fix the Network Gateway Validation  https://review.openstack.org/51861921:19
*** lblanchard has quit IRC21:20
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Fix devmode by right order of playbooks  https://review.openstack.org/51833621:20
*** deadnull has joined #tripleo21:23
*** itlinux has quit IRC21:26
*** itlinux has joined #tripleo21:31
*** jkilpatr_ has quit IRC21:32
*** ansmith has quit IRC21:35
*** ansmith_ has quit IRC21:35
*** bobh has quit IRC21:35
*** salmankhan has joined #tripleo21:39
*** gbarros has joined #tripleo21:39
jtomasekslagle: hi, do you have a link where I can read more on 'undercloud deploy' proposal please?21:41
mwhahahajtomasek: what undercloud deploy proposal? do you mean switching to a containerized undercloud?21:42
mwhahahaor the ci switch stuff from the ML21:42
slaglejtomasek: do you mean by reference on openstack-dev?21:42
slagle*my21:42
jtomasekslagle: yes21:42
slaglejtomasek: there was an earlier thread that EmilienM resurrected earlier in the week21:42
slaglejtomasek: about using undercloud deploy to test scenarios21:43
slaglejtomasek: in that situation, there will be no mistral. so we can't rely on it for any pre_deploy_workflows as shardy was proposing21:43
*** florianf has quit IRC21:44
*** iranzo1|AFK is now known as iranzo121:44
*** salmankhan has quit IRC21:44
jtomasekslagle: I'll try to find it and read up on it. It just sounds quite scary to me as it seems like skipping whole api (deployment configuration) layer we have21:44
slaglewe'd have coverage of the whole api in other tests21:45
mwhahahaer21:45
mwhahahawe'd still have a full deployment, just not for scenarios21:45
jtomasekslagle: just to make sure we're on the same page regarding the pre-deploy workflows. It is supposed to be a list of workflows which encapsulate some logic and output parameter values to environment files21:45
*** iranzo1 is now known as iranzo1|AFK21:46
mwhahahaie ovb would still do a full test with mistral, etc21:46
jtomasekI see, thanks21:46
mwhahahaand hopefully we can get those properly voting on rdo cloud, etc etc21:46
*** achadha_ has quit IRC21:46
mwhahahaso we shouldn't end up with less coverage, just a variety where we aren't doing a full multinode for basic sanity in the upstream and a full deployment via ovb21:46
*** gbarros has quit IRC21:46
slaglejtomasek: yes, i understand. but those no workflows are not strictly required to deploy with t-h-t today as evidenced by undercloud deploy21:46
slaglejtomasek: and what shardy was proposing was to expand that out to be used generally by any service21:47
slaglewhich sounds like...making it a requirement if we offer that interface21:47
*** adac has quit IRC21:49
slaglejtomasek: offering this general interface is what i'm not very supportive of21:49
fultonjslagle: is https://slagle.fedorapeople.org/tripleo-ansible-arch.png still accurate?21:49
jtomasekslagle: I see, hmm, right. they're not required because you can tweak t-h-t by hand. Those workflows are trying to make it more convenient by automating some parts of the tweaking. So they are not mandatory in any way.21:52
jtomasekI think we need to clear up how we tie the workflows to services.21:52
*** rcernin has joined #tripleo21:52
slaglejtomasek: right. so what I'm not supportive of is a general interface where any service author can specify a mistral workflow in pre_deploy, which we then have to honor in our deployment contract21:53
slaglethat would really tie our hands in terms of architecture decisions going forward21:53
mwhahahaabishop: so http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/13841da/job-output.txt.gz failed with the same tempest test but doesn't seem to match up with https://bugs.launchpad.net/tripleo/+bug/172925321:53
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)21:53
mwhahahaabishop: can you take a peak to confirm/deny it's the same issue?21:54
abishopmwhahaha: looking now...21:54
jtomasekslagle: I see, makes sense, so you're saying providing ansible task which does just that is fine?21:55
slaglejtomasek: i think it's more inline with the rest of t-h-t.21:56
slaglewe don't specify mistral workflows in t-h-t for deploy for upgrade, we specify ansible tasks and plays21:57
slaglethe output of Heat is those consolidated tasks/plays21:57
*** jkilpatr has joined #tripleo21:57
slaglethen...we offer mistral automation that uses those outputs21:57
slagleif desired21:57
jtomasekslagle: hmm, I need to think about this more, I am not sure I understand why it would be a problem to have mistral workflow to configure a service (in terms of setting up a template/environment file)21:57
jtomasek:))21:57
jtomasekwe use mistral for another bunch of tasks anyway, It never goes beyond deployment itsef. nor with this new proposal21:58
*** trown is now known as trown|outtypewww21:59
*** d0ugal_ has joined #tripleo21:59
slaglejtomasek: i think if we're just talking derived params, then maybe it's a non-issue22:00
slaglejtomasek: but if you read the 2nd paragraph of steve's mail it seems he's proposing a broader scope22:00
slaglewe can get him to clarify. but that's the part that concerns me22:01
*** d0ugal has quit IRC22:01
jtomasekslagle: ok, this needs to get clarified. IIUCThe proposal does not want to add mistral workflows into t-h-t but just create a map of services->workflows22:02
jtomasekslagle: in a file such as roles-data.yaml or completely separate one22:02
*** chlong has quit IRC22:02
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add various functionality to nodepool role  https://review.openstack.org/51856022:03
slaglejtomasek: i know he's not proposing adding workflows to t-h-t. it's the "calling any workflow that does whatever"22:04
slagleif it's *just* for deriving parameters, that's probably fine22:04
mwhahahaabishop: i don't think it's the same because it's booting but the vm isn't available in time so i guess i need to create a different bug22:07
abishopmwhahaha: definitely different issue. Here, cinder services all happy, tempest test failed when trying to access an instance http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/13841da/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-08_21_19_5422:07
abishopmwhahaha: exactly. here's the nova instance: http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/13841da/logs/subnode-2/var/log/containers/nova/nova-compute.log.txt.gz#_2017-11-08_21_19_20_87222:08
*** dougbtv__ has joined #tripleo22:08
*** d0ugal__ has joined #tripleo22:08
jtomasekslagle: ok, I think I understand the concern of allowing the workflow to do more than that which would add requirement to run a mistral workflows in testing. Ansible should be the tool shall there be such need22:08
*** itlinux has quit IRC22:09
*** d0ugal_ has quit IRC22:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432822:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925322:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958622:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011122:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103222:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)22:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)22:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]22:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]22:10
*** itlinux has joined #tripleo22:10
abishopmwhahaha: just to emphasize, cinder is happily creating volumes http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/13841da/logs/subnode-2/var/log/cinder/volume.log.txt.gz#_2017-11-08_21_13_55_04322:11
mwhahahaabishop: cool thanks, i'm creating a new bug22:12
abishopthe more the merrier :-/22:12
*** itlinux has quit IRC22:12
*** itlinux has joined #tripleo22:13
*** rwsu has quit IRC22:13
*** d0ugal has joined #tripleo22:14
*** d0ugal has quit IRC22:14
*** d0ugal has joined #tripleo22:14
*** d0ugal__ has quit IRC22:14
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Fix devmode by right order of playbooks  https://review.openstack.org/51833622:15
mwhahahaabishop: thanks for checking, i opened https://bugs.launchpad.net/tripleo/+bug/173106322:15
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]22:15
*** deadnull has quit IRC22:17
*** gbarros has joined #tripleo22:17
*** thrash is now known as thrash|g0ne22:18
*** rwsu has joined #tripleo22:20
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Refactor cellv2 host discovery logic to avoid races  https://review.openstack.org/51386722:21
Slowerslagle: around?22:22
*** mwhahaha changes topic to "CI status: MASTER: RED, STABLE/*: YELLOWish, For master jobs please do not approve things that are not critical. stable/* patches appear to be OK to merge for now | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/"22:23
*** abishop has quit IRC22:24
weshaymwhahaha, and the known issues are in alert ya?22:25
mwhahahayea22:25
weshayk22:25
*** gbarros has quit IRC22:25
*** ansmith_ has joined #tripleo22:27
*** ansmith has joined #tripleo22:28
weshaymwhahaha, did you see this in the gate? https://bugs.launchpad.net/tripleo/+bug/173103222:28
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]22:28
mwhahahaweshay: yea22:28
weshayok.. I thought gate jobs had gate in the url22:28
mwhahahaweshay: as inidcated by the url in that log http://logs.openstack.org/38/517038/4/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/f6ed2aa/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2017-11-08_17_48_0922:28
mwhahaha:D22:28
mwhahahait does22:28
weshayha, thanks22:28
* mwhahaha points to the gate nicely placed between 4/ and /legacy22:28
weshayjebus, this job is making me blind22:29
*** pradk has quit IRC22:29
mwhahahablind with rage :D22:30
*** gfidente|afk has quit IRC22:32
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: Don't calculate hashes for rh1 images  https://review.openstack.org/51862922:34
sshnaidmrlandy, ^^22:34
rlandysshnaidm: are you sure we don't want to?22:35
sshnaidmrlandy, 100%22:35
rlandysshnaidm: ok - checking other fetch-images jobs - should be set there22:35
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: Don't calculate hashes for rh1 images  https://review.openstack.org/51862922:36
*** gbarros has joined #tripleo22:36
sshnaidmrlandy, we just don't have these hashes on rh1 cloud22:37
sshnaidmrlandy, I mirror only image files and put them into current-tripleo directory22:37
sshnaidmrlandy, it should go with moving to rdo cloud, but till then...22:37
*** d0ugal_ has joined #tripleo22:39
openstackgerritK Jonathan Harker proposed openstack/diskimage-builder master: zypper: fix package removal  https://review.openstack.org/51765822:41
*** gbarros has quit IRC22:41
*** threestrands has joined #tripleo22:41
*** threestrands has joined #tripleo22:41
*** d0ugal has quit IRC22:41
*** achadha has joined #tripleo22:42
slagleSlower: yes22:46
*** achadha has quit IRC22:47
*** jmelvin has quit IRC22:48
Slowerslagle!22:49
Slowerslagle: so, bit of a tough problem if you're up for it..22:49
slaglewhat type of person would I be if I wasn't22:49
Slowerhehe22:49
Slowersec, finding links..22:50
*** itlinux has quit IRC22:50
Slowerslagle: http://logs.openstack.org/18/518118/2/check/legacy-tripleo-ci-centos-7-undercloud-containers/4eef40f/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2017-11-08_21_25_5422:50
Slowerslagle: so we've set up a job that uses quickstart to fire up the containerized undercloud instead of instack22:51
Slowerslagle: which is mostly working, cept now we've hit this networking error22:51
openstackgerritBen Nemec proposed openstack/tripleo-repos master: Update ceph and opstools repos  https://review.openstack.org/51772822:51
openstackgerritBen Nemec proposed openstack/tripleo-repos master: Implement Proxy Support  https://review.openstack.org/51812722:51
Slowerslagle: I think this is the NeutronPublicInterface it's trying to configure as eth122:51
sshnaidmbnemec, we have tripleo-repos??22:52
Slowerslagle: but there is no eth1, nor is there any kind of config value for the control plane bridged interface in undercloud.conf (that I can tell)22:52
bnemecsshnaidm: We've had it since the end of last cycle.  We couldn't use it in ci until it had support for the AFS mirrors though.22:52
Slowerslagle: obviously that's from https://review.openstack.org/#/c/518118/22:53
Slowerslagle: I'm hoping you know enough about the overcloud + instack to see how those two can be bridged22:54
Sloweror anyone else in here for that matter :)22:55
*** rwsu has quit IRC22:56
slagleSlower: for the multinode jobs, we don't bridge any physical interface to br-ctlplane22:57
slaglesince there is not always an eth122:57
slaglea bridge with no physical interfaces works just fine for our CI purposes22:57
Slowerhmm22:58
Slowerany thoughts on how to set that up with a heat-based undercloud?22:58
slagleSlower: here is how we set that on the instack-underclodu job:22:59
slaglehttps://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/config/testenv/multinode.yml#L2022:59
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260723:00
slagleSlower: yes. what you need to do is write a nic config template to configure the network however you want23:00
Slowerright23:00
Slowerand then add that to ..23:00
* Slower looks23:00
Slowercustom_env_files23:01
slaglei think i already did this for you23:01
slaglehttps://github.com/openstack-infra/tripleo-ci/commit/cb28c9bd6d46a9f5078db87d99885055f56c1fae#diff-1affdc8fbaff2ce4903a858ba511ece323:01
slagledo you have that?23:01
slaglei guess you'd have to add it to your new job23:02
Sloweryeah I would23:02
SlowerI'll check23:02
*** cdearborn has quit IRC23:03
slagleif you use that nic config, and add that mapping to your resource_registry, it should work23:03
slagleand you'll want to do that in your ci job, not in the undercloud deploy code directly23:04
Slowerslagle: awesome! thanks23:04
Slowerslagle: gotcha23:04
openstackgerritBen Nemec proposed openstack/tripleo-docs master: WIP Document use of tripleo-repos  https://review.openstack.org/51863123:04
*** jtomasek has quit IRC23:04
*** gbarros has joined #tripleo23:09
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432823:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172925323:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172958623:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173011123:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged]23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103223:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106323:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre)23:10
openstackLaunchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)23:10
openstackLaunchpad bug 1730111 in tripleo "Volume service hostgroup@tripleo_iscsi failed to start.: CappedVersionUnknown: Unrecoverable Error" [Critical,Triaged]23:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]23:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]23:10
*** ffiore has joined #tripleo23:12
*** hamzy_ has joined #tripleo23:17
*** gbarros has quit IRC23:21
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Refactor cellv2 host discovery logic to avoid races  https://review.openstack.org/51386723:21
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Update default cell_v2 cell when it already exists  https://review.openstack.org/50659523:25
*** ansiwen[q] has joined #tripleo23:34
*** gbarros has joined #tripleo23:35
*** newmember has joined #tripleo23:37
*** achadha has joined #tripleo23:44
*** achadha has quit IRC23:49
*** myoung is now known as myoung|afk23:50
*** ffiore has quit IRC23:53
*** liverpooler has joined #tripleo23:54
*** tosky has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!