| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK https://review.openstack.org/430277 | 00:00 |
|---|---|---|
| *** rook-ugh is now known as rook | 00:03 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo stable/ocata: Re-run ceilometer upgrade in step 5 https://review.openstack.org/447735 | 00:03 |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 00:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 00:10 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-docs master: WIP: Post deployment installation instructions for Octavia https://review.openstack.org/447496 | 00:15 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 00:21 |
| *** dmacpher-afk has quit IRC | 00:26 | |
| *** funzo has quit IRC | 00:30 | |
| *** tbonds has quit IRC | 00:35 | |
| *** limao has joined #tripleo | 00:37 | |
| *** dhill_ has quit IRC | 00:46 | |
| *** pmannidi has quit IRC | 00:51 | |
| *** pmannidi has joined #tripleo | 00:52 | |
| *** tbonds has joined #tripleo | 01:09 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 01:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 01:10 |
| *** dmacpher has joined #tripleo | 01:15 | |
| *** tbonds has quit IRC | 01:20 | |
| *** dsneddon has joined #tripleo | 01:22 | |
| EmilienM | bkero: what is the problem? | 01:28 |
| EmilienM | bkero: can you update the bug report? | 01:28 |
| bkero | EmilienM: It's just what michele said in comment 3. | 01:29 |
| EmilienM | bkero: and you have a fix you said? | 01:29 |
| bkero | apevec's #4 comment references a review, https://review.openstack.org/399578 which fixes this for tripleo.sh. The review just needs to be replicated in quickstart. | 01:29 |
| bkero | EmilienM: I don't have a fix yet, but I have replicated. | 01:30 |
| EmilienM | bkero: I also found that oooq often fail on : http://logs.openstack.org/38/444338/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/8f64d13/console.html#_2017-03-20_21_30_37_550712 | 01:30 |
| *** japestinho has joined #tripleo | 01:30 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 01:31 |
| bkero | EmilienM: Okay. | 01:31 |
| EmilienM | anyway, I'm out for today, if you can update the launchpad with how to fix the thing on newton please do | 01:31 |
| *** funzo has joined #tripleo | 01:31 | |
| bkero | I'll update it when I have a review that fixes it. | 01:31 |
| EmilienM | cool | 01:31 |
| EmilienM | we haven't merged backported since almost a week | 01:32 |
| EmilienM | this is really bad | 01:32 |
| *** funzo has quit IRC | 01:35 | |
| bkero | EmilienM: fixed | 01:38 |
| EmilienM | bkero: ? | 01:38 |
| *** rhallisey has quit IRC | 01:39 | |
| bkero | EmilienM: Submitting a review now | 01:39 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: build-test-packages: Fix branch name for dlrn building in newton https://review.openstack.org/447747 | 01:39 |
| bkero | EmilienM: ^ | 01:39 |
| EmilienM | I'm not sure to understand tbh | 01:40 |
| bkero | That results in built newton packages for me, for tripleo-heat-templates. | 01:40 |
| EmilienM | specially when there is no commit message | 01:40 |
| EmilienM | that explain what it does | 01:40 |
| bkero | Yeah, I'll add that in rev2. I just wanted to show you the fix. | 01:40 |
| EmilienM | well, I don't understand it :) | 01:41 |
| bkero | okay | 01:41 |
| EmilienM | I'll check that out tomorrow, if you can run a stable/newton patch in THT or something | 01:41 |
| EmilienM | so we can see that it actually works, thanks | 01:41 |
| EmilienM | I'm off from now, cheers and thanks | 01:41 |
| bkero | I did, locally. That's how I verified the fix. | 01:42 |
| bkero | I need to check it is the same for other projects though. | 01:42 |
| *** saneax-_-|AFK is now known as saneax | 01:49 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: build-test-packages: Fix branch name for dlrn building in newton https://review.openstack.org/447747 | 01:51 |
| bkero | Okay, updated with detailed commit message and commented on the bug. | 02:01 |
| *** limao has quit IRC | 02:01 | |
| *** limao_ has joined #tripleo | 02:02 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 02:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 02:10 |
| *** ipsecguy_ has quit IRC | 02:22 | |
| *** ipsecguy has joined #tripleo | 02:22 | |
| *** saneax is now known as saneax-_-|AFK | 02:29 | |
| *** fzdarsky__ has joined #tripleo | 02:32 | |
| *** funzo has joined #tripleo | 02:32 | |
| *** fzdarsky_ has quit IRC | 02:35 | |
| *** funzo has quit IRC | 02:37 | |
| *** saneax-_-|AFK is now known as saneax | 02:47 | |
| *** rlandy has quit IRC | 03:06 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 03:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 03:10 |
| *** bkopilov has quit IRC | 03:14 | |
| *** limao_ has quit IRC | 03:21 | |
| *** limao has joined #tripleo | 03:22 | |
| *** gbarros has quit IRC | 03:22 | |
| *** funzo has joined #tripleo | 03:33 | |
| *** fragatin_ has joined #tripleo | 03:34 | |
| *** funzo has quit IRC | 03:37 | |
| *** fragatina has quit IRC | 03:38 | |
| *** fragatin_ has quit IRC | 03:39 | |
| *** fragatina has joined #tripleo | 03:40 | |
| *** saneax is now known as saneax-_-|AFK | 03:40 | |
| *** psahoo has joined #tripleo | 03:42 | |
| *** fragatin_ has joined #tripleo | 03:43 | |
| *** links has joined #tripleo | 03:44 | |
| *** fragatina has quit IRC | 03:45 | |
| *** yamahata has quit IRC | 03:47 | |
| *** fragatin_ has quit IRC | 03:47 | |
| *** gkadam has joined #tripleo | 03:55 | |
| openstackgerrit | Feng Pan proposed openstack/os-net-config master: Add support for VPP interface https://review.openstack.org/424426 | 04:03 |
| *** fragatina has joined #tripleo | 04:03 | |
| *** fragatina has quit IRC | 04:03 | |
| *** fragatina has joined #tripleo | 04:04 | |
| *** tbonds has joined #tripleo | 04:07 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 04:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 04:10 |
| *** d0ugal has quit IRC | 04:16 | |
| *** d0ugal has joined #tripleo | 04:18 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-puppet-elements master: Enabled "tuned-profiles-cpu-partitioning" package for overcloud image https://review.openstack.org/418348 | 04:32 |
| *** skramaja has joined #tripleo | 04:32 | |
| *** bkopilov has joined #tripleo | 04:32 | |
| *** funzo has joined #tripleo | 04:34 | |
| *** saneax-_-|AFK is now known as saneax | 04:37 | |
| *** funzo has quit IRC | 04:39 | |
| *** ramishra has joined #tripleo | 04:45 | |
| *** janki has joined #tripleo | 04:45 | |
| *** saibarspeis has joined #tripleo | 04:46 | |
| *** udesale has joined #tripleo | 04:50 | |
| *** yamahata has joined #tripleo | 04:59 | |
| *** padkrish has joined #tripleo | 05:00 | |
| *** pmannidi has quit IRC | 05:00 | |
| *** pmannidi has joined #tripleo | 05:01 | |
| *** udesale has quit IRC | 05:03 | |
| *** artom has quit IRC | 05:04 | |
| *** udesale has joined #tripleo | 05:06 | |
| *** arxcruz has joined #tripleo | 05:08 | |
| *** psahoo has quit IRC | 05:08 | |
| *** udesale has quit IRC | 05:09 | |
| *** udesale has joined #tripleo | 05:09 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 05:10 |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 05:10 |
| *** ratailor has joined #tripleo | 05:13 | |
| *** dsneddon has quit IRC | 05:23 | |
| *** psahoo has joined #tripleo | 05:24 | |
| *** prateek has joined #tripleo | 05:27 | |
| *** artom has joined #tripleo | 05:34 | |
| *** funzo has joined #tripleo | 05:36 | |
| *** funzo has quit IRC | 05:40 | |
| *** rcernin has joined #tripleo | 05:43 | |
| *** psahoo_ has joined #tripleo | 05:49 | |
| *** yprokule has joined #tripleo | 05:49 | |
| *** psahoo has quit IRC | 05:49 | |
| *** saneax is now known as saneax-_-|AFK | 05:54 | |
| *** radeks has joined #tripleo | 05:55 | |
| *** rcernin has quit IRC | 05:57 | |
| *** dparkes_ has quit IRC | 06:06 | |
| *** udesale has quit IRC | 06:06 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Create dir on subnode for copying deploy.env https://review.openstack.org/447849 | 06:07 |
| *** udesale has joined #tripleo | 06:07 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 06:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 06:10 |
| *** zshi has joined #tripleo | 06:10 | |
| *** masco has joined #tripleo | 06:11 | |
| *** ealcaniz has joined #tripleo | 06:11 | |
| *** jprovazn has joined #tripleo | 06:16 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Support report by periodic multinode jobs https://review.openstack.org/447860 | 06:19 |
| *** jbadiapa has joined #tripleo | 06:30 | |
| *** saneax-_-|AFK is now known as saneax | 06:31 | |
| *** jaosorior has joined #tripleo | 06:31 | |
| *** ukalifon has joined #tripleo | 06:32 | |
| *** yolanda has quit IRC | 06:32 | |
| *** funzo has joined #tripleo | 06:36 | |
| *** lmiccini has joined #tripleo | 06:40 | |
| *** funzo has quit IRC | 06:41 | |
| *** janki is now known as janki|afk | 06:42 | |
| *** yolanda has joined #tripleo | 06:46 | |
| *** fragatina has quit IRC | 06:56 | |
| *** dparkes has joined #tripleo | 07:04 | |
| *** padkrish has quit IRC | 07:07 | |
| *** pgadiya has joined #tripleo | 07:08 | |
| *** pgadiya has quit IRC | 07:08 | |
| apetrich | jtomasek, I was thinking about the caching during the night | 07:08 |
| *** pgadiya has joined #tripleo | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 07:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 07:10 |
| chandankumar | sshnaidm: hello | 07:12 |
| jtomasek | apetrich: hi | 07:12 |
| sshnaidm | chandankumar, hi | 07:12 |
| chandankumar | shadower: mitaka tempest jobs are not running for this https://review.openstack.org/#/c/446868/ | 07:12 |
| chandankumar | please have a look. | 07:13 |
| apetrich | jtomasek, heya. right now the getparametersaction takes 15s. swift caching takes 3s and memory caching takes 1.5s is the swift fast enough for the front end? | 07:13 |
| jtomasek | apetrich: well, the faster the better | 07:14 |
| jtomasek | apetrich: it is operation which is done often, so it is affecting GUI usability | 07:15 |
| apetrich | jtomasek, I'm still not sure why even memory caching takes so long. I'm somewhat new to mistral | 07:16 |
| apetrich | jtomasek, I'm trying to think of a good ttl for the cache and if we get a better way to do it. | 07:17 |
| apetrich | jtomasek, do you call it as "mistral run-action tripleo.parameters.get" or in a different manner? | 07:19 |
| jtomasek | apetrich: we call it via mistral api | 07:20 |
| sshnaidm | chandankumar, mitaka is broken, it can't install undercloud | 07:21 |
| sshnaidm | chandankumar, so there is nothing to test there.. | 07:21 |
| chandankumar | sshnaidm: is anyone working on fixing it ? | 07:21 |
| jtomasek | apetrich: I need to relocate, I'll be back in ~2 hours | 07:22 |
| apetrich | jtomasek, no worries | 07:23 |
| *** jtomasek has quit IRC | 07:23 | |
| *** iranzo has joined #tripleo | 07:23 | |
| sshnaidm | chandankumar, not sure | 07:24 |
| *** skramaja_ has joined #tripleo | 07:34 | |
| *** tesseract has joined #tripleo | 07:35 | |
| bandini_ | morning | 07:36 |
| *** bandini_ is now known as bandini | 07:36 | |
| *** funzo has joined #tripleo | 07:37 | |
| *** skramaja has quit IRC | 07:37 | |
| *** funzo has quit IRC | 07:42 | |
| *** padkrish has joined #tripleo | 07:46 | |
| *** nyechiel has joined #tripleo | 07:55 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** bogdando has joined #tripleo | 08:00 | |
| *** skramaja_ is now known as skramaja | 08:00 | |
| *** zzzeek has joined #tripleo | 08:00 | |
| *** florianf has joined #tripleo | 08:05 | |
| *** padkrish has quit IRC | 08:05 | |
| *** suuuper has joined #tripleo | 08:06 | |
| *** shardy has joined #tripleo | 08:07 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 08:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 08:10 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Fix the regexp extracting the number of controllers https://review.openstack.org/447017 | 08:10 |
| *** pcaruana has joined #tripleo | 08:11 | |
| *** aufi has joined #tripleo | 08:11 | |
| *** snecklifter has quit IRC | 08:13 | |
| *** jlinkes has joined #tripleo | 08:15 | |
| *** jprovazn has quit IRC | 08:17 | |
| *** jprovazn has joined #tripleo | 08:18 | |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-heat-templates master: Use httpd in Zaqar docker service https://review.openstack.org/447476 | 08:22 |
| *** jprovazn has quit IRC | 08:24 | |
| *** shardy has quit IRC | 08:24 | |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-heat-templates master: Use httpd in Heat docker services https://review.openstack.org/447504 | 08:27 |
| *** dmacpher has quit IRC | 08:28 | |
| *** cylopez has joined #tripleo | 08:32 | |
| jaosorior | therve: is zaqar already running over httpd in the puppet service? | 08:32 |
| therve | That's a good question | 08:33 |
| jaosorior | therve: I don't think it is | 08:33 |
| jaosorior | therve: cause I didn't code it, and if you didn't code it, probably it wasn't done :/ | 08:34 |
| *** jpich has joined #tripleo | 08:34 | |
| therve | jaosorior, I guess I made most changes, so it just needs the service reload | 08:34 |
| therve | Well the puppet changes are in, and the undercloud changes too | 08:34 |
| therve | We may need a tht change too indeed | 08:34 |
| -openstackstatus- NOTICE: Wiki is broken with database problems, we are working to resolve it | 08:34 | |
| *** ChanServ changes topic to "Wiki is broken with database problems, we are working to resolve it" | 08:34 | |
| jaosorior | therve: right | 08:35 |
| jaosorior | therve: ah, puppet-tripleo already has the zaqar over httpd bits? | 08:35 |
| therve | puppet-tripleo? | 08:36 |
| therve | Good god how many times do we have to do the same change | 08:36 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-common master: Custom DHCP agent for DHCP-relay support https://review.openstack.org/447948 | 08:36 |
| *** jprovazn has joined #tripleo | 08:37 | |
| therve | Ah yeah that's for the overcloud | 08:37 |
| *** funzo has joined #tripleo | 08:38 | |
| *** panda|off is now known as panda|pto | 08:38 | |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud master: Custom DHCP agent for DHCP-relay support https://review.openstack.org/447214 | 08:39 |
| jaosorior | therve: so, ideally the undercloud should consume puppet-tripleo and configure a service that way as well. Sometimes that's not possible or it doesn't bring much benefit. But this is a case where it would have. | 08:40 |
| therve | jaosorior, Yeah except the swift backend for zaqar | 08:40 |
| therve | But anyway | 08:40 |
| therve | So I need to change puppet-tripleo to use apache | 08:41 |
| jaosorior | yeah | 08:41 |
| therve | And then tht for the upgrade task | 08:41 |
| *** amoralej|off is now known as amoralej | 08:41 | |
| jaosorior | therve: also the puppet/services/zaqar template has to "inherit" or use the ApacheBase one, as a reference you can check the barbican or keystone templates. | 08:41 |
| jaosorior | therve: if you need any help let me know | 08:41 |
| *** ChanServ changes topic to "TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt" | 08:43 | |
| -openstackstatus- NOTICE: Wiki problems have been fixed, it's up and running | 08:43 | |
| *** funzo has quit IRC | 08:43 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Add auth/authtoken configuration for novajoin https://review.openstack.org/447953 | 08:44 |
| *** jpena|off is now known as jpena | 08:44 | |
| openstackgerrit | Thomas Herve proposed openstack/puppet-tripleo master: Include zaqar apache module https://review.openstack.org/447957 | 08:52 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Add etcd image to overcloud containers https://review.openstack.org/445883 | 08:53 |
| *** jpena is now known as jpena|off | 08:57 | |
| *** pgadiya has quit IRC | 09:01 | |
| *** akrivoka has joined #tripleo | 09:02 | |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-heat-templates master: Run Zaqar with httpd in puppet service https://review.openstack.org/447963 | 09:02 |
| jaosorior | therve: looks about right | 09:03 |
| therve | jaosorior, There is a scenario that tests this? | 09:03 |
| *** openstackgerrit has quit IRC | 09:03 | |
| jaosorior | therve: scenario002 should do something basic | 09:04 |
| therve | OK :) | 09:04 |
| therve | I noticed that the websocket service is probably not deployed, but we can do that later | 09:04 |
| *** pgadiya has joined #tripleo | 09:04 | |
| *** jpena|off is now known as jpena | 09:04 | |
| jaosorior | works for me | 09:04 |
| jaosorior | therve: thanks for the patches dude | 09:05 |
| therve | jaosorior, No problem! | 09:05 |
| *** athomas has joined #tripleo | 09:06 | |
| *** pmannidi has quit IRC | 09:09 | |
| *** pmannidi has joined #tripleo | 09:10 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 09:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 09:10 |
| *** b00tcat has quit IRC | 09:12 | |
| *** openstackgerrit has joined #tripleo | 09:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/447270 | 09:12 |
| *** dtantsur|afk is now known as dtantsur | 09:13 | |
| *** zoli|gone is now known as zoli | 09:15 | |
| *** akrivoka has quit IRC | 09:17 | |
| *** akrivoka has joined #tripleo | 09:19 | |
| *** paramite has joined #tripleo | 09:22 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Add auth/authtoken configuration for novajoin https://review.openstack.org/447953 | 09:24 |
| *** akrivoka has quit IRC | 09:25 | |
| *** ckyriakidou has joined #tripleo | 09:26 | |
| *** mcornea has joined #tripleo | 09:26 | |
| *** akrivoka has joined #tripleo | 09:28 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: docker/keystone: Bind mount entire fernet keys repository https://review.openstack.org/446473 | 09:31 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: docker/keystone: Actually set fernet as the default token provider https://review.openstack.org/447971 | 09:31 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: test containers with single-nic-vlan https://review.openstack.org/447973 | 09:32 |
| jaosorior | sshnaidm: why is that a wIP? I think it's a very reasonable commit. | 09:33 |
| *** dsariel has joined #tripleo | 09:34 | |
| *** dsariel_ has joined #tripleo | 09:35 | |
| sshnaidm | jaosorior, just started to check.. not sure it works on ci | 09:35 |
| *** stendulker has joined #tripleo | 09:36 | |
| *** derekh has joined #tripleo | 09:36 | |
| *** karimb has joined #tripleo | 09:37 | |
| *** funzo has joined #tripleo | 09:39 | |
| remix_tj | Is there any doc, or blogpost or somewhat that explains how to implement a new openstack service in tripleo? | 09:40 |
| *** jtomasek has joined #tripleo | 09:42 | |
| *** funzo has quit IRC | 09:43 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Cleanup docker services templates https://review.openstack.org/443730 | 09:46 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Fix alert message in overcloud status check https://review.openstack.org/447977 | 09:46 |
| sshnaidm | jaosorior, can you please take a look? https://review.openstack.org/#/c/447977/1 | 09:48 |
| *** flepied has quit IRC | 09:48 | |
| *** b00tcat has joined #tripleo | 09:49 | |
| sshnaidm | Either we have a new format for successful stack completion, or something is wrong here: http://logs.openstack.org/30/447530/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/7439413/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 09:53 |
| sshnaidm | [overcloud]: CREATE_COMPLETE Stack CREATE completed successfully | 09:53 |
| *** tosky has joined #tripleo | 09:54 | |
| sshnaidm | jaosorior, EmilienM adarazs ^^ | 09:54 |
| adarazs | yep, I just noticed too :) | 09:54 |
| *** akrivoka has quit IRC | 09:54 | |
| sshnaidm | adarazs, maybe to ignore strings and check only status codes? | 09:55 |
| sshnaidm | because I suppose it fails a lot of jobs now | 09:55 |
| *** paramite has quit IRC | 09:56 | |
| adarazs | that could work. I don't remember why we were loooking for the string instead of the status code. I think it gave false results, succeeding but returning with nonzero | 09:57 |
| *** udesale has quit IRC | 09:57 | |
| sshnaidm | adarazs, yeah, I suppose it was such bug | 09:58 |
| adarazs | sshnaidm, EmilienM: I submitted the newton/ocata gate fix now, hopefully it will work. | 09:59 |
| *** limao has quit IRC | 10:00 | |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-heat-templates master: Use httpd in Zaqar docker service https://review.openstack.org/447476 | 10:01 |
| panda|pto | status code wasn't reliable | 10:01 |
| *** pgadiya has quit IRC | 10:02 | |
| sshnaidm | panda|pto, it seems like string checking is less reliable | 10:03 |
| therve | sshnaidm, What's wrong in that log file? | 10:03 |
| therve | I may have changed the output, though I tried not to | 10:03 |
| sshnaidm | therve, it should be "Stack overcloud CREATE_COMPLETE" and it's '[overcloud]: CREATE_COMPLETE Stack CREATE completed successfully' | 10:04 |
| *** dbecker has joined #tripleo | 10:04 | |
| therve | Ah, hum | 10:04 |
| *** salmankhan has joined #tripleo | 10:04 | |
| panda|pto | don't we look just for CREATE_COMPLETE ? | 10:05 |
| sshnaidm | panda|pto, no, it's a lot of them | 10:05 |
| bandini | remix_tj: http://tripleo.org/developer/tht_walkthrough/tht_walkthrough.html maybe? | 10:06 |
| remix_tj | bandini: i'll take a look thanks. | 10:07 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 10:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 10:10 |
| bandini | has anyone already opened a bug to track the issue with the new overcloud complete string breaking oooq? | 10:12 |
| therve | sshnaidm, Do you have an example of the previous output? | 10:12 |
| sshnaidm | therve, http://logs.openstack.org/30/447530/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/7439413/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 10:14 |
| therve | Thanks | 10:14 |
| *** jaosorior is now known as jaosorior_brb | 10:14 | |
| therve | sshnaidm, That's the same link? :) | 10:15 |
| *** pgadiya has joined #tripleo | 10:16 | |
| panda|pto | mh, tripleo.sh was checking the exit code | 10:16 |
| *** adarazs is now known as adarazs_brb | 10:16 | |
| sshnaidm | therve, sorry, http://logs.openstack.org/53/447653/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/6d23589/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 10:19 |
| sshnaidm | panda|pto, if it has wrong exit code - who cares? it will fail on pingtest | 10:19 |
| therve | sshnaidm, Ah, so it's missing the extra message at the end | 10:21 |
| *** udesale has joined #tripleo | 10:21 | |
| sshnaidm | therve, yeah, sometimes these messages could be mixed with others as well, not the most reliable part | 10:22 |
| *** flepied has joined #tripleo | 10:23 | |
| panda|pto | guess we didn't want to wait for the pingtest to fail, and if it fails early, you know more easily in what part it went wrong. At least that's what I do, see what was the last log create to understand where I should look first. | 10:23 |
| *** janki|afk is now known as janki | 10:27 | |
| *** ccamacho has joined #tripleo | 10:27 | |
| *** zoli is now known as zoli|lunch | 10:27 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Don't use string parsing when detecting deploy result https://review.openstack.org/447993 | 10:27 |
| sshnaidm | adarazs_brb, panda|pto ^^ | 10:28 |
| therve | sshnaidm, Is it possible that something changed the way the output is managed? | 10:29 |
| therve | Because it seems that I kept compatibility with that message | 10:29 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Use long format when print stack failures https://review.openstack.org/447994 | 10:30 |
| panda|pto | therve: compatibility in what way ? | 10:30 |
| therve | panda|pto, Well it should be returned as before | 10:30 |
| sshnaidm | therve, I don't think so, sometimes it's just garbaged/broken/cut on half, etc | 10:31 |
| therve | sshnaidm, Ah so it's not really a new issue | 10:31 |
| panda|pto | therve: still in standard output ? | 10:32 |
| *** yamahata has quit IRC | 10:32 | |
| therve | panda|pto, Yes | 10:32 |
| *** ealcaniz has quit IRC | 10:34 | |
| *** thrash|g0ne is now known as thrash | 10:35 | |
| *** salmankhan has quit IRC | 10:38 | |
| *** funzo has joined #tripleo | 10:39 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Disable exit on error for pacemaker commands for update flow https://review.openstack.org/439706 | 10:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Keep existing data for containerized ironic-conductor https://review.openstack.org/443173 | 10:40 |
| panda|pto | mh, why we're not even seeing the overcloud endpoint ? | 10:40 |
| panda|pto | overcloud endpoint message | 10:41 |
| sshnaidm | adarazs_brb, panda|pto - in your time pls: Create dir on subnode for copying deploy.env https://review.openstack.org/#/c/447849 | 10:42 |
| *** jkilpatr has quit IRC | 10:44 | |
| *** funzo has quit IRC | 10:44 | |
| *** shardy has joined #tripleo | 10:45 | |
| *** eck` is now known as eck`gone | 10:46 | |
| *** salmankhan has joined #tripleo | 10:47 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: [overcloud-prep-images] Update deprecated openstack command https://review.openstack.org/440518 | 10:52 |
| *** salmankhan has quit IRC | 10:52 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: [overcloud-deploy] Fix hosts file generation https://review.openstack.org/438625 | 10:52 |
| *** udesale has quit IRC | 10:54 | |
| lhinds | do we have a spec freeze for tripleo? | 10:55 |
| shardy | lhinds: No, but we're likely to respect a feature freeze from around pike-3 | 10:57 |
| shardy | https://releases.openstack.org/pike/schedule.html | 10:57 |
| *** aufi has quit IRC | 10:58 | |
| shardy | lhinds: that says the cycle-trailing FF is RC1, but I'd expect us to be focussing more on bugfixing by then, although every cycle there are late things we end up landing anyway | 10:58 |
| *** aufi has joined #tripleo | 10:58 | |
| lhinds | thx shardy , trying to work out what my hard dates are for this cycle | 10:59 |
| shardy | lhinds: tbh code reviews are more likely to be a blocker than specs | 10:59 |
| shardy | https://blueprints.launchpad.net/tripleo/pike | 10:59 |
| *** remix_tj has quit IRC | 11:00 | |
| shardy | we've got 46 blueprints targetted to pike - I doubt we'll land half of them, but if everyone did finish them we'd probably struggle with reviews | 11:00 |
| shardy | especially if folks decide to leave implementation late | 11:00 |
| lhinds | shardy: should have another patch on that aide service, inc ops-tools. | 11:00 |
| shardy | lhinds: IOW late specs or code are definitely taking a risk ;) | 11:00 |
| lhinds | shardy: ack | 11:00 |
| shardy | lhinds: ack, will try to check it out | 11:01 |
| *** milan has joined #tripleo | 11:01 | |
| *** udesale has joined #tripleo | 11:02 | |
| *** jkilpatr has joined #tripleo | 11:02 | |
| *** morazi has quit IRC | 11:03 | |
| *** stendulker has quit IRC | 11:04 | |
| *** salmankhan has joined #tripleo | 11:05 | |
| *** mburned_out is now known as mburned | 11:06 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 11:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 11:10 |
| *** zshi has quit IRC | 11:12 | |
| *** zshi has joined #tripleo | 11:12 | |
| EmilienM | hello | 11:15 |
| EmilienM | adarazs_brb, sshnaidm, have you seen https://review.openstack.org/#/c/447747/ ? | 11:16 |
| adarazs_brb | EmilienM: yes, so I'm sorry that I didn't notice that the gerrit-launchpad integration was not working. I made a fix for this in 30 minutes yesterday after taking the bug. | 11:17 |
| *** adarazs_brb is now known as adarazs | 11:17 | |
| adarazs | EmilienM: see my explanation here: https://bugs.launchpad.net/tripleo/+bug/1674067/comments/6 | 11:18 |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 11:18 |
| *** udesale has quit IRC | 11:18 | |
| *** udesale has joined #tripleo | 11:18 | |
| EmilienM | adarazs: ok... | 11:19 |
| EmilienM | adarazs: why gate-tripleo-ci-centos-7-nonha-multinode-oooq failed on your patch? | 11:19 |
| EmilienM | gate-tripleo-ci-centos-7-nonha-multinode-oooq looks super unstable | 11:20 |
| adarazs | EmilienM: package installation failed. I assume network hiccup. I'm trying to recheck it and it passed that already. | 11:20 |
| EmilienM | sounds like timeouts : http://logs.openstack.org/49/447849/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/309bac5/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 11:21 |
| EmilienM | sshnaidm: ^ on https://review.openstack.org/#/c/447849/ | 11:22 |
| sshnaidm | EmilienM, yeah, weird | 11:23 |
| adarazs | EmilienM: it was Error downloading packages: python2-keystoneauth1-2.18.0-0.20170120194254.9365dbe.el7.centos.noarch: [Errno 256] No more mirrors to try. | 11:23 |
| jistr | matbu: hi, how do you use the upgrade-only.yml playbook in OOOQ extras? i tried it but it doesn't do anything because it complains of hosts not being in inventory | 11:23 |
| adarazs | EmilienM: so it's not overcloud timeout this time | 11:23 |
| adarazs | http://logs.openstack.org/30/447530/1/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/814c855/logs/undercloud/home/jenkins/install_packages.sh.log.txt.gz | 11:23 |
| jistr | matbu: so i tried copy pasting the "add virthost/undercloud/overcloud hosts to the inventory" tasks to that playbook | 11:24 |
| EmilienM | adarazs: is it rdo server or openstack mirrors? | 11:24 |
| jistr | but instead of adding them to inventory | 11:24 |
| jistr | it destroyed my ssh.config.ansible | 11:24 |
| jistr | and now i can't do much :) | 11:24 |
| jistr | so i guess that wasn't the correct way | 11:24 |
| adarazs | EmilienM: it's from http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/python2-keystoneauth1-2.18.0-0.20170120194254.9365dbe.el7.centos.noarch.rpm: | 11:25 |
| adarazs | which works. so it was some temporary error with this CDN | 11:25 |
| EmilienM | it happens very very often | 11:26 |
| bandini | yeah I have seen it in a bunch of jobs as well | 11:27 |
| EmilienM | I'm reporting it on #rdo and see what we can do... | 11:27 |
| adarazs | not sure what to do about that, I really don't know why we are hosting these files there and what are the alternatives. | 11:28 |
| bandini | do we go through a proxy at least? | 11:28 |
| EmilienM | not sure | 11:28 |
| bandini | that would ameliorate these kind of issues, I'd hope | 11:28 |
| jistr | ameliorate! | 11:29 |
| EmilienM | adarazs: so if I change https://review.openstack.org/#/c/447229/ to depends-on https://review.openstack.org/#/c/447530/ - it should pass? | 11:29 |
| jistr | where's hewbrocca_afk when we need him | 11:29 |
| adarazs | EmilienM: yes. if the zuul depends on does its magic. :)_ | 11:29 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/newton: Make sure PrePuppet runs before any Deployment_Step https://review.openstack.org/447229 | 11:29 |
| EmilienM | bandini: ^ | 11:29 |
| bandini | ack, let's cross fingers | 11:30 |
| bandini | jistr: rereading 'ameliorate' and it sounds like something Harry Potter would say while using his magic wand ;) | 11:31 |
| bandini | "Ameliorate CI Upstreamus"! | 11:31 |
| jistr | :D yea | 11:31 |
| adarazs | :D | 11:31 |
| ccamacho | bandini++ | 11:31 |
| adarazs | we should have a karma bot here. :) | 11:32 |
| *** ealcaniz has joined #tripleo | 11:32 | |
| *** ealcaniz has quit IRC | 11:32 | |
| rnoriega | Hello guys! pliu and myself would need some help reviewing these two patches: https://review.openstack.org/#/c/447429/ and https://review.openstack.org/#/c/444050/3 | 11:32 |
| rnoriega | thanks tripleOers! | 11:33 |
| rnoriega | ;) | 11:33 |
| *** fzdarsky__ is now known as fzdarsky|lunch | 11:35 | |
| *** rbrady has joined #tripleo | 11:36 | |
| EmilienM | rnoriega: done | 11:38 |
| adarazs | EmilienM: do you know what this "groups: tripleo" does here? http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/projects.yaml#n6684 -- can this be the reason why the launchpad - gerrit integration is not working now for changes from the quickstart repos? | 11:38 |
| rnoriega | EmilienM, nice!! thanks | 11:39 |
| EmilienM | adarazs: yes it's launchpad... | 11:39 |
| EmilienM | adarazs: please update it, thanks! | 11:40 |
| adarazs | EmilienM: yep, will do. all this confusion could have been avoided if that worked. :) | 11:40 |
| adarazs | submitting a fix... | 11:40 |
| EmilienM | thanks | 11:41 |
| *** salmankhan has quit IRC | 11:42 | |
| dtantsur | EmilienM, hey! it's re https://review.openstack.org/351803 | 11:43 |
| dtantsur | do you think we can give it a try? | 11:43 |
| *** bfournie has quit IRC | 11:43 | |
| EmilienM | dtantsur: what patch fixed it in Ironic? I want to make sure we have it in the last promotion | 11:43 |
| EmilienM | (last promotion was 8 days ago) | 11:44 |
| dtantsur | EmilienM, no clue. last time we tried it was end of Newton, a hell lot of patches got in. | 11:44 |
| *** athomas has quit IRC | 11:44 | |
| *** salmankhan has joined #tripleo | 11:44 | |
| dtantsur | I dunno even if the fix was in ironic, not in oslo.msg or something else | 11:44 |
| EmilienM | dtantsur: last time we switched it, it was unstable. We could try again | 11:44 |
| dtantsur | EmilienM, it passed 3 rechecks /me shrugs | 11:45 |
| dtantsur | we can recheck a few times more | 11:45 |
| EmilienM | dtantsur: I would give a last recheck and +2 | 11:45 |
| *** zoli|lunch is now known as zoli | 11:45 | |
| EmilienM | dtantsur: we're already have many issues right now in our CI, I don't want to add a new one | 11:45 |
| dtantsur | EmilienM, feel free to block it until better times, if you expect them to come | 11:45 |
| dtantsur | just don't forget please :) | 11:46 |
| *** dsariel has quit IRC | 11:46 | |
| EmilienM | dtantsur: sure, thx | 11:46 |
| dtantsur | fwiw we have ironic-api running with wsgi in one of upstream jobs http://logs.openstack.org/49/437549/15/check/gate-ironic-dsvm-standalone-ubuntu-xenial-nv/9de4dd4/logs/apache/ironic-api-access.txt.gz | 11:47 |
| adarazs | EmilienM: https://review.openstack.org/448024 - Add tripleo-quickstart* repos to the tripleo group | 11:47 |
| rook | hey shardy -- well I have got past the *HostDeploy issue pretty regularly now (restarting os-collect-config on all the hosts seems to help?) however, now I am stuck here *NodesDeployment | 11:47 |
| EmilienM | dtantsur: is it voting now? | 11:48 |
| EmilienM | adarazs: thx | 11:48 |
| dtantsur | EmilienM, not yet (not because of wsgi, it's a new job overall) | 11:48 |
| EmilienM | so we would be voting before you ;-) | 11:49 |
| dtantsur | is it the first time? ;) | 11:49 |
| EmilienM | nope but just saying | 11:49 |
| dtantsur | but I agree, I can WIP it until we get it voting ourselves | 11:49 |
| *** lucasagomes is now known as lucas-brb | 11:50 | |
| *** fultonj_ has quit IRC | 11:50 | |
| adarazs | bandini, EmilienM: FYI the oooq gate job for https://review.openstack.org/#/c/447229/ just passed the package building step, so the newton/other stable branches are fixed after my change merges. | 11:51 |
| EmilienM | happy face | 11:51 |
| bandini | yay! | 11:51 |
| shardy | rook: hey, ugh, something must be wrong as that's basically the next step - and *AllNodesDeployment only write some hieradata | 11:52 |
| shardy | rook: on newton that suffers from the same issue as *HostsDeploy in that if there is an error, the deployment hangs | 11:52 |
| *** yolanda has quit IRC | 11:52 | |
| rook | ugh, I looked at services on the undercloud, i don't see anything that should be hanging this up | 11:52 |
| shardy | rook: that is also fixed in ocata | 11:52 |
| *** eck`gone is now known as eck` | 11:53 | |
| shardy | rook: looking at the IN_PROGRESS deployments, is it only a subset which aren't completing? | 11:53 |
| shardy | e.g consistently a bad node or something? | 11:53 |
| rook | The best solution I have to get more nodes is to use the undercloud to simply create the baremetal, then write a ansible script to update nova.conf to configure the nodes to join the cluster. | 11:53 |
| shardy | :( | 11:53 |
| rook | right | 11:53 |
| rook | I don't want to do that. | 11:53 |
| rook | however, this CNCF gear is going to go away soon | 11:54 |
| rook | and I have burnt so much time | 11:54 |
| *** yolanda has joined #tripleo | 11:54 | |
| shardy | rook: right, I just don't understand why something as simple as writing a hosts file, or writing hieradata is failing for you | 11:54 |
| rook | shardy yeah, it is the same symptom as *HostDeploy | 11:54 |
| shardy | hence suspecting something bad about the environment, but we've not managed to figure out what | 11:54 |
| *** abishop has joined #tripleo | 11:54 | |
| *** jayg|g0n3 is now known as jayg | 11:55 | |
| shardy | also yesterday it was running through to puppet, so I'm confused why this is inconsistent | 11:55 |
| rook | shardy: the only change has been the number of elements (bumped that) | 11:55 |
| rook | i feel like once I made that change, all hell broke loose. | 11:55 |
| rook | lose* | 11:55 |
| rook | ugh loose* lol | 11:56 |
| * rook drinks more | 11:56 | |
| *** pgadiya has quit IRC | 11:57 | |
| shardy | rook: I don't think increasing the yaql limits is related, but you could try reducing it if you think it correlated with the problems | 11:57 |
| rook | oh that is fantastic the swift.log on the undercloud is totally blank | 11:58 |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1646451 | 11:59 |
| openstack | Launchpad bug 1646451 in tripleo "Undercloud has no swift logs" [Medium,Triaged] | 11:59 |
| rook | haha | 11:59 |
| shardy | yeah I noticed that but we've not got around to fixing it yet | 11:59 |
| rook | I think I have seen this in the past | 11:59 |
| rook | I think it is due to the swift config | 11:59 |
| shardy | should be able to manually enable some logging | 11:59 |
| openstackgerrit | Dmitry Tantsur proposed openstack/puppet-tripleo stable/ocata: Explicitly configure credentials used by ironic to access other services https://review.openstack.org/448028 | 12:00 |
| honza | jtomasek: florianf: anything wrong with this? https://review.openstack.org/#/c/443191/ | 12:00 |
| jtomasek | honza, I plan to get to it today, sorry for the delay | 12:00 |
| honza | jtomasek: np! | 12:01 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add dracut-regenerate elements https://review.openstack.org/447047 | 12:01 |
| florianf | honza: Havent tested it yet, sorry. Will do it later. | 12:01 |
| matbu | jistr: hey sorry i was out for lunch | 12:01 |
| matbu | jistr: /me reads back | 12:01 |
| *** radez` is now known as radez | 12:01 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates stable/ocata: Explicitly configure credentials used by ironic to access other services https://review.openstack.org/448029 | 12:02 |
| matbu | jistr: so i think if you add the undercloud in your hosts inventory | 12:02 |
| matbu | jistr: it should works | 12:02 |
| jistr | matbu: like manually edit .quickstart/hosts file? | 12:02 |
| matbu | jistr: if you dpeloyed your env with quickstart, it should work easily | 12:03 |
| jistr | yes i did, but it did not work... | 12:03 |
| jistr | i mean the deployment worked | 12:03 |
| matbu | jistr: do you have the undercloud in the hosts file ? | 12:03 |
| jistr | but upgrade-only.yml didn't work | 12:03 |
| matbu | jistr: did you try to use it with ansible-playbook command or with quickstart.sh ? | 12:04 |
| jistr | matbu: right now i don't have anything, as my ssh.config.ansible (and probably hosts file too) was destroyed as i wrote above | 12:04 |
| matbu | jistr: yep :/ | 12:04 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates stable/ocata: DO NOT MERGE: testing ironic ocata https://review.openstack.org/448030 | 12:04 |
| jistr | matbu: i assumed that rerunning the "add virthost/uc/oc to inventory" tasks would be safe, but it isn't... | 12:04 |
| *** liverpooler has quit IRC | 12:05 | |
| *** liverpooler has joined #tripleo | 12:05 | |
| matbu | jistr: you use the quickstart.sh script ? | 12:05 |
| jistr | matbu: yes | 12:05 |
| jistr | is that good or bad? :) | 12:06 |
| matbu | jistr: hehe, well "it is" i guess, but im not very familiar with it | 12:07 |
| therve | rook, Swift logs are in the journal iirc | 12:07 |
| jistr | matbu: you're running without quickstart.sh usually? | 12:07 |
| adarazs | EmilienM: grrr, now I can't land https://review.openstack.org/447530 until https://review.openstack.org/#/c/447993/ lands because the output of the successful overcloud creation changed. | 12:07 |
| matbu | jistr: so i mostly make some stupid things and drop all my workspace :) | 12:07 |
| rook | therve: well, on this undercloud they are fubar. | 12:07 |
| EmilienM | lol | 12:07 |
| rook | therve: Error was encountered while opening journal files: Input/output error | 12:07 |
| matbu | jistr: if i want to run the upgrade-only on top of an existing dpeloyment yes | 12:07 |
| *** bkopilov has quit IRC | 12:07 | |
| EmilienM | adarazs: +2 | 12:08 |
| matbu | jistr: but i mostly use a full deploy+upgrade | 12:08 |
| openstackgerrit | Merged openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/447270 | 12:08 |
| matbu | jistr: the upgrade.yml playbook | 12:08 |
| *** pgadiya has joined #tripleo | 12:08 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: build-test-packages: fix distgit branch naming https://review.openstack.org/447530 | 12:08 |
| EmilienM | adarazs: rebased and approved | 12:08 |
| EmilienM | adarazs: so review https://review.openstack.org/#/c/447993/1 and we're good | 12:09 |
| jistr | matbu: yea that's ideal if things are already working, but for tuning every change this way, i'd probably only get like 2 or 3 tries per day i guess :D | 12:09 |
| jistr | so i need to iterate a bit faster | 12:09 |
| adarazs | EmilienM: done. | 12:09 |
| * adarazs goes to lunch while these jobs are running. | 12:09 | |
| therve | rook, /var/log/messages maybe? | 12:09 |
| *** adarazs is now known as adarazs_lunch | 12:09 | |
| rook | therve: nada | 12:10 |
| jistr | matbu: i might try running without quickstart.sh for the upgrade-only then, and see how far i get | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 12:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 12:10 |
| *** morazi has joined #tripleo | 12:10 | |
| *** fultonj has joined #tripleo | 12:10 | |
| matbu | jistr: yep, you will have to set the role path afair and perhaps the ansible_ssh_args variable | 12:10 |
| *** eck` is now known as eck`gone | 12:10 | |
| rook | therve: I have no data that swift is the issue here.. | 12:12 |
| rook | just trying to rule out everything | 12:12 |
| * therve nods | 12:13 | |
| rook | shardy: i did see : https://gist.githubusercontent.com/jtaleric/23c57f4aa7741f8eb73cd81ba65e03f4/raw/926ccf0cb472616f0ad5c2a084609cbe32c73d23/Heat-error_on-failed-node.log | 12:14 |
| rook | let me see if I see this again | 12:14 |
| *** eck`gone is now known as eck` | 12:15 | |
| therve | That's a mongo error | 12:15 |
| *** fultonj_ has joined #tripleo | 12:15 | |
| therve | But status code is 0, not sure it mattered | 12:15 |
| shardy | Also that's a message from running puppet, which happens after *AllNodesDeployment, so I'm confused | 12:16 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch trunk/cbs/buildlogs to use https https://review.openstack.org/448036 | 12:17 |
| rook | therve: I think it should be 1 don't you? | 12:17 |
| rook | i mean that is very misleading | 12:17 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Switch trunk/cbs/buildlogs to use https https://review.openstack.org/448037 | 12:17 |
| rook | unless it is something that can be ignored. | 12:17 |
| therve | rook, It should be 1 if it's actually an error, it doesn't seem to be | 12:18 |
| shardy | yeah the exception in stderr does make it look like an error tho | 12:18 |
| *** ansmith has joined #tripleo | 12:18 | |
| shardy | so perhaps we can raise a bug to clean that up if it's not actually an error | 12:19 |
| *** rhallisey has joined #tripleo | 12:19 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Switch trunk/cbs/buildlogs to use https https://review.openstack.org/448037 | 12:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Switch trunk/cbs/buildlogs to use https https://review.openstack.org/448041 | 12:20 |
| *** bfournie has joined #tripleo | 12:20 | |
| EmilienM | infra | 12:21 |
| EmilienM | oops | 12:21 |
| rook | shardy I have a deployment running atm, lets see where things choke :/ | 12:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-docs master: Switch trunk/cbs/buildlogs to use https https://review.openstack.org/448044 | 12:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-repos master: Add .gitreview https://review.openstack.org/448045 | 12:23 |
| dtantsur | shardy, hey. I remember you had a post on injecting a puppet module from source into overcloud deployment. do you have a link handy? also what about adding it to http://tripleo.org/developer/in_progress_review.html ? | 12:23 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-repos master: Switch trunk/cbs/buildlogs to use https https://review.openstack.org/448046 | 12:24 |
| shardy | dtantsur: yup it's http://hardysteven.blogspot.co.uk/2016/08/tripleo-deploy-artifacts-and-puppet.html | 12:25 |
| shardy | dtantsur: agreed it'd be good to update the docs, but I've not had time to edit that to a more concise version for tripleo-docs yet | 12:25 |
| dtantsur | ack | 12:25 |
| dtantsur | thanks! | 12:25 |
| shardy | dtantsur: the tl;dr for puppet modules is example 2 | 12:25 |
| dtantsur | yeah, now I remember | 12:26 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Switch trunk/cbs/buildlogs to use https https://review.openstack.org/448054 | 12:26 |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient master: Call validations in workflows https://review.openstack.org/448057 | 12:28 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Switch trunk/cbs/buildlogs to use https https://review.openstack.org/448058 | 12:28 |
| *** kjw3 has quit IRC | 12:29 | |
| *** kjw3 has joined #tripleo | 12:30 | |
| *** janki has quit IRC | 12:31 | |
| rook | shardy: what writes the hira data? | 12:32 |
| *** pradk has joined #tripleo | 12:32 | |
| rook | hiera* | 12:32 |
| shardy | rook: on newton it's this element https://github.com/openstack/tripleo-puppet-elements/blob/master/elements/hiera/40-hiera-datafiles | 12:34 |
| *** psahoo_ has quit IRC | 12:34 | |
| rook | shardy so, i see a ton of temp hiera files... in /tmp | 12:35 |
| rook | with prefixed start and stop | 12:35 |
| *** vpickard_ is now known as vpickard | 12:35 | |
| rook | and in each start/stop there is a /d./d | 12:37 |
| *** rlandy has joined #tripleo | 12:38 | |
| *** jaosorior_brb is now known as jaosorior | 12:38 | |
| rook | it seems like it is in an infinite loop of running the same thing over and over. | 12:38 |
| rook | https://gist.github.com/jtaleric/af4015022d8782b3d0cfb671479dfb1a | 12:38 |
| rook | shardy: ^ | 12:38 |
| rook | is that normal? | 12:38 |
| *** funzo has joined #tripleo | 12:41 | |
| shardy | rook: No, I think that's created by os-refresh-config which runs dib-run-parts, which creates the files, but AFAICS they should be in one of the profiledir.X dirs | 12:42 |
| rook | looking at what is going on (a node that mentions AllNodesDeployment https://gist.github.com/jtaleric/287f5029650a3e397f9cb38fd8144eb1 | 12:42 |
| rook | shardy: it just runs over and over -- never ending. | 12:42 |
| shardy | rook: are there a bazillion profiledir dirs, e.g has the mktemp failed due to lack of /tmp space or something | 12:42 |
| rook | There is plenty of space on this host. | 12:43 |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1673144 | 12:43 |
| openstack | Launchpad bug 1673144 in tripleo "lots of empty /tmp/profiledir directories not cleaned up" [High,Triaged] | 12:43 |
| rook | shardy: these aren't empty | 12:43 |
| EmilienM | bandini, adarazs_lunch, sshnaidm: FYI, I hope this series of patches will reduce the failures in our CI https://review.openstack.org/#/q/topic:cbs/https | 12:43 |
| rook | https://gist.github.com/jtaleric/af4015022d8782b3d0cfb671479dfb1a shardy | 12:43 |
| shardy | https://github.com/openstack/dib-utils/blob/master/bin/dib-run-parts#L118 | 12:43 |
| rook | they all have the start_ and end_ | 12:43 |
| rook | s/end/stop | 12:43 |
| shardy | rook: I'm trying to understand why the start/stop files are in /tmp and not in the $PROFILE_DIR | 12:44 |
| shardy | that seems wrong AFAICS | 12:44 |
| rook | oh, they aer shardy | 12:44 |
| rook | shardy: for example in the gist : profiledir.0mwXKI | 12:44 |
| shardy | rook: Ok, Ok I see | 12:44 |
| rook | it was actually profiledir.pic2JY | 12:44 |
| rook | I am just not sure why it just runs continuous... | 12:45 |
| *** trown|outtypewww is now known as trown | 12:45 | |
| rook | ohh | 12:45 |
| rook | shardy: i think i see the issue, maybe... | 12:46 |
| rook | shardy: i only see start_51-hosts | 12:46 |
| rook | never a end. | 12:46 |
| rook | s/end/stop | 12:46 |
| *** funzo has quit IRC | 12:46 | |
| *** karimb has quit IRC | 12:46 | |
| sshnaidm | EmilienM, wow.. starting to review | 12:46 |
| EmilienM | sshnaidm: well, see on #rdo - I just found out that we still have the problem | 12:47 |
| EmilienM | https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-pike/common/heat-cfntools-1.3.0-2.el7.noarch.rpm: [Errno 14] HTTPS Error 302 - Found | 12:47 |
| *** pkovar has joined #tripleo | 12:47 | |
| shardy | rook: yeah I'm wondering if that happens when it fails | 12:48 |
| *** dprince has joined #tripleo | 12:48 | |
| rook | everything else has stop_* but 51-hosts | 12:48 |
| shardy | https://github.com/openstack/os-refresh-config/blob/master/os_refresh_config/os_refresh_config.py#L155 | 12:49 |
| *** fultonj has quit IRC | 12:49 | |
| *** rhallisey has quit IRC | 12:49 | |
| *** fultonj_ is now known as fultonj | 12:50 | |
| shardy | rook: I'd create a script in /usr/libexec/os-refresh-config/error.d/ then we can perhaps prove that the script is failing, but we should see that log.error in the logs | 12:50 |
| shardy | the during $phase | 12:50 |
| EmilienM | tripleo meeting in 1h10, feel free to add items to our agenda: https://etherpad.openstack.org/p/tripleo-meeting-items | 12:50 |
| pradk | EmilienM, this broke gnocchi https://review.openstack.org/#/c/442793/6 | 12:51 |
| *** ratailor has quit IRC | 12:51 | |
| EmilienM | bnemec: fyi https://review.openstack.org/#/c/448045 | 12:51 |
| *** rhallisey has joined #tripleo | 12:51 | |
| EmilienM | pradk: maybe | 12:52 |
| EmilienM | pradk: it's hard to say when there is no test for gnocchi | 12:52 |
| EmilienM | pradk: mainly because of https://review.openstack.org/#/c/442795/ | 12:52 |
| rook | shardy what exactly goes there? That directory doesn't even exist. | 12:52 |
| rook | [root@overcloud-openstack-compute-67 os-refresh-config]# ls | 12:52 |
| rook | configure.d migration.d post-configure.d pre-configure.d | 12:52 |
| *** eck` is now known as eck`gone | 12:53 | |
| derekh | Is it just me or is introspection going way too fast to actually be doing something? http://logs.openstack.org/63/447963/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/6e1882e/console.html#_2017-03-21_10_34_42_436546 | 12:55 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Swift auth url should use a suffix https://review.openstack.org/448077 | 12:55 |
| jpich | florianf: May I get your eyes on the "\n" issue Ocata backport? https://review.openstack.org/#/c/447670/ | 12:55 |
| d0ugal | I wonder if we could require release notes for all patches unless they put something in the commit message to explicitly say it isn't needed. | 12:56 |
| florianf | jpich: sure! done. | 12:56 |
| jpich | derekh: It's sort of my fault, it's actually done a bit earlier: http://logs.openstack.org/63/447963/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/6e1882e/console.html#_2017-03-21_10_29_52_753209 | 12:56 |
| jpich | florianf: Thank you! | 12:56 |
| *** shardy has quit IRC | 12:57 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/tripleo-ci master: Do not merge: Test python-heat-agents package build https://review.openstack.org/439178 | 12:58 |
| jpich | d0ugal: I'd rather not tbh, we have a lot of dubious/unclear release notes already because it's seen as a checkbox | 12:58 |
| openstackgerrit | David Moreau Simard proposed openstack/tripleo-heat-templates master: Do not merge: Test python-heat-agents package build https://review.openstack.org/439179 | 12:58 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing novajoin authtoken https://review.openstack.org/446348 | 12:58 |
| d0ugal | jpich: fair enough - I just need some way to remind myself :) | 12:58 |
| *** jcoufal has joined #tripleo | 12:58 | |
| jpich | derekh: https://bugs.launchpad.net/tripleo/+bug/1659931 | 12:58 |
| openstack | Launchpad bug 1659931 in tripleo "No nodes being introspected in tripleo-ci jobs" [High,Fix released] - Assigned to Brad P. Crochet (brad-9) | 12:58 |
| jpich | d0ugal: No every single patch needs a release note :) | 12:59 |
| *** colonwq has quit IRC | 12:59 | |
| d0ugal | jpich: s/No/Not/? | 12:59 |
| jpich | yes! That word | 13:00 |
| jpich | damn typos | 13:00 |
| derekh | jpich: ok, got it, thanks | 13:00 |
| *** salmankhan has quit IRC | 13:03 | |
| fultonj | shardy: yesterday you suggested I temporarily abandon https://review.openstack.org/#/c/442969/ for https://review.openstack.org/#/c/411984/ to land. I did this yesterday, but don't understand why, because it looks like 411984 has already merged. If land != merge, would you mind letting me know what you mean for my own learning? | 13:03 |
| *** shardy has joined #tripleo | 13:03 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Introduce nodes assignment using NodePicker component https://review.openstack.org/445526 | 13:04 |
| *** ealcaniz has joined #tripleo | 13:04 | |
| fultonj | shardy: yesterday you suggested I temporarily abandon https://review.openstack.org/#/c/442969/ for https://review.openstack.org/#/c/411984/ to land. I did this yesterday, but don't understand why, because it looks like 411984 has already merged. If land != merge, would you mind letting me know what you mean for my own learning? (re-asking as it looks like you were not on earlier as you just joined) | 13:05 |
| trozet | shardy: hi | 13:05 |
| *** Goneri has joined #tripleo | 13:06 | |
| *** adarazs_lunch is now known as adarazs | 13:06 | |
| jaosorior | jistr: hey dude, can you help me out figuring out what I did wrong in this patch? https://review.openstack.org/446473 | 13:08 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 13:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | 13:10 |
| rook | shardy: not sure what script i need in that directory... looking at the code you sent, the os-refresh could try to push a error into the error.d directory, however that directory doesn't exist on any of my nodes./ | 13:11 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Run introspection 20 nodes at a time https://review.openstack.org/439568 | 13:12 |
| *** funzo has joined #tripleo | 13:13 | |
| *** [1]cdearborn has joined #tripleo | 13:14 | |
| *** lucas-brb is now known as lucasagomes | 13:14 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement verify profiles as a custom action https://review.openstack.org/429684 | 13:15 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check ironic boot configuration as a custom action https://review.openstack.org/428225 | 13:15 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check flavors as a custom action https://review.openstack.org/428178 | 13:15 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check node counts as a custom action https://review.openstack.org/429757 | 13:15 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add workflow wrapping all predeployment validations https://review.openstack.org/445914 | 13:15 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check boot images as a custom action https://review.openstack.org/428177 | 13:15 |
| jaosorior | dprince: is the puppet run what populates the /var/lib/kolla/config_files/src/ directory? | 13:16 |
| dprince | jaosorior: we bind mount that in from the host. It is populated via the heat json-file hook executed by os-collect-config | 13:17 |
| matbu | jistr: just fyi : you will just need to use that: https://paste.fedoraproject.org/paste/3PVoLj3aGVEQHRTVJRVTYV5M1UNdIGYhyRLivL9gydE= | 13:17 |
| *** janki has joined #tripleo | 13:18 | |
| matbu | jistr: then you will have probably to override some default/main.yml variables to stick to your env | 13:18 |
| *** pmannidi has quit IRC | 13:18 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Re-run gnocchi and ceilometer upgrade in step 5 https://review.openstack.org/447599 | 13:18 |
| jaosorior | dprince: what exactly does the json-file hook do? | 13:18 |
| dprince | jaosorior: takes heat metadata, writes it to a json-file | 13:19 |
| dtantsur | hmm, why don't we have ocata in https://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/ ? | 13:19 |
| jaosorior | dprince: also, when we run puppet for a specific service, how does that end up in a container? I thought that configuration was bind-mounted from the host to the container. | 13:19 |
| *** pmannidi has joined #tripleo | 13:19 | |
| dprince | jaosorior: it is a very useful thing in general, especially for debugging | 13:19 |
| dprince | jaosorior: puppet is run only for configuration. THis occurs early | 13:19 |
| trozet | jaosorior: do you mind reviewing this real quick: https://review.openstack.org/#/c/425934/ ? | 13:19 |
| openstackgerrit | Merged openstack/tripleo-docs master: Fixups for containers documentation https://review.openstack.org/447541 | 13:19 |
| *** ramishra has quit IRC | 13:19 | |
| dprince | jaosorior: then the services is started at a later time bind mounting in the generated configs | 13:20 |
| *** rhallisey has quit IRC | 13:20 | |
| *** dtantsur is now known as dtantsur|bbl | 13:20 | |
| *** amoralej is now known as amoralej|lunch | 13:20 | |
| *** psahoo has joined #tripleo | 13:21 | |
| *** rhallisey has joined #tripleo | 13:21 | |
| *** fzdarsky|lunch is now known as fzdarsky | 13:21 | |
| jaosorior | dprince: uhm... so, if a directory is created via puppet; is that accessible from the hosts filesystem at some point? | 13:21 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Allow complex upgrade deployment for N to O https://review.openstack.org/439598 | 13:21 |
| dprince | jaosorior: only if we "archive" it http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/docker-puppet.py#n158 | 13:22 |
| jaosorior | I see | 13:22 |
| jaosorior | where could I specify that it should get archived? | 13:23 |
| dprince | jaosorior: the TODO is to make eliminate the maintenance here with filesystem test, or 'docker diff' or something | 13:23 |
| dprince | jaosorior: right now, in that file | 13:23 |
| jaosorior | ok | 13:23 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart master: Add composable deployment config files and mixed release https://review.openstack.org/410831 | 13:23 |
| jaosorior | dprince: so, the deal is, I want to bind-mount the fernet keys directory instead of individual files https://review.openstack.org/#/c/446473/4/docker/services/keystone.yaml however, by doing what I'm doing in that patch, I get errors from keystone where it crashes saying there are no available keys | 13:24 |
| *** karimb has joined #tripleo | 13:24 | |
| *** dmacpher has joined #tripleo | 13:24 | |
| *** links has quit IRC | 13:24 | |
| shardy | rook: yeah you can create it and put a shell script in error.d, but I don't understand why you don't see this error: | 13:24 |
| shardy | https://github.com/openstack/os-refresh-config/blob/master/os_refresh_config/os_refresh_config.py#L154 | 13:24 |
| shardy | trozet: hi | 13:24 |
| jaosorior | dprince: like this http://logs.openstack.org/73/446473/4/experimental/gate-tripleo-ci-centos-7-undercloud-containers/f2fec8a/logs/var/log/undercloud_install.txt.gz#_2017-03-21_13_11_23_000 | 13:24 |
| dprince | jaosorior: I guess change docker-puppet.py then as part of this. Or improve docker-puppet.py so that it can dynamically figure out what was generated within the config container run | 13:24 |
| *** jmelvin has joined #tripleo | 13:25 | |
| shardy | fultonj: Hey, yeah perhaps I made a mistake, I thought you were chasing for reviews on the patch to master, which would have been blocked by the Depends-On which is not branch aware | 13:25 |
| *** cdearborn has joined #tripleo | 13:25 | |
| trozet | shardy: hey i happened to come across a nuance of inheriting one service in another and wanted to know what you think | 13:25 |
| *** prateek has quit IRC | 13:25 | |
| fultonj | shardy: ok, thanks. I will un-abandon those. | 13:25 |
| shardy | fultonj: ack, apologies for the confusion | 13:26 |
| *** akuznetsov has joined #tripleo | 13:26 | |
| fultonj | not at all. i learned from your advise yesterday about how to deal with a cherry-pick conflict. thank you! | 13:27 |
| jaosorior | dprince: uhm... I see that /etc is taken into account in docker-puppet though, that's where the directory I need is | 13:27 |
| EmilienM | sshnaidm: FYI https://bugs.launchpad.net/tripleo/+bug/1674681 | 13:27 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 13:27 |
| trozet | shardy: you see how https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/neutron-ovs-dpdk-agent.yaml#L58 that is inheriting regular neutron ovs agent template? | 13:27 |
| dprince | jaosorior: then it should be there already I think | 13:27 |
| dprince | jaosorior: unless perhaps there is some weird linking or somethign going on | 13:27 |
| dprince | jaosorior: you can run docker-puppet.py manually fwiw | 13:28 |
| dprince | jaosorior: https://dprince.github.io/docker-puppet.html | 13:28 |
| shardy | trozet: yup | 13:28 |
| *** jtomasek has quit IRC | 13:29 | |
| trozet | shardy: the thing is, it takes in all the config of the parent, including tripleo firewall rules...but because the service_name is set to neturon_ovs_dpdk_agent and not neutron_ovs_agent, the firewall rules will be written to hieradata, but never applied by TripleO::Firewall | 13:29 |
| adarazs | trown: FYI I made this after being bit by not having a launchpad <-> gerrit integration for our quickstart repos: https://review.openstack.org/448024 | 13:29 |
| adarazs | trown: we should probably migrate the remaining bugs from the tripleo-quickstart launchpad page. | 13:29 |
| jaosorior | dprince: I'll give it a try, thanks! | 13:30 |
| *** pgadiya has quit IRC | 13:30 | |
| trozet | shardy: so simply changing the service_name to be {get_attr: [NeutronOvsAgent, role_data, service_name]} works, and results in the service_name being modified to neutron_ovs_agent | 13:30 |
| trozet | shardy: so my question is...is the correct behavior to set the service name to be the same as the parent to apply firewall rules, or should we change tripleo firewall behavior? | 13:31 |
| shardy | trozet: Hmm, I'd probably prefer just writing a tripleo.neutron_ovs_dpdk_agent.firewall_rules key even if it duplicates a couple of lines? | 13:31 |
| shardy | trozet: yeah I guess it's a limitation of the "inheritance" mechanism we're using here | 13:31 |
| *** jpena is now known as jpena|lunch | 13:31 | |
| shardy | trozet: in general I think it'd be cleaner if we kept the service_name aligned with the yaml template names, e.g never have two templates which result in the same service_name | 13:32 |
| trozet | shardy: that makes sense | 13:32 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Run introspection 20 nodes at a time https://review.openstack.org/439568 | 13:32 |
| *** [1]cdearborn has quit IRC | 13:32 | |
| trozet | shardy: so if i added another tripleo firewall rules | 13:32 |
| trozet | shardy: can i use get attr or something to get the value from the parent firewall rules? | 13:33 |
| trozet | shardy: should be able to right? | 13:33 |
| shardy | trozet: yeah you can, but then you'll still get the tripleo.neutron_ovs_agent.firewall_rules set from the base template, if that matters | 13:33 |
| trozet | shardy: you mean still have both in hieradata? | 13:34 |
| shardy | trozet: yeah you will have both if you just add it via the map_merge | 13:34 |
| shardy | trozet: another approach is to use map_replace | 13:34 |
| trozet | shardy: ah | 13:34 |
| shardy | then you can replace tripleo.neutron_ovs_agent.firewall_rules with the new key | 13:34 |
| trozet | shardy: sounds like a better solution | 13:35 |
| shardy | https://docs.openstack.org/developer/heat/template_guide/hot_spec.html#map-replace | 13:35 |
| shardy | trozet: yeah thinking about it that may be best, example ^^ and in various places in t-h-t | 13:35 |
| trozet | shardy: ok i will file a bug ont aht dpdk file and submit a fix using map replace, thanks! | 13:35 |
| shardy | trozet: sounds good, thanks! | 13:36 |
| trozet | shardy: but i was able to get ovs 2.6 + DPDK working with newton | 13:36 |
| trozet | shardy: once i fixed that | 13:36 |
| openstackgerrit | John Fulton proposed openstack/puppet-tripleo stable/newton: WIP: Include nova::compute::libvirt::qemu from the libvirt profile https://review.openstack.org/442970 | 13:36 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DONT REVIEW: test consistent repo https://review.openstack.org/447522 | 13:38 |
| openstackgerrit | Mikhail S Medvedev proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 13:39 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DONT REVIEW: test consistent repo https://review.openstack.org/447522 | 13:39 |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates master: Disable core dump for setuid programs https://review.openstack.org/442458 | 13:40 |
| jistr | matbu: thanks :) i'm actually trying to amend the upgrade role to upgrade from non-containerized to containerized (ocata to pike) | 13:41 |
| jistr | not much success so far but i keep trying :D | 13:42 |
| matbu | jistr: cool, do you have a doc or a workflow to follow for a such upgrade ? | 13:42 |
| *** jtomasek has joined #tripleo | 13:43 | |
| matbu | jistr: maybe i can help | 13:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Remove certificate request bits from service profiles https://review.openstack.org/444891 | 13:43 |
| jistr | matbu: that would be great :) i don't have a doc yet but i can start an etherpad right now. So far the workflow is very simplistic | 13:44 |
| *** remix_tj has joined #tripleo | 13:44 | |
| matbu | jistr: cool, yep if you can, it sounds interesting :) | 13:44 |
| *** gbarros has joined #tripleo | 13:44 | |
| *** rwsu has quit IRC | 13:45 | |
| EmilienM | weekly meeting in 15min! | 13:45 |
| *** ckyriakidou has quit IRC | 13:46 | |
| jistr | matbu: this is what we have so far. No init/converge yet, just focused on the container migration itself: https://etherpad.openstack.org/p/pike-tripleo-upgrades | 13:49 |
| *** masco has quit IRC | 13:49 | |
| EmilienM | mcornea, marios : can we close https://bugs.launchpad.net/tripleo/+bug/1657175 now? | 13:49 |
| openstack | Launchpad bug 1657175 in tripleo "During Newton->Ocata upgrade we need to install python-heat-agent-* and ansible-pacemaker module" [Critical,In progress] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 13:49 |
| *** deadnull has joined #tripleo | 13:50 | |
| marios | EmilienM: yes | 13:50 |
| marios | EmilienM: thanks | 13:50 |
| *** artom has quit IRC | 13:50 | |
| deadnull | Currently doing a small deployment of ocata, ran into a package dependency issue with zeromq during the undercloud install. Any known issues with ocata? Should I stick to newton for now? | 13:50 |
| jaosorior | dprince: is there an intermediate container where puppet is run? or is that configuration done in the same container that furtherly runs the service? | 13:51 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/tripleo-ci master: Replace "ara generate" by "ara generate html" https://review.openstack.org/448108 | 13:51 |
| dmsimard | sshnaidm: ^ | 13:51 |
| dprince | jaosorior: usually the same as the services itself. But it is configurable per our docker/services templates | 13:52 |
| dprince | jaosorior: some services share the same common base container | 13:52 |
| *** amoralej|lunch is now known as amoralej | 13:53 | |
| *** lblanchard has joined #tripleo | 13:53 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Parameterize OVB stack flavors https://review.openstack.org/445163 | 13:53 |
| jaosorior | dprince: ok, I'm starting to get the picture. the only part that I still don't quite get is how the configuration files are populated into /var/lib/kolla/config_files/src/ (for instance /etc/keystone) is that before or after puppet runs? | 13:54 |
| *** salmankhan has joined #tripleo | 13:55 | |
| dprince | jaosorior: docker-puppet.py copies them into that directory via a bind mount | 13:55 |
| *** tosky has quit IRC | 13:55 | |
| *** morazi has quit IRC | 13:55 | |
| dprince | jaosorior: probably worth just reading docker-puppet.py for what you are doing. It is a fiarly small python file fwiw | 13:55 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/ocata: panko: Do db_sync in api manifest https://review.openstack.org/444338 | 13:55 |
| EmilienM | just removed the depends on^ | 13:55 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/ocata: Correct haproxy's stat unix socket path https://review.openstack.org/445479 | 13:56 |
| EmilienM | and rebasing ^ | 13:56 |
| lhinds | quick question, during the tripleo workshop, shardy showed a tripleo stack troubleshoot command, I can't recal the full syntax..anyone recall? | 13:56 |
| lhinds | openstack overcloud <..> <..? | 13:57 |
| shardy | deadnull: Hi, there are recent passing ocata CI jobs, so perhaps you can raise a bug describing the issue? | 13:57 |
| shardy | http://tripleo.org/cistatus.html | 13:57 |
| deadnull | shardy Layer 8 issue, conflict with epel release repo and delorean repo | 13:57 |
| *** bkopilov has joined #tripleo | 13:59 | |
| matbu | shardy: hey im looking at the cistatus, all upgrade jobs looks red | 14:00 |
| *** tosky has joined #tripleo | 14:01 | |
| jaosorior | dprince: so, on the archiving part, are we then copying the /etc/* from the hosts into the container then? why is that? | 14:03 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Change the relase name for the templates to pike. https://review.openstack.org/435345 | 14:03 |
| dprince | jaosorior: should be the opposite. From the container to the host | 14:04 |
| jaosorior | oh, right | 14:05 |
| jaosorior | dprince: so, then, modifying one of those files from the host, doesn't really affect the container, does it? since those files are just copies of the configuration | 14:07 |
| apetrich | jtomasek, d0ugal, just did some testing and using zlib and compressing what gets saved to the cache shaves an extra 0.5sec so I think keeping the swift is not terrible. unless we move the cache to the front of mistral | 14:07 |
| d0ugal | apetrich: agreed | 14:08 |
| dprince | jaosorior: it would effect the service container if you then launch it with those files | 14:08 |
| dprince | jaosorior: step 1: run docker (docker-puppet.py) to create config files. Those files currently get copied to the host | 14:08 |
| dprince | jaosorior: step 2: run docker-cmd services. This starts up actual OpenStack services and bind mounts in the config files from step 1 | 14:09 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: docker/keystone: Bind mount entire fernet keys repository https://review.openstack.org/446473 | 14:09 |
| dprince | jaosorior: if you modify any of the config files on the host before the services are launched (or restarted) it would effect them | 14:09 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 14:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 14:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 14:10 |
| dprince | jaosorior: right now the use of the host os's 'config_files' directory is just a staging area | 14:10 |
| *** eck`gone is now known as eck` | 14:11 | |
| *** saibarspeis has quit IRC | 14:12 | |
| jaosorior | dprince: thanks for the explanations dude | 14:12 |
| *** gkadam has quit IRC | 14:16 | |
| jtomasek | apetrich, +1 | 14:16 |
| *** gkadam has joined #tripleo | 14:16 | |
| jtomasek | apetrich, d0ugal: only thing I am quite afraid of is the need to explicitly invalidate the cache from selected action calls. It will cause bugs | 14:17 |
| *** gkadam has quit IRC | 14:17 | |
| *** chlong has joined #tripleo | 14:17 | |
| d0ugal | jtomasek: agreed also, but I don't see any other way really | 14:17 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates stable/newton: WIP: Increase libvirt/qemu.conf max_files and max_processes https://review.openstack.org/448122 | 14:17 |
| jtomasek | d0ugal, me neither | 14:17 |
| *** dmarlin has joined #tripleo | 14:18 | |
| apetrich | jtomasek, me too. I'm thinking about a 10 min TTL to the cache so we can minimise that | 14:18 |
| apetrich | d0ugal, ^ | 14:18 |
| apetrich | jtomasek, d0ugal every ten minutes one call takes longer :\ | 14:18 |
| jtomasek | apetrich, that won't really help imho, we update parameters often when interacting with GUI | 14:18 |
| apetrich | jtomasek, aye. | 14:19 |
| d0ugal | apetrich: I guess we could try doing some investigation to see why this bit of code is so slow | 14:20 |
| apetrich | d0ugal, that's pretty much what I'm doing now. | 14:20 |
| *** jkilpatr has quit IRC | 14:21 | |
| apetrich | d0ugal, that's how I got to zlib. we were storing over 1M of parameters compressed it is 91014 bytes | 14:23 |
| *** chlong has quit IRC | 14:24 | |
| *** tobias_fiberdata has quit IRC | 14:24 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: build-test-packages: Fix branch name for dlrn building in newton+ https://review.openstack.org/447747 | 14:25 |
| jtomasek | honza, do I need to update something else to get the CI passing after redux-form dependency has been added? | 14:26 |
| jtomasek | honza, specifically https://review.openstack.org/445526 | 14:26 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Parameterize OVB stack flavors https://review.openstack.org/445163 | 14:27 |
| honza | jtomasek: yes, steps 7 & 8 in this doc https://github.com/openstack/tripleo-ui/blob/master/docs/packaging.rst | 14:27 |
| honza | jtomasek: I can do that if you'd like | 14:27 |
| jtomasek | honza, that would be great! thanks:) | 14:28 |
| bnemec | deadnull: You shouldn't use epel with RDO packages. They aren't compatible. | 14:28 |
| honza | jtomasek: np, i have some other updates to do in those repos so this should be easy :) | 14:28 |
| d0ugal | thrash: You could just base https://review.openstack.org/#/c/448057/1 on master. It has everything you need AFAICT | 14:31 |
| thrash | d0ugal: oh... good point. :) | 14:31 |
| adarazs | bkero: I was thinking the launchpad - gerrit integration will comment for me but it didn't, because it was not correctly configured. this should help: https://review.openstack.org/448024 | 14:32 |
| bkero | adarazs: Yeah, I noticed that too, had to manually comment. | 14:33 |
| *** jkilpatr has joined #tripleo | 14:34 | |
| adarazs | bkero: so yeah, I made that fix within half an hour of taking the bug and then I forgot to check if it commented... | 14:34 |
| adarazs | bkero: sorry that you had to duplicate my change. | 14:35 |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient master: Call validations in workflows https://review.openstack.org/448057 | 14:35 |
| bkero | It's okay, I just wish I hadn't wasted my time on it after-work-hours yesterday. I was led to believe it wasn't solved and was extremely urgent. | 14:35 |
| rook | shardy: the os-refersh-config does that share the log with os-collect? | 14:36 |
| deadnull | bnemec yep, thx ... | 14:36 |
| shardy | rook: yeah the logs for both should be in /var/log/messages I think | 14:37 |
| shardy | os-collect-config calls os-refresh-config when metadata changes | 14:37 |
| shardy | which then calls dib-run-parts and all the scripts under /usr/libexec/os-refresh-config | 14:37 |
| rook | shardy: i wonder if 51-hosts never exits1 | 14:38 |
| *** pblaho has quit IRC | 14:39 | |
| shardy | rook: we'd see it in ps ax/pstree output in that case? | 14:39 |
| shardy | but yeah that would explain what you're seeing | 14:40 |
| apetrich | d0ugal, jtomasek these http://paste.openstack.org/show/603614/ are all the gets from a get parameters action. I don't think we can speed them easily | 14:41 |
| rook | shardy: but we should expect to see stop_51-hosts in that temp dir, and since we are not we can only assume 51-hosts is fubar. | 14:41 |
| rook | shardy: the only thought I have is to have 51-hosts exit 0 at the top - not sure we need host names | 14:41 |
| rook | since these are not controllers | 14:41 |
| d0ugal | apetrich, jtomasek: lol, yeah, so the problem is that it needs the full plan from swift. | 14:42 |
| *** jpena|lunch is now known as jpena | 14:42 | |
| d0ugal | apetrich, jtomasek - I think the most correct way is to calculate this on update and store it, rather than on-demand. | 14:42 |
| jtomasek | d0ugal, +1 | 14:43 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from provide workflow https://review.openstack.org/446049 | 14:44 |
| jtomasek | d0ugal, apetrich: are those gets the part which takes most of the time? how about templates processing and heat api call? | 14:44 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from introspect workflow https://review.openstack.org/446054 | 14:44 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from introspect manageable https://review.openstack.org/446065 | 14:44 |
| *** dhill_ has joined #tripleo | 14:44 | |
| *** gbarros has quit IRC | 14:44 | |
| shardy | rook: if you look at the o-r-c code I think there's no stop file if it exits with an error | 14:45 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test new ovb with images building https://review.openstack.org/445228 | 14:45 |
| shardy | rook: but yeah, if you add exit 0 and everything works, that will at least prove where the issue is | 14:45 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: New ovb playbook and settings https://review.openstack.org/431567 | 14:45 |
| shardy | rook: https://github.com/openstack/os-refresh-config/blob/master/os_refresh_config/os_refresh_config.py#L154 should tell us what the error was, which is why I'm confused we don't see that logged | 14:46 |
| *** tobias_fiberdata has joined #tripleo | 14:46 | |
| rook | lol shardy | 14:47 |
| rook | 2017-03-21 14:45:28Z [overcloud-openstack-ComputeAllNodesDeployment-4hwvwrhulljh.163]: SIGNAL_IN_PROGRESS Signal: deployment 5974d7dd-29ac-42d3-841b-4f045fb4c220 succeeded | 14:47 |
| rook | after changing 51-hosts to exit 0 | 14:47 |
| rook | ugh | 14:47 |
| *** tbonds has quit IRC | 14:47 | |
| *** psahoo has quit IRC | 14:48 | |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart-extras master: Add ethernet support on external network https://review.openstack.org/437376 | 14:48 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Bring back postci log https://review.openstack.org/448135 | 14:49 |
| *** gbarros has joined #tripleo | 14:50 | |
| sshnaidm | shardy, ^ | 14:50 |
| *** ratailor has joined #tripleo | 14:51 | |
| shardy | sshnaidm: thanks! :) | 14:51 |
| apetrich | jtomasek, d0ugal I'm not seeing a big culprit . heat does takes up 3 secs but the bulk is the huge number of calls | 14:51 |
| weshay_sick | trown, adarazs this is going to become important to RHEL 7.4 testing very soon https://review.openstack.org/#/c/442764/ | 14:51 |
| apetrich | jtomasek, d0ugal so caching we are faster than the heat alone. | 14:52 |
| apetrich | *part alone | 14:52 |
| adarazs | weshay_sick: oh, this is cool. | 14:52 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from introspect manageable https://review.openstack.org/446065 | 14:52 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from provide workflow https://review.openstack.org/446049 | 14:52 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from introspect workflow https://review.openstack.org/446054 | 14:52 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from configure workflow https://review.openstack.org/446337 | 14:52 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from create_raid_configuration https://review.openstack.org/446339 | 14:52 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from provide_manageable_nodes https://review.openstack.org/446081 | 14:53 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from configure_manageable_nodes https://review.openstack.org/446338 | 14:53 |
| *** ratailor has quit IRC | 14:53 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from create/update_deployment_plan https://review.openstack.org/446434 | 14:54 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from create_default_plan https://review.openstack.org/446433 | 14:54 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from package_update https://review.openstack.org/446435 | 14:54 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from delete_stack https://review.openstack.org/446436 | 14:54 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from get_passwords https://review.openstack.org/443053 | 14:55 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from deploy_plan https://review.openstack.org/446437 | 14:55 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Support multiple messages from delete_node https://review.openstack.org/446438 | 14:55 |
| trown | weshay_sick: cool, I will review that today, thanks for working on that | 14:55 |
| weshay_sick | thanks | 14:55 |
| *** morazi has joined #tripleo | 14:56 | |
| *** gbarros has quit IRC | 14:57 | |
| * adarazs is going to work on his compass today, looks like the bombardment of the critical bugs is decreasing. | 14:57 | |
| *** artom has joined #tripleo | 14:57 | |
| bkero | Let's hope. | 14:57 |
| bkero | The toci vxlan networking changes are getting build failures because of the bugs, so I'm blocked with those until the newton package building and pankodb-sync are resolved. | 14:58 |
| EmilienM | adarazs: well, we have a ton of critical bugs, I don't see how they are decreasing | 14:59 |
| *** karimb has quit IRC | 15:00 | |
| *** tbonds has joined #tripleo | 15:00 | |
| adarazs | EmilienM: I didn't have any more since noon. that's a win :P :) | 15:00 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/440124 | 15:02 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement verify profiles as a custom action https://review.openstack.org/429684 | 15:02 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check node counts as a custom action https://review.openstack.org/429757 | 15:02 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add workflow wrapping all predeployment validations https://review.openstack.org/445914 | 15:02 |
| EmilienM | it sounds like ovb jobs are timeouting, seen on https://review.openstack.org/#/c/447849/ | 15:04 |
| trozet | mwhahaha: hey can you reivew this please? https://review.openstack.org/#/c/443261/ | 15:05 |
| openstackgerrit | Peng Liu proposed openstack/puppet-tripleo master: Add l2 gateway Neutron service plugin profile https://review.openstack.org/444050 | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 15:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 15:10 |
| rook | shardy: the only thing I can think is maybe there is a limit on the size of what you can echo into a file? | 15:12 |
| *** karimb has joined #tripleo | 15:12 | |
| rook | shardy in one part of the code we cat the tmp file into the real file... and if we do not see HEAT_HOSTS we echo directly into the file | 15:13 |
| *** dtantsur|bbl is now known as dtantsur | 15:14 | |
| *** pblaho has joined #tripleo | 15:14 | |
| rook | which we should hit the first path: | 15:14 |
| rook | # HEAT_HOSTS_START - Do not edit manually within this section! | 15:14 |
| openstackgerrit | Andy Smith proposed openstack/puppet-tripleo master: Include oslo.messaging amqp support for rpc and notifications https://review.openstack.org/447319 | 15:15 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Silence some ignored errors https://review.openstack.org/448150 | 15:15 |
| *** colonwq2 has joined #tripleo | 15:15 | |
| *** padkrish has joined #tripleo | 15:15 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-heat-templates master: Remove useless trailing '\n' in /etc/hosts file. https://review.openstack.org/448152 | 15:16 |
| *** tbonds has quit IRC | 15:17 | |
| *** gbarros has joined #tripleo | 15:17 | |
| shardy | rook: but if a hard-error like that happened, we'd see it running 51-hosts manually? | 15:18 |
| shardy | rook: if we temporarily stop os-collect-config then run it manually, it works, doesn't it? | 15:19 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Silence some ignored errors https://review.openstack.org/448154 | 15:20 |
| *** karimb has quit IRC | 15:21 | |
| rook | shardy: i don't see a error no. and it appears to work. | 15:21 |
| rook | great, now I have one the odd errors I saw yesterday | 15:22 |
| *** florianf has quit IRC | 15:22 | |
| rook | https://gist.github.com/jtaleric/ef57fdd28fb77ee388af4a71ca3dfc3c -- so, ssh used to work to all the hosts... now this. | 15:22 |
| *** shardy is now known as coding_shardy | 15:23 | |
| dtrainor | Is there a way via API to get a list of available validations? | 15:25 |
| dtrainor | i'm toying around with the mistral cli and I can list and get workbooks, I think I'm getting close.. | 15:25 |
| d0ugal | dtrainor: I don't think there is a way | 15:25 |
| *** akrivoka has joined #tripleo | 15:26 | |
| dtrainor | okie dokie. thanks. | 15:26 |
| d0ugal | dtrainor: I take that back, there is. | 15:26 |
| jpich | dtrainor: Looks like there's a couple of methods to list them - https://github.com/openstack/tripleo-common/blob/master/workbooks/validations.yaml#L177-#L187 | 15:26 |
| d0ugal | dtrainor: http://paste.openstack.org/show/603625/ | 15:27 |
| *** florianf has joined #tripleo | 15:27 | |
| jpich | Nice! | 15:27 |
| dtrainor | perfect. good find, both of ya. Thanks so much. | 15:27 |
| dtrainor | I figured there was a way, at least for the purposes of ui, since I can't imagine the list of validations are baked in, rather they ones that are applicable in ui have a key that makes them applicable for ui | 15:29 |
| *** aufi has quit IRC | 15:29 | |
| *** karimb has joined #tripleo | 15:32 | |
| *** jrist has quit IRC | 15:33 | |
| *** ckyriakidou has joined #tripleo | 15:35 | |
| *** dsariel_ has quit IRC | 15:36 | |
| EmilienM | flaper87: thanks for the update | 15:38 |
| *** thrash is now known as thrash|bbl | 15:39 | |
| *** jrist has joined #tripleo | 15:40 | |
| trozet | EmilienM: can you review this please? https://review.openstack.org/#/c/443261/ | 15:41 |
| flaper87 | EmilienM: my pleasure :) | 15:41 |
| EmilienM | trozet: did you add the dependency in RDO packaging? | 15:41 |
| trozet | EmilienM: yeah let me get the link | 15:41 |
| trozet | EmilienM: https://review.rdoproject.org/r/#/c/5560/ | 15:42 |
| EmilienM | good, I approved it | 15:42 |
| *** yamahata has joined #tripleo | 15:44 | |
| *** udesale has quit IRC | 15:46 | |
| *** leanderthal has joined #tripleo | 15:46 | |
| *** zoli is now known as zoli|brb | 15:51 | |
| *** dsariel_ has joined #tripleo | 15:51 | |
| trozet | EmilienM: ty | 15:51 |
| *** karimb has quit IRC | 15:52 | |
| *** karimb has joined #tripleo | 15:52 | |
| *** ratailor has joined #tripleo | 15:53 | |
| *** karimb has quit IRC | 15:58 | |
| bnemec | EmilienM: It looks like -t 80 to the deploy command is not being respected anymore. | 15:58 |
| bnemec | I'm looking at http://logs.openstack.org/83/445883/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/05ee057/console.html | 15:58 |
| bnemec | Deploy started at 10:25, the job gets killed at 12:18. | 15:58 |
| jpich | florianf: Hi! https://review.openstack.org/#/c/447670/ is moving through the gate queue veeery slowly and I think I'll be in the plane by the time it merges :( Could you keep an eye on it and check tomorrow morning if http://status.openstack.org/openstack-health/#/job/tripleo-ui-upstream-translation-update is finally green? | 15:58 |
| bnemec | The deploy should have timed out before that so we could collect logs. | 15:59 |
| jpich | florianf: This should also mean that the translation percentages on https://translate.openstack.org/iteration/view/tripleo-ui/stable-ocata drop a bit as the new strings get unstuck | 15:59 |
| *** ratailor has quit IRC | 16:00 | |
| *** padkrish has quit IRC | 16:01 | |
| *** zoli|brb is now known as zoli | 16:03 | |
| EmilienM | bnemec: oops | 16:04 |
| *** ealcaniz has quit IRC | 16:05 | |
| EmilienM | bnemec: can you look at it please and file a bug? I'm already looking on other issues | 16:05 |
| jaosorior | are timeouts in the oooq jobs expected? | 16:05 |
| florianf | jpich: of course, I will | 16:06 |
| EmilienM | jaosorior: not afik. Where did you see it? | 16:06 |
| florianf | jpich: have a good flight! | 16:06 |
| jaosorior | EmilienM: quite a few commits | 16:06 |
| chandankumar | sshnaidm: any idea when will the mitaka issue which we got today will be fixed? | 16:06 |
| EmilienM | jaosorior: you need to be more specific and find out root cause or give links here | 16:06 |
| EmilienM | jaosorior: we might have different issues | 16:06 |
| *** karimb has joined #tripleo | 16:07 | |
| trown | weshay_sick: adarazs do you guys want to discuss the log landing page now? | 16:07 |
| jpich | florianf: Thank you!! Should there still be errors (no............... ;_;) AJaeger might be able to give pointers to the script causing issues | 16:07 |
| jaosorior | EmilienM: https://review.openstack.org/#/c/447971/ | 16:07 |
| sshnaidm | chandankumar, no idea unfortunately, afaik it's very close to EOL, so.. | 16:07 |
| florianf | jpich: cool, will try to catch him if I need to. | 16:07 |
| trown | weshay_sick: adarazs: I think we can actually have it at the same level as console without any infra change | 16:07 |
| jaosorior | EmilienM: that one will fail on the overcloud deploy (as I tested manually) but it's timing out there, and that's quite unrelated. | 16:07 |
| EmilienM | so it fails here: http://logs.openstack.org/71/447971/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/c9b984b/console.html#_2017-03-21_13_48_28_496096 | 16:07 |
| EmilienM | and there is no log for undercloud :( http://logs.openstack.org/71/447971/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/c9b984b/logs/ | 16:08 |
| *** nyechiel has quit IRC | 16:08 | |
| bnemec | chandankumar: sshnaidm: Which mitaka issue? I have a patch up to fix instack on mitaka. | 16:08 |
| EmilienM | jaosorior: so impossible to debug | 16:08 |
| jaosorior | EmilienM: wait a minute... that's not the one commit that should have failed... | 16:08 |
| chandankumar | bnemec: http://logs.openstack.org/68/446868/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha-mitaka/58e4697/ | 16:08 |
| sshnaidm | bnemec, it's about undercloud fail on mitaka, iirc | 16:09 |
| jaosorior | that one was passing before | 16:09 |
| jaosorior | EmilienM: anyway, that's one of the few I've seen timing out today | 16:09 |
| EmilienM | jaosorior: no clue now | 16:09 |
| weshay_sick | trown, sure | 16:09 |
| EmilienM | jaosorior: and I'm already multi tasking between different issues, I would use some help | 16:09 |
| jpich | florianf: ..........There won't be any more problems I'm sure | 16:09 |
| weshay_sick | trown, what is your idea? | 16:09 |
| jaosorior | EmilienM: same here to be honest. | 16:09 |
| bnemec | chandankumar: sshnaidm: Yeah, that should be fixed by https://review.openstack.org/#/c/447116 | 16:10 |
| * florianf makes a screenshot of what jpich just said. | 16:10 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 16:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 16:10 |
| chandankumar | bnemec: do we need to backport it to stable/mitaka branch? | 16:10 |
| jpich | florianf: :D | 16:11 |
| bnemec | chandankumar: No, there was no stable/mitaka branch for instack. That's why it broke. | 16:11 |
| trown | weshay_sick: adarazs we could put something in https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/common_functions.sh#L185-L279 that checks if the quickstart created log landing page exists and copy it to $WORKSPACE | 16:11 |
| *** nyechiel has joined #tripleo | 16:11 | |
| * chandankumar waits till it get merged. | 16:11 | |
| adarazs | trown: won't that still be under logs/? | 16:13 |
| trown | adarazs: no $WORKSPACE is above logs | 16:13 |
| trown | adarazs: stuff gets put in $WORKSPACE/logs in the postci function | 16:14 |
| *** tbonds has joined #tripleo | 16:17 | |
| adarazs | trown: I'm not sure what uploads the logs... that script might be outside of our CI scripts (at least I don't find where it happens in tripleo-ci) and it might be only uploading stuff from $WORKSPACE/logs + puts there the console log. | 16:18 |
| trown | adarazs: hmm true | 16:18 |
| trown | adarazs: worth a shot though | 16:18 |
| adarazs | trown: there would be a few more stuff uploaded from $WORKSPACE otherwise. | 16:19 |
| trown | adarazs: hmm ok... well we can still move it to $WORKSPACE/logs from quickstart collected logs in postci function | 16:21 |
| trown | would be nice to have at the top with console, but at the top of logs/ is still pretty easy to find | 16:21 |
| adarazs | trown, weshay_sick: yeah, that's a step in the right direction :) | 16:21 |
| trown | I think we can expect most folks will at least click the logs link :) | 16:21 |
| *** udesale has joined #tripleo | 16:21 | |
| *** karimb has quit IRC | 16:22 | |
| *** nyechiel has quit IRC | 16:22 | |
| adarazs | sshnaidm: heh, so overcloud deploy just returned 0 and it still failed miserably on the change the removes the checking... | 16:23 |
| adarazs | sshnaidm: http://logs.openstack.org/93/447993/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/3a5d2cb/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 16:23 |
| adarazs | (re https://review.openstack.org/#/c/447993/) | 16:23 |
| trown | adarazs: that looks like overcloud deploy failed | 16:24 |
| *** ckyriakidou has quit IRC | 16:24 | |
| adarazs | trown: the job continued and eventually the validate failed. | 16:24 |
| trown | adarazs: which means we still have tripleoclient bug if the deploy command returned 0 | 16:24 |
| *** ckyriakidou has joined #tripleo | 16:25 | |
| sshnaidm | that's bad | 16:25 |
| rook | coding_shardy https://bugs.launchpad.net/tripleo/+bug/1674732 -- I am going the route of having OOO launch the hosts and then preparing them by hand :( | 16:25 |
| openstack | Launchpad bug 1674732 in tripleo "51-hosts at scale fails to complete or report a bug" [Critical,Triaged] | 16:25 |
| trown | adarazs: looks like it failed downloading packages sigh | 16:25 |
| *** janki has quit IRC | 16:26 | |
| adarazs | yeah, that's the theme of the day :/ | 16:26 |
| adarazs | trown, sshnaidm: http://logs.openstack.org/93/447993/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/3a5d2cb/console.html#_2017-03-21_16_16_02_215315 | 16:26 |
| adarazs | we have "rc": 0 | 16:26 |
| sshnaidm | ok, seems like we need more flexible string logic then.. | 16:27 |
| dtrainor | I'm getting a deployment failure and not able to find any more information about it. There are no apparent failures on the Controller node. Here's the output of failures list http://paste.openstack.org/show/603632/ (note: I am using osp11 upstream but I don't think that's influencing the error) | 16:27 |
| trown | so either tripleoclient bug was never fixed, or was broken again at some point | 16:27 |
| sshnaidm | will try to fix it today, again | 16:27 |
| weshay_sick | trown, adarazs so I have it working at the top of logs/ | 16:27 |
| adarazs | trown: https://review.openstack.org/#/q/topic:cbs/https -- if we merge this it might help, but I'm skeptical about changing to https will really improve things. | 16:27 |
| weshay_sick | I'll poke at https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/common_functions.sh#L185-L279 | 16:28 |
| sshnaidm | trown, adarazs maybe because of that we didn't see failed_deployment logs? | 16:28 |
| weshay_sick | then I have to figure out why it's not working in centos | 16:28 |
| *** tbonds has quit IRC | 16:28 | |
| *** yprokule has quit IRC | 16:28 | |
| sshnaidm | adarazs, it doesn't | 16:28 |
| *** jaosorior has quit IRC | 16:29 | |
| openstackgerrit | Feng Pan proposed openstack/os-net-config master: Add support for VPP interface https://review.openstack.org/424426 | 16:29 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Only force http for repos that allow it https://review.openstack.org/448187 | 16:29 |
| bnemec | EmilienM: ^ | 16:30 |
| EmilienM | bnemec: mhh ok | 16:30 |
| openstackgerrit | Feng Pan proposed openstack/os-net-config master: Add support for VPP interface https://review.openstack.org/424426 | 16:30 |
| EmilienM | bnemec: I'll review it today | 16:31 |
| bnemec | EmilienM: No rush, it's not going to finish CI until tomorrow anyway. :-/ | 16:31 |
| bandini | EmilienM: sorry was in calls all afternoon. to answer your question of a few hours ago, no I don't need https://review.openstack.org/445479 backported to newton (we did not have it there so we're good) | 16:32 |
| *** nyechiel has joined #tripleo | 16:32 | |
| EmilienM | bandini: ok cool, np! | 16:33 |
| adarazs | weshay_sick: I'm trying to figure out why it's missing... | 16:33 |
| *** karimb has joined #tripleo | 16:34 | |
| *** milan has quit IRC | 16:34 | |
| Ng | is there a known bug with the undercloud-oooq CI job atm? http://paste.openstack.org/raw/603634/ | 16:34 |
| *** fragatina has joined #tripleo | 16:35 | |
| Ng | (seems to be affecting multinode-oooq too) | 16:35 |
| adarazs | Ng: yes. | 16:37 |
| adarazs | Ng: https://bugs.launchpad.net/tripleo/+bug/1674067 | 16:37 |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 16:37 |
| * Ng nods :) | 16:37 | |
| Ng | thanks | 16:37 |
| adarazs | Ng: trying to push it through the gates for a while now. | 16:37 |
| adarazs | Ng: it should work after this merges: https://review.openstack.org/447530 | 16:38 |
| Ng | wow, you got a -2 from jenkins. I don't think I've ever seen it do that before ;o | 16:38 |
| adarazs | yes, Jenkins seems to be angry with me all day. | 16:38 |
| openstackgerrit | Feng Pan proposed openstack/os-net-config master: Add support for VPP interface https://review.openstack.org/424426 | 16:40 |
| bandini | that is a weird -2 there is no depends-on | 16:40 |
| bandini | maybe it is rebased from a tip that filed to merge? | 16:41 |
| EmilienM | flaper87: when you have time, could you please look [deployment][tripleo] Next steps for configuration management in TripleO (and other tools?) thread please? | 16:41 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Parameterize OVB stack flavors https://review.openstack.org/445163 | 16:42 |
| EmilienM | flaper87: I want to make sure it's aligned with the work done on containers and get your feedback (you and others, maybe you can raise it to your next sync meeting) | 16:42 |
| *** dparkes has quit IRC | 16:43 | |
| *** jpich has quit IRC | 16:43 | |
| *** jcoufal has quit IRC | 16:43 | |
| *** tbonds has joined #tripleo | 16:44 | |
| flaper87 | EmilienM: yes, I read it yday but haven't gotten to reply yet. I will, though. Thanks for the reminder | 16:44 |
| EmilienM | flaper87: thx | 16:45 |
| *** rbrady has quit IRC | 16:47 | |
| *** jcoufal has joined #tripleo | 16:49 | |
| *** dsariel_ has quit IRC | 16:51 | |
| *** jkilpatr_ has joined #tripleo | 16:51 | |
| *** jkilpatr has quit IRC | 16:54 | |
| *** pblaho has quit IRC | 16:54 | |
| *** trown is now known as trown|lunch | 16:55 | |
| *** jaosorior has joined #tripleo | 16:55 | |
| *** udesale has quit IRC | 16:57 | |
| *** coding_shardy is now known as shardy | 16:57 | |
| *** thrash|bbl is now known as thrash | 16:58 | |
| *** jlinkes has quit IRC | 17:00 | |
| *** zoli is now known as zoli|gone | 17:01 | |
| *** aufi has joined #tripleo | 17:02 | |
| *** rbrady has joined #tripleo | 17:02 | |
| *** gbarros has quit IRC | 17:03 | |
| *** gbarros has joined #tripleo | 17:03 | |
| *** dsariel_ has joined #tripleo | 17:03 | |
| *** dsneddon has joined #tripleo | 17:05 | |
| *** gbarros has quit IRC | 17:06 | |
| *** gbarros has joined #tripleo | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 17:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 17:10 |
| EmilienM | amoralej: if you work on https://bugs.launchpad.net/tripleo/+bug/1674681 can you maybe assign it to you? | 17:10 |
| *** skramaja has quit IRC | 17:12 | |
| amoralej | EmilienM, at this point i think the issue is related to the squid proxy | 17:12 |
| *** gbarros has quit IRC | 17:12 | |
| *** gbarros has joined #tripleo | 17:13 | |
| amoralej | what's the impact of not using it for tripleo jobs?, what's the time gained by using it? | 17:13 |
| *** mcornea has quit IRC | 17:16 | |
| adarazs | weshay_sick: I'm looking all around but I don't understand either why that README_LOGS_LAYOUT.txt is not uploaded. I can see it gets created in the collect logs logs: https://ci.centos.org/view/rdo/view/tripleo-gate/job/tripleo-quickstart-extras-gate-master-tripleo-ci-delorean-full-minimal_pacemaker/953/artifact/collect_logs.txt | 17:16 |
| weshay_sick | ya.. weird right | 17:16 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Add httpchk for http services https://review.openstack.org/439052 | 17:17 |
| bnemec | amoralej: EmilienM: Only one of the repo failure results in the past 24 hours was an ovb job, and those are the only ones that use squid. | 17:19 |
| bnemec | Unless there's something going on in the multinode jobs that I don't know about, but I'm not sure how that would be possible. | 17:19 |
| bnemec | Those don't have access to our squid server. | 17:19 |
| *** artom has quit IRC | 17:19 | |
| amoralej | jpena ^ | 17:19 |
| *** artom has joined #tripleo | 17:19 | |
| amoralej | bnemec, thanks for that hint | 17:20 |
| jpena | amoralej: ack | 17:20 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add dracut-regenerate elements https://review.openstack.org/447047 | 17:20 |
| *** akuznetsov has quit IRC | 17:21 | |
| bnemec | I would actually bet that the lack of caching in the multinode jobs is part of the problem. | 17:22 |
| bnemec | I wonder if we're getting rate-limited by CBS. | 17:22 |
| bnemec | We've got to be driving an obscene amount of traffic. | 17:22 |
| adarazs | sshnaidm, trown|lunch, EmilienM: please keep an eye on https://review.openstack.org/447993 & https://review.openstack.org/447530 and try to get them merged. I can't wait for the gate job to finish. | 17:23 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 17:24 |
| *** pblaho has joined #tripleo | 17:24 | |
| EmilienM | adarazs: ok | 17:25 |
| EmilienM | adarazs: I'll take care of it | 17:25 |
| adarazs | thanks! | 17:25 |
| *** fragatina has quit IRC | 17:28 | |
| dtantsur | such a nice failure, my overcloud has: Error: /Stage[main]/Apache::Service/Service[httpd]: Could not evaluate: Cannot allocate memory - fork(2) | 17:29 |
| dtantsur | what is a requirement for controllers nowadays? | 17:29 |
| dtantsur | if 8Gi is not enough :) | 17:30 |
| *** lmiccini has quit IRC | 17:31 | |
| EmilienM | dtantsur: where did you see it? | 17:32 |
| dtantsur | EmilienM, locally, trying to deploy on 8Gi VMs | 17:32 |
| EmilienM | ok | 17:32 |
| shardy | chem, marios: Hey I guess you're near EOD now, but pls can you check https://bugs.launchpad.net/tripleo/+bug/1674767 tomorrow or when you have time | 17:33 |
| openstack | Launchpad bug 1674767 in tripleo "Minor update workflow doesn't work for custom roles" [High,Triaged] | 17:33 |
| dtantsur | now free -h shows available = 1.1Gi, but this is after it failed | 17:33 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates master: Fix usage of CinderNfsServers https://review.openstack.org/448208 | 17:33 |
| chem | shardy: looking | 17:33 |
| shardy | chem, marios: mhenkel has noticed this problem when testing with the contrail custom roles - I think it's probably just from old assumptions about roles | 17:33 |
| shardy | but I wanted to get clarification before hacking on any patches | 17:33 |
| shardy | chem: I suspect we can just add another boolean to roles_data which controls if package updates are applied | 17:34 |
| marios | shardy: ack will do thanks and indeed /me off for today | 17:34 |
| shardy | marios: np, I think there's an easy workaround for now, we can chat more tomorrow, thanks! | 17:35 |
| chem | shardy: sorry will look in ~15min, in a mtg right now | 17:35 |
| shardy | chem: np - I just wanted to draw attention to it, we can talk tomorrow, thanks! | 17:35 |
| bnemec | OOM might explain the timeouts we're seeing in ci. | 17:35 |
| chem | shardy: akc | 17:35 |
| *** artom has quit IRC | 17:35 | |
| *** rwsu has joined #tripleo | 17:36 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Don't track added_files in deploy environment processing https://review.openstack.org/446045 | 17:36 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Don't merge environments during plan creation https://review.openstack.org/448209 | 17:36 |
| *** artom has joined #tripleo | 17:37 | |
| *** lucasagomes is now known as lucas-afk | 17:38 | |
| *** yamahata has quit IRC | 17:40 | |
| beagles | shardy: I don't feel comfortable +A a spec but Jiri's https://review.openstack.org/#/c/438918/ has 3 +2's. I think you had the last substantive feedback on this one, so maybe you should be the one to pull the trigger? | 17:41 |
| shardy | beagles: ack, looking | 17:42 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: create a README_LOGS to document log files https://review.openstack.org/447653 | 17:42 |
| weshay_sick | ccamacho, https://bugs.launchpad.net/tripleo/+bug/1674711 | 17:42 |
| openstack | Launchpad bug 1674711 in tripleo "RFE, tripleo ci logs should include a README with log info" [High,Triaged] - Assigned to wes hayutin (weshayutin) | 17:42 |
| weshay_sick | fyi | 17:42 |
| ccamacho | weshay_sick thanks man | 17:43 |
| weshay_sick | sshnaidm, did you see my feedback on the new ovb playbook | 17:43 |
| *** jtomasek has quit IRC | 17:43 | |
| sshnaidm | weshay_sick, yeah | 17:44 |
| *** leanderthal has quit IRC | 17:44 | |
| sshnaidm | weshay_sick, I tried to push all tasks into roles, the few remaining are too special for tripleo-ci | 17:45 |
| openstackgerrit | Merged openstack/tripleo-specs master: Deployment plan management changes https://review.openstack.org/438918 | 17:46 |
| *** florianf has quit IRC | 17:46 | |
| *** trown|lunch is now known as trown | 17:47 | |
| *** beagles is now known as beagles_lunch | 17:48 | |
| *** florianf has joined #tripleo | 17:48 | |
| *** cylopez has quit IRC | 17:50 | |
| *** ckyriakidou has quit IRC | 17:50 | |
| *** derekh has quit IRC | 17:50 | |
| *** gkadam has joined #tripleo | 17:52 | |
| *** fragatina has joined #tripleo | 17:55 | |
| *** nyechiel has quit IRC | 17:55 | |
| *** florianf has quit IRC | 17:56 | |
| *** leanderthal has joined #tripleo | 17:56 | |
| sshnaidm | bnemec, do you know how many ovb jobs run in a day approximately ? | 17:57 |
| *** jaosorior has quit IRC | 17:57 | |
| *** dtantsur is now known as dtantsur|afk | 17:58 | |
| sshnaidm | bnemec, sorry, just saw your mail.. | 17:58 |
| *** rwsu has quit IRC | 17:59 | |
| *** rwsu has joined #tripleo | 18:00 | |
| *** flepied has quit IRC | 18:00 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Swift auth url should use a suffix https://review.openstack.org/448077 | 18:02 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement verify profiles as a custom action https://review.openstack.org/429684 | 18:04 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check ironic boot configuration as a custom action https://review.openstack.org/428225 | 18:04 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check node counts as a custom action https://review.openstack.org/429757 | 18:04 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add workflow wrapping all predeployment validations https://review.openstack.org/445914 | 18:04 |
| *** aufi has quit IRC | 18:05 | |
| *** artom_ has joined #tripleo | 18:06 | |
| *** karimb has quit IRC | 18:06 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient master: Call validations in workflows https://review.openstack.org/448057 | 18:07 |
| *** artom has quit IRC | 18:09 | |
| *** dprince has quit IRC | 18:10 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 18:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 18:10 |
| EmilienM | pradk: did you figure why gnocchi is broken on v3? | 18:12 |
| pradk | https://review.openstack.org/#/c/448077/ | 18:12 |
| pradk | on call | 18:12 |
| *** rbrady has quit IRC | 18:13 | |
| *** artom__ has joined #tripleo | 18:14 | |
| *** dbecker has quit IRC | 18:14 | |
| EmilienM | pradk: +2 | 18:15 |
| *** artom_ has quit IRC | 18:17 | |
| *** nyechiel has joined #tripleo | 18:18 | |
| *** shardy has quit IRC | 18:18 | |
| *** dsariel_ has quit IRC | 18:21 | |
| *** leanderthal has quit IRC | 18:23 | |
| mwhahaha | can some folks review this THT patch https://review.openstack.org/#/c/446029/ it's for bigswitch since we broke the integration back in newton | 18:24 |
| *** tbonds has quit IRC | 18:24 | |
| *** weshay_sick is now known as weshay_brb | 18:25 | |
| *** yamahata has joined #tripleo | 18:26 | |
| *** rbrady has joined #tripleo | 18:28 | |
| *** tbonds has joined #tripleo | 18:30 | |
| *** artom__ has quit IRC | 18:31 | |
| *** yamahata has quit IRC | 18:31 | |
| *** yamahata has joined #tripleo | 18:31 | |
| *** tesseract has quit IRC | 18:31 | |
| *** artom has joined #tripleo | 18:32 | |
| *** flepied has joined #tripleo | 18:33 | |
| *** beagles_lunch is now known as beagles | 18:33 | |
| *** pkovar has quit IRC | 18:35 | |
| EmilienM | mwhahaha: https://review.openstack.org/#/c/446029/13/puppet/services/neutron-bigswitch-agent.yaml | 18:37 |
| EmilienM | shouldn't we create a profile? | 18:37 |
| EmilienM | steps are done in puppet profiles | 18:37 |
| mwhahaha | i'm trying to make this backport as painless as possible | 18:37 |
| mwhahaha | i'd be happy to follow on with a profile | 18:37 |
| *** artom_ has joined #tripleo | 18:37 | |
| EmilienM | mwhahaha: wait | 18:38 |
| mwhahaha | but since this thing has to go all the way back to newton, i'd rather only have to backport a single patch in a single repo :D | 18:38 |
| EmilienM | mwhahaha: yeah but do we plan to have a profile on master? | 18:38 |
| *** salmankhan has quit IRC | 18:39 | |
| mwhahaha | EmilienM: i can follow on switching it to use a profile in a subsequent patch | 18:39 |
| *** artom has quit IRC | 18:39 | |
| mwhahaha | EmilienM: there's puppet-neutron work to be done as well | 18:39 |
| *** pkovar has joined #tripleo | 18:39 | |
| mwhahaha | EmilienM: see https://review.openstack.org/#/c/446029/13/puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml@36 | 18:39 |
| *** pcaruana has quit IRC | 18:39 | |
| *** jpena is now known as jpena|off | 18:39 | |
| EmilienM | mwhahaha: do you plan to backport this work? | 18:39 |
| mwhahaha | i only want to backport that patch | 18:40 |
| mwhahaha | nothing else | 18:40 |
| *** tosky has quit IRC | 18:40 | |
| mwhahaha | the profile/puppet-neutron updates are future work | 18:40 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Add "Save and Close" button to configuration modal dialog https://review.openstack.org/448229 | 18:40 |
| EmilienM | mwhahaha: to newton? | 18:40 |
| mwhahaha | EmilienM: yea | 18:40 |
| mwhahaha | EmilienM: wait what to newton? | 18:41 |
| EmilienM | mwhahaha: bigswitch | 18:41 |
| EmilienM | https://review.openstack.org/#/c/446029/ | 18:41 |
| mwhahaha | yes | 18:41 |
| EmilienM | ok | 18:41 |
| mwhahaha | because we broke it in 9->10 | 18:41 |
| *** ukalifon has quit IRC | 18:41 | |
| mwhahaha | and they just figured this out | 18:41 |
| EmilienM | mwhahaha: does it work to have the step thing in step_config? | 18:41 |
| EmilienM | I never saw it | 18:41 |
| mwhahaha | EmilienM: yea | 18:41 |
| EmilienM | k | 18:41 |
| mwhahaha | they tested it | 18:42 |
| *** weshay_brb is now known as weshay | 18:42 | |
| *** rwsu has quit IRC | 18:42 | |
| EmilienM | (tm) | 18:43 |
| EmilienM | mwhahaha: +2 and commented | 18:43 |
| mwhahaha | pretty much | 18:43 |
| *** dprince has joined #tripleo | 18:43 | |
| *** suuuper has quit IRC | 18:44 | |
| EmilienM | mwhahaha: please file a bug in launchpad about this profile thing, to make sure we don't miss it in pike. thanks | 18:44 |
| mwhahaha | yea i'll post some patches in a minute as well | 18:45 |
| *** bogdando has quit IRC | 18:46 | |
| *** leanderthal has joined #tripleo | 18:46 | |
| *** leanderthal has quit IRC | 18:46 | |
| EmilienM | ok | 18:48 |
| dtrainor | honza++ re: "Save and Close" | 18:48 |
| *** gbarros has quit IRC | 18:49 | |
| honza | dtrainor: \o/ | 18:49 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add dracut-regenerate elements https://review.openstack.org/447047 | 18:55 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Create bigswitch agent profile https://review.openstack.org/448234 | 18:55 |
| *** karimb has joined #tripleo | 18:56 | |
| *** gkadam has quit IRC | 18:57 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Use tripleo profile for bigswitch agent https://review.openstack.org/448236 | 18:59 |
| *** nyechiel has quit IRC | 19:00 | |
| *** pkovar has quit IRC | 19:04 | |
| *** jprovazn has quit IRC | 19:10 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 19:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 19:10 |
| *** rwsu has joined #tripleo | 19:12 | |
| *** chlong has joined #tripleo | 19:12 | |
| *** bswartz has quit IRC | 19:13 | |
| *** karimb has quit IRC | 19:16 | |
| *** rwsu has quit IRC | 19:17 | |
| *** eck` is now known as eck`gone | 19:18 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 19:20 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize Redis service https://review.openstack.org/442151 | 19:21 |
| *** ukalifon has joined #tripleo | 19:21 | |
| *** ckyriakidou has joined #tripleo | 19:23 | |
| rook | ugh! | 19:28 |
| rook | shardy left! | 19:28 |
| rook | might be late his time | 19:28 |
| openstackgerrit | Feng Pan proposed openstack/os-net-config master: Add support for VPP interface https://review.openstack.org/424426 | 19:30 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 19:31 |
| *** gbarros has joined #tripleo | 19:35 | |
| *** akrivoka has quit IRC | 19:35 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 19:39 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 19:39 |
| stevebaker | morning | 19:43 |
| *** amoralej is now known as amoralej|off | 19:44 | |
| *** rhallisey has quit IRC | 19:45 | |
| *** rhallisey has joined #tripleo | 19:48 | |
| EmilienM | bnemec: have you seen https://review.openstack.org/#/c/448045/ ? | 19:50 |
| *** eck`gone is now known as eck` | 19:51 | |
| bnemec | EmilienM: +2 | 19:52 |
| EmilienM | bnemec: I have no idea why ci jobs failed tbh | 19:53 |
| bnemec | EmilienM: I'd just throw a dart at the critical bugs list and blame whatever you hit. :-) | 19:54 |
| *** ukalifon has quit IRC | 19:54 | |
| *** salmankhan has joined #tripleo | 19:58 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 19:59 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 20:00 |
| *** iranzo has quit IRC | 20:01 | |
| *** salmankhan has quit IRC | 20:02 | |
| *** artom__ has joined #tripleo | 20:04 | |
| slagle | does anyone know why we don't create the heat cfn endpoint in the undercloud? | 20:04 |
| *** artom_ has quit IRC | 20:07 | |
| bnemec | slagle: We don't create a cfn endpoint in the overcloud either, do we? | 20:08 |
| bnemec | I think the idea is that cfn is only used for signalling, and a registered endpoint isn't needed for that. | 20:08 |
| bnemec | But that's mostly speculation on my part. | 20:09 |
| openstackgerrit | Merged openstack/tripleo-ui stable/ocata: Remove '\n' linebreak from localised string https://review.openstack.org/447670 | 20:09 |
| slagle | bnemec: we do create it in the overcloud | 20:09 |
| *** pkovar has joined #tripleo | 20:09 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 20:10 |
| bnemec | slagle: Hmm, that must have changed. We don't have it on rh1. | 20:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 20:10 |
| bnemec | slagle: In that case I have no idea. :-) | 20:10 |
| slagle | i was hoping to do the signaling over the public endpoint (hostname), when using deployed servers and the ctlplane is not routable | 20:10 |
| slagle | but...i could also switch the default signaling to swift | 20:11 |
| *** karimb has joined #tripleo | 20:11 | |
| *** radeks has quit IRC | 20:16 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:16 | |
| *** dsneddon has quit IRC | 20:17 | |
| *** chlong has quit IRC | 20:21 | |
| *** dparkes has joined #tripleo | 20:21 | |
| *** rhallisey has quit IRC | 20:22 | |
| *** jkilpatr_ has quit IRC | 20:28 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Add log capturing support workflows https://review.openstack.org/405002 | 20:28 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 20:30 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Add log capturing support workflows https://review.openstack.org/405002 | 20:30 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 20:31 |
| *** bogdando has joined #tripleo | 20:33 | |
| *** dsneddon has joined #tripleo | 20:33 | |
| *** salmankhan has joined #tripleo | 20:33 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 20:34 |
| weshay | trown, I gave your idea a go, I think this was what you were suggesting, but it didn't work https://review.openstack.org/#/c/447653/ | 20:39 |
| trown | weshay: ya, I think we have to do $WORKSPACE/logs | 20:39 |
| weshay | k | 20:40 |
| trown | weshay: I think infra only collects logs ... sorry | 20:40 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Add multinode deploy arg to the pre-deploy tasks https://review.openstack.org/448274 | 20:40 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Add multinode deploy arg to the pre-deploy tasks https://review.openstack.org/448274 | 20:42 |
| openstackgerrit | Mikhail S Medvedev proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 20:43 |
| *** oneswig has joined #tripleo | 20:43 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-common master: Custom DHCP agent for DHCP-relay support https://review.openstack.org/447948 | 20:44 |
| openstackgerrit | Mikhail S Medvedev proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 20:44 |
| *** jkilpatr has joined #tripleo | 20:44 | |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud master: Custom DHCP agent for DHCP-relay support https://review.openstack.org/447214 | 20:44 |
| *** gbarros has quit IRC | 20:44 | |
| *** chlong has joined #tripleo | 20:47 | |
| *** ansmith has quit IRC | 20:48 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart master: Add deploy_multinode option to the featureset multinode config https://review.openstack.org/448278 | 20:49 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-common master: Custom DHCP agent for DHCP-relay support https://review.openstack.org/447948 | 20:50 |
| *** gbarros has joined #tripleo | 20:50 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: create a README_LOGS to document log files https://review.openstack.org/447653 | 20:51 |
| *** gbarros has quit IRC | 20:51 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Add multinode deploy arg to the pre-deploy tasks https://review.openstack.org/448274 | 20:53 |
| *** trown is now known as trown|outtypewww | 20:54 | |
| *** achadha has joined #tripleo | 20:57 | |
| *** morazi has quit IRC | 20:58 | |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud master: Tripleo routed networks ironic inspector, and Undercloud https://review.openstack.org/437544 | 21:00 |
| openstackgerrit | John Eckersberg proposed openstack/puppet-tripleo master: WIP qdr HA mesh https://review.openstack.org/448279 | 21:02 |
| *** karimb has quit IRC | 21:04 | |
| *** athomas has joined #tripleo | 21:05 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 21:05 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 21:06 |
| *** Goneri has quit IRC | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 21:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 21:10 |
| *** lblanchard has quit IRC | 21:10 | |
| *** salmankhan1 has joined #tripleo | 21:11 | |
| *** salmankhan has quit IRC | 21:11 | |
| *** salmankhan1 is now known as salmankhan | 21:11 | |
| openstackgerrit | James Slagle proposed openstack/instack-undercloud master: Configurable [clients]/endpoint_type for heat.conf https://review.openstack.org/448281 | 21:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 21:13 |
| *** morazi has joined #tripleo | 21:14 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 21:14 |
| *** mburned is now known as mburned_out | 21:16 | |
| *** mburned_out is now known as mburned | 21:16 | |
| *** eck` is now known as eck`gone | 21:19 | |
| *** salmankhan has quit IRC | 21:21 | |
| *** nyechiel has joined #tripleo | 21:21 | |
| openstackgerrit | James Slagle proposed openstack/instack-undercloud master: Create Heat API CFN endpoint https://review.openstack.org/448287 | 21:24 |
| *** abishop has quit IRC | 21:24 | |
| *** bfournie has quit IRC | 21:25 | |
| *** gbarros has joined #tripleo | 21:26 | |
| *** colonwq2 has quit IRC | 21:28 | |
| EmilienM | slagle: I'm surprised we don't deploy the endpoint but we do deploy the service :/ | 21:31 |
| EmilienM | slagle: could you please add a release note? | 21:31 |
| slagle | EmilienM: ok sure. didn't realize we were -1'ing for release notes now | 21:34 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct https://review.openstack.org/446934 | 21:34 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-heat-templates stable/ocata: TEST CHANGE, IGNORE https://review.openstack.org/446865 | 21:35 |
| EmilienM | slagle: well, after a bunch of repeat and reviews without -1, some folks still forget the importance of documenting our work | 21:35 |
| EmilienM | slagle: so yeah, I'm -1'ing patches without release notes when I think it could be added | 21:35 |
| EmilienM | that's the only way I found to make sure it becomes a reflex in the future | 21:36 |
| *** nyechiel has quit IRC | 21:36 | |
| EmilienM | slagle: but if you think it's not fair, I can stop doing it (/me looking for feedback really) | 21:37 |
| openstackgerrit | James Slagle proposed openstack/instack-undercloud master: Create Heat API CFN endpoint https://review.openstack.org/448287 | 21:39 |
| EmilienM | slagle: +2 | 21:40 |
| slagle | EmilienM: i was asking because i only recalled discussion around not -1'ing | 21:40 |
| slagle | EmilienM: i don't really care | 21:40 |
| slagle | thanks for your review | 21:40 |
| EmilienM | I don't understand how we missed this endpoint though | 21:40 |
| EmilienM | bnemec: is it the timeout thing? http://logs.openstack.org/22/447522/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/7e971e3/console.html#_2017-03-21_21_00_11_660948 | 21:42 |
| openstackgerrit | James Slagle proposed openstack/instack-undercloud master: Configurable [clients]/endpoint_type for heat.conf https://review.openstack.org/448281 | 21:43 |
| bnemec | EmilienM: Holy cow, the introspection step took 42 minutes in that job! http://logs.openstack.org/22/447522/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/7e971e3/console.html#_2017-03-21_20_01_11_667633 | 21:50 |
| bnemec | No, wait. | 21:50 |
| * bnemec can't read | 21:50 | |
| EmilienM | we're testing a patch in nova | 21:50 |
| EmilienM | if you look at the patch itsemf | 21:50 |
| EmilienM | itself* | 21:50 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Generate/inject instack.json if othercloud nodes https://review.openstack.org/448294 | 21:51 |
| bnemec | EmilienM: We've regressed performance pretty badly again over the past month: https://66.187.229.172/S/H | 21:51 |
| EmilienM | the page doesn't load for me | 21:52 |
| EmilienM | ah it works now | 21:53 |
| *** ckyriakidou has quit IRC | 21:53 | |
| bnemec | EmilienM: It's probably just slow. There's a lot of data to crunch. | 21:53 |
| EmilienM | well, we keep adding stuffs | 21:54 |
| EmilienM | i would be surprised if our CI would be faster | 21:54 |
| EmilienM | like I already said, we should reduce the OVB jobs to strick minimum | 21:54 |
| *** dprince has quit IRC | 21:54 | |
| EmilienM | I don't even know why we deploy telemetry on OVB jobs (for example) since we have it covered on scenario001 | 21:54 |
| bnemec | EmilienM: Because if we drop telemetry from OVB then we lose things like net-iso and ipv6 coverage. | 21:55 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Add multinode deploy arg to the pre-deploy tasks https://review.openstack.org/448274 | 21:55 |
| *** jcoufal has quit IRC | 21:55 | |
| EmilienM | yeah but by testing the world, we are busting ourselves | 21:56 |
| *** karimb has joined #tripleo | 21:56 | |
| slagle | kind of seems like the ovb containers job push us over the edge | 21:57 |
| bnemec | Well, the other side of it is that we've now noted a 10 minute regression in deployment time over only one month. | 21:57 |
| bnemec | As much as I would like to blame the containers job, I'm not sure that's it either. The newton branch hasn't slowed down, so it's not additional load on the environment causing this. | 21:58 |
| bnemec | That said, the containers job is absolutely responsible for the egregious queue we have right now. | 21:58 |
| slagle | well, that, and all the patches that need to be rechecked when it failed 100% of the time for weeks | 21:58 |
| EmilienM | we have a bunch of critical bugs open, that would all make CI more robust | 21:59 |
| * bnemec points out that it's non-voting | 21:59 | |
| slagle | but but but, i've been politely asked not to ignore it :) | 21:59 |
| EmilienM | like https://bugs.launchpad.net/tripleo/+bug/1674681 for example | 21:59 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 21:59 |
| * bnemec politely declines ;-) | 21:59 | |
| bnemec | That isn't actually affecting the ovb jobs much, AFAICT. | 22:00 |
| bnemec | Hooray for our shady squid server. | 22:00 |
| mwhahaha | s/shady/awesome | 22:01 |
| EmilienM | instead of adding more jobs and making them voting, we should also invest in stabilize the existing ones | 22:01 |
| EmilienM | some folks (me included) have been spending almost all the time dealing with new CI issues every day | 22:01 |
| EmilienM | and folks keep pushing patches like if nothing happens | 22:01 |
| mwhahaha | but features | 22:01 |
| EmilienM | during the weekly meeting this morning I asked for some help and not much folks volunteered | 22:01 |
| bkero | slagle: re: 446934 would you rather see that implemented as env vars as conditionals in setup_multinode() and additional calls to tripleo.sh? AFAIK it's a goal of quickstart to modularize and document some of the duties of tripleo.sh. | 22:02 |
| bnemec | <mwhahaha> but features <-- Yes, that | 22:02 |
| EmilienM | if we don't push more efforts in stabilizing ci, things will never change | 22:02 |
| mwhahaha | should be a 1 feature to 10 bugs quota :D | 22:03 |
| slagle | bkero: i'd rather see the code moved to a common function that could be sourced by both tripleo.sh and this quickstart role | 22:03 |
| *** oneswig has quit IRC | 22:03 | |
| slagle | bkero: the code is significant enough, i'd rather not be having to maintain it in 2 places | 22:03 |
| achadha | guys, this is probably not the best time to bring up this issue: Seeing failure while trying to update a stack. Says "ERROR: Request limit exceeded: JSON body size (1048684 bytes) exceeds maximum allowed size (1048576 bytes)." | 22:03 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Rename 'uploads' key to 'container_images' https://review.openstack.org/447323 | 22:03 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Library wrapper around kolla-build tool https://review.openstack.org/448296 | 22:03 |
| achadha | I've manually patched https://review.openstack.org/#/c/437918/3/elements/puppet-stack-config/puppet-stack-config.yaml.template | 22:04 |
| achadha | and added max_json_body_size in hieradata, but no luck | 22:05 |
| mwhahaha | achadha: https://bugs.launchpad.net/tripleo/+bug/1667697 ? | 22:05 |
| openstack | Launchpad bug 1667697 in tripleo "Default heat's max_json_body_size value is too low to deploy undercloud" [High,Fix released] - Assigned to Martin André (mandre) | 22:05 |
| mwhahaha | achadha: when you say you patched it was that before you deployed? | 22:05 |
| achadha | after | 22:05 |
| achadha | trying to update an existing stack (after adding some roles) | 22:06 |
| mwhahaha | achadha: the hieradata only works before you install/upgrade | 22:06 |
| mwhahaha | achadha: to patch it on an already live system go edit heat.conf and added max_json_body_size under the [DEFAULT] block with your value and restart heat | 22:07 |
| *** jmelvin has quit IRC | 22:08 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 22:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 22:10 |
| *** morazi has quit IRC | 22:10 | |
| *** gbarros has quit IRC | 22:10 | |
| achadha | mwhahaha: that did the trick (changing param in heat.conf). Thanks! | 22:10 |
| bkero | slagle: common_functions.sh currently is only sourced in tripleo.sh for delorean_setup. Would you prefer a global sourcing on top of tripleo.sh? | 22:11 |
| bkero | Or just gut the functions that need it, source it inside the tripleo.sh function, and call it from there? | 22:11 |
| slagle | bkero: it doesn't have to be in common_functions.sh if that is not already sourced by quickstart. you could just add it to a new multinode_function.sh or something, and then yea, have tripleo.sh source that inside the multinode_setup function | 22:13 |
| slagle | bkero: and you could also use it in the role | 22:13 |
| bkero | slagle: quickstart doesn't use common_functions.sh. So instead, you'd suggest making scripts/multinode-setup.sh and having tripleo.sh and quickstart call that? I could do that, but there's the problem that it uses some things from tripleo.sh that aren't in common_functions.sh, like log() and clone() | 22:15 |
| *** athomas has quit IRC | 22:16 | |
| bkero | I could move those, but then it would require a global sourcing of common_functions.sh inside tripleo.sh | 22:16 |
| slagle | i would source mulitnode_setup.sh, which has a function defined it | 22:18 |
| slagle | then call the function | 22:18 |
| bkero | I can do that in tripleo.sh and it will work because tripleo.sh has log() and clone() defined. That is not true for quickstart though. | 22:19 |
| slagle | i think that it would be realtively easy to refactor things in a way such that if the log function were defined, then use it, else use echo | 22:19 |
| bkero | Yeah, log is trivial to log "THING" || echo "THING" | 22:19 |
| slagle | or define a log function in the quickstart script that is just a passthrough to echo | 22:19 |
| bkero | Clone is a bit more complicated though. | 22:20 |
| *** dparkes has quit IRC | 22:21 | |
| slagle | quickstart also defines a clone | 22:22 |
| slagle | line 39 here https://review.openstack.org/#/c/446934/22/roles/undercloud-setup/templates/toci_vxlan_networking.sh.j2, right? | 22:22 |
| bkero | That's also copied for the same reason. :/ Ideally that would be eliminated. | 22:22 |
| slagle | i'm less concerned about that being copied | 22:22 |
| *** dmarlin has quit IRC | 22:22 | |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: New command "overcloud container image build" https://review.openstack.org/448300 | 22:23 |
| * bkero pushes some code around | 22:23 | |
| slagle | bkero: the clone is only needed if you intend to this to be used outside of ci, which i do with tripleo.sh, which is why i added it | 22:24 |
| slagle | bkero: otherwise, those repos will already exist under $TRIPLEO_ROOT (/opt/stack in a ci job) | 22:24 |
| slagle | or is /opt/stack/new. one of those | 22:24 |
| *** bogdando has quit IRC | 22:24 | |
| bkero | slagle: I believe these roles have to work outside CI as well | 22:25 |
| *** chlong has quit IRC | 22:27 | |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: Split multinode_setup.sh out into its own script for use with quickstart https://review.openstack.org/448302 | 22:36 |
| bkero | slagle: You suggest something like this? ^ | 22:37 |
| bkero | That works, but means that tripleo-ci is required outside CI for any multinode deployment, and is totally opaque to quickstart's self-generated docs | 22:38 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Enable quickstart to work with clean base OS images. https://review.openstack.org/442764 | 22:48 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 22:53 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 22:53 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Support multiple messages from register_or_update https://review.openstack.org/446432 | 23:10 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674067 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674236 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674681 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs) | 23:10 |
| openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| openstack | Launchpad bug 1674681 in tripleo "centos.org repositories are unreliable" [Critical,Triaged] | 23:10 |
| *** bfournie has joined #tripleo | 23:13 | |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates master: Restrict Access to Kernel Message Buffer https://review.openstack.org/447407 | 23:21 |
| *** rwsu has joined #tripleo | 23:22 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 23:31 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 23:31 |
| *** soc_off_ has quit IRC | 23:39 | |
| *** soc_off has joined #tripleo | 23:40 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 23:49 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Test boot partition code https://review.openstack.org/448099 | 23:50 |
| *** bswartz has joined #tripleo | 23:52 | |
| *** tbonds has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!