| *** yuanying has joined #tripleo | 00:12 | |
| tchaypo | it's quiet in here today | 00:37 |
|---|---|---|
| *** xuhaiwei has joined #tripleo | 00:54 | |
| *** yamahata has joined #tripleo | 01:13 | |
| greghaynes | *crickets* | 01:16 |
| *** penick has joined #tripleo | 01:26 | |
| *** ddieterl_ has joined #tripleo | 01:29 | |
| *** panda has quit IRC | 01:31 | |
| *** panda has joined #tripleo | 01:31 | |
| *** ddieterly has quit IRC | 01:33 | |
| *** ci-overcloud has joined #tripleo | 02:04 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 02:04 |
| *** ci-overcloud has quit IRC | 02:04 | |
| tchaypo | yes. failed because neutron can't validate the token it gets passed. still. | 02:06 |
| *** penick has quit IRC | 02:25 | |
| *** Yanyanhu has joined #tripleo | 02:36 | |
| *** ramishra has joined #tripleo | 02:48 | |
| *** untriaged-bot has joined #tripleo | 03:01 | |
| untriaged-bot | Untriaged bugs so far: | 03:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 03:01 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 03:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 03:01 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 03:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 03:01 |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 03:01 |
| *** untriaged-bot has quit IRC | 03:01 | |
| *** ramishra has quit IRC | 03:10 | |
| *** ddieterl_ has quit IRC | 03:17 | |
| *** lazy_prince is now known as killer_prince | 03:21 | |
| tchaypo | okay, that's fun. Retru the same command and now 5 of my servers work... difference being that in between I'd restarted keystone and ironic-conductor | 03:22 |
| *** ramishra has joined #tripleo | 03:24 | |
| *** pcrews has joined #tripleo | 03:26 | |
| tchaypo | It's odd that every single time I get exactly 5 successful builds | 03:29 |
| *** jml has quit IRC | 03:30 | |
| lifeless | tchaypo: have you uppped your scheduler retry count? | 03:32 |
| *** jml has joined #tripleo | 03:35 | |
| *** tzumainn has joined #tripleo | 03:36 | |
| *** pcrews has quit IRC | 03:42 | |
| *** nikunj2512 has joined #tripleo | 03:50 | |
| *** killer_prince is now known as lazy_prince | 03:58 | |
| *** pensu has joined #tripleo | 04:01 | |
| *** ci-overcloud has joined #tripleo | 04:03 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 04:03 |
| *** ci-overcloud has quit IRC | 04:03 | |
| *** lazy_prince is now known as killer_prince | 04:17 | |
| tchaypo | lifeless: no, I've been trying to track down the auth problem first | 04:28 |
| tchaypo | but I wouldn't expect that issue to give me exactly 5 hosts every time | 04:28 |
| *** nikunj2512 has quit IRC | 04:30 | |
| *** nikunj2512 has joined #tripleo | 04:30 | |
| *** morazi has quit IRC | 04:33 | |
| *** ramishra has quit IRC | 04:35 | |
| *** ramishra has joined #tripleo | 04:35 | |
| lifeless | tchaypo: auth problem? | 04:39 |
| *** rushiagr_away is now known as rushiagr | 04:40 | |
| *** noslzzp has joined #tripleo | 04:51 | |
| *** akuznetsov has quit IRC | 04:52 | |
| *** loki184 has joined #tripleo | 04:52 | |
| *** akuznetsov has joined #tripleo | 04:56 | |
| *** bvivek has joined #tripleo | 05:05 | |
| *** loki184 has quit IRC | 05:07 | |
| *** loki184 has joined #tripleo | 05:14 | |
| *** tzumainn has quit IRC | 05:15 | |
| *** ddieterly has joined #tripleo | 05:23 | |
| *** Yanyan has joined #tripleo | 05:24 | |
| *** Yanyanhu has quit IRC | 05:26 | |
| *** ddieterly has quit IRC | 05:27 | |
| StevenK | tchaypo: Your questions are terrible, and I'm bored with pawing through neutron code. | 05:31 |
| *** pradeep has joined #tripleo | 05:31 | |
| tchaypo | StevenK: if you're pawing through neutron code it sounds like the answer is "this isn't something setup-neutron does, it's magic from neutron" | 05:45 |
| StevenK | tchaypo: So I could probably explain what 'physical', 'float' and 'external' are better, which may address your concerns. | 05:46 |
| StevenK | tchaypo: Since the network type includes some magic, like shared = True | 05:47 |
| tchaypo | Concernts is a strong word. Confusion perhaps? | 05:47 |
| tchaypo | I think it's likely someone more familiar with neutron/tripleo would just nod and say "of course, that's sensible" | 05:48 |
| tchaypo | the first few times I read that stuff the constant use of "net" instead of "network" bothered me, but since neutron has "net-list" not "network-list" i eventually decided it was correct | 05:48 |
| StevenK | tchaypo: Neutron has the concept of nets and subnets | 05:49 |
| *** mrunge has joined #tripleo | 05:52 | |
| tchaypo | yeah. which i dislike, but *shrug* it's the thing we have | 05:54 |
| tchaypo | lifeless: I'm going to start ignoring the auth issue now. | 05:54 |
| tchaypo | Well, I'll test it one more time ot make sure it's reproducible, but then I'll ignore it | 05:54 |
| tchaypo | and by "ignore" i mean "file a bug and drive on" | 05:55 |
| tchaypo | .. on third thoughts, let me start from the beginning instead of the end | 05:56 |
| tchaypo | the original symptom is that ironic-conductor tells neutron to configure a port for a node and supplies a token for auth; neutron checks the token against keystone and sees that it's invalid so it rejects the request. this goes back up to nova which tries twice more, then gives up; which goes back up to heat, which tries a few more times, but eventually starts running into nova errors which make the deletes | 05:58 |
| tchaypo | fail, so it can't keep retrying, so eventually heat gives up too | 05:58 |
| tchaypo | as far as i can tell the token times out 2 hours after ironic-conductor restarts; I expected the token lifetime to start when the heat stack-create kicks off, but it seems to happen about 38 minutes after that happens | 06:00 |
| tchaypo | I'm fairly certain it's ironic-conductor's token, because (A) restarting ironic-conductor seems to fix the problem (the next 5 hosts work fine) and (B) 2 hours is a nice neat timeframe. | 06:01 |
| *** akuznetsov has quit IRC | 06:05 | |
| *** bvivek has quit IRC | 06:06 | |
| *** akuznetsov has joined #tripleo | 06:11 | |
| *** sdake has joined #tripleo | 06:12 | |
| *** sdake has quit IRC | 06:12 | |
| *** sdake has joined #tripleo | 06:12 | |
| *** k4n0 has joined #tripleo | 06:20 | |
| *** ddieterly has joined #tripleo | 06:23 | |
| *** bvivek has joined #tripleo | 06:24 | |
| *** ddieterly has quit IRC | 06:28 | |
| *** jprovazn has joined #tripleo | 06:32 | |
| *** killer_prince is now known as lazy_prince | 06:37 | |
| *** jtomasek has joined #tripleo | 06:49 | |
| *** bvivek has quit IRC | 06:49 | |
| *** ifarkas has joined #tripleo | 06:52 | |
| *** bvivek has joined #tripleo | 06:53 | |
| lifeless | tchaypo: so thats a bug in ironic | 06:56 |
| lifeless | tchaypo: possibly in how it manages/uses its neutron client | 06:56 |
| lifeless | tchaypo: file it for sure | 06:57 |
| lifeless | tchaypo: since we really can't release J like that :) | 06:57 |
| lifeless | tchaypo: you should be making it a release critical bug and get it socialised ASAP | 06:57 |
| *** jml has quit IRC | 07:05 | |
| *** dtantsur|afk is now known as dtantsur | 07:05 | |
| *** jml has joined #tripleo | 07:07 | |
| *** jcoufal has joined #tripleo | 07:10 | |
| *** Chao_Yan has joined #tripleo | 07:13 | |
| *** ddieterly has joined #tripleo | 07:17 | |
| *** lazy_prince has quit IRC | 07:22 | |
| *** ddieterly has quit IRC | 07:22 | |
| tchaypo | Just as soon as I can confirm that it's totes deffo a bug in ironic | 07:32 |
| tchaypo | and not a 38-minute timeout in heat or something | 07:32 |
| *** killer_prince has joined #tripleo | 07:36 | |
| *** killer_prince is now known as lazy_prince | 07:36 | |
| tchaypo | based on the discussions on -dev recently though, I suspect we're going to end up increasing the timeout to something ludicrious; from what I've read on the list it sounds like that's what everyone does in production | 07:53 |
| *** jistr has joined #tripleo | 07:53 | |
| tchaypo | based on the discussions on -dev recently though, I suspect we're going to end up increasing the timeout to something ludicrious; from what I've read on the list it sounds like that's what everyone does in production | 07:53 |
| *** lazy_prince has quit IRC | 07:57 | |
| *** lazy_prince has joined #tripleo | 07:58 | |
| *** derekh has joined #tripleo | 08:04 | |
| *** gfidente has joined #tripleo | 08:08 | |
| *** athomas has joined #tripleo | 08:10 | |
| *** yuanying has quit IRC | 08:12 | |
| *** yuanying has joined #tripleo | 08:13 | |
| *** yuanying has quit IRC | 08:17 | |
| *** mrunge has quit IRC | 08:18 | |
| *** bvivek has quit IRC | 08:19 | |
| *** mrunge has joined #tripleo | 08:19 | |
| *** mrunge has quit IRC | 08:21 | |
| *** mrunge has joined #tripleo | 08:21 | |
| *** lazy_prince has quit IRC | 08:23 | |
| *** rdopieralski has joined #tripleo | 08:25 | |
| *** mrunge has quit IRC | 08:32 | |
| *** mrunge has joined #tripleo | 08:32 | |
| *** k4n0 has quit IRC | 08:33 | |
| loki184 | please review https://review.openstack.org/#/c/103483/ | 08:33 |
| *** pblaho has joined #tripleo | 08:35 | |
| *** bvivek has joined #tripleo | 08:41 | |
| *** lucasagomes has joined #tripleo | 08:41 | |
| *** ci-overcloud has joined #tripleo | 08:42 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 08:42 |
| *** ci-overcloud has quit IRC | 08:42 | |
| *** yuanying has joined #tripleo | 08:43 | |
| *** rushiagr is now known as rushiagr_away | 08:44 | |
| *** rushiagr_away is now known as rushiagr | 08:45 | |
| *** ramishra has quit IRC | 08:48 | |
| *** k4n0 has joined #tripleo | 08:50 | |
| *** rushiagr is now known as rushiagr_away | 08:56 | |
| *** akrivoka has joined #tripleo | 08:59 | |
| *** untriaged-bot has joined #tripleo | 09:00 | |
| untriaged-bot | Untriaged bugs so far: | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 09:00 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 09:00 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 09:00 |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 09:00 |
| *** untriaged-bot has quit IRC | 09:00 | |
| *** ramishra has joined #tripleo | 09:09 | |
| *** yuanying has quit IRC | 09:18 | |
| *** rushiagr_away is now known as rushiagr | 09:23 | |
| *** Yanyan has quit IRC | 09:27 | |
| *** pradeep has quit IRC | 09:27 | |
| *** pradeep has joined #tripleo | 09:28 | |
| *** rlandy has joined #tripleo | 09:31 | |
| *** Yanyanhu has joined #tripleo | 09:36 | |
| *** pensu has quit IRC | 09:36 | |
| *** jp_at_hp has joined #tripleo | 09:38 | |
| *** pradeep1 has joined #tripleo | 09:40 | |
| *** pradeep has quit IRC | 09:40 | |
| *** k4n0 has quit IRC | 09:44 | |
| *** pblaho has quit IRC | 09:48 | |
| *** pblaho has joined #tripleo | 09:49 | |
| *** pensu has joined #tripleo | 09:52 | |
| *** Yanyanhu has quit IRC | 09:59 | |
| *** andreaf_ is now known as andreaf | 10:01 | |
| *** pradeep1 has quit IRC | 10:04 | |
| *** athomas has quit IRC | 10:10 | |
| *** pradeep has joined #tripleo | 10:11 | |
| *** yuanying has joined #tripleo | 10:13 | |
| *** Chao_Yan has quit IRC | 10:16 | |
| *** athomas has joined #tripleo | 10:19 | |
| *** k4n0 has joined #tripleo | 10:29 | |
| *** yamahata has quit IRC | 10:37 | |
| *** yuanying has quit IRC | 10:47 | |
| *** zigo has quit IRC | 10:50 | |
| *** pmyers has joined #tripleo | 10:51 | |
| *** zigo has joined #tripleo | 10:52 | |
| *** dtantsur is now known as dtantsur|brb | 11:08 | |
| *** Daviey has quit IRC | 11:09 | |
| *** Daviey has joined #tripleo | 11:14 | |
| *** mrunge has quit IRC | 11:15 | |
| *** mrunge has joined #tripleo | 11:16 | |
| *** bvivek has quit IRC | 11:19 | |
| *** bvivek has joined #tripleo | 11:19 | |
| *** yuanying has joined #tripleo | 11:25 | |
| *** loki184 has quit IRC | 11:26 | |
| *** lucasagomes is now known as lucas-hungry | 11:27 | |
| *** nikunj2512 has quit IRC | 11:31 | |
| *** Chao_Yan has joined #tripleo | 11:37 | |
| *** Chao_Yan has quit IRC | 11:44 | |
| *** yuanying has quit IRC | 11:59 | |
| *** marzif__ has quit IRC | 12:02 | |
| *** ddieterly has joined #tripleo | 12:07 | |
| *** jang1 has joined #tripleo | 12:10 | |
| *** jdob has joined #tripleo | 12:11 | |
| *** pradeep has quit IRC | 12:11 | |
| *** pradeep has joined #tripleo | 12:12 | |
| *** openstackgerrit has joined #tripleo | 12:14 | |
| *** pradeep has quit IRC | 12:17 | |
| *** pradeep has joined #tripleo | 12:17 | |
| *** rhallisey has joined #tripleo | 12:21 | |
| *** rhallisey has quit IRC | 12:21 | |
| *** rhallisey has joined #tripleo | 12:22 | |
| *** jdob has quit IRC | 12:25 | |
| *** jdob has joined #tripleo | 12:25 | |
| *** weshay has joined #tripleo | 12:27 | |
| *** dtantsur|brb is now known as dtantsur | 12:28 | |
| *** denis_makogon has left #tripleo | 12:33 | |
| *** lucas-hungry is now known as lucasagomes | 12:34 | |
| *** noslzzp has quit IRC | 12:35 | |
| *** sdake has quit IRC | 12:35 | |
| *** rushiagr is now known as rushiagr_away | 12:35 | |
| *** morazi has joined #tripleo | 12:38 | |
| *** andreaf is now known as andreaf_ | 12:38 | |
| *** mrunge has quit IRC | 12:39 | |
| *** rdopieralski has quit IRC | 12:40 | |
| *** mrunge has joined #tripleo | 12:42 | |
| *** rdopieralski has joined #tripleo | 12:46 | |
| *** sballe has joined #tripleo | 12:47 | |
| *** ramishra has quit IRC | 12:51 | |
| *** mrunge has quit IRC | 12:52 | |
| *** yuanying has joined #tripleo | 12:55 | |
| *** tzumainn has joined #tripleo | 12:56 | |
| *** dprince has joined #tripleo | 12:58 | |
| *** bvivek has quit IRC | 13:08 | |
| *** mestery has quit IRC | 13:09 | |
| *** mestery has joined #tripleo | 13:09 | |
| *** jehb has joined #tripleo | 13:10 | |
| *** killer_prince has joined #tripleo | 13:11 | |
| *** killer_prince is now known as lazy_prince | 13:11 | |
| *** jdob has quit IRC | 13:11 | |
| *** jdob has joined #tripleo | 13:11 | |
| *** sballe has quit IRC | 13:12 | |
| *** julim has joined #tripleo | 13:16 | |
| *** morazi has quit IRC | 13:20 | |
| *** ddieterly has quit IRC | 13:24 | |
| *** yuanying has quit IRC | 13:29 | |
| *** morazi has joined #tripleo | 13:34 | |
| *** sballe_ has quit IRC | 13:37 | |
| *** sballe has joined #tripleo | 13:37 | |
| openstackgerrit | Ryan Brady proposed a change to openstack/tripleo-image-elements: WIP - RHEL7 Registration https://review.openstack.org/114009 | 13:41 |
| *** pblaho has quit IRC | 13:42 | |
| derekh | Hmm so check-tripleo-novabm-overcloud-precise-nonha keeps failing because its running out of memory | 13:54 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 13:58 |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Updated autodiscovery form to allow IPMI driver option https://review.openstack.org/121834 | 14:00 |
| *** ramishra has joined #tripleo | 14:02 | |
| *** gfidente has quit IRC | 14:04 | |
| *** ddieterly has joined #tripleo | 14:08 | |
| *** ddieterl_ has joined #tripleo | 14:10 | |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Fix for typo in overview form https://review.openstack.org/123139 | 14:12 |
| *** ddieterly has quit IRC | 14:13 | |
| *** ChuckC_ has quit IRC | 14:14 | |
| *** ddieterl_ has quit IRC | 14:14 | |
| *** andreaf has joined #tripleo | 14:15 | |
| *** gfidente has joined #tripleo | 14:16 | |
| *** ddieterly has joined #tripleo | 14:19 | |
| *** jcooley has quit IRC | 14:19 | |
| *** k4n0 has quit IRC | 14:19 | |
| *** sballe_ has joined #tripleo | 14:21 | |
| *** yamahata has joined #tripleo | 14:23 | |
| *** sballe_ has quit IRC | 14:23 | |
| *** jcooley has joined #tripleo | 14:24 | |
| *** sballe has quit IRC | 14:24 | |
| *** yuanying has joined #tripleo | 14:25 | |
| *** jistr has quit IRC | 14:30 | |
| *** andreaf has quit IRC | 14:31 | |
| *** jistr has joined #tripleo | 14:33 | |
| openstackgerrit | Ryan Brady proposed a change to openstack/diskimage-builder: WIP - Enable RHEL Registration https://review.openstack.org/123145 | 14:37 |
| *** Kiall has quit IRC | 14:39 | |
| *** Kiall has joined #tripleo | 14:43 | |
| *** sballe has joined #tripleo | 14:48 | |
| *** jprovazn has quit IRC | 14:49 | |
| *** parkerr_ has joined #tripleo | 14:52 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Enable automatic failover of l3 routers https://review.openstack.org/121134 | 14:54 |
| *** yuanying has quit IRC | 14:59 | |
| *** wfoster_ is now known as wfoster | 14:59 | |
| *** untriaged-bot has joined #tripleo | 15:01 | |
| untriaged-bot | Untriaged bugs so far: | 15:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 15:01 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 15:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 15:01 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 15:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 15:01 |
| *** untriaged-bot has quit IRC | 15:01 | |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 15:01 |
| *** sballe_ has joined #tripleo | 15:06 | |
| *** rdopieralski has quit IRC | 15:08 | |
| *** sballe has quit IRC | 15:09 | |
| openstackgerrit | Lorcan Browne proposed a change to openstack/tripleo-image-elements: Add pkg-map to swift-ring element dependency file https://review.openstack.org/123153 | 15:11 |
| *** jistr has quit IRC | 15:12 | |
| *** jistr has joined #tripleo | 15:13 | |
| *** sballe_ has quit IRC | 15:21 | |
| openstackgerrit | Jon-Paul Sullivan (jp_at_hp) proposed a change to openstack/tripleo-image-elements: Create os-cloud-config element https://review.openstack.org/123157 | 15:22 |
| openstackgerrit | Jon-Paul Sullivan (jp_at_hp) proposed a change to openstack/tripleo-image-elements: Enable git install of os-cloud-config https://review.openstack.org/123158 | 15:22 |
| *** pcrews has joined #tripleo | 15:23 | |
| *** sballe has joined #tripleo | 15:28 | |
| *** ChuckC_ has joined #tripleo | 15:31 | |
| *** morazi has quit IRC | 15:34 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: WIP : Reduce mysql innodb pool size https://review.openstack.org/123164 | 15:43 |
| *** morazi has joined #tripleo | 15:47 | |
| *** yuanying has joined #tripleo | 15:55 | |
| *** sballe has quit IRC | 16:02 | |
| *** sballe has joined #tripleo | 16:06 | |
| *** loki184 has joined #tripleo | 16:06 | |
| loki184 | please look at this patch https://review.openstack.org/#/c/103483/ | 16:09 |
| *** sballe has quit IRC | 16:12 | |
| *** ddieterly has quit IRC | 16:17 | |
| *** derekh has quit IRC | 16:19 | |
| *** pradeep has quit IRC | 16:19 | |
| *** rhallisey has quit IRC | 16:21 | |
| *** yuanying has quit IRC | 16:29 | |
| openstackgerrit | A change was merged to openstack/os-cloud-config: Allow use custom ssh_virt_type when register node https://review.openstack.org/120708 | 16:36 |
| *** loki184 has quit IRC | 16:36 | |
| *** saurabhs has joined #tripleo | 16:40 | |
| *** jistr has quit IRC | 16:43 | |
| *** akrivoka has quit IRC | 16:47 | |
| *** rhallisey has joined #tripleo | 16:50 | |
| *** loki184 has joined #tripleo | 16:53 | |
| *** athomas has quit IRC | 16:54 | |
| *** sdake has joined #tripleo | 16:56 | |
| *** sdake has quit IRC | 16:56 | |
| *** sdake has joined #tripleo | 16:56 | |
| *** bvivek has joined #tripleo | 17:03 | |
| *** ifarkas has quit IRC | 17:06 | |
| *** loki184 has quit IRC | 17:08 | |
| *** yamahata has quit IRC | 17:08 | |
| *** eghobo has joined #tripleo | 17:09 | |
| *** marun has joined #tripleo | 17:10 | |
| *** jp_at_hp has quit IRC | 17:15 | |
| *** bvivek has quit IRC | 17:16 | |
| *** ramishra has quit IRC | 17:20 | |
| *** eghobo has quit IRC | 17:25 | |
| *** yuanying has joined #tripleo | 17:25 | |
| *** penick has joined #tripleo | 17:30 | |
| *** morazi has quit IRC | 17:31 | |
| *** bvivek has joined #tripleo | 17:37 | |
| *** morazi has joined #tripleo | 17:46 | |
| *** gfidente has quit IRC | 17:52 | |
| *** jcoufal has quit IRC | 17:56 | |
| *** pradeep has joined #tripleo | 17:57 | |
| *** yuanying has quit IRC | 17:59 | |
| *** woozygeek has quit IRC | 18:00 | |
| *** pradeep has quit IRC | 18:01 | |
| *** penick has quit IRC | 18:13 | |
| *** marun has quit IRC | 18:18 | |
| *** jml has quit IRC | 18:20 | |
| *** jml has joined #tripleo | 18:22 | |
| *** sballe has joined #tripleo | 18:35 | |
| *** cwolferh has joined #tripleo | 18:41 | |
| *** bvivek has quit IRC | 18:44 | |
| *** cwolferh_ has quit IRC | 18:44 | |
| *** openstack has joined #tripleo | 18:56 | |
| *** rwsu has joined #tripleo | 19:10 | |
| *** jml has quit IRC | 19:11 | |
| *** penick has joined #tripleo | 19:15 | |
| *** jml has joined #tripleo | 19:15 | |
| *** sballe_ has joined #tripleo | 19:21 | |
| *** sballe has quit IRC | 19:24 | |
| *** rushiagr_away is now known as rushiagr | 19:28 | |
| *** yuanying has quit IRC | 19:29 | |
| *** dvossel has joined #tripleo | 19:30 | |
| *** dvossel has left #tripleo | 19:30 | |
| *** dvossel has joined #tripleo | 19:30 | |
| *** sballe_ has quit IRC | 19:32 | |
| *** jtomasek has quit IRC | 19:32 | |
| *** weshay has quit IRC | 19:38 | |
| *** jml has quit IRC | 19:48 | |
| *** jtomasek has joined #tripleo | 19:49 | |
| *** jml has joined #tripleo | 19:49 | |
| *** weshay has joined #tripleo | 19:51 | |
| *** marun has joined #tripleo | 19:51 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Remove Devananda from the cd-admins list https://review.openstack.org/120602 | 19:54 |
| *** yuanying has joined #tripleo | 19:55 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Add pkg-map to swift-ring element dependency file https://review.openstack.org/123153 | 19:56 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Add a pep8 testenv https://review.openstack.org/119671 | 19:57 |
| *** dprince has quit IRC | 20:02 | |
| *** dtantsur is now known as dtantsur|afk | 20:03 | |
| *** rushiagr is now known as rushiagr_away | 20:05 | |
| *** morazi has quit IRC | 20:06 | |
| *** morazi has joined #tripleo | 20:18 | |
| *** andreaf has joined #tripleo | 20:21 | |
| *** weshay has quit IRC | 20:25 | |
| *** jehb has quit IRC | 20:28 | |
| *** yuanying has quit IRC | 20:29 | |
| *** lazy_prince is now known as killer_prince | 20:42 | |
| *** parkerr_ has quit IRC | 20:43 | |
| *** rlandy has quit IRC | 20:44 | |
| *** spzala has joined #tripleo | 20:45 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Fix ansible element to build standalone https://review.openstack.org/118329 | 20:48 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Seed: use Nova compute Ironic driver https://review.openstack.org/119650 | 20:59 |
| *** noslzzp has joined #tripleo | 20:59 | |
| *** penick has quit IRC | 20:59 | |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | Untriaged bugs so far: | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 21:00 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 21:00 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 21:00 |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| *** jdob has quit IRC | 21:01 | |
| *** penick has joined #tripleo | 21:03 | |
| *** penick has quit IRC | 21:04 | |
| *** jtomasek has quit IRC | 21:19 | |
| *** ChuckC_ has quit IRC | 21:20 | |
| *** penick has joined #tripleo | 21:24 | |
| *** penick has quit IRC | 21:24 | |
| *** yuanying has joined #tripleo | 21:26 | |
| *** penick has joined #tripleo | 21:26 | |
| *** penick has quit IRC | 21:26 | |
| jogo | so nova-bm | 21:34 |
| jogo | devananda: ^ | 21:34 |
| jogo | devananda: AFAIK ironic has graduated right? | 21:34 |
| jogo | so can tripleo stop using nova-bm now? | 21:34 |
| *** penick has joined #tripleo | 21:36 | |
| devananda | jogo: IMO it should have stopped a long time ago | 21:38 |
| devananda | jogo: also yes, ironic graduated | 21:38 |
| jogo | devananda: re: https://review.openstack.org/#/c/121155/ | 21:38 |
| devananda | nova-bm is officially deprecated and will, afaik, be deleted from nova in a few weeks | 21:38 |
| devananda | ie, as soon as Kilo opens | 21:38 |
| jogo | devananda: so that patch shouldn't land | 21:39 |
| devananda | jogo: that makes me cry | 21:39 |
| devananda | especially that it has a +2 on it | 21:39 |
| lifeless | tripleo switched its defaults a while back | 21:40 |
| jogo | lifeless: that is what I sort of assumed, I think its time to drop nova-bm all together soon | 21:41 |
| lifeless | stopping folk from using a nova component thats in an openstack release isn't really in the openstack culture | 21:41 |
| lifeless | I think its fine to delete it from nova | 21:41 |
| lifeless | then tripleo can (if folk care) support running it with the last release | 21:41 |
| lifeless | and if noone cares we can stop testing it and or delete support for it | 21:42 |
| lifeless | but we're like tempest | 21:42 |
| devananda | if tripleo wants to maintain testing of it on stable/Juno , I agree - that's reasonable and a thing they should be able to do | 21:42 |
| devananda | also, if Nova doesn't want to spend cycles reviewing bug fixes for a driver they're about to delete, I think that's reasonable for them | 21:43 |
| devananda | those two statements are unfortunately in conflict with each other though | 21:43 |
| lifeless | I don't see the conflict | 21:44 |
| lifeless | delete the driver from Nova | 21:44 |
| lifeless | stable support teams should be getting the bug fix proposals from that point on | 21:44 |
| lifeless | devananda: btw have you seen the auth issue tchaypo has? | 21:44 |
| jogo | lifeless: so you think this change should land so it gets into stable/juno | 21:45 |
| lifeless | devananda: ironic isn't refreshing its neutron token and conductors stop working 2 hours after startup - e.g. exactly at the keystone token lifetime | 21:45 |
| lifeless | devananda: I have encouranged tchaypo to file a bug - I think this is release critical | 21:46 |
| tchaypo | morning | 21:46 |
| lifeless | jogo: I don't care if it lands in trunk or not; but there has to be a story for fixing issues in code that is deleted from trunk, in stable branches | 21:46 |
| jogo | lifeless: there is, just fix in stable | 21:46 |
| *** pensu has quit IRC | 21:47 | |
| devananda | lifeless: interesting. I was just looking at that code this morning and had a related concern | 21:51 |
| tchaypo | I tested it again last night and this time it was an hour after ironic was restarted | 21:51 |
| tchaypo | but both times it was a little over 30 minutes after heat started building the stack | 21:52 |
| devananda | tchaypo: mornin. can you pastebin me the [neutron] section of your ironic conf file? | 21:52 |
| tchaypo | 38 minutes the first time, 35 the second time (measured from first CREATE_IN_PROGRESS to first ResourceInerror) | 21:52 |
| tchaypo | devananda: no | 21:53 |
| tchaypo | devananda: it doesn't have one | 21:53 |
| devananda | ok :) | 21:53 |
| devananda | tchaypo: so ironic doesn't, afaict, cache the token | 21:53 |
| devananda | tchaypo: heat is running a single operation for the duration of this call? | 21:54 |
| tchaypo | sorry for the timing, but my instructor has just turned up and I need to go and play in traffic | 21:54 |
| devananda | tchaypo: is it possible that _heat_s token has outlived that expiration? | 21:54 |
| tchaypo | yes, that's possible too. If it aquires a token just before the first create_in_progress and passes tthat token down to nova which passes it to ironic... | 21:55 |
| tchaypo | the error we're seeing at the bottom of the pile is that ironic tells neutron to configure a port for a node; neutron checks the token and discovers that it's invalid | 21:56 |
| tchaypo | it's *always* at that point that the token is found to be invalid, and *only* at that point | 21:56 |
| devananda | from the pxe driver.... | 21:56 |
| devananda | 325 provider = dhcp_factory.DHCPFactory(token=task.context.auth_token) | 21:56 |
| devananda | 326 provider.update_dhcp(task, dhcp_opts) | 21:56 |
| tchaypo | ie, even after that failure nothing else reports any auth failures, just that one operation | 21:56 |
| devananda | ironic is using the context's auth_token and passing that to neutron | 21:56 |
| devananda | that context should be the one passed to Ironic from Nova | 21:56 |
| devananda | so I'm thinking that, if Heat is caching a single token and using it for all the calls in a long-running stack-create operation | 21:57 |
| devananda | the deploys will start to fail as soon as that token expires | 21:57 |
| tchaypo | which i believe it is, see threads on openstack-dev over the last week | 21:57 |
| devananda | yep | 21:57 |
| devananda | I've been watching that thread | 21:57 |
| devananda | also, ironic should be checking the token too -- if it's not, that's a bug | 21:58 |
| tchaypo | the standard production fix seems to be to just set the token expiry to something ludicrous | 21:58 |
| devananda | (it could be valid when ironic accepts it, then invalid when it gets to neutron? maybe? but that seems like a really small window by comparison) | 21:58 |
| tchaypo | yes, and I'd think that nova should check it when heat retries, and nova should be noticing it's invalid and refusing to retry | 21:58 |
| devananda | tchaypo: yup | 21:58 |
| tchaypo | .... although now that I think about it - after we see neutron refusing to accept it a few times, nova stops deleting instances when heat asks it to | 21:58 |
| devananda | heh | 21:59 |
| tchaypo | so maybe that's actually because it *is* rejecting the token | 21:59 |
| *** yuanying has quit IRC | 21:59 | |
| tchaypo | have to go before driving instructor starts getting grumpy. back in an hour. | 21:59 |
| devananda | o/ | 21:59 |
| *** noslzzp has quit IRC | 22:06 | |
| *** noslzzp has joined #tripleo | 22:08 | |
| *** ChuckC_ has joined #tripleo | 22:13 | |
| *** noslzzp has quit IRC | 22:21 | |
| *** jml has quit IRC | 22:43 | |
| *** marun has quit IRC | 22:46 | |
| *** jml has joined #tripleo | 22:46 | |
| *** marun has joined #tripleo | 22:48 | |
| *** yuanying has joined #tripleo | 22:55 | |
| *** penick has quit IRC | 23:02 | |
| *** sballe has joined #tripleo | 23:15 | |
| *** sballe has quit IRC | 23:15 | |
| tchaypo | back | 23:18 |
| tchaypo | so like, even if that issue is heat, ironic isn't *completely* off the hook | 23:18 |
| *** penick has joined #tripleo | 23:18 | |
| tchaypo | because it still means that something is failing on all but 5 nodes of my ~23 for until the token expires | 23:19 |
| tchaypo | in other news - looking at keystone.conf | 23:19 |
| tchaypo | under [token] | 23:20 |
| tchaypo | we have | 23:20 |
| tchaypo | expiration = 14400 | 23:20 |
| tchaypo | so it's odd that the token is expiring 30 minutes into the stack-create | 23:20 |
| tchaypo | 90 minutes after heat-api was restarted | 23:20 |
| *** dvossel has quit IRC | 23:26 | |
| *** yuanying has quit IRC | 23:29 | |
| *** lucasagomes has quit IRC | 23:31 | |
| *** yamahata has joined #tripleo | 23:34 | |
| *** morazi has quit IRC | 23:37 | |
| openstackgerrit | A change was merged to openstack/tripleo-specs: TripleO network configuration https://review.openstack.org/97859 | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!