| *** e0ne has quit IRC | 00:00 | |
| *** e0ne has joined #tripleo | 00:00 | |
| *** e0ne has quit IRC | 00:05 | |
| *** morazi has quit IRC | 00:18 | |
| *** edmund has quit IRC | 00:20 | |
| *** bauzas has quit IRC | 00:39 | |
| *** UtahDave has quit IRC | 01:04 | |
| StevenK | lifeless: http://paste.openstack.org/show/61604/ | 01:10 |
|---|---|---|
| cody-somerville | mordred: fyi, I have your laptop's charger. | 01:12 |
| mikal | cody-somerville: its ok, mordred doesn't have the laptop any more | 01:12 |
| cody-somerville | It's for his HP laptop | 01:12 |
| cody-somerville | the one I think he's using now ;) | 01:12 |
| cody-somerville | I'll give it back to you in Brussels (along with a short blue Ethernet cable that belongs in Sunnyvale). | 01:13 |
| lifeless | StevenK: roughly yes, but copy to the chosen output please, not /tmp/predictablepath | 01:14 |
| lifeless | e.g. if I pass -o foo give me foo.qcow2, foo.kernel, foo.ramdisk | 01:15 |
| StevenK | lifeless: Pass -o to a cleanup.d how? | 01:15 |
| StevenK | lifeless: And I'm guessing you don't want it under $TARGET_ROOT either | 01:15 |
| lifeless | its coming *from* target_root :) | 01:17 |
| lifeless | you have the variable in the disk-image-create script already, and it's exported, I think inherited by | 01:17 |
| StevenK | Right, $IMAGE_NAME | 01:18 |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Switch test environment users https://review.openstack.org/67614 | 01:20 |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Add unstyled overcloud resource category page https://review.openstack.org/67632 | 01:29 |
| *** greghaynes has quit IRC | 01:40 | |
| *** xuhanp_ has joined #tripleo | 01:41 | |
| *** weshay has joined #tripleo | 01:43 | |
| *** coolsvap has quit IRC | 01:51 | |
| *** e0ne has joined #tripleo | 01:57 | |
| *** nosnos has joined #tripleo | 01:59 | |
| *** jrist has quit IRC | 02:01 | |
| *** e0ne has quit IRC | 02:02 | |
| lifeless | hmm | 02:16 |
| lifeless | /dev/nbd15p1 1.4G 1.1G 273M 80% /tmp/openstack-vfs-localfsAvkjEz | 02:16 |
| lifeless | methinks we haven't turned file injection off properly :) | 02:16 |
| mordred | lifeless: config-drive? | 02:19 |
| lifeless | shouldn't need nbd for config drive | 02:20 |
| mordred | nod | 02:20 |
| lifeless | and it contains the full fs when I list it | 02:20 |
| lifeless | and look | 02:25 |
| lifeless | suddently things boot about 5 billion times faster | 02:25 |
| *** vkozhukalov has joined #tripleo | 02:34 | |
| *** sdake has quit IRC | 02:49 | |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Add unstyled overcloud resource category page https://review.openstack.org/67632 | 02:52 |
| *** e0ne has joined #tripleo | 02:57 | |
| *** e0ne has quit IRC | 03:02 | |
| *** weshay has quit IRC | 03:05 | |
| *** ramishra has joined #tripleo | 03:06 | |
| *** ramishra_ has joined #tripleo | 03:08 | |
| *** ramishra has quit IRC | 03:11 | |
| *** greghaynes has joined #tripleo | 03:40 | |
| *** e0ne has joined #tripleo | 03:57 | |
| *** marun has joined #tripleo | 04:00 | |
| *** e0ne has quit IRC | 04:02 | |
| *** marun has quit IRC | 04:12 | |
| *** marun has joined #tripleo | 04:13 | |
| *** marun has quit IRC | 04:14 | |
| StevenK | lifeless: http://paste.openstack.org/show/61605/ | 04:14 |
| SpamapS | hm | 04:15 |
| SpamapS | 2014-01-21 04:12:51.987 2497 TRACE heat.openstack.common.rpc.common error: [Errno 104] Connection reset by peer | 04:16 |
| SpamapS | seeing errors for rabbitmq RPC | 04:16 |
| *** marun has joined #tripleo | 04:16 | |
| SpamapS | n/m | 04:17 |
| SpamapS | that is just early boot crapola | 04:17 |
| SpamapS | hm cfn reqs to seed are failing auth | 04:24 |
| SpamapS | 2014-01-21 04:24:24.149 12222 WARNING keystone.openstack.common.db.sqlalchemy.session [-] This application has not enabled MySQL traditional mode, which means silent data corruption may occur. Please encourage the application developers to enable this mode. | 04:24 |
| SpamapS | that is a LOVELY warning | 04:24 |
| StevenK | ... | 04:30 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/diskimage-builder: Create a new baremetal element https://review.openstack.org/68019 | 04:31 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/diskimage-builder: Create a new baremetal element https://review.openstack.org/68019 | 04:33 |
| *** coolsvap has joined #tripleo | 04:38 | |
| *** ramishra_ has quit IRC | 04:40 | |
| *** ramishra has joined #tripleo | 04:55 | |
| *** e0ne has joined #tripleo | 04:57 | |
| *** e0ne has quit IRC | 05:01 | |
| *** marun has quit IRC | 05:08 | |
| StevenK | lifeless: Thanks for the review. In terms of -get-kernel, you think a comment, or a echo with "DEPRECATED: Please consider using the `baremetal` element." ? | 05:33 |
| lifeless | StevenK: deprecate that shit :) | 05:34 |
| *** rpodolyaka1 has joined #tripleo | 05:35 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/diskimage-builder: Create a new baremetal element https://review.openstack.org/68019 | 05:42 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/diskimage-builder: Create a new baremetal element https://review.openstack.org/68019 | 05:48 |
| *** e0ne has joined #tripleo | 05:57 | |
| *** akuznetsov has quit IRC | 05:59 | |
| *** e0ne has quit IRC | 06:02 | |
| *** tzumainn has quit IRC | 06:07 | |
| *** kpavel has joined #tripleo | 06:22 | |
| *** akuznetsov has joined #tripleo | 06:26 | |
| *** akuznetsov has quit IRC | 06:37 | |
| *** akuznetsov has joined #tripleo | 06:37 | |
| *** ramishra has quit IRC | 06:38 | |
| *** mrunge has joined #tripleo | 06:43 | |
| *** mrunge has quit IRC | 06:44 | |
| *** mrunge has joined #tripleo | 06:44 | |
| *** vkozhukalov has quit IRC | 06:47 | |
| *** e0ne has joined #tripleo | 06:57 | |
| *** e0ne has quit IRC | 07:02 | |
| *** e0ne has joined #tripleo | 07:04 | |
| *** akuznets_ has joined #tripleo | 07:05 | |
| *** mrunge has quit IRC | 07:07 | |
| *** mrunge has joined #tripleo | 07:08 | |
| *** akuznetsov has quit IRC | 07:08 | |
| *** rdopieralski has joined #tripleo | 07:30 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/tuskar-ui: Deployment design form https://review.openstack.org/67155 | 07:32 |
| *** rlandy has joined #tripleo | 07:37 | |
| *** ifarkas has joined #tripleo | 07:41 | |
| *** rpodolyaka1 has quit IRC | 07:43 | |
| lifeless | back for evening | 07:45 |
| *** jprovazn has joined #tripleo | 07:52 | |
| *** jtomasek has joined #tripleo | 07:56 | |
| *** sdake has joined #tripleo | 07:58 | |
| *** pblaho has joined #tripleo | 08:03 | |
| *** bauzas has joined #tripleo | 08:08 | |
| *** e0ne has quit IRC | 08:09 | |
| GheRivero | morning TripleO! | 08:10 |
| marios | o/ | 08:11 |
| *** jcoufal has joined #tripleo | 08:13 | |
| *** jistr has joined #tripleo | 08:14 | |
| rlandy | ifarkas: hello - building the undercloud with tuskar went well but I deploying the undercloud failed with a WaitConditionTimeout. Should I have added anything additional to the "heat stack-create" command? | 08:14 |
| rlandy | side note: devtest was undergoing some changes last night (possible that the error is unrelated to tuskar addition) | 08:15 |
| rlandy | rebuilding from scratch this morning | 08:15 |
| ifarkas | rlandy, not that I know of. | 08:15 |
| ifarkas | rlandy, can you find anything useful in the logs? | 08:16 |
| rlandy | other than the timeout error - looked like authentication problem | 08:17 |
| rlandy | which is why I was wondering if I needed to pass another password parameter for the extra element | 08:18 |
| rpodolyaka | morning all | 08:22 |
| *** vkozhukalov has joined #tripleo | 08:28 | |
| ifarkas | morning rpodolyaka! | 08:36 |
| jprovazn | rlandy: can you ssh to undercloud? | 08:38 |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/diskimage-builder: Setup ccache symlinks on openSUSE https://review.openstack.org/67965 | 08:43 |
| rlandy | jprovazn: not anymore - I killed it to start again :) | 08:45 |
| rlandy | to answer your question - I could ping the undercloud | 08:45 |
| rlandy | issuing the keystone commands - failed with could not connect | 08:46 |
| jprovazn | rlandy: I think wait condition is not matched until undercloud boots and openstack services are running on it, so it might be 1) your VM failed to boot or 2) after boot setup (the whole os-collect-config process) failed. Option 2) is more probable | 08:47 |
| rlandy | jprovazn: makes sense (nova showed the vm as running - heat stack-list showed it as CREATE_FAILED). I am doing the whole devtest process with tuskar in the undercloud again. If the failure is repeatable, we can inspect the machine/logs | 08:51 |
| jprovazn | rlandy: OK, you can use '-c' to use cached images to make it much more quicker | 08:52 |
| *** boris-42 has quit IRC | 09:00 | |
| *** derekh has joined #tripleo | 09:00 | |
| *** boris-42 has joined #tripleo | 09:00 | |
| derekh | morning | 09:02 |
| rpodolyaka | o/ | 09:02 |
| Ng | morning | 09:09 |
| openstackgerrit | lifeless proposed a change to openstack-infra/tripleo-ci: Set TRIPLEO_ROOT https://review.openstack.org/68045 | 09:17 |
| lifeless | derekh: ^ is what you need I think | 09:18 |
| derekh | lifeless: yup, probably and it would explain why I did this https://review.openstack.org/#/c/67939/3/toci_devtest.sh | 09:20 |
| *** markmc has joined #tripleo | 09:20 | |
| derekh | lifeless: will abandon mine | 09:20 |
| ifarkas | I tried to deploy the undercloud but it failed for me. It seems that os-config-refresh couldn't fetch heat metadata from seed. Here's the log: http://pastebin.com/wthMqvns | 09:21 |
| ifarkas | Anyone has any idea what could be the problem? | 09:21 |
| derekh | ifarkas: I got no idea, gonna run it locally to see if I can reproduce | 09:23 |
| lifeless | derekh: I've reviewed all your toci patches | 09:23 |
| lifeless | derekh: if you can fix the stack up (basically delete two patches and tweak the heat template) we can land it all | 09:24 |
| ifarkas | derekh, thanks, let me know if it works for you | 09:24 |
| lifeless | ifarkas: whats that log from ? | 09:24 |
| derekh | lifeless: ok, on it now | 09:24 |
| lifeless | ifarkas: it makes little sense to me :) | 09:24 |
| ifarkas | lifeless, sorry, the undercloud log is the output of os-collect-config -d and seed log is from journalctl | 09:25 |
| rpodolyaka | ifarkas: I think, I'm seeing this too | 09:26 |
| ifarkas | rpodolyaka, cool, at least I am not the only one :-) | 09:26 |
| rpodolyaka | though I have no idea why this is happening :( | 09:27 |
| lifeless | oh, ok | 09:27 |
| lifeless | so that thats a heat service auth failure | 09:27 |
| lifeless | with presigned request | 09:28 |
| lifeless | ifarkas: check the heat-api log | 09:28 |
| *** viktors has joined #tripleo | 09:28 | |
| rpodolyaka | yeah | 09:28 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Fixup the handling of ssh keys https://review.openstack.org/67938 | 09:30 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Fix the server name https://review.openstack.org/67599 | 09:30 |
| ifarkas | lifeless, alright, I am on it | 09:30 |
| derekh | lifeless: done ^, heat template tweaked, the tweak is different to the one you suggested but I think its right | 09:32 |
| ifarkas | lifeless, there's nothing in the log on the seed vm from heat if I run os-collect-config on the undercloud machine | 09:33 |
| rpodolyaka | ifarkas: http://paste.openstack.org/show/61611/ , sounds familiar? | 09:33 |
| derekh | lifeless: at least its what I used to create the testenv stack last night | 09:33 |
| lifeless | Ng: https://review.openstack.org/#/c/67599/ please | 09:37 |
| ifarkas | rpodolyaka, yeah, I have the same | 09:37 |
| *** athomas has joined #tripleo | 09:37 | |
| Ng | lifeless: sure | 09:37 |
| openstackgerrit | Dirk Mueller proposed a change to openstack/diskimage-builder: Add a service mapping for openSUSE https://review.openstack.org/68051 | 09:37 |
| ifarkas | lifeless, I've just set heat to debug and I have the same error in the log as rpodolyaka | 09:37 |
| lifeless | ifarkas: rpodolyaka: I thinkDate: Tue Jan 21 08:36:17 2014 +0000 | 09:40 |
| lifeless | Merge "Use oslo crypto" | 09:40 |
| lifeless | looks suspect to me | 09:40 |
| lifeless | IMBW | 09:41 |
| lifeless | I think try running without that patch - e.g. use 834eb42e2e1aa0048a10708b5a598204da98ec1b as your commitish for heat | 09:41 |
| rpodolyaka | lifeless: thanks! will try | 09:42 |
| lifeless | failing that, keystone becomes the culprit. looking in it's log for clues | 09:42 |
| ifarkas | lifeless, yeah, thanks, I will test it too | 09:42 |
| lifeless | Merge "Replacing python-oauth2 by oauthlib" | 09:44 |
| lifeless | might do it | 09:44 |
| rpodolyaka | yeah, so actually keystone error is http://paste.openstack.org/show/61612/ | 09:44 |
| rpodolyaka | and the only place where this exception is raised in keystone is here https://github.com/openstack/keystone/blob/master/keystone/contrib/ec2/controllers.py#L119 | 09:47 |
| lifeless | hah | 09:47 |
| lifeless | which is the oauthlib patch I just mentioned | 09:47 |
| rpodolyaka | hehe | 09:48 |
| lifeless | so yeah, I'd back that patch out, restart keystone, heat stack-delete heat stack-create undercloud.... | 09:48 |
| rpodolyaka | lifeless: thanks for pointing this out! | 09:48 |
| lifeless | rpodolyaka: I'm just free associating with git logs :) | 09:49 |
| rpodolyaka | sure, but I'm struggling to understrand how this is related to oauth :) | 09:49 |
| * rpodolyaka was sure ec2/amazon used custom auth | 09:49 | |
| lifeless | it looks like heat grabs a token then uses it to sign a request and hands that request (but not the token) into the metadata | 09:50 |
| lifeless | Ng: https://review.openstack.org/#/c/67868/ and all its dependents | 09:51 |
| Ng | lifeless: also sure. the previous stack is all Approved, whenever Jenkins decides to stop drinking beer and do some bending | 09:51 |
| StevenK | It's Zuul, not Jenkins? :-) | 09:52 |
| openstackgerrit | Jiri Tomasek proposed a change to openstack/tuskar-ui: Number picker input widget and angular directive https://review.openstack.org/68053 | 09:53 |
| *** e0ne has joined #tripleo | 09:53 | |
| Ng | StevenK: all these robots look the same to me ;) | 09:54 |
| * StevenK confuses Ng by adding gearman into the mix. | 09:54 | |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: SSL support for openstack services. https://review.openstack.org/67343 | 09:55 |
| StevenK | I see Adobe are taking leaves from Amazon's book. "Thank you and enjoy Flash Player. Here is another product that might interest you." | 09:56 |
| *** kwwii_ has joined #tripleo | 09:59 | |
| *** kwwii_ has left #tripleo | 10:00 | |
| *** kpavel has quit IRC | 10:00 | |
| *** martyntaylor has joined #tripleo | 10:07 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Create a new baremetal element https://review.openstack.org/68019 | 10:07 |
| openstackgerrit | Jiri Tomasek proposed a change to openstack/tuskar-ui: Number picker input widget and angular directive https://review.openstack.org/68053 | 10:08 |
| *** coolsvap has quit IRC | 10:09 | |
| *** coolsvap has joined #tripleo | 10:12 | |
| openstackgerrit | Jiri Tomasek proposed a change to openstack/tuskar-ui: Number picker input widget and angular directive https://review.openstack.org/68053 | 10:13 |
| Ng | lifeless: ok, both stacks approved up and down | 10:21 |
| lifeless | \o/ | 10:22 |
| lifeless | Ng: might want to have a look for ancilliary fixes | 10:22 |
| *** coolsvap has quit IRC | 10:23 | |
| *** coolsvap has joined #tripleo | 10:27 | |
| *** xuhanp_ has quit IRC | 10:29 | |
| *** martyntaylor has quit IRC | 10:35 | |
| *** akrivoka has joined #tripleo | 10:37 | |
| lifeless | gnight | 10:37 |
| *** boris-42 has quit IRC | 10:38 | |
| *** ramishra has joined #tripleo | 10:49 | |
| *** martyntaylor has joined #tripleo | 10:51 | |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Add more detail to the json file https://review.openstack.org/67686 | 10:56 |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Fixup the handling of ssh keys https://review.openstack.org/67938 | 10:56 |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Fix the server name https://review.openstack.org/67599 | 10:56 |
| *** max_lobur_afk is now known as max_lobur | 11:05 | |
| openstackgerrit | Ronelle Landy proposed a change to openstack/tripleo-image-elements: Moving iptables rules to the relevant elements https://review.openstack.org/68075 | 11:08 |
| rlandy | derekh: ^^ https://review.openstack.org/68075 (marked workinprogress) | 11:09 |
| derekh | rlandy: cool, I'll try it out later today | 11:11 |
| *** akrivoka has quit IRC | 11:14 | |
| *** e0ne has quit IRC | 11:15 | |
| *** e0ne has joined #tripleo | 11:15 | |
| rlandy | rpodolyaka: ifarkas: are you able to deploy the undercloud now? | 11:17 |
| lsmola_ | rpodolyaka: hello | 11:17 |
| rpodolyaka | rlandy: unfortunately, no | 11:17 |
| *** akrivoka has joined #tripleo | 11:17 | |
| lsmola_ | rdopieralski: i have the same problem | 11:17 |
| lsmola_ | rlandy: nope | 11:17 |
| rpodolyaka | rlandy: I tried to downgrade heat/keystone on seed vm but with no sucess | 11:17 |
| *** jcoufal has quit IRC | 11:17 | |
| lsmola_ | rlandy: i have reverted keystone and heat, restart the services and it is not working | 11:18 |
| lsmola_ | rlandy: any other ideas? | 11:18 |
| rpodolyaka | lsmola_: no, looking through the keystone code trying to understand what is going on | 11:19 |
| rpodolyaka | lsmola_: and how it is supposed to work | 11:19 |
| rlandy | lsmola: rpodolyaka: can't say I have another bright idea right now ... I tripped over this last night doing a bug verification - so I wanted to check the standard deploy worked before I threw in the bug-related changes | 11:19 |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Add unstyled overcloud resource category page https://review.openstack.org/67632 | 11:19 |
| lsmola_ | rdopieralski: ok | 11:20 |
| * rdopieralski grins | 11:20 | |
| lsmola_ | rdopieralski: damn | 11:20 |
| lsmola_ | :-D | 11:20 |
| lsmola_ | rpodolyaka: I keep exchanging you with rdopieralski :-) | 11:21 |
| rlandy | did anyone manage to deploy the undercloud yesterday? | 11:21 |
| lsmola_ | rpodolyaka: ok, I trying to look what has changed on tripleo side | 11:21 |
| rlandy | this was my first attempt at devtest since last week - when it worked | 11:21 |
| lsmola_ | rpodolyaka: I swear it was working for me on friday | 11:22 |
| rlandy | ok - we got a check on Friday | 11:22 |
| *** lucasagomes has joined #tripleo | 11:22 | |
| lsmola_ | rlandy: I think i was able to deploy it the whole last week, did it several times | 11:23 |
| rlandy | yeah - so do I | 11:23 |
| rpodolyaka | yeah, worked for me last wee to | 11:23 |
| rlandy | did | 11:23 |
| rlandy | ok - so what happened on Friday night -> Monday? | 11:23 |
| jprovazn | sounds like a pretty good hangover :) | 11:29 |
| lsmola_ | lol | 11:29 |
| rpodolyaka | :) | 11:30 |
| rlandy | :) | 11:30 |
| lsmola_ | rlandy: can I test this with curl like this ? http://paste.openstack.org/show/61615/ | 11:34 |
| lsmola_ | rlandy: or I am missing something? | 11:34 |
| rlandy | sec - on call ... brb | 11:34 |
| rlandy | lsmola_: just fyi ... I'm a newbie on the team and figuring this out as I go ... you may want to consult with others as well ;) | 11:35 |
| lsmola_ | rlandy: :-) that sounds familiar to me | 11:36 |
| *** martyntaylor has quit IRC | 11:42 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Upgrade setuptools to >=1 in a newly created virtualenv https://review.openstack.org/67688 | 11:42 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Read env-num from TE_DATAFILE https://review.openstack.org/67689 | 11:42 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Replace rsync with the copyseed command https://review.openstack.org/67867 | 11:42 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Replace usage of qemu+ssh with ssh <command> https://review.openstack.org/67868 | 11:42 |
| *** jcoufal has joined #tripleo | 11:42 | |
| derekh | lsmola_: rlandy so running tcpdump on the seed "tcpdump -i lo -A -nn port 5000" while running that curl command you get | 11:44 |
| derekh | http://paste.fedoraproject.org/70276/30456813/ | 11:44 |
| derekh | looks like the request is missing the auth token | 11:44 |
| *** rlandy is now known as rlandy|brb | 11:45 | |
| derekh | surley there should be a X-Auth-Token heater in that request | 11:45 |
| lsmola_ | derekh: hm, ok | 11:46 |
| lsmola_ | derekh: so it is missing also in call that the os-colelct-config is doing | 11:46 |
| lsmola_ | derekh: or it is being added there | 11:46 |
| lsmola_ | derekh: ? | 11:47 |
| lsmola_ | derekh: the log I posted is from os-collect-config -d | 11:47 |
| derekh | lsmola_: I *think* the call from os-collect-config to heat is ok (could be wrong), then heat tries to talk to keystone and isn't sending its auth token | 11:48 |
| rpodolyaka | seems so | 11:48 |
| derekh | lsmola_: not sure though, will have to dig a bit more, I'll let you knwo if I find anything out | 11:48 |
| *** martyntaylor has joined #tripleo | 11:48 | |
| lsmola_ | derekh: hm, I am getting os_collect_config.cfn [-] 403 Client Error: AccessDenied , when running os-collect-config -d in my undercloud machine | 11:52 |
| lsmola_ | derekh: i will try to tcpdump the call | 11:52 |
| derekh | lsmola_: yup, getting that here also | 11:53 |
| lsmola_ | derekh: do you see AUTH token in the header? | 11:53 |
| derekh | lsmola_: nope, but don't think you should see it in that request | 11:57 |
| lsmola_ | derekh: ok | 12:00 |
| derekh | unless of course I'm wrong :-) | 12:00 |
| *** boris-42 has joined #tripleo | 12:05 | |
| lsmola_ | derekh: ok, so I have the same as you using os-collect-config -d http://paste.openstack.org/show/61617/ | 12:05 |
| lsmola_ | derekh: so the question is, what patch could broke the token passing | 12:06 |
| *** morazi has joined #tripleo | 12:06 | |
| lsmola_ | derekh: cause it was working on friday | 12:06 |
| derekh | lsmola_: dunno, ok time to start sticking print statements in code | 12:10 |
| derekh | lsmola_: dunno, time to start stick print statments in code | 12:10 |
| lsmola_ | derekh: ok | 12:12 |
| derekh | lsmola_: ok, looks like I'm going down the wrong path, here is the code that makes that request from heat to keystone | 12:20 |
| derekh | https://github.com/openstack/heat/blob/master/heat/api/aws/ec2token.py#L188-L193 | 12:20 |
| derekh | lookes like no attempt is made to add an auth token to the request and the file hasn't changed in 2 months | 12:21 |
| lsmola_ | derekh: interesting | 12:21 |
| lsmola_ | derekh: btw. it might be something on seed keystone side | 12:21 |
| derekh | lsmola_: yup, could be | 12:21 |
| lsmola_ | derekh: cause it works for jprovazn with ovecloud <-> undercloud communication | 12:22 |
| *** rlandy|brb is now known as rlandy | 12:22 | |
| lsmola_ | derekh: and he uses old undercloud | 12:22 |
| rlandy | lsmola_: sorry back ... catching up ^^ | 12:22 |
| *** coolsvap has quit IRC | 12:22 | |
| lsmola_ | derekh: so maybe Keystone ignored this till recently? | 12:23 |
| rpodolyaka | hmm, is it ok that request to keystone is one on behalf of undercloud-User-ihb73467qruc? | 12:23 |
| rpodolyaka | *is done | 12:23 |
| derekh | rpodolyaka: so your saying that request to keystone looks ok? | 12:25 |
| derekh | so we gotta find out why keystone isn't allowing it | 12:25 |
| rpodolyaka | derekh: don't know, but it passes validation up to this point https://github.com/openstack/keystone/blob/master/keystone/contrib/ec2/controllers.py#L119 | 12:25 |
| rpodolyaka | derekh: which makes me believe we have a problem with user/roles | 12:26 |
| *** tserong has quit IRC | 12:27 | |
| lsmola_ | rpodolyaka: btw. I dont see this exception in my seed journal | 12:27 |
| rpodolyaka | lsmola_: hmm, this is what I see in keystone log on seed vm - http://paste.openstack.org/show/61618/ | 12:28 |
| rpodolyaka | lsmola_: and grep "User not valid for tenant" led me to https://github.com/openstack/keystone/blob/master/keystone/contrib/ec2/controllers.py#L119 | 12:28 |
| lsmola_ | rdopieralski: where do I find this log? | 12:29 |
| *** tserong has joined #tripleo | 12:31 | |
| rpodolyaka | lsmola_: /var/log/upstart/keystone.log on ubuntu :) | 12:31 |
| lsmola_ | rpodolyaka: hmm I don't see that one anywhere | 12:34 |
| lsmola_ | rdopieralski: but I have reverted my keystone a way back to 05808a95e3058ea66dc1af1c09b0fdab3744215c | 12:36 |
| lsmola_ | rpodolyaka: eh | 12:36 |
| lsmola_ | rdopieralski: sorry :-D | 12:36 |
| rpodolyaka | lsmola_: do you have debug=True in keystone? | 12:36 |
| rpodolyaka | *conf | 12:36 |
| lsmola_ | rpodolyaka: nope | 12:36 |
| lsmola_ | rpodolyaka: but that warning should be visible anyway right? | 12:36 |
| rpodolyaka | lsmola_: I think so | 12:37 |
| derekh | I think on fedora we are leaking keystone logs into nothingness | 12:40 |
| derekh | I print statment confirms to me that this exception is being raised | 12:40 |
| derekh | only if I start keystone on command line | 12:41 |
| derekh | sudo su -s /bin/bash -c '/opt/stack/venvs/keystone/bin/python /opt/stack/venvs/keystone/bin/keystone-all --config-dir /etc/keystone' | 12:41 |
| lsmola_ | derekh: ok | 12:44 |
| lsmola_ | derekh: should I ask why "we are leaking keystone logs into nothingness" :-) | 12:45 |
| rlandy | just recently? | 12:45 |
| derekh | lsmola_ rlandy don't know to both questions | 12:46 |
| lsmola_ | :-) | 12:46 |
| *** ramishra has quit IRC | 12:46 | |
| derekh | ok, got os-collect-config to run again with this | 12:47 |
| derekh | http://paste.openstack.org/show/61619/ | 12:47 |
| derekh | the two ID's are my admin and Member roles | 12:48 |
| derekh | so we gotta find out why roles is empty | 12:48 |
| rpodolyaka | oh, nice :) | 12:49 |
| rpodolyaka | hmm, is this user created by heat? | 12:50 |
| derekh | rpodolyaka: user is | c621ee19476046db85d8304885572d0e | undercloud-User-ggcjcybrzoai | True | | | 12:50 |
| derekh | so running this on the seed solved the problem | 12:55 |
| derekh | keystone user-role-add --user undercloud-User-ggcjcybrzoai --tenant admin --role admin | 12:55 |
| derekh | keystone user-role-add --user undercloud-User-ggcjcybrzoai --tenant admin --role Member | 12:55 |
| derekh | not sure if both commands are needed | 12:55 |
| rpodolyaka | just did the same thing - works for me too | 12:56 |
| rpodolyaka | but the point is, isn't it heat who must do this? | 12:56 |
| derekh | rpodolyaka: yup, was just testing things out | 12:58 |
| derekh | so this did change recently | 12:58 |
| derekh | https://review.openstack.org/#/c/66646/2 | 12:58 |
| derekh | going to ask on the heat channel | 12:59 |
| openstackgerrit | A change was merged to openstack/tripleo-heat-templates: Add license file https://review.openstack.org/67806 | 13:00 |
| lsmola_ | derekh: great, that sounds promising | 13:01 |
| rlandy | lsmola_: you rolled back keystone but not heat, right? | 13:01 |
| lsmola_ | rlandy: I rolled heat, though not that far I assume | 13:02 |
| lsmola_ | rlandy: checking | 13:02 |
| rpodolyaka | I rolled back heat, but didn't actually tried to do stack-delete/stack-create | 13:02 |
| rlandy | Jan 19 doesn't seem that far away - but I guess there are a lot of commits per day | 13:03 |
| derekh | when you rolled back did you start from scratch again ? | 13:03 |
| lsmola_ | rlandy: hmm, so I don't have it in my seed | 13:04 |
| lsmola_ | rlandy: is it the right place? | 13:05 |
| rpodolyaka | derekh: nope, so my 'roll back' is not really demonstrative in case the commit you pointed is to blame | 13:05 |
| lsmola_ | rdopieralski: eh, I didn't rolled back the client | 13:06 |
| *** pblaho has quit IRC | 13:07 | |
| lsmola_ | rdopieralski: eh, again :-) | 13:07 |
| lsmola_ | rpodolyaka: rlandy I didn't rolled back the client :-) trying | 13:07 |
| rpodolyaka | lsmola_: me too | 13:08 |
| lsmola_ | rpodolyaka: wait, the code is under heat | 13:09 |
| *** Shrews_ has joined #tripleo | 13:10 | |
| rpodolyaka | lsmola_: yep :) | 13:10 |
| lsmola_ | rdopieralski: so unless there is some magic, I did rolled it back, I dont see it in my git log | 13:11 |
| lsmola_ | rdopieralski: come on, I have to cut my fingers of, and replace them with new ones | 13:11 |
| *** Shrews has quit IRC | 13:12 | |
| *** Shrews_ is now known as Shrews | 13:12 | |
| lsmola_ | rpodolyaka: ^^ keep confusing you too :-D heeelp | 13:12 |
| rpodolyaka | lsmola_: yeah, I just meant, I didn't do stack-delete/stack-create, so the use wasn't really recreated and this wasn't helpful in case of missing user roles | 13:12 |
| rpodolyaka | *user | 13:13 |
| *** Shrews_ has joined #tripleo | 13:13 | |
| lsmola_ | rpodolyaka: hm I did | 13:13 |
| rpodolyaka | щр | 13:13 |
| rpodolyaka | oh | 13:13 |
| *** Shrews_ has quit IRC | 13:14 | |
| rlandy | are you rolling back on the seed vm? | 13:14 |
| *** Shrews_ has joined #tripleo | 13:15 | |
| *** Shrews has quit IRC | 13:16 | |
| *** Shrews_ has quit IRC | 13:16 | |
| lsmola_ | rlandy: yes | 13:16 |
| lsmola_ | rlandy: then restarting 4 heat services and 1 keystone | 13:16 |
| *** Shrews has joined #tripleo | 13:19 | |
| lsmola_ | derekh: so we should see the 2 roles with 'keystone user-role-list' on seed? | 13:25 |
| derekh | lsmola_: I got 6 roles | 13:26 |
| lsmola_ | derekh: i have only one now | 13:26 |
| derekh | lsmola_: sorry, missunderstood | 13:26 |
| derekh | lsmola_: you should only see the heat_stack_user role for that user | 13:27 |
| *** e0ne_ has joined #tripleo | 13:27 | |
| *** e0ne has quit IRC | 13:28 | |
| derekh | lsmola_: just to confirm the code you roled back was in /opt/stack/venvs/heat | 13:29 |
| *** nosnos has quit IRC | 13:30 | |
| *** jdob has joined #tripleo | 13:31 | |
| *** nosnos has joined #tripleo | 13:31 | |
| lsmola_ | derekh: ha | 13:31 |
| lsmola_ | derekh: no, I rolled back the /opt/stack/heat | 13:32 |
| lsmola_ | derekh: that could be the problem :-) | 13:32 |
| derekh | lsmola_: ok, that wouldn't do anything, that directory is just there to confuse people | 13:32 |
| rpodolyaka | I just installed it to venv using pip install -e :) | 13:32 |
| lsmola_ | derekh: lol | 13:33 |
| rpodolyaka | but yes, we should probably remove those dirs | 13:33 |
| derekh | rpodolyaka: yup, | 13:33 |
| lsmola_ | derekh: so how to I roll it back in there? | 13:33 |
| derekh | rpodolyaka: although I do use them a lot | 13:34 |
| rpodolyaka | derekh: me too :) | 13:34 |
| rpodolyaka | derekh: for pip install -e and trying different commits easily | 13:35 |
| derekh | lsmola_: I usuaully do this | 13:35 |
| derekh | [root@localhost heat]# cd /opt/stack/venvs/heat/lib/python2.7/site-packages/ | 13:35 |
| derekh | [root@localhost site-packages]# mv heat heat_ | 13:35 |
| derekh | [root@localhost site-packages]# ln -s /opt/stack/heat/heat | 13:35 |
| derekh | [root@localhost site-packages]# cd !$ | 13:35 |
| derekh | cd /opt/stack/heat/heat | 13:35 |
| derekh | [root@localhost heat]# git reset --hard <IDOFCOMMITIWANT> | 13:35 |
| lsmola_ | derekh: ok | 13:36 |
| *** nosnos has quit IRC | 13:38 | |
| *** martyntaylor has quit IRC | 13:40 | |
| derekh | ok, it may be only reproducable on tripleo only (or maybe only with latest keystone), going to try that minimal test that shardy linked to on #heat | 13:42 |
| *** ramishra has joined #tripleo | 13:43 | |
| *** ramishra has quit IRC | 13:45 | |
| *** ramishra has joined #tripleo | 13:46 | |
| *** ramishra_ has joined #tripleo | 13:47 | |
| *** julim has joined #tripleo | 13:49 | |
| *** ramishra has quit IRC | 13:51 | |
| derekh | shardy: so we start looking at keystone http://paste.openstack.org/show/61624/ | 13:52 |
| *** martyntaylor has joined #tripleo | 13:54 | |
| shardy | derekh: Hmm, OK - just to confirm, the credentials you have sourced are for the admin tenant right? | 13:55 |
| derekh | shardy: yup | 13:55 |
| derekh | OS_USERNAME=admin | 13:55 |
| derekh | OS_TENANT_NAME=admin | 13:55 |
| shardy | derekh: Ok, sounds like I need to pull latest keystone and try again | 13:55 |
| shardy | derekh: No clues in the keystone log? | 13:55 |
| *** noslzzp has joined #tripleo | 13:56 | |
| derekh | shardy: nope, just deprecated warnings | 13:58 |
| *** ramishra_ has quit IRC | 13:59 | |
| shardy | derekh: Hmm, I just pulled latest keystone and keystoneclient and it still works.. | 13:59 |
| * shardy is confused | 13:59 | |
| shardy | there must be some other difference but I'm not sure what atm | 14:00 |
| *** dprince has joined #tripleo | 14:00 | |
| *** lblanchard has joined #tripleo | 14:00 | |
| *** coolsvap has joined #tripleo | 14:00 | |
| openstackgerrit | Ryan Brady proposed a change to openstack/tripleo-image-elements: Move cinder iptables rules into scoped elements https://review.openstack.org/68118 | 14:01 |
| shardy | derekh: I'll build a fresh devstack VM and try there | 14:01 |
| *** tzumainn has joined #tripleo | 14:01 | |
| derekh | shardy: ok, I'm looking into our keystone, we may have the v3 api configured wrong or something | 14:03 |
| *** max_lobur is now known as max_lobur_afk | 14:04 | |
| shardy | derekh: What does your keystone endpoint URL look like? | 14:04 |
| derekh | that would probably do it | 14:04 |
| derekh | | 91485fd74b6a4daba6667c1584ad0d99 | regionOne | http://192.0.2.1:5000/v2.0 | http://192.0.2.1:5000/v2.0 | http://192.0.2.1:35357/v2.0 | 9ee566a842b64c28ae9ce14c824ed5c2 | | 14:04 |
| shardy | derekh: That should be OK, I substitute the v2.0 with v3 in the code | 14:05 |
| derekh | shardy: ok | 14:05 |
| rlandy | derekh: shardy: if it's helpful I have a lab machine with tripleo install (undercloud deploy just failed) - nothing rolled back yet. | 14:08 |
| *** jdob_ has joined #tripleo | 14:08 | |
| rlandy | can pm the creds to view/try changes | 14:09 |
| rbrady | how are the docs in tripleo-incubator updated from the repo to http://docs.openstack.org/developer/tripleo-incubator/devtest_overcloud.html ? | 14:09 |
| derekh | rlandy: thanks but thats pretty much what I have here, shardy might be able to make use of it though | 14:09 |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/diskimage-builder: Setup ccache symlinks on openSUSE https://review.openstack.org/67965 | 14:09 |
| *** jrist has joined #tripleo | 14:11 | |
| *** jrist has quit IRC | 14:12 | |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Install glance-manage into /usr/local/bin https://review.openstack.org/67956 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to horizon https://review.openstack.org/67952 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to heat https://review.openstack.org/67953 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Install heat-manage into /usr/local/bin https://review.openstack.org/67954 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to glance https://review.openstack.org/67955 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to neutron https://review.openstack.org/67949 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Install nova-baremetal-manage into /usr/local/bin https://review.openstack.org/67948 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Install keystone-manage into /usr/local/bin https://review.openstack.org/67951 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to keystone https://review.openstack.org/67950 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to novnc https://review.openstack.org/67945 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Install nova-manage into /usr/local/bin https://review.openstack.org/67947 | 14:12 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to nova https://review.openstack.org/67946 | 14:12 |
| marios | abort... we've been slagled | 14:13 |
| rdopieralski | O.o | 14:13 |
| hewbrocc` | patch bomb! take cover! | 14:13 |
| akrivoka | wow impressive :) | 14:14 |
| *** jrist has joined #tripleo | 14:15 | |
| lsmola_ | derekh: hmm, have you ever seen 'Got: ImportError('Class sslmiddleware_filter cannot be found', when I revert and restart Heat-API | 14:17 |
| *** jrist has quit IRC | 14:17 | |
| derekh | lsmola_: nope that one is new to me | 14:17 |
| *** jrist has joined #tripleo | 14:18 | |
| shardy | derekh: Aha, I've just reproduced, looks like a keystone regression but still investigating | 14:18 |
| shardy | The roles.list(name='heat_stack_user') filter by name is not working | 14:19 |
| shardy | derekh: DOH! I had https://review.openstack.org/#/c/65985/ applied in my local test env which is not yet merged | 14:20 |
| slagle | marios: hehe | 14:20 |
| shardy | derekh: I'll raise a heat bug and propose a workaround, sorry for the inconvenience | 14:20 |
| slagle | is there a way to resubmit just one change from a patchset to gerrit? | 14:21 |
| rpodolyaka | slagle: only, not rebase on master afaik (git review -R) | 14:21 |
| rpodolyaka | slagle: though, this won't help much if you need to change something in the middle of the series | 14:21 |
| slagle | right, that was the problem | 14:22 |
| lsmola_ | shardy: cool, I am glad you have been able to find it so fas | 14:22 |
| lsmola_ | shardy: fast | 14:22 |
| *** jayg|g0n3 is now known as jayg | 14:23 | |
| *** e0ne has joined #tripleo | 14:24 | |
| *** CaptTofu has joined #tripleo | 14:25 | |
| *** martyntaylor1 has joined #tripleo | 14:26 | |
| derekh | slagle: you could if you wanted just resubmit the updated patch and leave the later patches depended on the outdated version, | 14:26 |
| derekh | slagle: although probably only a good idea if you suspect it will have multiple itterations | 14:27 |
| *** max_lobur_afk is now known as max_lobur | 14:27 | |
| *** e0ne_ has quit IRC | 14:27 | |
| derekh | slagle: once it merges then you could rebase the rest | 14:27 |
| lsmola_ | shardy: so it should be enough just to patch the keystoneclient ,right? | 14:29 |
| *** martyntaylor has quit IRC | 14:29 | |
| slagle | derekh: how do i resubmit just the updated patch though? make it a whole new change you mean? | 14:29 |
| derekh | shardy: cool, thanks | 14:29 |
| shardy | lsmola_: No, I've been trying to get that patch accepted into keystoneclient, and they're being, uh, difficult, so I'll have to work around it in heat for now | 14:29 |
| shardy | lsmola_: I'm working on a fix | 14:30 |
| rbrady | Does anyone know the process for docs update in tripleo-incubator? the docs dir in the repo is explicitly ignored in .gitignore | 14:30 |
| lsmola_ | shardy: ok, great | 14:30 |
| rbrady | derekh, slagle: ^^ | 14:30 |
| openstackgerrit | Dirk Mueller proposed a change to openstack/tuskar-ui: Remove dependencies on pep8, pyflakes and flake8 https://review.openstack.org/68126 | 14:30 |
| slagle | rbrady: you update the devtest*.sh scripts themselves as the docs are generated from those scripts | 14:31 |
| rbrady | slagle: when does the doc actually get updated? commit hook? | 14:31 |
| rpodolyaka | rbrady: ... by a Zuul job that pushes them to http://docs.openstack.org/developer/tripleo-incubator/ | 14:31 |
| rbrady | slagle: thanks! | 14:31 |
| slagle | rbrady: yes | 14:31 |
| rpodolyaka | rbrady: after a change is merged | 14:31 |
| derekh | slagle: for that patchset if you wanted to just update https://review.openstack.org/#/c/67946/ you could do $ git review -d 67946 | 14:32 |
| derekh | then git review | 14:32 |
| derekh | slagle: non of the later commits would update but out be outdated | 14:32 |
| derekh | slagle: so they couldn't merge until you rebase them | 14:33 |
| *** matty_dubs|gone is now known as matty_dubs | 14:33 | |
| derekh | slagle: this would only really be usefull if you expected you would have to update https://review.openstack.org/#/c/67946/ multiple times | 14:33 |
| derekh | slagle: then once its merged you could rebase the rest | 14:33 |
| openstackgerrit | Ryan Brady proposed a change to openstack/tripleo-incubator: Update cinder usage https://review.openstack.org/68129 | 14:33 |
| slagle | derekh: ah, makese sense. thanks | 14:33 |
| *** jdob_ has quit IRC | 14:34 | |
| rbrady | rpodolyaka: thanks | 14:35 |
| rpodolyaka | rbrady: btw, if you need to build docs locally (e.g. if you are fixing docs), you could do: tox -evenv -- python setup.py build_sphinx | 14:35 |
| *** kwwii_ has joined #tripleo | 14:36 | |
| *** kwwii_ has left #tripleo | 14:36 | |
| *** coolsvap_away has joined #tripleo | 14:45 | |
| *** coolsvap has quit IRC | 14:47 | |
| *** coolsvap_away is now known as coolsvap | 14:48 | |
| *** bauzas has quit IRC | 14:51 | |
| *** bauzas has joined #tripleo | 14:53 | |
| openstackgerrit | James Slagle proposed a change to openstack/diskimage-builder: Add ability to use local cloud image https://review.openstack.org/68133 | 14:53 |
| *** boris-42 has quit IRC | 14:54 | |
| *** jcoufal has quit IRC | 15:02 | |
| *** jcoufal_ has joined #tripleo | 15:02 | |
| *** spzala has joined #tripleo | 15:03 | |
| shardy | derekh, lsmola_: https://review.openstack.org/68135 | 15:04 |
| *** vkozhukalov has quit IRC | 15:04 | |
| lsmola_ | shardy: excellent | 15:04 |
| lsmola_ | shardy: thank you | 15:05 |
| derekh | lsmola_: thanks, will try it out | 15:05 |
| derekh | shardy: thanks ^ | 15:05 |
| shardy | lsmola_, derekh: np, sorry for breaking it ;) | 15:05 |
| derekh | shardy: no prob | 15:06 |
| shardy | If you can +1 the review if it works for you that would be much appreciated :) | 15:06 |
| lsmola_ | shardy: np, you are not first one who break it :-) | 15:06 |
| lsmola_ | shardy: actually breaking it is our motto :-) | 15:07 |
| derekh | shardy: will do | 15:07 |
| ccrouch | Why in overcloud-source.yaml do we pass CeilometerDSN, aka ceilometer.db, into nova-compute-instance.yaml when its not used by any of the elements used to build the overcloud compute image? | 15:10 |
| ccrouch | https://github.com/openstack/tripleo-image-elements/search?q=ceilometer.db&type=Code | 15:10 |
| ccrouch | is this in anticipation of the ceilometer-agent-compute element going into the overcloud compute image? | 15:10 |
| *** mrunge has quit IRC | 15:20 | |
| *** martyntaylor1 has quit IRC | 15:22 | |
| *** marun has joined #tripleo | 15:25 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Write public key to authorized_keys https://review.openstack.org/68139 | 15:27 |
| ccrouch | rwsu: shadower: any thoughts on my question above? | 15:27 |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Set TRIPLEO_ROOT https://review.openstack.org/68045 | 15:28 |
| *** lucasagomes is now known as lucas-hungry | 15:33 | |
| *** UtahDave has joined #tripleo | 15:37 | |
| *** martyntaylor has joined #tripleo | 15:38 | |
| *** martyntaylor has quit IRC | 15:38 | |
| *** martyntaylor has joined #tripleo | 15:38 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Destroy the remote seed domain https://review.openstack.org/68143 | 15:46 |
| *** lblanchard has quit IRC | 15:48 | |
| *** lblanchard has joined #tripleo | 15:51 | |
| *** jprovazn has quit IRC | 15:52 | |
| *** jcoufal_ has quit IRC | 15:57 | |
| *** boris-42 has joined #tripleo | 16:02 | |
| *** rdopieralski has quit IRC | 16:12 | |
| *** lucas-hungry is now known as lucasagomes | 16:25 | |
| *** panda has joined #tripleo | 16:27 | |
| *** LinuxJedi is now known as LinuxJedi__ | 16:29 | |
| *** panda_ has quit IRC | 16:30 | |
| *** panda_ has joined #tripleo | 16:31 | |
| *** taps has joined #tripleo | 16:33 | |
| *** panda has quit IRC | 16:33 | |
| *** LinuxJedi__ has quit IRC | 16:44 | |
| *** bauzas has quit IRC | 16:45 | |
| *** Aarongr_afk is now known as AaronGr | 16:46 | |
| *** markmc has quit IRC | 16:46 | |
| *** ftcjeff has joined #tripleo | 16:49 | |
| *** rlandy is now known as rlandy|bbl | 16:53 | |
| *** vkozhukalov has joined #tripleo | 17:06 | |
| *** CaptTofu_ has joined #tripleo | 17:07 | |
| *** julim has quit IRC | 17:09 | |
| *** hewbrocc` has quit IRC | 17:09 | |
| rwsu | ccrouch: the only parameter that is needed is the metering secret, i don't know how tomas was planning to use the other params | 17:09 |
| *** ccrouch has quit IRC | 17:09 | |
| *** CaptTofu has quit IRC | 17:09 | |
| *** shadower has quit IRC | 17:09 | |
| *** lifeless has quit IRC | 17:09 | |
| *** ccrouch has joined #tripleo | 17:09 | |
| *** shadower_ has joined #tripleo | 17:09 | |
| *** lifeless1 has joined #tripleo | 17:09 | |
| *** jeblair has quit IRC | 17:09 | |
| *** ccrouch has quit IRC | 17:09 | |
| *** ccrouch has joined #tripleo | 17:09 | |
| *** julim has joined #tripleo | 17:10 | |
| *** jeblair has joined #tripleo | 17:10 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: set DIB_REPOLOCATION_<project> for projects https://review.openstack.org/68178 | 17:10 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Write public key to authorized_keys https://review.openstack.org/68139 | 17:10 |
| *** matty_dubs is now known as matty_dubs|lunch | 17:14 | |
| *** jistr has quit IRC | 17:15 | |
| *** e0ne has quit IRC | 17:16 | |
| *** martyntaylor has quit IRC | 17:16 | |
| *** CaptTofu_ has quit IRC | 17:19 | |
| *** wendar has joined #tripleo | 17:27 | |
| *** morazi has quit IRC | 17:29 | |
| greghaynes | Morning, TripleO | 17:30 |
| cody-somerville | greghaynes: Hi Greg! | 17:30 |
| cody-somerville | greghaynes: Welcome! | 17:30 |
| greghaynes | Hello, thanks! | 17:30 |
| *** viktors has left #tripleo | 17:30 | |
| *** marun has quit IRC | 17:38 | |
| *** marun has joined #tripleo | 17:44 | |
| *** jistr has joined #tripleo | 17:47 | |
| SpamapS | g'morning tripleo | 17:48 |
| greghaynes | morning | 17:48 |
| SpamapS | greghaynes: ahoy there! | 17:49 |
| greghaynes | heyhey | 17:50 |
| greghaynes | finally made it :) | 17:50 |
| *** matty_dubs|lunch is now known as matty_dubs | 17:52 | |
| *** mrunge has joined #tripleo | 17:53 | |
| *** CaptTofu has joined #tripleo | 17:53 | |
| *** coolsvap is now known as coolsvap_away | 17:54 | |
| *** mestery has quit IRC | 17:55 | |
| Ng | hey greghaynes :) | 17:56 |
| greghaynes | Hi there | 17:56 |
| *** derekh has quit IRC | 17:58 | |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Added initial tuskar api tests https://review.openstack.org/67940 | 18:01 |
| *** athomas has quit IRC | 18:04 | |
| *** max_lobur is now known as max_lobur_afk | 18:07 | |
| *** martyntaylor has joined #tripleo | 18:09 | |
| SpamapS | https://review.openstack.org/#/c/68135/ <-- breaking TripleO | 18:16 |
| SpamapS | cherry pick that into your heat to get devtest to work again | 18:16 |
| *** martyntaylor has left #tripleo | 18:20 | |
| *** morazi has joined #tripleo | 18:22 | |
| *** akrivoka has quit IRC | 18:26 | |
| *** wendar has quit IRC | 18:28 | |
| *** wendar has joined #tripleo | 18:28 | |
| mordred | look it's a greghaynes ! | 18:29 |
| greghaynes | yep yep | 18:31 |
| mordred | greghaynes: great. now that you're here - first task is "fix everything" | 18:34 |
| * mordred is now done for the day having imparted that wisdom | 18:34 | |
| pleia2 | hah | 18:34 |
| *** hewbrocca has joined #tripleo | 18:34 | |
| greghaynes | sounds easy enough! | 18:35 |
| SpamapS | mordred: you have one other task. | 18:41 |
| mordred | SpamapS: wait - what is _my_ task? | 18:41 |
| SpamapS | mordred: _BREAK EVERYTHING_ | 18:41 |
| mordred | SpamapS: DONE | 18:41 |
| SpamapS | http://www.quickmeme.com/img/e9/e919eb2ca92fc654a462564623691e576a2f179e252bffe4e447269193c870c3.jpg | 18:41 |
| *** mrunge has quit IRC | 18:43 | |
| *** ccrouch has quit IRC | 18:43 | |
| greghaynes | ok, amateur devtest q incoming | 18:45 |
| *** jprovazn has joined #tripleo | 18:47 | |
| greghaynes | heat stack-create results in errord undercloud, got any tips on where to poke around to reverse engineer this set of tracebacks? https://gist.github.com/anonymous/8545751/raw/b91a8875d66604e82210ee9a0f9ac1f8a4c432f1/gistfile1.txt | 18:47 |
| *** marun has quit IRC | 18:48 | |
| *** marun has joined #tripleo | 18:49 | |
| slagle | it's look like ssh back to your devtest hold failed | 18:50 |
| cody-somerville | greghaynes: Looks like the virtual power drive is failing to connect via ssh. Is ssh running on your host? and is the network setup so it can ssh in? | 18:50 |
| slagle | yea, that :) | 18:50 |
| greghaynes | yep, thats a winner | 18:51 |
| greghaynes | thanks! | 18:51 |
| * greghaynes considers adding this to devtest docs | 18:51 | |
| slagle | making sure ssh is started and enabled would be a good patch for install-dependencies | 18:51 |
| slagle | :) | 18:51 |
| greghaynes | oo, even better | 18:52 |
| *** marun has quit IRC | 18:54 | |
| *** rpodolyaka1 has joined #tripleo | 18:55 | |
| *** jcoufal-mobile has joined #tripleo | 18:57 | |
| openstackgerrit | Jiri Tomasek proposed a change to openstack/tuskar-ui: Number picker input widget and angular directive https://review.openstack.org/68053 | 18:57 |
| lifeless1 | morning | 18:58 |
| greghaynes | Morning, lifeless1 | 18:58 |
| rpodolyaka1 | o/ | 18:58 |
| *** akuznets_ has quit IRC | 18:59 | |
| *** lifeless1 is now known as lifeless | 18:59 | |
| lifeless | greghaynes: hi | 18:59 |
| lifeless | meeting time everyone :) | 18:59 |
| *** shadower_ is now known as shadower | 19:02 | |
| *** rlandy|bbl is now known as rlandy | 19:02 | |
| *** ccrouch has joined #tripleo | 19:05 | |
| *** spzala has quit IRC | 19:07 | |
| *** ccrouch1 has joined #tripleo | 19:09 | |
| *** ccrouch has quit IRC | 19:10 | |
| *** e0ne has joined #tripleo | 19:11 | |
| openstackgerrit | Ryan Brady proposed a change to openstack/tripleo-image-elements: Fixes cinder element dependencies https://review.openstack.org/68204 | 19:18 |
| openstackgerrit | Ryan Brady proposed a change to openstack/tripleo-incubator: Update cinder usage https://review.openstack.org/68129 | 19:24 |
| openstackgerrit | Aaron Weitekamp proposed a change to openstack/diskimage-builder: Update default RHEL guest image https://review.openstack.org/68206 | 19:27 |
| openstackgerrit | Aaron Weitekamp proposed a change to openstack/diskimage-builder: Add support Red Hat Network (RHN) https://review.openstack.org/68207 | 19:27 |
| openstackgerrit | Aaron Weitekamp proposed a change to openstack/diskimage-builder: Support list of Red Hat channels and repos https://review.openstack.org/68208 | 19:27 |
| openstackgerrit | Aaron Weitekamp proposed a change to openstack/diskimage-builder: Add support for Red Hat Satellite https://review.openstack.org/68209 | 19:27 |
| *** vkozhukalov has quit IRC | 19:27 | |
| SpamapS | ooo shiny RHEL stuff | 19:28 |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Deployment design form https://review.openstack.org/67155 | 19:29 |
| jdob | marios: ping | 19:29 |
| *** pblaho has joined #tripleo | 19:31 | |
| marios | jdob: hi | 19:41 |
| *** mestery has joined #tripleo | 19:41 | |
| jdob | marios: hey, you mentioned your overcloud.yaml thing was blocked by the controllers and models, but i'm not sure why. the models are in place and the controllers shouldn't matter to what you're doing | 19:42 |
| jdob | eventually it'll need to be wired to a controller, but 99% of the actual work of building the template doesn't care about the controller | 19:42 |
| marios | jdob: right. but i'm making assumptions about what the controllers will be (i.e where template generation will be invoked from). also, models are just in, and i expect may yet change | 19:43 |
| marios | jdob: is all i meant | 19:43 |
| jdob | but what you're doing is pretty much algorithmic, we can get that checked in and just not wired up to a controller | 19:43 |
| jdob | i agree the models may change, but all the more reason for you to start beating on them now to find the holes :) | 19:43 |
| marios | jdob: yes i agree. and i am | 19:43 |
| jdob | oh, ok, i misread then, I thought you were saying you were blocked | 19:44 |
| jdob | sounds like we're on the same page | 19:44 |
| marios | jdob: yeah, didn't say i was blocked mate, just that models/controllers are tbd | 19:45 |
| marios | jdob: thanks | 19:45 |
| jdob | my bad :) | 19:45 |
| marios | jdob: heh np :) | 19:45 |
| jdob | i blame my distraction as I stare out the window and want to play in the snow | 19:45 |
| marios | jdob: its almost 2200 here... ;) | 19:46 |
| jdob | where are you out of? | 19:46 |
| marios | jdob: cyprus | 19:46 |
| jdob | oh wow, that's a new one for me | 19:46 |
| marios | jdob: for most people ;) (small place) | 19:47 |
| jdob | the "mate" thing threw me off, I didnt know of many places that use that term | 19:47 |
| *** marun has joined #tripleo | 19:51 | |
| hewbrocca | jdob: marios was in the UK for quite a while | 19:53 |
| jdob | ahh, ok | 19:54 |
| *** e0ne has quit IRC | 19:54 | |
| *** GheRivero has quit IRC | 19:57 | |
| *** GheRivero has joined #tripleo | 19:58 | |
| ccrouch1 | can someone point me at where to look to get a better understanding of how the stateful upgrades are going to work ? | 20:00 |
| marios | jdob: yeah - was in the uk for many years and in fact i was born there cos my parents used to live there. | 20:00 |
| marios | goodnight tripleo | 20:00 |
| lifeless | ccrouch1: hey, https://trello.com/b/0jIoMrdo/tripleo under current MVP | 20:01 |
| *** markmc has joined #tripleo | 20:01 | |
| lifeless | ccrouch1: have a look within those three cards | 20:01 |
| ccrouch1 | i think i've got a reasonable grasp of the current flow to push a new deployment to a new node with glance, nova-bm-helper the node being provisioned etc | 20:01 |
| ccrouch1 | I'm trying to figure out the equivalent for upgrades | 20:01 |
| SpamapS | perhaps we should install horizon and tuskar in the seed and undercloud | 20:01 |
| ccrouch1 | lifeless: thanks, looking... | 20:02 |
| *** pblaho has quit IRC | 20:02 | |
| rlandy | SpamapS: did you try out the heat fix (https://review.openstack.org/#/c/68135/)? | 20:02 |
| SpamapS | rlandy: building now | 20:03 |
| rlandy | SpamapS: k ... would be interested to know if it worked for you. I did the quick/sketchy thing - imported the fix to the seed, restarted heat and keystone services and deployed the undercloud. Maybe not surprisingly, still got the error. | 20:08 |
| *** bauzas has joined #tripleo | 20:08 | |
| *** jistr has quit IRC | 20:11 | |
| *** epim has joined #tripleo | 20:16 | |
| *** jcoufal-mobile has quit IRC | 20:18 | |
| SpamapS | rlandy: did you see the error in the seed's heat-engine log? | 20:23 |
| *** ftcjeff_ has joined #tripleo | 20:24 | |
| SpamapS | rlandy: for me, on seed, keystone user-role-list --user=undercloud-User-$randomthing shows heat_stack_user | 20:25 |
| SpamapS | meaning problem is solved | 20:25 |
| *** lucasagomes has quit IRC | 20:26 | |
| dprince | rlandy: That fix works for me too. If you apply it to the /opt/stack/venv/heat in your seed VM and the 'service restart heat-engine' you should be good to go on the next heat stack-create for the undercloud | 20:27 |
| *** jdob has quit IRC | 20:35 | |
| *** jdob has joined #tripleo | 20:36 | |
| rlandy | dprince: SpamapS: ok - will try it again .... good to know it works | 20:37 |
| dprince | rlandy: or just set these and rebuild your seed too: | 20:38 |
| dprince | export DIB_REPOLOCATION_heat=https://review.openstack.org/openstack/heat | 20:38 |
| dprince | export DIB_REPOREF_heat=refs/changes/35/68135/1 | 20:38 |
| *** mrunge has joined #tripleo | 20:38 | |
| dprince | rlandy: either should work... | 20:38 |
| *** ftcjeff_ has quit IRC | 20:39 | |
| *** spzala has joined #tripleo | 20:42 | |
| *** e0ne has joined #tripleo | 20:43 | |
| *** e0ne has quit IRC | 20:44 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Make neutron-openvswitch-agent run cleanup. https://review.openstack.org/68261 | 20:45 |
| *** rpodolyaka1 has quit IRC | 20:45 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Move Babel and argparse from test-requirements.txt https://review.openstack.org/67070 | 20:46 |
| *** marun has quit IRC | 20:46 | |
| *** jdob has quit IRC | 20:47 | |
| *** jdob has joined #tripleo | 20:47 | |
| lifeless | greghaynes: StevenK: oh btw - you'll want to update your affiliation mappings soonish :0 | 20:52 |
| SpamapS | hah yeah lets not repeat the havana mess | 20:55 |
| Ng | lifeless: SpamapS: I'm confused by the commentary on https://review.openstack.org/#/c/65492/3/elements/ssh-ephemeral/install.d/10-sshd-host-keys-ephemeral,unified | 20:57 |
| Ng | I thought it was ok to register a state path for things that don't yet exist? | 20:57 |
| Ng | but if I don't want to register the not-yet-existing host keys, why would I then want to register all of /etc/ssh? At that point wouldn't modifying the config to store in /mnt/state be sufficient? | 20:58 |
| lifeless | Ng: give me a sec, will look at it | 20:58 |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/diskimage-builder: Setup ccache symlinks on openSUSE https://review.openstack.org/67965 | 21:01 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Only configure DHCP for real interfaces https://review.openstack.org/68265 | 21:04 |
| lifeless | Ng: ok so | 21:04 |
| lifeless | Ng: register-state-path is not a requirement for state paths | 21:04 |
| lifeless | Ng: its specific plumbing to help with recalcitrant situations | 21:04 |
| lifeless | Ng: a missing file is not such | 21:04 |
| SpamapS | Note that cloud-init will just always write to /etc/ssh | 21:05 |
| lifeless | Ng: so thats the first bit on 'missing files' | 21:05 |
| SpamapS | which Ng patched out.. | 21:05 |
| *** lblanchard has quit IRC | 21:05 | |
| SpamapS | but which counts as recalcitrant and is why I suggest just taking all of /etc/ssh into /mnt/state | 21:05 |
| *** e0ne has joined #tripleo | 21:06 | |
| lifeless | note that cloud-init also deletes the initial keys from the image | 21:06 |
| SpamapS | The bad part of that is that we now own all the sshd configs.. but I think that may be O-K. :p | 21:06 |
| lifeless | its fundamentally broken if you ever reboot a cloud instance | 21:06 |
| SpamapS | lifeless: uh no | 21:06 |
| lifeless | oh, cfg option | 21:06 |
| lifeless | if cfg.get("ssh_deletekeys", True): | 21:06 |
| SpamapS | lifeless: it only does that if instance-id changes | 21:06 |
| lifeless | SpamapS: no | 21:06 |
| lifeless | line 61 | 21:07 |
| lifeless | defaults to enabled | 21:07 |
| SpamapS | lifeless: right it only runs that code path if there is a new un-handled instance id | 21:07 |
| lifeless | SpamapS: that doesn't match my observed behaviour | 21:07 |
| SpamapS | which btw, suggests that we need to put /var/lib/cloud in state :) | 21:07 |
| lifeless | SpamapS: IMBW of course | 21:07 |
| * SpamapS this close to saying /var/lib should be bind mounted into /mnt/state | 21:08 | |
| lifeless | does ssh-keygen follow symlinks ? | 21:08 |
| SpamapS | except dpkg .. ugh | 21:08 |
| lifeless | SpamapS: theres a tonne of cruft we don't want | 21:08 |
| lifeless | because distros also conflate what the user wants and what the system wants | 21:08 |
| SpamapS | lifeless: I agree that keys are deleted before they are regenerated. That is important. But I am not agreeing that keys are generated on every reboot. | 21:09 |
| lifeless | SpamapS: sure, I moved on | 21:09 |
| lifeless | SpamapS: if the keys are deleted, a symlink will be deleted | 21:09 |
| SpamapS | right | 21:09 |
| lifeless | SpamapS: if we disable key deletion.... | 21:09 |
| greghaynes | lifeless: I just joined the openstack foundation the other day and added HP affiliation, is there anywhere else I should be worried about affiliation mappings? | 21:09 |
| lifeless | SpamapS: and ssh-keygen follows symlinks | 21:09 |
| *** julim has quit IRC | 21:09 | |
| lifeless | greghaynes: about 3 dozen places [exaggerating] | 21:09 |
| lifeless | greghaynes: git-dm for sure | 21:10 |
| SpamapS | lifeless: or.. /etc/ssh -> /mnt/state/etc/ssh and we just deal with owning sshd_config and the like. | 21:10 |
| lifeless | SpamapS: thats a possibility too | 21:10 |
| lifeless | SpamapS: we're exploring it ;) | 21:10 |
| SpamapS | I also think smoser posted a patch to let us configure cloud-init to set the ssh data dir | 21:11 |
| SpamapS | but that is more to deal with us patching it directly in the image build | 21:11 |
| lifeless | SpamapS: to answer Ng's question, the proposal to move all of /etc is to do an end-run around cloud-init | 21:11 |
| lifeless | by making /etc resolve to the stateful location, so whatever it thinks its doing will be in the right place | 21:11 |
| lifeless | as long as key generation happens after ephemeral volume mounting | 21:11 |
| lifeless | otherwise, it will happen in / and be hidden after /mnt is mounted | 21:12 |
| SpamapS | Yeah that may be a tricky thing but it definitely _should_ | 21:12 |
| lifeless | so, I have no objection to having templates for all of /etc/ssh | 21:12 |
| lifeless | erm, I mean all of /etc/openssh or whatever, *not* all of /etc | 21:12 |
| lifeless | if we're going to own it, I think we should own it and make it consistent everywhere | 21:13 |
| lifeless | slagle: ^ in the same way I'm conerned about service names | 21:13 |
| lifeless | SpamapS: but I think we should try disabling key deletion as a first stab | 21:13 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Rename generate-interfaces-file.sh... https://review.openstack.org/68266 | 21:13 |
| Ng | I'm not 100% sure why, but taking over all of /etc/ssh makes me feel a bit queasy | 21:13 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Rename generate-interfaces-file.sh... https://review.openstack.org/68266 | 21:14 |
| lifeless | SpamapS: because owning all of openssh configuration is a pretty big hammer | 21:14 |
| SpamapS | lifeless: looks like ephemeral is part of cloud-config and thus after ssh key generation | 21:15 |
| Ng | is the objection to just patching cc_ssh.py, that it's kinda rude and potentially fragile? | 21:15 |
| SpamapS | Ng: well it is really fragile IMO | 21:15 |
| SpamapS | Ng: also we have a patch now.. so worst case we upgrade cloud-init on all our images. | 21:16 |
| slagle | lifeless: what's the concern? having all our services not named the same? | 21:18 |
| lifeless | slagle: documentation for ops of a tripleo environment | 21:18 |
| lifeless | SpamapS: so the element just won't work right now | 21:19 |
| lifeless | Ng: ^ | 21:19 |
| Ng | oh? | 21:19 |
| lifeless | 10:15 < SpamapS> lifeless: looks like ephemeral is part of cloud-config and thus after ssh key generation | 21:19 |
| lifeless | Ng: e.g. it will get written to /mnt and then /mnt will get mounted | 21:19 |
| Ng | hmm | 21:19 |
| lifeless | SpamapS: you have a workaround for /mnt not being mounted by cloud config, right ? | 21:19 |
| slagle | lifeless: i added a os-svc-restart...so we could just document that. the service name will be the same regardless of distro or source/package install | 21:19 |
| Ng | the test instances I booted ended up with the setup I was expecting | 21:19 |
| lifeless | slagle: the logs are affected too though, right? | 21:20 |
| Ng | but it's possible I did something wronglike | 21:20 |
| lifeless | slagle: so any complexity is going to propogate all the way down into logstash and analysis tools | 21:20 |
| lifeless | slagle: I like os-svc-restart | 21:20 |
| lifeless | slagle: but I don't think it solves the issue | 21:20 |
| lifeless | Ng: were they vms or physical? Did they have a /mnt partition mounted? | 21:21 |
| Ng | hmm, actually they were probably bad test cases due to a lack of a separate partition being mounted | 21:21 |
| Ng | nuts | 21:21 |
| *** rlandy has quit IRC | 21:21 | |
| SpamapS | lifeless: yes.. https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements/use-ephemeral/os-refresh-config/pre-configure.d/00-fix-ephemeral-mount | 21:21 |
| slagle | lifeless: yea, true. some services don't log to files at all right now, they just use journald | 21:22 |
| slagle | vs upstart service logs | 21:22 |
| Ng | well, if I can replace the entire file with register-state-path /etc/ssh and make all this go away, I guess we're owning /etc/ssh ;) | 21:22 |
| slagle | but, we control the logging configuration. even in the package installs | 21:22 |
| SpamapS | lifeless: I thought of one other thing.. we _could_ try to get ephemeral in the fstab. Doesn't it get a label? That would happen well before cloud-init tries to do any writing to disk. | 21:23 |
| *** dprince has quit IRC | 21:23 | |
| * Ng hopes that our children will get to live in a world where there is only one init system on Linux distros | 21:23 | |
| lifeless | Ng: that wouldn't fix the mount race thats in discussion | 21:23 |
| lifeless | Ng: 4 for ever | 21:23 |
| lifeless | (cloud-init, systemd, upstart, orc :P) | 21:23 |
| *** spzala has quit IRC | 21:23 | |
| lifeless | all 4 all at once | 21:23 |
| SpamapS | \o/ .. devtest actually worked ( with the heat patch cherry picked ) | 21:23 |
| Ng | lifeless: I hate your future | 21:24 |
| lifeless | Ng: SpamapS: I am going to have to context switch; I suggest that in principle disabling deletion is a better fix than taking all of /etc - less scope creep | 21:24 |
| lifeless | and that we may need to work with cloud init to support /mnt , both the mount timing issue and data store for the keys | 21:25 |
| lifeless | both seem like upstream patches to me | 21:25 |
| *** e0ne has quit IRC | 21:27 | |
| *** markmc has quit IRC | 21:30 | |
| *** e0ne has joined #tripleo | 21:31 | |
| ccrouch1 | lifeless: https://etherpad.openstack.org/p/tripleo-image-updates | 21:33 |
| ccrouch1 | was very helpful. | 21:33 |
| ccrouch1 | I think i have a better idea how the upgrade-with-reboot will work, i.e. just leave the ephemeral partition alone when laying down new image. | 21:33 |
| ccrouch1 | Are there more details about the upgrade-without-reboot flow? | 21:33 |
| ccrouch1 | I see reference to "takeovernode ", is that https://github.com/openstack/tripleo-incubator/blob/master/scripts/takeovernode | 21:33 |
| ccrouch1 | but that still includes a reboot | 21:33 |
| lifeless | yes | 21:33 |
| lifeless | but only needs one if the kernel changed | 21:34 |
| lifeless | its not finished, obviously | 21:34 |
| jomara_ | tzumainn: kicked off irc internal? | 21:35 |
| *** jprovazn has quit IRC | 21:37 | |
| bnemec | jomara_: Yeah, looks like there was a problem with it. | 21:37 |
| bnemec | I just reconnected, but I was off for quite a while. | 21:38 |
| ccrouch1 | lifeless: got it, I see the reboot OR...now in the todo. thx | 21:38 |
| lifeless | slagle: so look, I dunno | 21:38 |
| lifeless | slagle: I can see that we can tweak logs and we could tell folk to know about os-svc-restart | 21:38 |
| lifeless | slagle: but I worry that doing that - persistently masking users from the things their distro of choice installed | 21:39 |
| lifeless | slagle: basically makes us a virtual linux distro | 21:39 |
| lifeless | slagle: I'm speculating that a narrower focus - drill down on openstack - and say 'it will be X for our supported platforms' might avoid that | 21:40 |
| lifeless | slagle: so rather than documenting new-tool-Y, we document 'it will use your init system, but be called nova-api' | 21:40 |
| jomara_ | bnemec: yeah it just popped me back on too | 21:40 |
| slagle | lifeless: and what would nova-api do? | 21:41 |
| slagle | lifeless: detect the init system restart the service correctly? | 21:41 |
| *** CaptTofu has quit IRC | 21:41 | |
| *** e0ne has quit IRC | 21:43 | |
| lifeless | slagle: I meant that we'd call the unit/job/$thing nova-api consistently | 21:43 |
| slagle | oh i see | 21:44 |
| slagle | i actually was going to go with a similar approach using symlinks for the systemd services | 21:44 |
| slagle | but since systemd uses symlinks to disable/enable, you run into all sorts of problems | 21:45 |
| slagle | we could rename the service files, but then we're modifying packaged files | 21:45 |
| lifeless | is that a concern? Since we're not using the package manager to upgrade... | 21:48 |
| slagle | i'm not super concerned | 21:50 |
| slagle | i would probably need to check if there is a concern from a support perspective | 21:51 |
| lifeless | ok | 21:51 |
| slagle | or maybe we could package something that fixes the names | 21:51 |
| slagle | tbh, i wish fedora would just stick to upstream names in general. it makes searching for packages so much easier | 21:52 |
| bnemec | slagle: Yeah, symlinks for service files are an issue. We discussed a workaround for mariadb here: https://bugzilla.redhat.com/show_bug.cgi?id=1014311 | 21:53 |
| lifeless | slagle: +1 :) | 21:54 |
| lifeless | I spoke to stewart @ LCA, he says that we can use the binary package for anything they don't have native repos | 21:54 |
| lifeless | so we should be good to go on that migration | 21:54 |
| *** jcoufal has joined #tripleo | 21:56 | |
| slagle | bnemec: oh, nice so we could package up a set of complimentary service files then | 21:57 |
| bnemec | slagle: Yeah, it seems like that should work fine. | 21:58 |
| bnemec | Sadly they're dropping the mysql service compatibility for mariadb in rhel7. :-( | 21:58 |
| *** jdob has quit IRC | 21:58 | |
| *** lblanchard has joined #tripleo | 22:02 | |
| *** CaptTofu has joined #tripleo | 22:05 | |
| ccrouch1 | I see from overcloud-source.yaml that gre is the tunneling protocol neutron uses in the overcloud. I presume that means vxlan or vlan options are untested? | 22:09 |
| lifeless | totally | 22:11 |
| lifeless | more fun... | 22:13 |
| lifeless | https://bugs.launchpad.net/neutron/+bug/1271344 | 22:13 |
| lifeless | pleia2: hey so | 22:14 |
| lifeless | pleia2: you seemed a little unsure where the tripleo-ci stuff was at; perhaps if we had a brief chat that might clear thinsg up? I'm inferring (wrongly?) that as a result you aren't sure what to hack on... | 22:15 |
| ccrouch1 | next random question :-) | 22:18 |
| ccrouch1 | does anyone see any benefit from having neutron run on its own node in the overcloud separate from the other controller services? | 22:18 |
| *** vipul is now known as vipul-away | 22:19 | |
| ccrouch1 | iiuc we've seen customer deployments that have/desire this separation | 22:20 |
| ccrouch1 | i'm just wondering if anyone else sees a preference for that? | 22:20 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-incubator: Document stable branches in devtest https://review.openstack.org/68274 | 22:22 |
| pleia2 | lifeless: a chat would be great | 22:22 |
| lifeless | pleia2: gg+ ? | 22:23 |
| slagle | ccrouch1: i believe you would need to do that if you had true isolated networks and desired public internet access for your tenant instances | 22:23 |
| lifeless | slagle: ccrouch1: hmm? no, its not needed for that | 22:23 |
| slagle | no? i thought i saw it on a diagram...somewhere in the docs | 22:23 |
| lifeless | its a scaling bottleneck though, and folk generally want network to be as fast as possible :) | 22:24 |
| lifeless | slagle: HP region has internal network, external network | 22:24 |
| lifeless | two NICs, done | 22:24 |
| slagle | ok :) | 22:24 |
| lifeless | so breaking it out is a thing | 22:24 |
| pleia2 | lifeless: yeah, just grabbing my headphones :) | 22:24 |
| lifeless | just not required by that use case | 22:25 |
| *** vipul-away is now known as vipul | 22:25 | |
| *** vipul is now known as vipul-away | 22:25 | |
| pleia2 | lifeless: ready when you are | 22:26 |
| ccrouch1 | lifeless: so its a potential roadmap item to support the heat template that splits neutron out? | 22:26 |
| lifeless | sure | 22:26 |
| lifeless | split all the things | 22:26 |
| ccrouch1 | :-) | 22:26 |
| slagle | this is what i was thinking of: http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#network-connectivity | 22:28 |
| slagle | but, yea, that wouldn't be a requirement | 22:28 |
| *** jwalls has joined #tripleo | 22:30 | |
| *** ftcjeff has quit IRC | 22:30 | |
| SpamapS | ccrouch1: just to add on.. splitting things out right now is painful because we have to keep things mergable or duplicate effort. With the hot software config changes we should be able to write-once-use-anywhere for template snippets that just include metadata. | 22:44 |
| *** morazi has quit IRC | 22:46 | |
| *** spzala has joined #tripleo | 22:47 | |
| *** jayg is now known as jayg|g0n3 | 22:47 | |
| ccrouch1 | SpamapS: good to know | 22:52 |
| lifeless | SpamapS: well, also tuits; we haven't really pushed to make it be done | 23:00 |
| *** morazi has joined #tripleo | 23:01 | |
| *** mrunge has quit IRC | 23:02 | |
| *** CaptTofu has quit IRC | 23:04 | |
| *** jcoufal has quit IRC | 23:04 | |
| *** lblanchard has quit IRC | 23:07 | |
| SpamapS | lifeless: I've prioritized reviews that pave the way for it. :) | 23:12 |
| *** matty_dubs is now known as matty_dubs|gone | 23:13 | |
| *** jtomasek has quit IRC | 23:19 | |
| *** jwalls has quit IRC | 23:28 | |
| *** lblanchard has joined #tripleo | 23:29 | |
| *** ccrouch1 has quit IRC | 23:41 | |
| *** blamar has quit IRC | 23:47 | |
| *** lblanchard has quit IRC | 23:52 | |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/diskimage-builder: Add Copyright and License header to debian element https://review.openstack.org/66975 | 23:57 |
| StevenK | Bleh, I can think of a few more places where I should do the same | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
