| dhellmann | that should be easy to fix | 00:00 |
|---|---|---|
| adam_g | erm, also defaults | 00:00 |
| adam_g | yea | 00:00 |
| adam_g | nova/openstack/common/log.py gets synced in, no? | 00:00 |
| *** dwalleck has joined #openstack-ironic | 00:01 | |
| dhellmann | yeah | 00:03 |
| dhellmann | if the default has "nova" or "ironic" in it, then you won't be able to make them match | 00:03 |
| adam_g | dhellmann, yeah--the logging stuff seems easys enough to fix but not so much for others ,eg rootwrap_config | 00:05 |
| dhellmann | you're not using oslo.rootwrap, I guess? | 00:07 |
| dhellmann | (the lib) | 00:07 |
| dhellmann | adam_g: I have to take off. I'll try to help some more tomorrow. | 00:08 |
| adam_g | dhellmann, hmm, listed in the requirements.txt | 00:08 |
| dhellmann | I guess if you both have options to specify the default file, though... | 00:08 |
| adam_g | dhellmann, thanks, have a goood one | 00:09 |
| *** dwalleck has quit IRC | 00:10 | |
| openstackgerrit | David Shrewsbury proposed a change to openstack/ironic: Refactor nova.virt.ironic.driver get_host_stats https://review.openstack.org/83853 | 00:12 |
| * NobodyCam starts a test with 83853 and walks afk for a bit.... | 00:15 | |
| Shrews | NobodyCam: make sure you're using that latest patchset | 00:16 |
| NobodyCam | Shrews: xport DIB_REPOREF_ironic=refs/changes/53/83853/5 :-p lazy late in the day test | 00:17 |
| Shrews | ++ | 00:18 |
| NobodyCam | :) | 00:18 |
| Shrews | So, this unit test memory hog issue is consistenly repeatable on my hp vm, but I have not seen it on my laptop. HP vm has 2 cores, running 12.04. Laptop has 4 and 13.10. | 00:35 |
| Shrews | I suspect the # of cores might be the key there since the test runner appears to split up tests for each core | 00:36 |
| Shrews | adam_g: how many cpu cores on the machine you saw this problem? | 00:37 |
| adam_g | Shrews, 4 | 00:38 |
| adam_g | Shrews, i was able to hit it with --concurrency=1 as well | 00:38 |
| adam_g | FWIW | 00:38 |
| Shrews | hrm. os? | 00:38 |
| adam_g | Shrews, ubuntu 14.04 | 00:41 |
| Shrews | i'm seeing it with concurrency=1 too | 00:42 |
| Shrews | *sigh* | 00:42 |
| *** rloo has quit IRC | 01:10 | |
| *** eghobo has joined #openstack-ironic | 01:17 | |
| *** nosnos has joined #openstack-ironic | 01:33 | |
| NobodyCam | Shrews: [2014-04-03 18:47:20] undercloud : 1939 s | 01:54 |
| NobodyCam | worked :) | 01:54 |
| Shrews | woohoo! | 01:54 |
| NobodyCam | and woo hoo the tripleo tests are running again :) | 01:55 |
| *** eghobo has quit IRC | 01:56 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack/ironic: Update tox.ini to also run nova tests https://review.openstack.org/84033 | 01:58 |
| openstackgerrit | Adam Gandelman proposed a change to openstack/ironic: Update tox.ini to also run nova tests https://review.openstack.org/84033 | 01:58 |
| adam_g | devananda, ^ slightly better approach that gets rid of that shell script enitrely | 01:59 |
| adam_g | still a bit hacky in tox.ini | 01:59 |
| Shrews | adam_g: i find that i can reproduce that memory bug by running this: discover -t ./ -v | 02:01 |
| adam_g | Shrews, before the tests even run? | 02:01 |
| Shrews | the api tests all fail for some reason, then it just hangs | 02:01 |
| Shrews | during | 02:02 |
| Shrews | i'm no unit testing framework expert, so learning how this stuff works as i go | 02:02 |
| *** mynameisdeleted has joined #openstack-ironic | 02:02 | |
| mynameisdeleted | hi all | 02:02 |
| adam_g | Shrews, im with you, just spent the last hour or so deep into testr/subunit/testtools | 02:03 |
| mynameisdeleted | so.. ironic is good for a giant-storage ubuntu machien provisioning like 30 gaming pc's in an internet cafe? | 02:03 |
| mynameisdeleted | I want to use adobe for an hour.. I can buy time, and have an adobe image sent to me whcih I rent which has a cloud license for adobe | 02:03 |
| mynameisdeleted | I want to rn the latest video game I rent an image iwth that installed also licensed for cloud node use | 02:04 |
| mynameisdeleted | I want linux I can tell the front desk person to send me that | 02:04 |
| mynameisdeleted | does this project launch windows on iscsi targets well? | 02:05 |
| adam_g | Shrews, runnig discover gets me hung up at ironic.tests.drivers.test_seamicro.SeaMicroPrivateMethodsTestCase.test__power_off_fail ... | 02:06 |
| adam_g | Shrews, ive run the suite a bunch via tox directly without issue, at least out of that nova unit test review above | 02:06 |
| Shrews | weird | 02:07 |
| *** dwalleck has joined #openstack-ironic | 02:14 | |
| *** _LXXIII_ has quit IRC | 02:14 | |
| *** eghobo has joined #openstack-ironic | 02:28 | |
| *** eghobo has quit IRC | 02:47 | |
| *** matsuhas_ has joined #openstack-ironic | 02:53 | |
| *** matsuhashi has quit IRC | 02:53 | |
| *** matsuhas_ has quit IRC | 02:54 | |
| *** matsuhashi has joined #openstack-ironic | 02:56 | |
| *** matsuhashi has quit IRC | 03:02 | |
| *** matsuhashi has joined #openstack-ironic | 03:04 | |
| *** hemna_ has quit IRC | 03:11 | |
| *** nosnos has quit IRC | 03:17 | |
| openstackgerrit | Russell Haering proposed a change to openstack/ironic: Version agent lookup payloads https://review.openstack.org/85228 | 03:19 |
| openstackgerrit | Adam Gandelman proposed a change to openstack/ironic: Update tox.ini to also run nova tests https://review.openstack.org/84033 | 03:33 |
| *** eghobo has joined #openstack-ironic | 03:33 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack/ironic: Update tox.ini to also run nova tests https://review.openstack.org/84033 | 03:34 |
| *** eghobo has quit IRC | 03:58 | |
| *** eghobo has joined #openstack-ironic | 04:00 | |
| *** matsuhashi has quit IRC | 04:01 | |
| *** eghobo has quit IRC | 04:02 | |
| *** harlowja is now known as harlowja_away | 04:03 | |
| *** matsuhashi has joined #openstack-ironic | 04:05 | |
| *** rameshg87 has joined #openstack-ironic | 04:06 | |
| *** dwalleck has quit IRC | 04:20 | |
| *** nosnos has joined #openstack-ironic | 04:20 | |
| openstackgerrit | Jenkins proposed a change to openstack/ironic: Updated from global requirements https://review.openstack.org/83471 | 04:20 |
| openstackgerrit | Jenkins proposed a change to openstack/ironic-python-agent: Updated from global requirements https://review.openstack.org/85233 | 04:20 |
| *** zelenyuk has joined #openstack-ironic | 04:31 | |
| *** killer_prince2 has joined #openstack-ironic | 04:34 | |
| *** killer_prince2 is now known as lazy_prince | 04:46 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack/ironic: Update tox.ini to also run nova tests https://review.openstack.org/84033 | 04:57 |
| *** harlowja_away has quit IRC | 05:29 | |
| *** vkozhukalov has joined #openstack-ironic | 05:31 | |
| openstackgerrit | Jenkins proposed a change to openstack/ironic: Imported Translations from Transifex https://review.openstack.org/83956 | 06:09 |
| *** mrda is now known as mrda_weekend | 06:47 | |
| *** vkozhukalov has quit IRC | 06:50 | |
| *** romcheg has joined #openstack-ironic | 06:53 | |
| *** zelenyuk has quit IRC | 06:57 | |
| *** romcheg has quit IRC | 07:03 | |
| *** matsuhashi has quit IRC | 07:07 | |
| *** saju_m has joined #openstack-ironic | 07:12 | |
| *** ifarkas has joined #openstack-ironic | 07:16 | |
| *** max_lobur has joined #openstack-ironic | 07:34 | |
| openstackgerrit | Yuriy Zveryanskyy proposed a change to openstack/ironic: Fix messages formatting for _sync_power_states https://review.openstack.org/85044 | 07:34 |
| dtantsur | Morning Ironic | 07:47 |
| lifeless | o/ | 07:53 |
| Mikhail_D_wk | Morning all! :) | 07:53 |
| *** matsuhashi has joined #openstack-ironic | 08:04 | |
| *** martyntaylor has joined #openstack-ironic | 08:04 | |
| *** martyntaylor has quit IRC | 08:09 | |
| *** jistr has joined #openstack-ironic | 08:10 | |
| *** derekh has joined #openstack-ironic | 08:10 | |
| *** vkozhukalov has joined #openstack-ironic | 08:15 | |
| *** max_lobur has quit IRC | 08:24 | |
| *** martyntaylor has joined #openstack-ironic | 08:25 | |
| *** lucasagomes has joined #openstack-ironic | 08:26 | |
| Mikhail_D_wk | comstud: Are you here? :) | 08:26 |
| *** Mikhail_D_wk has left #openstack-ironic | 08:34 | |
| *** stevehuang has quit IRC | 08:34 | |
| *** stevehuang has joined #openstack-ironic | 08:34 | |
| *** matsuhashi has quit IRC | 08:35 | |
| *** Mikhail_D_wk has joined #openstack-ironic | 08:36 | |
| *** athomas has joined #openstack-ironic | 08:37 | |
| openstackgerrit | Yuriy Zveryanskyy proposed a change to openstack/ironic: Remove 'persistent' parameter for boot device in PXE driver https://review.openstack.org/85267 | 08:52 |
| *** matsuhashi has joined #openstack-ironic | 08:53 | |
| *** max_lobur has joined #openstack-ironic | 09:07 | |
| *** max_lobur1 has joined #openstack-ironic | 09:07 | |
| lucasagomes | lifeless, if you have a time can you take a look at 85267? I put a comment there, but I'd like to see the opnion of someone with a more sysadim background | 09:09 |
| *** max_lobur has quit IRC | 09:11 | |
| openstackgerrit | A change was merged to openstack/ironic: Change admin_url help in ironic driver https://review.openstack.org/84148 | 09:19 |
| *** matsuhashi has quit IRC | 09:47 | |
| *** matsuhashi has joined #openstack-ironic | 09:47 | |
| lifeless | lucasagomes: reviewed | 09:54 |
| lucasagomes | lifeless, ta much! | 09:55 |
| *** saju_m has quit IRC | 09:56 | |
| *** saju_m has joined #openstack-ironic | 10:09 | |
| *** overlayer has joined #openstack-ironic | 10:11 | |
| *** saju_m has quit IRC | 10:14 | |
| *** matsuhashi has quit IRC | 10:23 | |
| *** saju_m has joined #openstack-ironic | 10:30 | |
| *** matsuhashi has joined #openstack-ironic | 10:30 | |
| *** nosnos has quit IRC | 10:33 | |
| *** saju_m has quit IRC | 10:35 | |
| *** saju_m has joined #openstack-ironic | 10:35 | |
| *** jistr has quit IRC | 10:37 | |
| *** matsuhashi has quit IRC | 10:38 | |
| *** nosnos has joined #openstack-ironic | 10:39 | |
| *** saju_m has quit IRC | 10:40 | |
| *** saju_m has joined #openstack-ironic | 10:40 | |
| *** matsuhas_ has joined #openstack-ironic | 10:41 | |
| openstackgerrit | A change was merged to openstack/ironic: Reduce logging output from non-Ironic libraries https://review.openstack.org/84496 | 10:49 |
| *** matsuhas_ has quit IRC | 10:52 | |
| *** matsuhashi has joined #openstack-ironic | 10:52 | |
| *** yuriyz has quit IRC | 10:54 | |
| *** matsuhashi has quit IRC | 10:57 | |
| *** jistr has joined #openstack-ironic | 10:58 | |
| rameshg87 | hello lucasagomes: | 11:03 |
| lucasagomes | rameshg87, hi there! | 11:03 |
| rameshg87 | regarding the bug https://bugs.launchpad.net/ironic/+bug/1301975 | 11:03 |
| rameshg87 | Fix sleep() workaround for the "device is busy" problem | 11:04 |
| rameshg87 | i think we can make use of https://pypi.python.org/pypi/psutil/ | 11:04 |
| rameshg87 | how about using the library, it provides lsof functionality | 11:04 |
| lucasagomes | rameshg87, ah, looks good! We also think that maybe adding a new dependency to the project just because of that small problem might be overkill | 11:06 |
| lucasagomes | also have to think* | 11:06 |
| lucasagomes | rameshg87, I can't reproduce the device is busy problem in my env tho :( even removing that sleep things works | 11:07 |
| lucasagomes | but someone might have hit this problem | 11:07 |
| rameshg87 | yeah, i agree | 11:11 |
| rameshg87 | so do you suggest waiting for more votes who have faced this issue ? | 11:12 |
| rameshg87 | even i haven't faced this issue :-) | 11:12 |
| lucasagomes | rameshg87, oh no, I mean, as I haven't hit this problem and don't have the right experience/knowledge to know how to best avoid the problem I just opened the bug and left it there | 11:15 |
| lucasagomes | but if you feel like you can fix it in a nice way | 11:15 |
| lucasagomes | that would be great | 11:15 |
| *** yuriyz has joined #openstack-ironic | 11:15 | |
| lucasagomes | rameshg87, I'm sure sleeping is not the right way to avoid it | 11:16 |
| lucasagomes | heh | 11:16 |
| rameshg87 | lucasagomes, :-). okay, i just thought of checking with you since you submitted the bug | 11:16 |
| lucasagomes | rameshg87, :) | 11:17 |
| *** athomas has quit IRC | 11:18 | |
| *** rameshg87 has left #openstack-ironic | 11:23 | |
| *** lucasagomes is now known as lucas-hungry | 11:27 | |
| openstackgerrit | David Shrewsbury proposed a change to openstack/ironic: Encapsulate Ironic client retry logic https://review.openstack.org/83105 | 11:29 |
| *** nosnos has quit IRC | 11:30 | |
| *** jbjohnso_ has joined #openstack-ironic | 11:33 | |
| *** athomas has joined #openstack-ironic | 11:33 | |
| *** eghobo has joined #openstack-ironic | 11:36 | |
| *** saju_m has quit IRC | 11:38 | |
| openstackgerrit | Jenkins proposed a change to openstack/ironic: Updated from global requirements https://review.openstack.org/83471 | 11:49 |
| Shrews | Happy Friday, everyone. A good reason to be excited. Another good reason... only 2 days until Game of Thrones season premiere. | 12:03 |
| openstackgerrit | Mikhail Durnosvistov proposed a change to openstack/ironic: Old value 'updated_at' field returned after update https://review.openstack.org/75430 | 12:04 |
| *** saju_m has joined #openstack-ironic | 12:10 | |
| *** saju_m has quit IRC | 12:14 | |
| *** romcheg has joined #openstack-ironic | 12:15 | |
| *** saju_m has joined #openstack-ironic | 12:15 | |
| *** eghobo has quit IRC | 12:21 | |
| *** linggao has joined #openstack-ironic | 12:22 | |
| romcheg | Morning guys | 12:39 |
| romcheg | Please don't approve 85135, it still does not fix compatibility errors for mac | 12:40 |
| lucas-hungry | morning romcheg Shrews | 12:40 |
| lucas-hungry | Shrews, GOT stills good? | 12:40 |
| lucas-hungry | romcheg, ack, can u -1 it please? I didn't know hw to test it on mac | 12:41 |
| lucas-hungry | so | 12:41 |
| lucas-hungry | romcheg, and I saw some projects already approved that so I +2 with that warning | 12:41 |
| *** lucas-hungry is now known as lucasagomes | 12:41 | |
| romcheg | lucas-hungry: I'm going to -1 it after I detect where the problem is | 12:41 |
| * lucasagomes is not hungry anymore | 12:41 | |
| lucasagomes | romcheg, ack | 12:41 |
| lucasagomes | romcheg, I will remove my vote | 12:41 |
| Shrews | lucasgomes: yes. i'm hoping for even better things this season :) | 12:41 |
| romcheg | lucasagomes: aparently check_uptodate.sh fails | 12:42 |
| lucasagomes | Shrews, :) I read the first book | 12:42 |
| lucasagomes | Shrews, and started the second... but tihngs changed a lot | 12:43 |
| lucasagomes | I prefered when they didn't have a lot of magic stuff and it was more about the human behavior | 12:43 |
| lucasagomes | romcheg, ack | 12:43 |
| lucasagomes | romcheg, changed my vote there | 12:43 |
| *** jdob has joined #openstack-ironic | 12:44 | |
| lazy_prince | lucasagomes: can you review https://blueprints.launchpad.net/ironic/+spec/uefi-gpt-support | 12:49 |
| lucasagomes | lazy_prince, sure :) thanks for that | 12:49 |
| lazy_prince | welcome.. :) | 12:50 |
| lucasagomes | lazy_prince, I haven't played with UEFI at all, but the steps there looks good | 12:51 |
| lazy_prince | thanks.. i guess, if there is any change in future, we can always update the bp.. | 12:52 |
| *** dwalleck has joined #openstack-ironic | 12:52 | |
| lucasagomes | yup | 12:53 |
| *** saju_m has quit IRC | 12:57 | |
| *** dwalleck_ has joined #openstack-ironic | 13:00 | |
| *** dwalleck has quit IRC | 13:04 | |
| *** dwalleck has joined #openstack-ironic | 13:04 | |
| *** dwalleck_ has quit IRC | 13:05 | |
| dtantsur | Guys, how do you run unit tests manually? Running tox recreates virtualenv every time, running " setup.py testr -t test_name" results in terrible binary backtrace | 13:13 |
| romcheg | dtantsur: tox should not recreate venvs | 13:13 |
| dtantsur | romcheg, well, it does. And always fails the first time. Any ideas, why? | 13:13 |
| dtantsur | so, first run "OSError: [Errno 2] No such file or directory" | 13:14 |
| dtantsur | the second run: py27 installdeps: -r/home/dtantsur/ironic/requirements.txt, -r/home/dtantsur/ironic/test-requirements.txt (for some time) | 13:14 |
| romcheg | Maybe some deps are not installed? | 13:14 |
| romcheg | Could you please post a log to paste.openstack.org | 13:14 |
| dtantsur | where is it's log? | 13:14 |
| Shrews | dtantsur: it shouldn't be recreating each time. i've found that if i have multiple venvs, using tox from a different venv will cause it to recreate. so just be consistent | 13:14 |
| dtantsur | Shrews, I'm in it's own venv, I think | 13:15 |
| dtantsur | (at least it's own from previous runs) | 13:16 |
| Shrews | dtantsur: i follow the steps from the quick start (http://docs.openstack.org/developer/ironic/dev/dev-quickstart.html). works fine for me | 13:17 |
| Shrews | and the OSError seems very suspicious | 13:17 |
| dtantsur | Shrews, me too. I'll try again. I think OSError is because when recreating it kills something it needs. Not sure though | 13:18 |
| dtantsur | Shrews, ok I followed guide, and on the "tox" step got again: py27 recreate: /home/dtantsur/ironic/.tox/py27 | 13:22 |
| Shrews | dtantsur: let it finish, then run tox again | 13:22 |
| Shrews | if it still recreates, then there is definitely a problem | 13:23 |
| *** krtaylor has joined #openstack-ironic | 13:24 | |
| dtantsur | I think I found the cause in pip logs. It tries to recreate some egg-info (setuptools, brrrr) and fails because this file already exists and is owned by root | 13:24 |
| dtantsur | It's hard to tell how I like the practice of using the same temporary files for every invocation... | 13:25 |
| dtantsur | Well, it wasn't owned by root, nut anyway it no longer seems to recreate the env, so thanks everyone, problem is somehow solved | 13:27 |
| *** matty_dubs|gone is now known as matty_dubs | 13:28 | |
| dtantsur | BUT I still get "binary" log, even from tox :( | 13:29 |
| Haomeng|2 | lucasagomes: morning | 13:33 |
| openstackgerrit | Vladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute util https://review.openstack.org/85344 | 13:34 |
| Haomeng|2 | lucasagomes: I am working on API i18n support for our ironic, want to check with you do you know if our api have common handler to catch the api exceptions? | 13:34 |
| Haomeng|2 | lucasagomes: this patch - https://review.openstack.org/#/c/84362/ , it still missing the gettextutils.get_localized_message call for the locale on the api request | 13:37 |
| Haomeng|2 | lucasagomes: my question is, if we have common api exception handler to process api exception, that we have chance to call gettextutils.get_localized_message to translate the api exception error message based on the request Accept-Language parameter | 13:38 |
| Haomeng|2 | lucasagomes: I debuged our api call for the case which node id is not existing one - http://paste.openstack.org/show/75083/ | 13:40 |
| Haomeng|2 | lucasagomes: we can see the exception stack, it is raised to high level code - /opt/stack/ironic/ironic/api/app.py(85)__call__() | 13:41 |
| Haomeng|2 | lucasagomes: take other project as reference, they have common wsgi exception handler - https://github.com/openstack/glance/blob/master/glance/common/wsgi.py#L647 which can forward the translate_exception | 13:43 |
| Haomeng|2 | lucasagomes: so any ideas about our api exception translation enablement solutions? | 13:44 |
| lucasagomes | Haomeng|2, hey buddy, 1sec I will read it in 10 min (I'm on a hangout right now) | 13:45 |
| Haomeng|2 | lucasagomes: no rush, thanks | 13:45 |
| Haomeng|2 | lucasagomes: when you have time, leave offline words to me:) | 13:46 |
| Haomeng|2 | lucasagomes: :) | 13:47 |
| *** mdbooth has joined #openstack-ironic | 13:56 | |
| mdbooth | I just posted this on #openstack-nova, but I'll repeat it here | 13:57 |
| mdbooth | grep suggests that HostMaanger.service_states is never set, except to {}. This means that capabilities in get_all_host_states will always be None. That seems to be the only place a host state is created, which means that the capabilities passed to new_host_state in baremetal_host_manager will always be None, which means they will never contain 'baremetal_driver', which means that a BaremetalNodeState will never | 13:57 |
| mdbooth | be instantiated | 13:58 |
| mdbooth | It's quite possible I missed something in the above chain | 13:58 |
| mdbooth | However, if I didn't this might have some interesting implications | 13:58 |
| mdbooth | Potentially including the scheduler attempting to put more than 1 instance on a single baremetal host | 13:59 |
| Haomeng|2 | lucasagomes: the point is that we need to take back the error - pecan.response.translatable_error = error before any exception raised during api call | 14:07 |
| Haomeng|2 | lucasagomes: I will be away, leave ideas here and I will check on my Saturday morning, nice weekend:) | 14:09 |
| NobodyCam | Good Morning Ironic and TGIF!!! | 14:10 |
| romcheg | Morning NobodyCam | 14:12 |
| *** derekh has quit IRC | 14:13 | |
| NobodyCam | Good Morning romcheg :) | 14:13 |
| Mikhail_D_wk | NobodyCam: morning! :) | 14:13 |
| NobodyCam | morning Mikhail_D_wk :) | 14:13 |
| *** jgrimm has joined #openstack-ironic | 14:15 | |
| *** jdob_ has joined #openstack-ironic | 14:16 | |
| *** linggao has quit IRC | 14:20 | |
| *** linggao has joined #openstack-ironic | 14:20 | |
| * Shrews pokes NobodyCam in the eye with an anchovie | 14:22 | |
| NobodyCam | lol | 14:27 |
| NobodyCam | huh | 14:27 |
| NobodyCam | heheheheh | 14:27 |
| NobodyCam | good morning Shrews :) | 14:28 |
| * NobodyCam wounders why Shrews has anchovie's | 14:29 | |
| jroll | morning ironic | 14:29 |
| NobodyCam | good morning jroll :) | 14:29 |
| *** jdob_ has quit IRC | 14:30 | |
| yuriyz | morning/evening all | 14:30 |
| NobodyCam | morning yuriyz :) | 14:30 |
| *** saju_m has joined #openstack-ironic | 14:31 | |
| *** ndipanov_ has quit IRC | 14:32 | |
| jroll | NobodyCam, devananda: would love to hear your thoughts on vkozhukalov's email about the agent and rootwrap | 14:32 |
| NobodyCam | jroll: just saw it... | 14:33 |
| NobodyCam | need coffee ... | 14:33 |
| jroll | no rush, I'm just checking in on things before heading to the office :) | 14:33 |
| NobodyCam | :) | 14:34 |
| jroll | and coffee+++++++ | 14:34 |
| *** saju_m has quit IRC | 14:34 | |
| *** saju_m has joined #openstack-ironic | 14:35 | |
| lucasagomes | morning NobodyCam yuriyz jroll | 14:38 |
| NobodyCam | lucasagomes: your testing failed with https://review.openstack.org/#/c/83471 too.. did you get a chance to look at why yet? | 14:38 |
| NobodyCam | and good morning :) | 14:38 |
| lucasagomes | Haomeng|2, right so yeah we have a common handle to catch the exceptions in the api | 14:39 |
| lucasagomes | Haomeng|2, but that lives in the wsme code | 14:39 |
| lucasagomes | Haomeng|2, https://github.com/stackforge/wsme/blob/master/wsmeext/pecan.py#L74-L107 | 14:39 |
| jroll | morning lucas :) | 14:39 |
| *** jistr is now known as jistr|biab | 14:40 | |
| lucasagomes | Haomeng|2, basically this wsexpose decorator will caputre all the exception, serialize them and return as a response :/ | 14:40 |
| lucasagomes | NobodyCam, oh not yet, I had too meetings in a row today | 14:40 |
| lucasagomes | NobodyCam, didn't have much time, I will take a look | 14:40 |
| lucasagomes | two* | 14:41 |
| NobodyCam | was just checking | 14:41 |
| lucasagomes | Haomeng|2, so pecan does have a hook called on_error() which would allow u to handle the error message | 14:42 |
| *** mkerrin has quit IRC | 14:42 | |
| *** dwalleck has quit IRC | 14:42 | |
| lucasagomes | Haomeng|2, but when used with wsme, this hook never gets trigged :( I opened a bug about it awhile ago https://bugs.launchpad.net/wsme/+bug/1256042 | 14:42 |
| lucasagomes | it's confirmed but there's no fix for that yet | 14:42 |
| lucasagomes | :/ | 14:43 |
| lucasagomes | we might need to fix that in wsme before | 14:43 |
| lucasagomes | Haomeng|2, I see there's a guy assigned to that bug, maybe worth talking to him | 14:43 |
| *** mdbooth has left #openstack-ironic | 14:49 | |
| *** lsmola_ has quit IRC | 14:56 | |
| NobodyCam | hummm setting debug=True in our conf file didn't seem to give me the logging I was looking for... looks in why | 14:58 |
| Shrews | NobodyCam: what output are you looking for? | 15:00 |
| Shrews | NobodyCam: my log cleanup change merged today. wondering if it's related | 15:01 |
| *** ewindisch has quit IRC | 15:04 | |
| openstackgerrit | Chris Krelle proposed a change to openstack/ironic: Add INFO level logging to ssh.py https://review.openstack.org/85124 | 15:05 |
| devananda | g'morning, all | 15:06 |
| devananda | jroll: simply, "no." | 15:06 |
| NobodyCam | Shrews: nope. my log had no debug in it at all | 15:06 |
| NobodyCam | good morning devananda :) | 15:06 |
| *** ewindisch has joined #openstack-ironic | 15:07 | |
| devananda | jroll: i have said since the project started, ironic's responsibility ends where the host OS begins | 15:07 |
| jroll | devananda: +1 | 15:07 |
| jroll | devananda: figured you were on board with that sentiment, but wanted to double check | 15:07 |
| * devananda sits down with a latte and a GF muffin | 15:08 | |
| NobodyCam | :) | 15:08 |
| NobodyCam | devananda: which office are you in this morning | 15:08 |
| devananda | NobodyCam: Victrola on cap hill | 15:09 |
| devananda | NobodyCam: if by "office" you mean which cafe | 15:09 |
| NobodyCam | :) oh that's a nice one... | 15:09 |
| NobodyCam | :) hear they have really good coffee there | 15:09 |
| NobodyCam | :) | 15:09 |
| lucasagomes | morning devananda | 15:10 |
| devananda | yep | 15:10 |
| devananda | afternoon, lucasagomes ! | 15:10 |
| NobodyCam | lucasagomes: https://review.openstack.org/#/c/83471 just deployed for me. but I can not atest to what my test env is atm... so I am going to rebuild and test again | 15:13 |
| lucasagomes | NobodyCam, ack I'm finishin one patch and then I will give it a go here as well | 15:13 |
| *** dwalleck has joined #openstack-ironic | 15:22 | |
| *** ilives has quit IRC | 15:25 | |
| openstackgerrit | Dmitry Tantsur proposed a change to openstack/ironic: Implement caching for master images https://review.openstack.org/85387 | 15:25 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Add a blacklist mechanism for drivers https://review.openstack.org/85388 | 15:27 |
| openstackgerrit | Vladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute util https://review.openstack.org/85344 | 15:27 |
| openstackgerrit | Vladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute util https://review.openstack.org/85344 | 15:28 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Add a blacklist mechanism for drivers https://review.openstack.org/85388 | 15:29 |
| dtantsur | lucasagomes, may I reupload https://review.openstack.org/#/c/84396/1 with offset = 1? That will save you a tiny bit of time, I guess :) | 15:29 |
| lucasagomes | dtantsur, <slaps my head> yeah if u can do that would help | 15:29 |
| lucasagomes | sorry forgot :/ | 15:29 |
| dtantsur | ack | 15:29 |
| lucasagomes | I'm using the other patch in my env | 15:29 |
| NobodyCam | :-p | 15:30 |
| NobodyCam | is that going to be backported? | 15:30 |
| lucasagomes | heh we don't know yet | 15:31 |
| lucasagomes | possible | 15:31 |
| lucasagomes | I will mark it as WIP after dtantsur fix it | 15:31 |
| NobodyCam | if we're not going to BP it.. I think the other set of parted patches is what we should loand | 15:32 |
| NobodyCam | land even | 15:32 |
| openstackgerrit | Dmitry Tantsur proposed a change to openstack/ironic: Replace sfdisk with parted https://review.openstack.org/84396 | 15:33 |
| devananda | ya'll might want to see a recent email to -dev by Matt Booth | 15:33 |
| devananda | we just talked in -nova and it looks like he may be right | 15:33 |
| dtantsur | NobodyCam, I'm not sure that without this landed partitioning will work... | 15:33 |
| NobodyCam | ack | 15:34 |
| NobodyCam | devananda: link by chance? | 15:34 |
| *** ifarkas has quit IRC | 15:35 | |
| lucasagomes | devananda, about the host_manager yeah I saw | 15:35 |
| lucasagomes | didn't reply yet tho | 15:35 |
| *** saju_m has quit IRC | 15:35 | |
| NobodyCam | reading now | 15:35 |
| dtantsur | lucasagomes, the side effect of my upload is that it assigned bug on me. Feel free to assign back on you | 15:38 |
| lucasagomes | dtantsur, heh ack will do | 15:40 |
| lucasagomes | dtantsur, thanks for the fix :D | 15:41 |
| dtantsur | lucasagomes, np :) | 15:41 |
| *** martyntaylor has left #openstack-ironic | 15:42 | |
| NobodyCam | i am being told that it is bbt... lol... so I will brb.. | 15:51 |
| devananda | so, i just had a terrible idea | 15:54 |
| devananda | and i'd like someone to tell me why it's bad | 15:54 |
| devananda | (you know, to convince me it's actually as bad as I think it is) | 15:54 |
| *** coolsvap has joined #openstack-ironic | 15:54 | |
| *** jistr|biab is now known as jistr | 15:55 | |
| devananda | what if the deploy ramdisk used $(insert favorite configuration management service) | 15:55 |
| openstackgerrit | A change was merged to openstack/ironic: Refactor nova.virt.ironic.driver get_host_stats https://review.openstack.org/83853 | 15:55 |
| devananda | for the hardware config bits | 15:55 |
| NobodyCam | humm | 15:57 |
| *** Mikhail_D_ltp has joined #openstack-ironic | 15:58 | |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Add a blacklist mechanism for drivers https://review.openstack.org/85388 | 15:59 |
| NobodyCam | devananda: my consern with $(insert favorite configuration management service here) is that is more the configuration side and not deployment | 16:01 |
| lucasagomes | devananda, what are the hardware config bits? | 16:02 |
| devananda | flash firmware, change bios, build raid | 16:02 |
| devananda | that sort of thing | 16:02 |
| NobodyCam | i can see how it would be helpful for us | 16:02 |
| NobodyCam | but would be a fine line | 16:02 |
| devananda | CMs already have tooling for doing hw config, right? the IPA folks are creating tooling for doing hw config, too. | 16:03 |
| *** eghobo has joined #openstack-ironic | 16:04 | |
| devananda | why are we recreating it? why not just reuse it? | 16:04 |
| * lucasagomes don't know whether CMs does have it or not | 16:04 | |
| lucasagomes | but if they do sounds like a good plan | 16:05 |
| devananda | jroll: g'morning! | 16:05 |
| lucasagomes | one problem is, it's a ramdisk | 16:05 |
| comstud | Mikhail_D_wk: I am now :) | 16:05 |
| lucasagomes | so! unless you want to do an image that uses squahfs or something like that | 16:05 |
| lucasagomes | it's going to run everythin on the memory | 16:05 |
| lucasagomes | squashfs | 16:05 |
| devananda | lucasagomes: i don't see that as a problem | 16:05 |
| *** comstud is now known as bearhands | 16:06 | |
| *** russellb is now known as rustlebee | 16:06 | |
| devananda | lucasagomes: as it is, IPA is fairly large. they are chaining into iPXE and fetching the image over HTTP(S) | 16:06 |
| lucasagomes | it's not if you have enough memory available | 16:06 |
| NobodyCam | fyi lucasagomes 83471 is working for me now. | 16:06 |
| lucasagomes | devananda, yeah its required to be http when transferring the image | 16:07 |
| lucasagomes | I'm thinking about it running, and depending on idk 2GB ram to run a ramdisk | 16:07 |
| lucasagomes | which might be fine :/ | 16:07 |
| NobodyCam | what do the gate vm's have for memory | 16:08 |
| devananda | NobodyCam: 8 today | 16:09 |
| lucasagomes | devananda, well yeah doesn't sounds like a terrible idea :) | 16:11 |
| lucasagomes | I'm not expert in CMs tho | 16:11 |
| lucasagomes | but at a first glance sounds good, not reinventing the wheel | 16:12 |
| NobodyCam | as long as we stay out of the use salt, no use puppet, no no use chef, no no no use CF_engine wars | 16:13 |
| lucasagomes | yeah | 16:13 |
| devananda | heh | 16:13 |
| lucasagomes | we can have a pluggable design and support one of them | 16:13 |
| devananda | lucasagomes: you're supposed to tell me why it's bad | 16:13 |
| devananda | :) | 16:13 |
| lucasagomes | devananda, :P I tried | 16:13 |
| NobodyCam | what kinda size wouldwe be adding to the ramdisk for this... | 16:14 |
| NobodyCam | picking salt as an example only | 16:14 |
| NobodyCam | how much blot would be need to carry | 16:15 |
| devananda | dunno | 16:15 |
| NobodyCam | lol google keeps taking me to http://www.saltstack.com | 16:17 |
| NobodyCam | lol | 16:17 |
| bearhands | i'm guessing when teeth team wakes up, we'll be able to confirm how bad we think it is | 16:17 |
| bearhands | :) | 16:17 |
| * bearhands rubs his eyes. | 16:17 | |
| NobodyCam | bearhands: watch your eye's Shrews is poking people with anchovie's | 16:18 |
| NobodyCam | :-p | 16:18 |
| openstackgerrit | Vladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute util https://review.openstack.org/85344 | 16:18 |
| lucasagomes | heh | 16:19 |
| bearhands | hah | 16:19 |
| devananda | lucasagomes: https://review.openstack.org/#/c/85044/2 -- did you test this before the patch? I thought it worked fine | 16:19 |
| bearhands | devananda, lucasgomes: The format before the patch should work fine | 16:20 |
| bearhands | and actually tends to be what we prefer in nova | 16:20 |
| devananda | yea | 16:20 |
| bearhands | there's a speciifc reason why too, but I don't recall it | 16:20 |
| devananda | parse time interpolation | 16:20 |
| devananda | vs run time | 16:21 |
| devananda | iirc | 16:21 |
| bearhands | oh | 16:21 |
| devananda | er, not parse time | 16:21 |
| bearhands | actually, that's broken, i guess | 16:21 |
| devananda | wrong word | 16:21 |
| bearhands | it's msg = | 16:21 |
| bearhands | not in the LOG() | 16:21 |
| devananda | ahh yea. that's wrong | 16:21 |
| bearhands | LOG(msg, {}) is what is fine | 16:21 |
| bearhands | hehe | 16:21 |
| devananda | it should be msg=_(), LOG(msg, {...}) | 16:22 |
| devananda | right | 16:22 |
| bearhands | generally, except it's used for last_error here | 16:23 |
| bearhands | the 2nd fix in this patch is correct | 16:23 |
| *** matty_dubs is now known as matty_dubs|lunch | 16:23 | |
| bearhands | (as well) | 16:23 |
| lucasagomes | devananda, I think the "," can be used with logging | 16:24 |
| lucasagomes | but not with msg = | 16:24 |
| * bearhands throws a +1 on | 16:24 | |
| lucasagomes | I did a quick testing | 16:24 |
| * lucasagomes pastes | 16:24 | |
| openstackgerrit | Aleksandr Gordeev proposed a change to openstack/ironic-python-agent: Add timeout param for execution_thread.join https://review.openstack.org/85411 | 16:24 |
| openstackgerrit | Aleksandr Gordeev proposed a change to openstack/ironic-python-agent: Add FlowExtension https://review.openstack.org/85412 | 16:24 |
| lucasagomes | devananda, http://paste.openstack.org/show/75094/ | 16:24 |
| bearhands | lucasagomes: yes | 16:24 |
| bearhands | it was creating a tuple before | 16:24 |
| lucasagomes | exactly | 16:24 |
| russell_h | JoshNang: at risk of contradicting my own code, on 84303 IMO there should be only one jitter parameter | 16:26 |
| lucasagomes | NobodyCam, 83471 worked for me as well | 16:26 |
| NobodyCam | lucasagomes: I have +2'd it | 16:26 |
| lucasagomes | [stack@localhost devstack]$ pip list | grep keystoneclient | 16:26 |
| lucasagomes | python-keystoneclient (0.7.1.18.gb6cdfff, /opt/stack/python-keystoneclient) | 16:26 |
| lucasagomes | and I deployed a machine | 16:26 |
| Shrews | NobodyCam: I'm too busy obtaining lunch to poke any eyes with tasty dish right now. :) | 16:26 |
| lucasagomes | right I will remove my -1 | 16:26 |
| Shrews | fish, too | 16:27 |
| devananda | lucasagomes: oi ... | 16:27 |
| devananda | adam_g: "from nova.objects.flavor import Flavor as flavor_obj" ? | 16:27 |
| JoshNang | russell_h: would it still use a random number related to the jitter then? like a range around the jitter number? | 16:27 |
| russell_h | JoshNang: something similar to this: http://twistedmatrix.com/trac/browser/tags/releases/twisted-13.2.0/twisted/internet/protocol.py#L330 | 16:27 |
| NobodyCam | Shrews: LOL :) | 16:27 |
| russell_h | JoshNang: http://twistedmatrix.com/trac/browser/tags/releases/twisted-13.2.0/twisted/internet/protocol.py#L398 | 16:27 |
| lucasagomes | devananda, heh oi == hi, in portuguese | 16:27 |
| russell_h | I dont' even know what normalvariate does | 16:27 |
| pquerna | russell_h: it does math. | 16:28 |
| russell_h | exactly | 16:28 |
| JoshNang | ah that is much cleaner | 16:28 |
| pquerna | russell_h: http://golang.org/src/pkg/math/rand/normal.go | 16:28 |
| bearhands | haha | 16:28 |
| bearhands | http://en.wikipedia.org/wiki/Normal_distribution | 16:29 |
| bearhands | i hate math | 16:29 |
| russell_h | JoshNang: pquerna is right, you should rewrite the agent in go | 16:30 |
| JoshNang | hackday project! | 16:30 |
| russell_h | var fn = [128]float32 | 16:30 |
| russell_h | makes sense | 16:30 |
| russell_h | pquerna: pretty sure these constants were generated with a python script | 16:31 |
| *** romcheg has quit IRC | 16:31 | |
| openstackgerrit | Vladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute util https://review.openstack.org/85344 | 16:31 |
| pquerna | russell_h: http://hg.python.org/cpython/file/2.7/Lib/random.py#l381 <- probaly | 16:31 |
| jroll | morning devananda and ironic | 16:36 |
| bearhands | devananda: thoughts on https://review.openstack.org/#/c/84862/1/ironic/conductor/manager.py,unified ? | 16:41 |
| bearhands | devananda: if this is the case, there's actually an existing bug here | 16:41 |
| bearhands | (race in syncing power states grabbing lock vs deploy wanting lock) | 16:42 |
| *** yuriyz has quit IRC | 16:43 | |
| NobodyCam | devananda: just lost connection | 16:44 |
| NobodyCam | he'll be bck in 1/2 an hour or so | 16:44 |
| bearhands | ok | 16:44 |
| JayF | I really don't think config management software overlaps much if at all with what IPA is doing. Most of what I've used config management for, and what it's built to do, is maintain configs over a period of time | 16:44 |
| JayF | and is usually used to manage a 'host OS' rather than managing the hardware itself | 16:44 |
| bearhands | that was my basically my reaction.. I'm not sure I've seen it used to manage hardware in this way | 16:47 |
| bearhands | i'm not sure i view "firmware" and "config" as the same thing. | 16:48 |
| bearhands | :) | 16:48 |
| JayF | +9001 | 16:48 |
| jroll | that number brings back memories of ISO certification | 16:49 |
| * jroll shudders | 16:49 | |
| JayF | jroll: the point was more that it's over 9000 | 16:50 |
| jroll | sure | 16:50 |
| jroll | you haven't seen the horrors that I've seen :P | 16:50 |
| *** jistr is now known as jistr|afk | 16:58 | |
| *** harlowja has joined #openstack-ironic | 17:04 | |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic-python-agent: Add BackOffLoopingCall with jitter https://review.openstack.org/84303 | 17:05 |
| *** matty_dubs|lunch is now known as matty_dubs | 17:11 | |
| *** lucasagomes is now known as lucas-afk | 17:25 | |
| *** romcheg has joined #openstack-ironic | 17:29 | |
| *** lazy_prince has quit IRC | 17:32 | |
| NobodyCam | quick walkies time... brb | 17:34 |
| *** romcheg has quit IRC | 17:43 | |
| *** romcheg has joined #openstack-ironic | 17:49 | |
| devananda | back | 17:49 |
| devananda | that walk took longer than i expected, but it's beautiful outside, so i'm not complaining | 17:50 |
| *** romcheg has quit IRC | 17:50 | |
| bearhands | devananda: /win 38 | 17:58 |
| bearhands | oops | 17:58 |
| bearhands | lol, not exactly what i meant | 17:58 |
| bearhands | i have no idea what your window 38 is | 17:58 |
| *** openstackgerrit has quit IRC | 18:01 | |
| *** openstackgerrit has joined #openstack-ironic | 18:02 | |
| bearhands | devananda: so, i have a question regarding do_node_deploy() grabbing the lock in conductor | 18:06 |
| bearhands | devananda: periodic tasks could have the node locked when it is called. it seems like maybe it should retry, or we need some additional synchronization or something. | 18:07 |
| *** Mikhail_D_ltp has quit IRC | 18:07 | |
| *** romcheg has joined #openstack-ironic | 18:08 | |
| *** romcheg has quit IRC | 18:08 | |
| bearhands | devananda: additionally, the sync power states periodic task tries to not lock if in DEPLOYWAIT, but there's a race there as well with a callback trying to lock | 18:08 |
| bearhands | The latter can be solved if we put some optional additional constraints on the reserve_nodes DB call | 18:09 |
| adam_g | devananda, from nova.objects.flavor import Flavor as flavor_obj <- a change in nova requires this. similar to https://review.openstack.org/#/c/71364/ | 18:09 |
| bearhands | devananda: thoughts? | 18:10 |
| adam_g | actually https://review.openstack.org/#/c/78686/ | 18:10 |
| devananda | adam_g: ah, thanks | 18:12 |
| devananda | bearhands: there are probably some races in there which you're finding... we definitely need better retry in the clients | 18:12 |
| bearhands | it seems like this should be on the server side | 18:13 |
| bearhands | after all, it's periodic tasks causing this not 2 clients talking simultaneously | 18:13 |
| bearhands | (i'm not sure that retries are the right solution server side -- one of these can be done with better constraints on the DB call that reserves) | 18:14 |
| bearhands | i could see maybe deploy failing because the node is locked being acceptable | 18:15 |
| bearhands | although it's kinda crappy because it's just a periodic task trying to do a power state sync | 18:15 |
| bearhands | best solution is if we can avoid locking in the periodic task | 18:17 |
| devananda | so from client's perspective, what's the difference if | 18:17 |
| devananda | request fails because another client already locked the node | 18:17 |
| devananda | or server locked the node for some maintenance task? | 18:17 |
| bearhands | i guess it would be 'maintenance', but yeah | 18:17 |
| bearhands | for something extremely simple like syncing power state | 18:17 |
| bearhands | But also.. what makes the callback in a DEPLOYWAIT state.. is it the client? | 18:18 |
| bearhands | I view the callback case for DEPLOYWAIT is worse than just a new deploy | 18:18 |
| bearhands | is/as | 18:19 |
| devananda | deploy driver may set state to deploywait if it depends on an agent to finish the work | 18:20 |
| devananda | agent then POSTs back to initiate the deploy driver continuing the work | 18:20 |
| bearhands | so the agent is the client in that case | 18:20 |
| devananda | yea | 18:20 |
| bearhands | This has come up because of that extra DB query we're doing in _sync_power_states | 18:22 |
| bearhands | I put up a review to remove it, however, it makes the race condition potential larger | 18:22 |
| bearhands | https://review.openstack.org/#/c/84862/1/ironic/conductor/manager.py,unified | 18:22 |
| bearhands | well, larger for the DEPLOYWAIT case. although this is the case that can be solved by adding a constraint to the reserve_nodes DB call to say.. 'only lock if provision_state != DEPLOYWAIT' | 18:26 |
| *** vkozhukalov has quit IRC | 18:30 | |
| *** jistr|afk has quit IRC | 18:32 | |
| *** romcheg has joined #openstack-ironic | 18:38 | |
| *** coolsvap has quit IRC | 18:42 | |
| NobodyCam | ahh ha! | 18:51 |
| * NobodyCam thinks he may have found his logging issue. | 18:51 | |
| *** overlayer has quit IRC | 18:54 | |
| *** rwsu has quit IRC | 18:57 | |
| *** dwalleck has quit IRC | 19:00 | |
| *** dwalleck has joined #openstack-ironic | 19:08 | |
| *** rwsu has joined #openstack-ironic | 19:13 | |
| *** romcheg has quit IRC | 19:14 | |
| *** max_lobur1 has quit IRC | 19:22 | |
| *** praefect has joined #openstack-ironic | 19:24 | |
| *** mgagne1 has joined #openstack-ironic | 19:24 | |
| *** mgagne1 is now known as mgagne | 19:25 | |
| NobodyCam | devananda: fyi: https://review.openstack.org/85455 | 19:27 |
| praefect | Hi guys, there's a bit of confusion here regarding the future of ironic, is it suppose to become a full-fledge baremetal system that could be used to sell baremetal to clients in a provider environment or will it become a bare metal provisioning system geared more towards undercloud provisioning..? I don't wanna get all philosophical here but I'd really like to know.. | 19:35 |
| jroll | praefect: yes. :) | 19:35 |
| praefect | I was in HK summit and I think I've heard stuff there that might be responsible for my confusion.. | 19:36 |
| praefect | jroll: thanks, that's what I was looking for =) | 19:36 |
| jroll | :) | 19:36 |
| praefect | seriously is it still a question you guys are pondering? | 19:36 |
| jroll | to clarify, there is a ton of things to work on and consider for real multi-tenant deployments | 19:37 |
| praefect | jroll: any pointer as to where I could read on these obstacles? the only problem I remember is the possibility for a client to poison the BIOS and the resulting security implications | 19:40 |
| jroll | yes, there's that, as well as poisoning firmware on other devices (e.g. disks) | 19:42 |
| jroll | and networking is hard (TM) | 19:43 |
| jroll | I'm not sure if there's anything formal written up | 19:43 |
| praefect | jroll: thanks | 19:43 |
| NobodyCam | praefect: het | 19:43 |
| NobodyCam | hey | 19:43 |
| NobodyCam | even | 19:43 |
| NobodyCam | full-fledge baremetal system ?? not sure I get that | 19:44 |
| jroll | praefect: of course :) | 19:44 |
| praefect | NobodyCam: I mean, not just to build an openstack cloud from metal but more to rent the resulting bare metal server to a client... | 19:44 |
| NobodyCam | but there are many many security concerns with muli users on barematel | 19:45 |
| jroll | NobodyCam: we were just talking about this :) | 19:45 |
| NobodyCam | :) | 19:46 |
| NobodyCam | just so many explots that baremetal is up agenst | 19:46 |
| jroll | yeah, it depends a lot on operational things | 19:46 |
| praefect | I'm not security guys but beside the firmware situation (which could be fixed by reflashing the BIOS in the disk scrubbing process...) I don't see many other security issues, but jroll is right, it depends on operational things... if you end up with a baremetal in its own VLAN and no access to any iscsi target.. then it's safe (I think) | 19:47 |
| NobodyCam | praefect: no it can not | 19:48 |
| NobodyCam | I can write a FW that tells you it is flashing while bit bucketing any data you send to it | 19:48 |
| mgagne | NobodyCam: how is it different from what thousands of providers around the world has been doing for years, renting baremetal servers. Is it the ability to massively compromise/rootkit all servers of a provider in a short period of time? | 19:48 |
| jroll | NobodyCam: what if your BIOS only accepts signed updates? :) | 19:49 |
| NobodyCam | I'll use the nic cards, or the usb port, or the fan controller | 19:49 |
| jroll | praefect: btw, you might be interested to hear that we're working on an agent-based deployment method that can do things like firmware updates: https://wiki.openstack.org/wiki/Ironic-python-agent | 19:50 |
| jroll | NobodyCam: right right | 19:50 |
| JayF | I think the answer is; just like with dedicated server hosting that /isn't/ automated, there's a heck of a lot of attack vectors to think abuot and mitigate. Having software do that isn't any more or less scary than having humans do it. | 19:50 |
| *** dkehn_ has joined #openstack-ironic | 19:50 | |
| JayF | Crytography + signed updates helps a ton though | 19:50 |
| jroll | NobodyCam: we went through this at the mid-cycle already :) | 19:50 |
| NobodyCam | heheheh | 19:51 |
| NobodyCam | jroll: your laptop got a mic... https://www.security.nl/posting/366329/Onderzoeker+ontdekt+mysterieuze+BIOS-malware <- ultra sonic virus | 19:52 |
| jroll | yeah, security is hard | 19:53 |
| JayF | everything is hard. Security is hard and dangerous ;) | 19:53 |
| NobodyCam | hummm may just have to attend http://openstacksummitmay2014atlanta.sched.org/event/754c3678d31f9f74e020b9a1e6f4dece#.Uz8N6tzL8Xc | 19:54 |
| *** dkehnx has quit IRC | 19:54 | |
| *** eguz has joined #openstack-ironic | 19:54 | |
| jroll | JayF: idk, this chair is pretty soft | 19:54 |
| jroll | so, clearly not everything is hard | 19:55 |
| jroll | NobodyCam: you should, that's russell_h speaking | 19:55 |
| NobodyCam | JayF: just so many attack vectors on baremetal | 19:55 |
| praefect | will definitely attend... | 19:56 |
| NobodyCam | I suspose one could deply the end user in a lxc container as the only tenant on the box. | 19:56 |
| jroll | that still leaves you open to kernel exploits, AIUI | 19:58 |
| NobodyCam | but all that said .. the HW mfg's are dealing with many of these issuses as we type. | 19:58 |
| jroll | yep :) | 19:58 |
| JayF | NobodyCam: does HP do signed firmware/bios updates yet? | 19:58 |
| *** eghobo has quit IRC | 19:58 | |
| NobodyCam | we support currently and i think is going to be default soon | 19:59 |
| JayF | nice | 19:59 |
| *** dwalleck has quit IRC | 20:00 | |
| NobodyCam | ofc tat depends on what gen servers your running | 20:00 |
| JayF | I was just asking generally, i.e. if stuff shipping today did | 20:01 |
| jbjohnso_ | incidently, that presumably applies to your core firmware and such, I don't imagine you have all firmware updates similarly protected | 20:03 |
| NobodyCam | supports it but you have check the enable box I believe... (don't hold me to that) I have not looked at what is currently shipping inthe server line :-p | 20:03 |
| JayF | It's fine, was just a general question :) | 20:04 |
| jbjohnso_ | e.g. IBM servers now always have signed firmware, no option, but if you pop a network card in, that firmware cannot be reasonably protected | 20:04 |
| NobodyCam | yep | 20:04 |
| *** vkozhukalov has joined #openstack-ironic | 20:04 | |
| jbjohnso_ | that's one of the challenges of baremetal, even as the core platform is intact, there are still lots of potentially sneaky places for stuff to hide | 20:05 |
| *** dwalleck has joined #openstack-ironic | 20:06 | |
| jbjohnso_ | I personally am doubtful the x86 space can be safe for untrusted baremetal tenants without losing the soul of the x86 space | 20:09 |
| jbjohnso_ | E.g. secureboot hands the keys of the kingdom to MS, who in turn is signing shims they can't possibly realistically validate the full stack of | 20:09 |
| jbjohnso_ | so you have all the inconvenience of a security attempt with a pretty high likelihood of it being circumventable | 20:10 |
| russell_h | jbjohnso_: I mean, it doesn't need to involve MS | 20:10 |
| NobodyCam | i dont want my computer to phone home jsut to boot up! | 20:10 |
| russell_h | jbjohnso_: but yeah, someone needs to sign things and its difficult to validate everything | 20:10 |
| russell_h | jbjohnso_: I don't see this as unique to hardware though. If a determined attacker with time and resources _wants_ to pwn you, you're going to get pwned | 20:11 |
| russell_h | like if you buy a server, install ubuntu on it and run a java app | 20:12 |
| NobodyCam | russell_h: ++ so true | 20:12 |
| russell_h | I could compromise your hardware manufacturer, who probably emails firmwares about internally for signing | 20:12 |
| jbjohnso_ | russell_h, yeah, though a software environment has some pretty well defined 'start from scratch' state | 20:12 |
| russell_h | I could compromise your OS at any of a hundred spots | 20:12 |
| russell_h | I could compromise the JVM | 20:12 |
| russell_h | I could compromise a maven server (afaik most maven packages are still unsigned) | 20:13 |
| jbjohnso_ | russell_h, for server hardware/firmware, it theoretically also exists, but practically speaking the number of non-volatile storage places | 20:13 |
| russell_h | and so on | 20:13 |
| jbjohnso_ | is not as well characterized | 20:13 |
| russell_h | yeah | 20:13 |
| russell_h | I guess I see it as more that hardware companies are still a bit ghettoer in terms of how they handle software development, packaging and distribution | 20:14 |
| jbjohnso_ | and trying to bring all those under some 'trusted' authority while at the same time that authority meaningfully auditing the ecosystem is a pretty big beast for something like x86 | 20:14 |
| *** dwalleck has quit IRC | 20:14 | |
| JayF | Well you have to do it p2p, not centralized style | 20:14 |
| JayF | where you choose what entities you trust rather than having that selected for me | 20:15 |
| JayF | for instance, NobodyCam said HP signs firmwares. To accept that as secure, I have to trust HP | 20:15 |
| JayF | or with MS secureboot, you trust MS to keep the keys secure, etc | 20:15 |
| jbjohnso_ | right, but the x86 ecosystem gets convoluted | 20:15 |
| russell_h | right | 20:15 |
| jbjohnso_ | so you trust HP, but you have a QLogic SoC in there as part of an adapter | 20:15 |
| russell_h | really the problem is that to successfully boot a computer, I need a bunch of hardware and firmware | 20:16 |
| JayF | So what you're saying is, don't buy that cheap Intel NIC being sold on eBay from SleeperAgent34? | 20:16 |
| russell_h | and realistically I have no way to actually validate most of it | 20:16 |
| russell_h | and even if HP, for example, isn't actively backdooring their gear | 20:16 |
| russell_h | someone is probably doing it for them at the factory | 20:16 |
| jbjohnso_ | right. Secureboot made the fundamental mistake of having the firmware hard-bake one vendor key rather than having a vendor claim it at install time | 20:16 |
| jbjohnso_ | so I get the decentralized thing, but there's a lot of entry points to cover... it's a massive ecosystem that in order to work was permissive by default for decades | 20:17 |
| jbjohnso_ | not in terms of network access, but in terms of things like PCI conversations, SMBus stuff, all sorts of nifty things | 20:18 |
| russell_h | and HP is probably like the 4th most trustworthy of a hundred entities in whom I'm placing similar trust | 20:18 |
| jbjohnso_ | yeah, though it's also true that being a tenant on a baremetal system is roughly like buying used server equipment | 20:18 |
| jbjohnso_ | both can hypothetically be trojaned in a way a user is unlikely to detect | 20:19 |
| russell_h | eh, I'd argue that being a tenant on a baremetal system is roughly like using any other computer in the world | 20:20 |
| russell_h | in that you certainly could be compromised if someone cares enough to do so, but you probably aren't | 20:20 |
| russell_h | like its not like the NSA is going to go pwning some baremetal thing | 20:20 |
| russell_h | but throw up their hands when you buy a brand new server | 20:20 |
| jbjohnso_ | well, NSA and such are one thing | 20:21 |
| jbjohnso_ | they can get in to the supply chain | 20:21 |
| jbjohnso_ | there are a class of potential attackers for whom supply chain isn't as feasible | 20:21 |
| russell_h | yeah, but so can most anyone else | 20:21 |
| russell_h | sure, I mean I guess its fair that there are a ton of people who can't reasonably attack the supply chain | 20:21 |
| jbjohnso_ | well, I personally for example wouldn't be able to infect HP systems off their manufacturing line | 20:22 |
| jbjohnso_ | it's a matter of risk mitigation rather than elimination in that case | 20:22 |
| *** dwalleck has joined #openstack-ironic | 20:23 | |
| russell_h | fair | 20:24 |
| russell_h | the real problem, to me, is that at some point you're going to put software on your box | 20:25 |
| russell_h | and if someone can't compromise you at a lower level, they still have plenty of surface area left to focus on | 20:25 |
| jbjohnso_ | that is true | 20:25 |
| russell_h | its actually absurd that (we believe) firmware is a larger surface area than software | 20:26 |
| jbjohnso_ | but at least on the higher levels, you can proverbially throw the disks out and start over | 20:26 |
| jbjohnso_ | but of course that relies upon the assertion that you even know there is a problem | 20:26 |
| russell_h | right | 20:26 |
| russell_h | and that you have a way to get clean software when you start over | 20:27 |
| jbjohnso_ | I think it's not that the surface area is perceived as 'larger', just that it is perceived as a bit sneakier | 20:27 |
| russell_h | my take, we just need to drag hardware into the present so we can focus on interesting problems | 20:27 |
| russell_h | well, yeah, and its a lot less transparent | 20:27 |
| russell_h | like its not like I can compile my own firmware for most gear | 20:28 |
| *** praefect has quit IRC | 20:31 | |
| openstackgerrit | Jay Faulkner proposed a change to openstack/ironic-python-agent: Make tests pass for Python 3.3 https://review.openstack.org/85481 | 20:32 |
| *** romcheg has joined #openstack-ironic | 20:33 | |
| *** romcheg has quit IRC | 20:35 | |
| NobodyCam | humm | 20:35 |
| russell_h | JayF: isn't that going to break the global requirements stuff? | 20:38 |
| openstackgerrit | Jay Faulkner proposed a change to openstack/ironic-python-agent: Make tests pass for Python 3.3 https://review.openstack.org/85481 | 20:40 |
| *** dwalleck has quit IRC | 20:41 | |
| JayF | russell_h: global reqs want eventlet=>0.13, which this technically would be | 20:41 |
| JayF | russell_h: but if you look I commented we might not wanna merge it yet for that reason | 20:41 |
| *** jdob has quit IRC | 20:47 | |
| russell_h | JayF: huh, cool | 20:49 |
| *** linggao has quit IRC | 20:50 | |
| openstackgerrit | A change was merged to openstack/ironic-python-agent: Add timeout param for execution_thread.join https://review.openstack.org/85411 | 20:51 |
| openstackgerrit | Jay Faulkner proposed a change to openstack/ironic-python-agent: Make tests pass for Python 3.3 https://review.openstack.org/85481 | 20:55 |
| JayF | russell_h: ^ even if we pull out the eventlet requirements.txt change, I think we should merge the stuff that adds python 3.3 compat, even though we know it won't be complete until eventlet fully supports py33 | 20:56 |
| JayF | Or I could even create a separate requirements.txt file for python 3.3 | 20:56 |
| JoshNang | JayF: I think I've seen that in a few projects | 20:58 |
| JayF | eventlet still isn't python 3.3 compat though, according to that issue, which means while tests pass, I'm not sure I'd want to ship an ramdisk with 3.3 on it until after they have a version released that declares support for 3.3 | 20:59 |
| JayF | that being said, getting our tests passing and adding a python33 job that passed would be great in insuring we keep 3.3 support in our code while eventlet finishes up their fixes | 20:59 |
| jroll | +1 | 20:59 |
| *** florentflament has quit IRC | 21:00 | |
| *** florentflament has joined #openstack-ironic | 21:02 | |
| NobodyCam | lol devananda I was just about to approve 85044 | 21:03 |
| NobodyCam | :-p | 21:03 |
| devananda | heh | 21:03 |
| devananda | that's adam_g's patch? I can add the comment in a follow on | 21:04 |
| devananda | wait, no, that's a different # | 21:04 |
| NobodyCam | htat was yuriy's fix missing % | 21:04 |
| devananda | ah | 21:04 |
| NobodyCam | :-p | 21:04 |
| *** matty_dubs is now known as matty_dubs|gone | 21:05 | |
| *** romcheg has joined #openstack-ironic | 21:08 | |
| openstackgerrit | A change was merged to openstack/ironic: Fix messages formatting for _sync_power_states https://review.openstack.org/85044 | 21:18 |
| NobodyCam | brb | 21:23 |
| *** romcheg has quit IRC | 21:26 | |
| russell_h | devananda: what do you want to do on https://review.openstack.org/#/c/81919/ | 21:27 |
| russell_h | s/to do/me to do/ | 21:27 |
| openstackgerrit | A change was merged to openstack/ironic: Updated from global requirements https://review.openstack.org/83471 | 21:27 |
| *** romcheg has joined #openstack-ironic | 21:29 | |
| openstackgerrit | A change was merged to openstack/ironic-python-agent: Added execute util https://review.openstack.org/85344 | 21:32 |
| *** romcheg has quit IRC | 21:33 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack/ironic: Update tox.ini to also run nova tests https://review.openstack.org/84033 | 21:36 |
| adam_g | devananda, note ^ after making ironic/nova/tests/* importable, i had to exclude ironic/nova/* from flake8, which also does an 'import everything under the sun at load' similar to testr (but without the config flexibility) and hits the same oslo.config issues | 21:37 |
| adam_g | need to run out | 21:38 |
| *** jrist is now known as jrist-afk | 21:41 | |
| *** jrist-afk is now known as jrist | 21:41 | |
| openstackgerrit | Chris Krelle proposed a change to openstack/ironic: Add Logging. https://review.openstack.org/85124 | 21:41 |
| devananda | adam_g: right, i saw that. not too keen on it as i'd like to flake8 those files as well as run unit tests | 21:51 |
| devananda | adam_g: but adding unit tests is a good improvement from where it's at | 21:51 |
| NobodyCam | brb | 21:51 |
| devananda | bearhands: audit-level logging should be done for actual changes, not for requested changes, right? | 21:59 |
| devananda | bearhands: eg, RPC request for change_node_power_state vs. the state was actually successfully changed. | 22:00 |
| devananda | first case => debug log. second case => audit log. | 22:00 |
| devananda | yesno? | 22:00 |
| *** vkozhukalov has quit IRC | 22:06 | |
| * devananda steps away for a bit | 22:16 | |
| *** eguz has quit IRC | 22:20 | |
| *** eghobo has joined #openstack-ironic | 22:21 | |
| *** jgrimm has quit IRC | 22:24 | |
| NobodyCam | bearhands: its ? on 85124 | 22:32 |
| openstackgerrit | Jay Faulkner proposed a change to openstack/ironic-python-agent: Compatibility fixes for Python 3.3 https://review.openstack.org/85481 | 23:08 |
| NobodyCam | oh now this is strange. http://paste.openstack.org/show/ElhQeS9M5Ojr4RzSXITX/ | 23:18 |
| NobodyCam | ok this the error in gate undercloud http://paste.openstack.org/show/SglUwAkk8PDNEEl4Da4S | 23:41 |
| NobodyCam | are the nodes in the gatetests left on... | 23:42 |
| *** eghobo has quit IRC | 23:43 | |
| bearhands | devananda: I'm not aware of any 'rule' regarding audit logging | 23:44 |
| NobodyCam | why yes they do! | 23:48 |
| *** eghobo has joined #openstack-ironic | 23:57 | |
| *** eghobo has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!