| *** IlyaE has quit IRC | 00:33 | |
| *** ErikB1 has quit IRC | 01:18 | |
| *** witlessb has joined #openstack-sahara | 01:56 | |
| *** witlessb has quit IRC | 02:01 | |
| *** ErikB1 has joined #openstack-sahara | 02:49 | |
| *** ErikB1 has quit IRC | 03:28 | |
| *** Ch00k has joined #openstack-sahara | 03:44 | |
| *** ErikB has joined #openstack-sahara | 03:44 | |
| *** RockKuo has joined #openstack-sahara | 03:46 | |
| *** ErikB has quit IRC | 03:59 | |
| *** akuznetsov has joined #openstack-sahara | 04:30 | |
| *** akuznetsov has quit IRC | 04:42 | |
| *** akuznetsov has joined #openstack-sahara | 04:43 | |
| *** IlyaE has joined #openstack-sahara | 04:57 | |
| *** IlyaE has quit IRC | 05:05 | |
| *** akuznetsov has quit IRC | 05:08 | |
| *** Ch00k has quit IRC | 05:12 | |
| *** akuznetsov has joined #openstack-sahara | 05:19 | |
| *** Ch00k has joined #openstack-sahara | 05:27 | |
| *** Ch00k has quit IRC | 05:35 | |
| *** akuznetsov has quit IRC | 05:46 | |
| *** IlyaE has joined #openstack-sahara | 05:49 | |
| openstackgerrit | Jenkins proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/82719 | 06:06 |
|---|---|---|
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: [IDH302] Restoring cluster parameters after scaling https://review.openstack.org/84364 | 06:26 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: [IDH] Integration tests for IDH 3.0.2 https://review.openstack.org/79719 | 06:26 |
| *** tnovacik has joined #openstack-sahara | 06:28 | |
| *** Ch00k has joined #openstack-sahara | 06:37 | |
| *** IlyaE has quit IRC | 06:51 | |
| *** IlyaE has joined #openstack-sahara | 07:17 | |
| openstackgerrit | A change was merged to openstack/sahara: Fix default repo links and tarball links for IDH https://review.openstack.org/84118 | 07:35 |
| *** IlyaE has quit IRC | 07:37 | |
| *** witlessb has joined #openstack-sahara | 07:59 | |
| *** vrovachev has joined #openstack-sahara | 09:02 | |
| *** dmitryme has joined #openstack-sahara | 09:08 | |
| *** tosky has joined #openstack-sahara | 09:52 | |
| *** Ch00k_ has joined #openstack-sahara | 10:03 | |
| *** Ch00k has quit IRC | 10:05 | |
| *** akuznetsov has joined #openstack-sahara | 10:16 | |
| *** akuznetsov has quit IRC | 10:20 | |
| *** akuznetsov has joined #openstack-sahara | 10:34 | |
| *** tellesnobrega has quit IRC | 10:38 | |
| *** tellesnobrega has joined #openstack-sahara | 10:44 | |
| *** Ch00k_ has quit IRC | 11:08 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Fix check active nodemanagers for vanilla 2 plugin https://review.openstack.org/84228 | 11:23 |
| *** Ch00k has joined #openstack-sahara | 11:26 | |
| *** ErikB has joined #openstack-sahara | 11:27 | |
| *** _crobertsrh has quit IRC | 11:32 | |
| *** dmitryme has quit IRC | 11:33 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: [IDH302] Restoring cluster parameters after scaling https://review.openstack.org/84364 | 11:41 |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: [IDH] Integration tests for IDH 3.0.2 https://review.openstack.org/79719 | 11:41 |
| *** themistymay has joined #openstack-sahara | 11:41 | |
| openstackgerrit | Sofiia Kostiuchenko proposed a change to openstack/sahara: Fixed tests failures when SKIP_ALL_TESTS_FOR_PLUGIN=True https://review.openstack.org/84417 | 11:54 |
| *** tosky has quit IRC | 11:56 | |
| *** dmitryme has joined #openstack-sahara | 12:03 | |
| *** RockKuo has quit IRC | 12:18 | |
| *** crobertsrh has joined #openstack-sahara | 12:37 | |
| *** elmiko has joined #openstack-sahara | 12:52 | |
| *** skolekonov has quit IRC | 12:56 | |
| *** sballe has joined #openstack-sahara | 13:02 | |
| *** tosky has joined #openstack-sahara | 13:16 | |
| openstackgerrit | Chad Roberts proposed a change to openstack/sahara: Updating dashboard user guide doc for icehouse https://review.openstack.org/84169 | 13:17 |
| *** tmckay has joined #openstack-sahara | 13:20 | |
| *** elmiko is now known as elmiko_ | 13:24 | |
| *** akuznetsov has quit IRC | 13:26 | |
| *** akuznetsov has joined #openstack-sahara | 13:29 | |
| openstackgerrit | Sofiia Kostiuchenko proposed a change to openstack/sahara: Fixed tests failures when SKIP_ALL_TESTS_FOR_PLUGIN=True https://review.openstack.org/84417 | 13:38 |
| *** sballe has quit IRC | 13:43 | |
| tmckay | elmiko, so this "how to add a new migration script" -- is that a new doc, or updating the README in migrations? Do you know? | 13:46 |
| tmckay | elmiko_, ^^ | 13:46 |
| tmckay | aignatov, SergeyLukjanov, ^^ do you know what we're aiming for here? New section to the online docs, or review of the README? | 13:47 |
| crobertsrh | I was lost on that one too | 13:47 |
| tmckay | I think I survived on the README and common sense | 13:48 |
| tmckay | it wasn't too hard | 13:48 |
| ErikB | Can anyone comment on Code Freeze for IH? | 13:52 |
| *** bob_nettleton has joined #openstack-sahara | 13:54 | |
| *** themistymay has joined #openstack-sahara | 13:59 | |
| *** bob_nettleton has left #openstack-sahara | 14:08 | |
| tmckay | crobertsrh, hmm, looks to me like we just need a short section under developer guide "HowTo". The installation guide under the "venv from pypy" section already has an instruction that will upgrade the database to "head". If you're using RDO or Fuel, I'm gonna assume that the package handling/openstack does the right thing. | 14:18 |
| tmckay | And if you're in a weird space, the dev guide is still there | 14:18 |
| tmckay | probably rip off most of the README with a few more words | 14:19 |
| * tmckay crobertsrh == souding board :) | 14:20 | |
| tmckay | sounding, even | 14:20 |
| crobertsrh | Which page is this? | 14:20 |
| dmitryme | tmckay: regarding docs for migrations: user still needs to issue DB update command for RDO | 14:21 |
| dmitryme | right now it is missing in the docs, I am going to fix it soon | 14:22 |
| tmckay | dmitryme, okay. I was thinking it might be done by the rpm install, but what you say makes sense. Are you okay with me adding a migration script section to the dev guide? | 14:23 |
| dmitryme | tmckay: sure, currently I have only Installation guide on me | 14:23 |
| tmckay | pretty much just the readme, with a few words about the revision numbers. We have been using sequential, but that's just a convention | 14:23 |
| tmckay | okay | 14:24 |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Update REST api docs https://review.openstack.org/84249 | 14:28 |
| *** sballe has joined #openstack-sahara | 14:31 | |
| *** bradd1 has joined #openstack-sahara | 14:46 | |
| *** IlyaE has joined #openstack-sahara | 14:55 | |
| aignatov | ErikB: what do you mean exactly? | 15:04 |
| aignatov | regarding feature freeze | 15:04 |
| ErikB | aignatov, yes feature freeze | 15:06 |
| openstackgerrit | Denis Egorenko proposed a change to openstack/sahara-image-elements: Add hadoop v2 support for IDH plugin https://review.openstack.org/83062 | 15:06 |
| aignatov | ErikB: what is your question about Feature freeze? :) | 15:12 |
| *** _mattf is now known as mattf | 15:13 | |
| *** RockKuo has joined #openstack-sahara | 15:19 | |
| *** Ch00k has quit IRC | 15:21 | |
| ErikB | aignatov, simply when | 15:21 |
| mattf | ErikB, we're in it... | 15:22 |
| aignatov | ErikB: feature freeze since Mar 4 | 15:22 |
| aignatov | as far as i remember | 15:22 |
| ErikB | ok | 15:22 |
| ErikB | thanks | 15:22 |
| mattf | ErikB, there's a pass for code w/i plugins tho, w/i reason | 15:23 |
| mattf | tho, imho we should be locked down for the past 2 weeks | 15:23 |
| IvanBerezovskiy | mattf, hi. could you please take a look on https://review.openstack.org/#/c/83062/ ? | 15:24 |
| mattf | sure | 15:24 |
| mattf | oh man, idh | 15:25 |
| *** Ch00k has joined #openstack-sahara | 15:26 | |
| *** Ch00k has quit IRC | 15:26 | |
| mattf | IvanBerezovskiy, did you hear that idh is essentially dead in favor of cdh? | 15:26 |
| IvanBerezovskiy | mattf, yes, of course. Denis fixed it | 15:27 |
| mattf | IvanBerezovskiy, it's on my radar for later today, thanks | 15:28 |
| IvanBerezovskiy | mattf, thanks | 15:28 |
| dmitryme | mattf: am I right assuming that after Sahara RDO installation 'sahara-api' and 'sahara-db-manage' binaries are set in user's PATH? | 15:32 |
| IvanBerezovskiy | mattf, I meant that Denis fixed the problem with mirrors for IDH :) | 15:35 |
| tmckay | dmitryme, I have a question about the alembic stuff, since we're making edits. In the README it says if you want to migrate from Icehouse to a later release you have to add version tracking to the database with " sahara-db-manage --config-file /path/to/sahara.conf stamp icehouse" | 15:35 |
| tmckay | dmitryme, but, there is no "icehouse" revision. That won't work. | 15:35 |
| mattf | dmitryme, that's a very reasonable assumption. if it's not the case it's worth a BZ | 15:36 |
| tmckay | and if you've been using icehouse, don't you already have versioning in the db? | 15:36 |
| dmitryme | tmckay: sorry, I am not the right person to ask | 15:36 |
| mattf | dmitryme, are you working through an rdo install? | 15:36 |
| tmckay | dmitryme, okay, I'll use git blame :) | 15:36 |
| dmitryme | mattf: right, adding sahara-db-manage call to installation instructions | 15:37 |
| tmckay | ah, aignatov, my friend :) Called out by git blame | 15:37 |
| *** sreshetnyak has joined #openstack-sahara | 15:39 | |
| aignatov | tmckay: I just renamed it :) | 15:39 |
| tmckay | aignatov, I don't understand this line in the database migration README, "sahara-db-manage --config-file /path/to/sahara.conf stamp icehouse". There is no icehouse revision. | 15:39 |
| tmckay | aignatov, okay | 15:39 |
| tmckay | looks like sreshetnyak, then | 15:39 |
| mattf | dmitryme, most rdo installs have been fresh (no db upgrade), so you may stumble on issues w/ savanna-db-manage. BZ them and we'll get them resolved. | 15:40 |
| mattf | the hope is to have new sahara packages by the end of next week | 15:40 |
| mattf | and we can fold in fixes | 15:41 |
| aignatov | tmckay: I think this was done by example from other projects | 15:41 |
| tmckay | aignatov, okay. Well, the "stamp" command is valid, but you have to stamp with a current revision. | 15:41 |
| aignatov | I think SergeyLukjanov is in progress to send patch about icehouse revision | 15:42 |
| tmckay | so, I suppose if you have a havanna database, you have to give it an initial empty stamp | 15:42 |
| aignatov | Since we don't have havana compatibility and alebik scripts were done by icehouse dev cycle | 15:43 |
| aignatov | we can leave it as is :) | 15:43 |
| aignatov | imho | 15:43 |
| tmckay | aignatov, okay. Should we remove that line from the README? Or do we add an empty icehouse revision, and then make 001 migrate from icehouse? | 15:44 |
| tmckay | aignatov, I'm using the README as the basis for a devguide page, that' | 15:44 |
| tmckay | that's why I'm asking | 15:45 |
| tosky | when are stable branches created usually? (stable/icehouse in this case, like stable/havana) | 15:46 |
| tmckay | truth is, 001 migrates from None, so I don't think an initial stamp is even needed | 15:46 |
| *** vrovachev has quit IRC | 15:47 | |
| aignatov | tmckay: 001 will create initial database | 15:49 |
| *** elmiko_ is now known as elmiko | 15:50 | |
| elmiko | tmckay: just got back, did you get an answer? | 15:51 |
| tmckay | elmiko, I think so. | 15:51 |
| elmiko | cool | 15:51 |
| tmckay | aignatov, are we combining all revisions into a single revision for icehouse? | 15:51 |
| tmckay | Even so, I think we can change the README to say how to use stamp in general, but I don't think we need the "if you want to migrate from Icehouse" part. Don't want to confuse users | 15:52 |
| aignatov | tmckay: I'm not sure if will combine them, since we have not compatibility from havana I'd prefer to have 001 script only just for creating new icehouse database wit already all updated fields | 15:54 |
| aignatov | icehouse fields | 15:54 |
| aignatov | I agree about some confusing here | 15:54 |
| aignatov | actually in icehouse there will not be any migrations :) | 15:55 |
| tmckay | :) Okay, I'll post a README change as part of adding the doc page, and folks can comment | 15:55 |
| aignatov | yes, I'm ok with any changes right now ^) | 15:55 |
| aignatov | :) | 15:55 |
| tmckay | aignatov, thanks | 15:55 |
| *** IvanBerezovskiy has left #openstack-sahara | 15:55 | |
| *** sballe_ has joined #openstack-sahara | 16:05 | |
| *** tnovacik has quit IRC | 16:05 | |
| *** sballe has quit IRC | 16:05 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Fix db management: don't autocreate db on start https://review.openstack.org/82750 | 16:09 |
| openstackgerrit | Matthew Farrellee proposed a change to openstack/sahara-image-elements: Actually abort on unrecognized platforms https://review.openstack.org/84485 | 16:09 |
| mattf | SergeyLukjanov, when are we opening for J dev? | 16:11 |
| SergeyLukjanov | mattf, after the rc1 | 16:12 |
| * SergeyLukjanov reading scrollback | 16:12 | |
| mattf | SergeyLukjanov, i noticed that nova already opened | 16:15 |
| *** RockKuo has quit IRC | 16:17 | |
| SergeyLukjanov | mattf, they already have rc1 | 16:19 |
| elmiko | mattf: you realize that change you are proposing will totally break my el6 builder ;) | 16:20 |
| SergeyLukjanov | mattf, in fact Juno dev is open right after the rc1 cut (Icehouse dev moved to the milestone-proposed branch after the rc1) | 16:20 |
| SergeyLukjanov | rc1 is planned for Thu | 16:21 |
| mattf | elmiko, i do! | 16:21 |
| *** Ch00k has joined #openstack-sahara | 16:22 | |
| mattf | elmiko, i have a followup change that enables rhel...when we have it working | 16:22 |
| mattf | SergeyLukjanov, ok, so the 3rd we open for J? | 16:23 |
| tosky | just to be sure, does it mean "branch stable/icehouse created" ? | 16:23 |
| SergeyLukjanov | mattf, we should if everything will work not bad | 16:25 |
| elmiko | mattf: cool | 16:25 |
| SergeyLukjanov | tosky, nope, it's a milestone-proposed branch | 16:25 |
| SergeyLukjanov | tosky, stable/icehouse will be created after the Icehouse release | 16:25 |
| SergeyLukjanov | april 17 | 16:25 |
| tosky | SergeyLukjanov: will it be created from the milestone branch or from master? Otherwise I don't see how master can be used for Juno | 16:26 |
| SergeyLukjanov | tosky, it'll be created from the m-p branch | 16:26 |
| tosky | make sense, thanks | 16:26 |
| SergeyLukjanov | tosky, np | 16:27 |
| SergeyLukjanov | elmiko, I remember about my part in docs, should push it to review today I hope | 16:29 |
| mattf | elmiko, http://ur1.ca/gyjnk will unbreak you if you accept the "exit 2" | 16:31 |
| tosky | dmitryme: you probably know about it, but there is a reference to the unified guest agent in the latest meeting minutes for Marconi | 16:31 |
| SergeyLukjanov | re migration - I'm going right now to push the squashed "icehouse" migration | 16:32 |
| dmitryme | tosky: yep , I did a little speech here :-) | 16:32 |
| dmitryme | but thanks for letting me know | 16:32 |
| tosky | dmitryme: ah, ok, I didn't check the full logs :) | 16:32 |
| tosky | dmitryme: any interesting outcome (like interested people), if I can ask? | 16:32 |
| dmitryme | tosky: I am visiting different projects right now, seeing if they find the idea of agent reasonable | 16:33 |
| dmitryme | Marconi is supposed to be a backend for the agent | 16:33 |
| elmiko | SergeyLukjanov: sounds good | 16:34 |
| *** sballe_ has quit IRC | 16:34 | |
| dmitryme | basically Marconi welcome that idea | 16:35 |
| dmitryme | which is not a surprise since agent seems to a regular app consuming Marconi | 16:35 |
| dmitryme | and every project would love to see it being used | 16:36 |
| dmitryme | s/it/itself/ | 16:36 |
| tosky | eheh, right | 16:36 |
| dmitryme | at least that is my feeling for Sahara | 16:36 |
| dmitryme | right now i am sitting on Solum meeting | 16:36 |
| dmitryme | they might be interested in consuming the agent | 16:37 |
| dmitryme | am, I guess 'using' is a better word | 16:37 |
| *** ErikB has quit IRC | 16:53 | |
| *** Ch00k has quit IRC | 16:55 | |
| *** ErikB has joined #openstack-sahara | 16:58 | |
| *** crobertsrh has quit IRC | 17:00 | |
| openstackgerrit | A change was merged to openstack/sahara: Fix check active nodemanagers for vanilla 2 plugin https://review.openstack.org/84228 | 17:01 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Compact all Icehouse migrations into single one https://review.openstack.org/84498 | 17:14 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Reserve 5 migrations for backports https://review.openstack.org/84499 | 17:14 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Open Juno dev https://review.openstack.org/84503 | 17:19 |
| *** bradd1 has quit IRC | 17:27 | |
| *** bradd1 has joined #openstack-sahara | 17:35 | |
| SergeyLukjanov | tmckay, I see https://bugs.launchpad.net/bugs/1277584 on you | 17:36 |
| SergeyLukjanov | tmckay, could you fix it before rc1? | 17:36 |
| tmckay | SergeyLukjanov, yes, I can do that as part of doc day | 17:37 |
| SergeyLukjanov | tmckay, thank you! | 17:37 |
| tmckay | np | 17:41 |
| openstackgerrit | Sofiia Kostiuchenko proposed a change to openstack/sahara: Added parameters to configure a list of node group processes https://review.openstack.org/84510 | 17:55 |
| *** crobertsrh has joined #openstack-sahara | 18:18 | |
| mattf | ErikB, is the root-passwd element required in hdp image building, or is it just for debugging access ? | 18:20 |
| mattf | tosky, elmiko ^^ | 18:20 |
| ErikB | mattf, just for debug afaik | 18:21 |
| tosky | yep, good question - it should be always possible to use the cloud-init user | 18:21 |
| mattf | we should consider removing it from the elements line in create-diskimage | 18:22 |
| mattf | ErikB, would ^^ be ok w/ you? you can always add it back for debugging | 18:22 |
| tosky | ErikB: this is the issues we have found: https://bugs.launchpad.net/sahara/+bug/1292614 | 18:22 |
| mattf | imho, it's kinda bad to provide a root backdoor to everyone | 18:22 |
| mattf | we may or may not have spent 2 weeks stumbling over it | 18:22 |
| * mattf grins | 18:23 | |
| ErikB | mattf, hang on, let me get Bob on the line... | 18:23 |
| *** dmitryme has quit IRC | 18:23 | |
| openstackgerrit | A change was merged to openstack/sahara: [IDH302] Restoring cluster parameters after scaling https://review.openstack.org/84364 | 18:33 |
| *** bob_nettleton has joined #openstack-sahara | 18:41 | |
| bob_nettleton | mattf, tosky, elmiko, ErikB just mentioned you were discussing https://bugs.launchpad.net/sahara/+bug/1292614 | 18:43 |
| bob_nettleton | if you'd like, we can modify the HDP images to remove this root-pwd element usage by default, but maybe provide a configuration flag to allow us to generate images with the root-pwd enabled? | 18:44 |
| bob_nettleton | it's very useful for image debugging, but sounds like it might make sense to remove the usage in the general case. | 18:44 |
| tosky | bob_nettleton: but you already have a user with sudo permissions, the one defined through/with cloud-init | 18:45 |
| bob_nettleton | tosky, ok, so this means you just want it removed completely? why is this element available in sahara-image-elements if using it is so problematic? | 18:46 |
| tosky | bob_nettleton: https://bugs.launchpad.net/sahara/+bug/1292614 | 18:46 |
| mattf | bob_nettleton, the element is designed for debugging, not deployment | 18:48 |
| tosky | as elmiko and mattf can testify, we spent quite a bit of time on that; my idea is that it could be possible to mount the /selinux directory if you run it from centos/rhel/fedora, but maybe it's simply not worth of | 18:48 |
| mattf | and it's old, may not be needed anymore | 18:48 |
| bob_nettleton | mattf, ok, then I'm ok with changing this to only be used in debugging mode, which wouldn't be the common case. | 18:50 |
| bob_nettleton | mattf, tosky, one thing that I'm not clear about though: if using root-pwd is a security backdoor, isn't the default cloud-init user basically the same? As I mentioned, I'm ok with removing the usage of root-pwd, but just wasn't sure about this last point. | 18:51 |
| mattf | the cloud-user should be protected by the keypair you give to the cluster | 18:52 |
| bob_nettleton | mattf, ok, thanks for clearing that up. | 18:53 |
| mattf | my pleasure | 18:53 |
| bob_nettleton | mattf, tosky, ok, so I can create a patch for this (to remove the usage of root-pwd) from the HDP image generation, unless one of you has already created a patch, in that case I can review it if you'd like. either is fine with me. just let me know... | 18:54 |
| tosky | no patches so far, we were waiting to understand the reason for the password before trying to kill it :) | 18:55 |
| tosky | thanks | 18:55 |
| mattf | bob_nettleton, happy to review the patch and give you the commit | 18:56 |
| mattf | (make that hwx share of pie bigger) | 18:56 |
| bob_nettleton | tosky, ok. thanks for the heads-up on this. | 18:57 |
| bob_nettleton | mattf, tosky, I'll create the patch then. As you mentioned, it will help me increase my newbie statistics on OpenStack. :) | 18:58 |
| bob_nettleton | mattf, tosky, I'll treat the patch as a fix for https://bugs.launchpad.net/sahara/+bug/1292614, unless anyone has an objection to that. | 18:59 |
| tosky | no objection from me | 19:02 |
| mattf | bob_nettleton, it doesn't fix the root-passwd bug, just lets us avoid root-passwd. pls leave the bug. | 19:06 |
| tosky | SergeyLukjanov: thanks for jumping in; I was planning to coordinate, I guess it will happen sooner than what I expected | 19:07 |
| SergeyLukjanov | tosky, it was started in december I think | 19:08 |
| SergeyLukjanov | tosky, by adding API tests to tempest as the first step, then simple CLI tests | 19:08 |
| SergeyLukjanov | tosky, me and ylobankov working on this tempest stuff | 19:09 |
| tosky | that was my next question (who should I bother :) | 19:09 |
| SergeyLukjanov | tosky, in two words plan is to have scenarios in sahara repo /contrib/tempest/scenario/data_processing that'll run the same integration tests as now | 19:10 |
| SergeyLukjanov | and run them in sahara-ci instead of current ones | 19:10 |
| tosky | SergeyLukjanov: so not in the tempest repository? | 19:10 |
| SergeyLukjanov | tosky, it's not really good to place tests that could not be executed in current openstack infra into tempest | 19:11 |
| tosky | I see | 19:11 |
| SergeyLukjanov | but intention is to move them to tempest when will be able to run them there | 19:11 |
| tosky | so the difference is that the new ones will follow the "tempest" structure, being in fact tempest test, but just living in a different place - is that correct? | 19:11 |
| SergeyLukjanov | and we'll run them in sahara-ci just by "cp sahara/contrib/tempest/* tempest/" | 19:12 |
| SergeyLukjanov | tosky, yup | 19:12 |
| tosky | ack | 19:12 |
| tosky | ok, I need to commute now, thanks again for the help | 19:13 |
| SergeyLukjanov | tosky, sure, np | 19:13 |
| SergeyLukjanov | tosky, there was a plan that ylobankov will work on it in Juno | 19:13 |
| bob_nettleton | mattf, tosky, so I should file a separate bug for removing the usage of root-pwd from the HDP image generation? | 19:13 |
| mattf | bob_nettleton, gut says a good commit message should be enough | 19:14 |
| tosky | SergeyLukjanov: as I will have to work on <some scenario tests>, if you don't mind, I can cooperate/accelerate it | 19:14 |
| SergeyLukjanov | tosky, on the other side I'd like to have fake plugin and test the whole provisioning layer in the tempest in gate and make heat gating on our heat-based provisioning | 19:14 |
| SergeyLukjanov | tosky, it'll be great, we should collaborate on it | 19:14 |
| SergeyLukjanov | tosky, when do you want to start working on it? | 19:15 |
| bob_nettleton | mattf, ok, I was under the impression that a patch had to be either a blueprint impl or bug fix, but if I don't need it for this one, that's fine. thanks! | 19:15 |
| tosky | SergeyLukjanov: as soon as I can have a working setup in my environment, which is something we were discussing ^^ :) | 19:15 |
| *** Ch00k has joined #openstack-sahara | 19:15 | |
| SergeyLukjanov | tosky, okay, so, we can chat with you and ylobankov when you'll be ready | 19:16 |
| tosky | SergeyLukjanov: but I can talk for sure about the planning, on the "how it should look like" and "what are the steps" | 19:16 |
| SergeyLukjanov | anyway, testing/docs could be the better effort before the release :) | 19:16 |
| tosky | eh, having a working setup is already a testing (and we did some fix - ok, centos-oriented, but still fixes :) | 19:17 |
| SergeyLukjanov | tosky, yup, sure ;) | 19:20 |
| tellesnobrega | SergeyLukjanov: hi, im starting to do some research in sahara, and i'm aiming to enable real-time processing in sahara, do you have any opinions on this? if it's a good idea, not possible, desirable?] | 19:26 |
| *** akuznetsov has quit IRC | 19:28 | |
| *** dmitryme has joined #openstack-sahara | 19:30 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Add a page to the developer guide on Alembic migrations https://review.openstack.org/84530 | 19:39 |
| tmckay | elmiko, weeha, another one | 19:39 |
| elmiko | nice :) | 19:39 |
| *** tosky has quit IRC | 19:41 | |
| openstackgerrit | Sofiia Kostiuchenko proposed a change to openstack/sahara: Added parameters to configure a list of node group processes https://review.openstack.org/84510 | 19:42 |
| SergeyLukjanov | tellesnobrega, hey | 19:44 |
| SergeyLukjanov | tellesnobrega, I was working a lot with Twitter Storm (when it was Twitter yet) | 19:45 |
| SergeyLukjanov | tellesnobrega, so, I'm thinking that we'll support it someday in sahara | 19:45 |
| tellesnobrega | SergeyLukjanov: that is good news | 19:46 |
| SergeyLukjanov | tellesnobrega, but currently, our EDP API supports only Oozie, so, we're limited to the features it provides | 19:46 |
| tellesnobrega | SergeyLukjanov: i'm starting my masters degree now, and its related to real-time processing. My idea is to implement real-time in sahara. I'm not familiar with Oozie and how it limit the functionalities, but do you it is feasible? | 19:48 |
| tellesnobrega | SergeyLukjanov: i'm completely new to sahara, up to now i have only read about it | 19:48 |
| SergeyLukjanov | tellesnobrega, I think that you should try to go through the docs to install and play with sahara as the first step | 19:54 |
| SergeyLukjanov | tellesnobrega, we have some thoughts to make oozie pluggable, so, we'll be able to use other (self-written too) jobs management tools | 19:54 |
| SergeyLukjanov | tellesnobrega, to support any data processing framework for ex. | 19:54 |
| tellesnobrega | SergeyLukjanov: sounds good. I will start looking into it. And work so we can get it forward. I'm sure that we can get this done | 19:56 |
| SergeyLukjanov | tellesnobrega, I think it's really depends only on EDP pluggability that will be discussed on Atlanta summit I think | 19:57 |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Add a paragraph discouraging modification of upstream files https://review.openstack.org/84536 | 19:57 |
| tellesnobrega | SergeyLukjanov: sure, i will be there, i will find out when this discussion is going to take place and try to attend it | 19:58 |
| *** akuznetsov has joined #openstack-sahara | 20:01 | |
| *** mattf is now known as _mattf | 20:07 | |
| elmiko | SergeyLukjanov: i saw your comment about local.conf vs localrc, should we mention both since devstack.org is recommending using local.conf in their docs? | 20:08 |
| *** themistymay has quit IRC | 20:08 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Add a page to the developer guide on Alembic migrations https://review.openstack.org/84530 | 20:11 |
| SergeyLukjanov | elmiko, you can enable sahara only in the localrc I think | 20:12 |
| SergeyLukjanov | elmiko, by enable_service sahara or ENABLED_SERVICES+=,sahara | 20:12 |
| *** akuznetsov has quit IRC | 20:13 | |
| elmiko | or you can put it in local.conf | 20:13 |
| elmiko | devstack is migrating from localrc to local.conf | 20:13 |
| SergeyLukjanov | elmiko, hm, sounds like I miss something :) | 20:13 |
| elmiko | at least is was working that way yesterday from devstack | 20:13 |
| elmiko | but for the docs, maybe we should mention using either localrc or local.conf? | 20:14 |
| SergeyLukjanov | elmiko, ++ | 20:19 |
| elmiko | ok, will do | 20:19 |
| tellesnobrega | SergeyLukjanov: can you explain what is the role of Oozie in sahara? | 20:26 |
| tellesnobrega | SergeyLukjanov: its the scheduler right? for map-reduce | 20:27 |
| tmckay | tellesnobrega, Oozie is a service that lets you describe map reduce jobs as xml in a workflow.xml file | 20:28 |
| tmckay | tellesnobrega, it has lots and lots of functionality. Sahara wraps jobs in Oozie workflows and submits them to the Oozie server | 20:29 |
| tmckay | so, yes, it's the scheduler. | 20:29 |
| SergeyLukjanov | tellesnobrega, sorry, it's already 12:29 am for me, so, I'm mostly lurking | 20:29 |
| tellesnobrega | SergeyLukjanov: no problem | 20:29 |
| tmckay | Generating workflows for Oozie allowed us to provide basic job types very easily, without worrying about managing hadoop commands directly | 20:30 |
| tmckay | tellesnobrega, but Sahara does not expose all the power that Oozie has. One thing we need to add imho is the ability to upload Oozie workflows created outside of Sahara | 20:30 |
| *** tellesnobrega has left #openstack-sahara | 20:31 | |
| SergeyLukjanov | tmckay, yup, it's a good idea | 20:31 |
| tmckay | he left :) | 20:31 |
| *** tellesnobrega has joined #openstack-sahara | 20:31 | |
| tellesnobrega | tmckay: sorry | 20:32 |
| tellesnobrega | im having some computer problems, it closed the irc | 20:32 |
| tmckay | np :) | 20:33 |
| tmckay | tellesnobrega, anyway, I've worked on EDP a bunch, and the Oozie stuff, so I can maybe answer some of your questions. I'll be in Atlanta, too | 20:33 |
| tellesnobrega | tmckay: so, every job given to sahara today is given to oozie? | 20:33 |
| tmckay | yes | 20:33 |
| tmckay | there is a piece called the job_manager that generates workflows and submits them to Oozie | 20:34 |
| tellesnobrega | tmckay: for what im planning, the idea is to use marconi as the queue, give to a scheduler and put the messages to the instances for processing in real-time | 20:36 |
| *** Ch00k_ has joined #openstack-sahara | 20:37 | |
| tellesnobrega | im not sure if it fits in sahara, or should be something diferent , reusing maybe the instance provisioning part | 20:38 |
| *** Ch00k__ has joined #openstack-sahara | 20:40 | |
| *** Ch00k has quit IRC | 20:41 | |
| *** Ch00k_ has quit IRC | 20:42 | |
| *** IlyaE has quit IRC | 20:44 | |
| tmckay | tellesnobrega, I'm not familiar with marconi but I should take a look. I think like Sergey noted, if we can make EDP pluggable we should be able to do anything | 20:45 |
| tmckay | tellesnobrega, sahara really only has a few abstractions. A job wraps a collection of binaries (apps, supporting libs), and references an input and output data source. That gets submitted to a cluster. That's about it. | 20:47 |
| tmckay | tellesnobrega, so, there are some details, like naming the main class for a Java job. But in general, as long as we can pass off apps and specify paths to input and output, we should be able to map to anything. | 20:47 |
| *** sballe has joined #openstack-sahara | 20:49 | |
| *** bob_nettleton has left #openstack-sahara | 20:49 | |
| tellesnobrega | tmckay: sounds good to me. I will keep going on my research, we can discuss this with more details at the summit, and when i have some more questions i will be back here | 20:51 |
| tmckay | tellesnobrega, okay, I'm here pretty much every day :) | 20:52 |
| tellesnobrega | tmckay: sure, thanks | 20:52 |
| tmckay | np | 20:52 |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Updating the setup development environment docs for icehouse https://review.openstack.org/84206 | 20:54 |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Updating the vanilla image building docs https://review.openstack.org/84547 | 21:04 |
| *** IlyaE has joined #openstack-sahara | 21:07 | |
| *** Ch00k__ has quit IRC | 21:09 | |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Updating the vanilla image building docs https://review.openstack.org/84547 | 21:14 |
| *** crobertsrh has quit IRC | 21:20 | |
| elmiko | i'm trying to start a cluster and i get this error in my sahara logs "sahara.context BadRequest: Error. Unable to associate floating ip (HTTP 400)" has anyone seen that before? | 21:34 |
| dmitryme | elmiko: nope, but did you try to manually assign floating IPs to instances? | 21:35 |
| dmitryme | maybe its broken | 21:35 |
| elmiko | dmitryme: would i need to turn off use_floating_ips in my config then? | 21:36 |
| elmiko | it's not even starting the instances | 21:36 |
| dmitryme | elmiko: hmmm, why do you think it is not even starting the instances? Floating IPs are assigned after the instances are started | 21:37 |
| elmiko | oh, i don't see the instances in dashboard when before i could see them starting | 21:37 |
| dmitryme | my guess would be that the following occurs: | 21:38 |
| dmitryme | 1. You spin up cluster | 21:38 |
| dmitryme | 2. Sahara starts instances | 21:38 |
| dmitryme | 3. Sahara tries to assigne floating IPs to instances | 21:38 |
| dmitryme | 4. Sahara fails and rolls back deleting all the resources allocated for the cluster, instances included | 21:39 |
| elmiko | ahh, that makes sense | 21:39 |
| elmiko | i see that it created a floating ip port, it just fails to assign it | 21:39 |
| dmitryme | so I would propose to start an instance and try to assign floating IP to it via Horizon | 21:40 |
| elmiko | i'll give it a shot | 21:40 |
| elmiko | when i spin up the instance it gives an ip address, but when i try to assign a new floating ip i see an error about the external network not being reachable from the subnet, hmm | 21:42 |
| *** ruhe has quit IRC | 21:54 | |
| *** openstackgerrit has quit IRC | 21:54 | |
| *** ruhe has joined #openstack-sahara | 21:55 | |
| *** sballe has quit IRC | 22:15 | |
| *** witlessb has quit IRC | 22:20 | |
| *** juice has joined #openstack-sahara | 22:50 | |
| *** tmckay has quit IRC | 22:52 | |
| juice | dmitryme: are you still around? | 23:43 |
| dmitryme | juice: yep | 23:43 |
| juice | great - so we have this issue that we are trying to solve for trove | 23:44 |
| juice | the basic issue is that as a service, our instances are visible in nova to the end user | 23:44 |
| juice | there in nova, they can do as they please with the instance that we (trove) are trying to manage for them | 23:44 |
| juice | we considered creating/owning all instances in nova with the same system-user | 23:45 |
| dmitryme | juice: frankly, we don't feel like that is an issue for us. And what is the problem with the approach you outlined? | 23:46 |
| juice | well the primary issue is that there are "secrets" on the guest instance that would do not want the user getting access to | 23:47 |
| dmitryme | aha, yep I understand your reason | 23:48 |
| juice | also doing things like "snapshot/restore" could cause problems with two agents running on the vms | 23:48 |
| dmitryme | so, why creating instances in a dedicated tenant/domain does not help? | 23:48 |
| juice | it would and we can do something like that but some folks in trove suggested adding a "managed" tag in nova instances to prevent end user fiddling | 23:50 |
| juice | so I am here to see if that is something that would be useful for you guys | 23:50 |
| juice | that is, once the instance is tagged, all interactions must come from the corresponding service. | 23:50 |
| dmitryme | juice: I am afraid it is not. You see, unlike Trove Sahara gives full access to the instances anyway | 23:51 |
| juice | dmitryme: I see well - looks like we are all alone on this one :) | 23:51 |
| dmitryme | regarding using tags, try proposing that in the list | 23:52 |
| juice | dmitryme: so just one quick question | 23:52 |
| juice | if a user logs into one of the nodes in the cluster and just does something wrong that compromises the whole cluster...is that just a "at your own risk" type of issue? | 23:53 |
| dmitryme | my opinion is yes. I always viewed cluster as a single entity and never thought of setting up an 'inner' security | 23:55 |
| dmitryme | sure besides some regular things which work for outside world as well. Firewall for instance | 23:56 |
| juice | dmitryme: ok that is cool - it's good to talk about this stuff to better understand how other components in openstack line up | 23:56 |
| juice | thanks for your time | 23:56 |
| dmitryme | juice: right | 23:56 |
| dmitryme | thanks for your time as well | 23:57 |
| juice | oh when do you think there will be a skeleton of the unified guest agent so we can look around at the code? | 23:57 |
| dmitryme | that is a good question | 23:57 |
| dmitryme | right now I am spending my time partially on advertising the agent and partially on other Sahara stuff | 23:58 |
| dmitryme | I think 1-2 month later | 23:58 |
| *** RockKuo has joined #openstack-sahara | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!