| *** tosky has quit IRC | 00:06 | |
| *** openstackgerrit has quit IRC | 00:18 | |
| *** openstackgerrit has joined #openstack-sahara | 00:19 | |
| *** hdd has quit IRC | 00:22 | |
| openstackgerrit | Andrew Lazarev proposed openstack/sahara: Fixed scaling with new node group with auto sg https://review.openstack.org/139285 | 00:35 |
|---|---|---|
| *** tnovacik has quit IRC | 00:35 | |
| *** ViswaV_ has quit IRC | 00:36 | |
| *** Networkn3rd has quit IRC | 00:44 | |
| oikawa | https://bugs.launchpad.net/sahara/+bug/1390414 How about this bug report ? This patch is a little large (30KB). I want to start working as soon as possible If triage of this report is bug. | 01:06 |
| uvirtbot | Launchpad bug 1390414 in sahara "SwiftNativeFileSystem's directory is incompatible with Swift and Horizon" [Undecided,New] | 01:06 |
| *** ViswaV has joined #openstack-sahara | 01:12 | |
| *** ViswaV_ has joined #openstack-sahara | 01:14 | |
| elmiko | oikawa: i think i've seen that issue when running pig jobs and storing output to a Swift container | 01:16 |
| *** ViswaV has quit IRC | 01:17 | |
| elmiko | oikawa: https://bugs.launchpad.net/sahara/+bug/1390414 is triaged and assigned to you | 01:20 |
| uvirtbot | Launchpad bug 1390414 in sahara "SwiftNativeFileSystem's directory is incompatible with Swift and Horizon" [Medium,Confirmed] | 01:20 |
| *** hdd has joined #openstack-sahara | 01:22 | |
| *** ViswaV_ has quit IRC | 01:32 | |
| *** weiting has joined #openstack-sahara | 01:56 | |
| *** weiting has quit IRC | 01:59 | |
| *** Networkn3rd has joined #openstack-sahara | 02:11 | |
| *** hdd has quit IRC | 02:32 | |
| *** witlessb has joined #openstack-sahara | 02:56 | |
| *** Poornima has joined #openstack-sahara | 03:01 | |
| *** witlessb has quit IRC | 03:01 | |
| oikawa | elmiko: thanks! | 03:21 |
| *** tellesnobrega_ has quit IRC | 03:33 | |
| openstackgerrit | Jeremy Stanley proposed openstack/python-saharaclient: Workflow documentation is now in infra-manual https://review.openstack.org/139383 | 03:51 |
| openstackgerrit | Jeremy Stanley proposed openstack/sahara: Workflow documentation is now in infra-manual https://review.openstack.org/139387 | 03:52 |
| openstackgerrit | Jeremy Stanley proposed openstack/sahara-dashboard: Workflow documentation is now in infra-manual https://review.openstack.org/139388 | 03:52 |
| openstackgerrit | Jeremy Stanley proposed openstack/sahara-extra: Workflow documentation is now in infra-manual https://review.openstack.org/139389 | 03:52 |
| openstackgerrit | Jeremy Stanley proposed openstack/sahara-image-elements: Workflow documentation is now in infra-manual https://review.openstack.org/139390 | 03:52 |
| openstackgerrit | Jeremy Stanley proposed openstack/sahara-specs: Workflow documentation is now in infra-manual https://review.openstack.org/139391 | 03:52 |
| *** ViswaV has joined #openstack-sahara | 03:55 | |
| *** ViswaV has quit IRC | 03:59 | |
| *** ViswaV has joined #openstack-sahara | 04:00 | |
| openstackgerrit | Jeremy Stanley proposed stackforge/sahara-guestagent: Workflow documentation is now in infra-manual https://review.openstack.org/139514 | 04:12 |
| openstackgerrit | Kazuki OIKAWA proposed openstack/sahara: Add edp.java.adapt_for_oozie config for Java Action https://review.openstack.org/115884 | 04:39 |
| *** ViswaV has quit IRC | 04:51 | |
| *** Networkn3rd has quit IRC | 04:59 | |
| *** chandankumar has joined #openstack-sahara | 05:08 | |
| *** chandankumar has quit IRC | 05:09 | |
| *** ghenriks has quit IRC | 05:10 | |
| *** chandankumar has joined #openstack-sahara | 05:24 | |
| *** Poornima has quit IRC | 05:26 | |
| openstackgerrit | Kazuki OIKAWA proposed openstack/sahara-extra: Add main function wrapper for edp.java.adapt_for_oozie config https://review.openstack.org/139522 | 05:50 |
| *** Longgeek has joined #openstack-sahara | 05:52 | |
| *** Poornima has joined #openstack-sahara | 05:56 | |
| openstackgerrit | Kazuki OIKAWA proposed openstack/sahara-extra: Add main function wrapper for edp.java.adapt_for_oozie config https://review.openstack.org/139522 | 06:23 |
| *** ViswaV has joined #openstack-sahara | 06:25 | |
| *** ViswaV has quit IRC | 06:25 | |
| *** ViswaV has joined #openstack-sahara | 06:25 | |
| *** ghenriks has joined #openstack-sahara | 06:27 | |
| *** Longgeek has quit IRC | 06:30 | |
| *** Longgeek has joined #openstack-sahara | 06:32 | |
| *** openstackgerrit has quit IRC | 06:49 | |
| *** openstackgerrit has joined #openstack-sahara | 06:49 | |
| *** ViswaV_ has joined #openstack-sahara | 07:07 | |
| *** ViswaV has quit IRC | 07:08 | |
| *** tnovacik has joined #openstack-sahara | 07:44 | |
| *** tnovacik has quit IRC | 07:58 | |
| *** tellesnobrega_ has joined #openstack-sahara | 08:00 | |
| *** tellesnobrega_ has quit IRC | 08:04 | |
| *** witlessb has joined #openstack-sahara | 08:17 | |
| openstackgerrit | Ken Chen proposed openstack/sahara-specs: Add more services into CDH https://review.openstack.org/139574 | 08:50 |
| openstackgerrit | Ken Chen proposed openstack/sahara-specs: Add more services into CDH https://review.openstack.org/139574 | 08:52 |
| *** ViswaV_ has quit IRC | 09:09 | |
| *** skolekonov has joined #openstack-sahara | 09:11 | |
| *** stannie has joined #openstack-sahara | 09:17 | |
| *** IvanBerezovskiy1 has joined #openstack-sahara | 09:22 | |
| *** tellesnobrega_ has joined #openstack-sahara | 09:35 | |
| *** tellesnobrega_ has quit IRC | 10:00 | |
| *** chandankumar has quit IRC | 10:02 | |
| *** chandankumar has joined #openstack-sahara | 10:09 | |
| openstackgerrit | Merged openstack/sahara-image-elements: Workflow documentation is now in infra-manual https://review.openstack.org/139390 | 10:51 |
| openstackgerrit | Merged openstack/sahara-extra: Workflow documentation is now in infra-manual https://review.openstack.org/139389 | 10:53 |
| *** tosky has joined #openstack-sahara | 10:53 | |
| openstackgerrit | Merged openstack/sahara-specs: Workflow documentation is now in infra-manual https://review.openstack.org/139391 | 10:55 |
| openstackgerrit | Merged openstack/python-saharaclient: Workflow documentation is now in infra-manual https://review.openstack.org/139383 | 11:19 |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: Fixed scaling with new node group with auto sg https://review.openstack.org/139285 | 11:56 |
| tosky | looking at the tests executed by jenkins, I noticed that data_processing tests (so API and CLI) in tempest are not executed for the stable/icehouse branch | 12:01 |
| tosky | in fact the line sahara=True is not enabled in the [service_available] section of the generated tempest.conf (by devstack?) | 12:02 |
| tosky | is it expected, as Sahara was not fully integrated for IceHouse? | 12:02 |
| SergeyLukjanov | tosky, yup, because sahara started to be integrated from juno | 12:03 |
| tosky | SergeyLukjanov: so should I report a bug if I find that some tempest tests fail in icehouse? | 12:04 |
| tosky | or not? | 12:04 |
| tosky | or should I ask on #openstack-qa? | 12:04 |
| SergeyLukjanov | tosky, nope, tests were done for sahara juno+ I think | 12:04 |
| tosky | ack, thanks | 12:05 |
| tosky | I was unsure as tempest should support all the released version, but Sahara was not official official | 12:05 |
| tosky | tempest master* | 12:05 |
| SergeyLukjanov | tosky, yeah, only starting juno | 12:06 |
| SergeyLukjanov | for sahara | 12:06 |
| openstackgerrit | Sergey Lukjanov proposed openstack/sahara: Update conf sample after oslo.messaging release https://review.openstack.org/139046 | 12:37 |
| *** Longgeek has quit IRC | 12:52 | |
| *** chandankumar has quit IRC | 13:05 | |
| *** hdd has joined #openstack-sahara | 13:10 | |
| *** Longgeek has joined #openstack-sahara | 13:17 | |
| *** tmckay has quit IRC | 13:29 | |
| *** ylobankov has joined #openstack-sahara | 13:33 | |
| openstackgerrit | Merged openstack/sahara: Fixed Fake plugin for Fedora image https://review.openstack.org/138943 | 13:38 |
| *** skolekonov has quit IRC | 13:42 | |
| *** miqui_ has joined #openstack-sahara | 14:04 | |
| *** _crobertsrh is now known as crobertsrh | 14:14 | |
| *** tmckay has joined #openstack-sahara | 14:31 | |
| *** hdd has quit IRC | 14:37 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara-specs: [EDP] Add options supporting DataSource identifiers in job_configs https://review.openstack.org/138809 | 14:57 |
| *** egafford has joined #openstack-sahara | 15:03 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Fix HIVESERVER2 number issue https://review.openstack.org/139658 | 15:15 |
| *** Poornima has quit IRC | 15:17 | |
| *** hdd has joined #openstack-sahara | 15:27 | |
| crobertsrh | Anyone else have thoughts on https://review.openstack.org/#/c/136727/ given that the work to actually support this feature is coming very soon? | 15:31 |
| crobertsrh | We just released a new version of saharaclient and it will most likely be fully supported by the next release (unless we were to release a version just to remove these methods). | 15:32 |
| SergeyLukjanov | crobertsrh, I'm ok if it'll be implemented in ~ month | 15:33 |
| crobertsrh | I think it will be | 15:34 |
| SergeyLukjanov | crobertsrh, no real need to remove updates, release, than add updates and release again :) | 15:34 |
| crobertsrh | Unless we need to practice :) | 15:34 |
| *** Networkn3rd has joined #openstack-sahara | 15:37 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Fix HIVESERVER2 number issue https://review.openstack.org/139658 | 15:42 |
| egafford | tmckay, elmiko: I was thinking about our discussion on mapping data sources (and other args) to job executions, and about the lack of any abstraction layer to unify the way we pass args to jobs. It occurred to me that if we moved the definition of a param/arg/config map to the job itself, things might make a lot more sense. Basically, the idea would be to allow the creation of a consistent, human-readable definition for any specif | 16:07 |
| egafford | So the job definition could take a map of {argument_name: {validation_type: _, injection_rule: _, required: _, default: _)}. (Or just default with a false-ish sentinel value for required.) | 16:07 |
| egafford | Validation_type could be string, number, data_type (which could take a URL, id, or name, as discussed), or others. | 16:07 |
| egafford | Injection_rule could be argument[index], param['name'], or config['name']. | 16:07 |
| egafford | It could allow the job itself to define what it needs, regardless of engine, and give us the glue to make it all work, readably, at the execution definition. It's definitely more heavyweight than what we were talking about yesterday, but could allow some really nice wizarding once wired up, and does feel to me like the right final goal if we want to be all things to all people. Thoughts, and apologies if this has already been disc | 16:07 |
| openstackgerrit | Merged openstack/sahara: Update conf sample after oslo.messaging release https://review.openstack.org/139046 | 16:09 |
| tmckay | so a job carries a template that needs to be filled in at execution time, and the UI is driven by it | 16:09 |
| elmiko | egafford: interesting idea | 16:10 |
| tmckay | crobertsrh, ^^ | 16:10 |
| egafford | tmckay: Right. | 16:10 |
| crobertsrh | makes sense...I'll think about it more...seems good though | 16:11 |
| egafford | tmckay: But since the template really is attached, permanently, to the job, and different people might well define and run the jobs, it's a useability win to put the mapping on the job itself. | 16:11 |
| tmckay | agreed | 16:11 |
| egafford | It also makes a lot of potential collision problems on data source names and such mostly go away (if the arg is defined as a data type, and you get a string that's not a guid, your path to finding that thing is unambiguous.) | 16:12 |
| tmckay | thinking about scope (does it fit in kilo?), and compatibility issues (what about existing jobs in a Sahara database) | 16:13 |
| egafford | tmckay: Sure; I'd hope that this mapping would be an optional feature, and that old-school config/arg/params would override templating. | 16:14 |
| egafford | As to scope, very hard to say. | 16:14 |
| tmckay | I think it's a good idea -- you're right, it's clear from our discussions that what we ultimately need is a mapping mechanism | 16:15 |
| tmckay | egafford, will you write up a blueprint and a spec when you have a chance? | 16:15 |
| egafford | We need a unified job argument definition scheme, certainly, and the only structure that works is a map. Sure; I'd be happy to. | 16:16 |
| elmiko | egafford: i agree with tmckay, write it up as a spec. | 16:19 |
| egafford | tmckay, elmiko: Definitely, needs all kinds of review. Just wanted to hash it out here a bit to make certain I was on a road that hadn't already been traveled before I showed up. | 16:20 |
| tmckay | egafford, nope. Historically, EDP grew up being very hadoop centric, and the first supported job type was mapreduce I believe, with a traditional single input dir and output dir. | 16:21 |
| elmiko | egafford: i don't think so, but i am curious to see how we contain the mapping | 16:21 |
| tmckay | development was really quick, with emphasis on other (constrained) job types | 16:21 |
| tmckay | It's only recently that we've really started to focus on flexibility and generality. Issues started to pop up when we added Java (this past Spring) | 16:22 |
| egafford | tmckay: Sure, it makes sense that development to date would've taken a "SUPPORT MORE PLUGINS NOW" focus. | 16:22 |
| tmckay | yeah. Time to shift focus and iterate, now that we've got some experience with it and we've found some holes | 16:23 |
| egafford | Yup. Power users can make it work at all, thanks to what's already been done. Now we lower the bar. | 16:24 |
| egafford | (BTW, "power users can make it work at all" is intended as recognition of a major accomplishment, given the timelines involved and the number of different platforms supported.) | 16:27 |
| tmckay | :) thanks. Hindsight is always 20/20. Early efforts did not consider general paradigms like Java/Spark, and now we have a "support the old stuff and don't wreck the API" constraint | 16:29 |
| tmckay | But, we have the technology. We can rebuild it. | 16:29 |
| egafford | Hey, efforts to date allowed me to reasonably quickly script out the initialization and kickoff of jobs in four different data processing frameworks simultaneously, and they reliably succeeded by the end. That's not bad for government work there, even without wizards. | 16:31 |
| tmckay | we're supposed to be better than the government | 16:31 |
| egafford | I've worked for the government, albeit briefly. We are. | 16:32 |
| * tmckay waves to the listeners | 16:32 | |
| elmiko | lol | 16:33 |
| egafford | Yes, there's that. And I'm sure there are a ton of very competent, engineering focused government initiatives out there. | 16:33 |
| egafford | So many. | 16:33 |
| tmckay | absolutely | 16:33 |
| egafford | Anyway, this's gotten awkward, so I'm gonna go wrestle with packages and think about a spec. Thanks all. | 16:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara: Updated from global requirements https://review.openstack.org/139209 | 16:40 |
| *** IvanBerezovskiy1 has left #openstack-sahara | 16:41 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Use first_run to Start Services https://review.openstack.org/134471 | 17:02 |
| *** ViswaV has joined #openstack-sahara | 17:24 | |
| *** ViswaV has quit IRC | 17:29 | |
| *** ViswaV has joined #openstack-sahara | 17:31 | |
| openstackgerrit | Matthew Farrellee proposed openstack/sahara: Update oslo-incubator log https://review.openstack.org/139148 | 17:53 |
| openstackgerrit | Matthew Farrellee proposed openstack/sahara: Update oslo-incubator policy https://review.openstack.org/139149 | 17:53 |
| openstackgerrit | Matthew Farrellee proposed openstack/sahara: Update oslo-incubator threadgroup https://review.openstack.org/139150 | 17:53 |
| openstackgerrit | Matthew Farrellee proposed openstack/sahara: Update oslo-incubator periodic_task https://review.openstack.org/139151 | 17:53 |
| openstackgerrit | Matthew Farrellee proposed openstack/sahara: Removed _i18n module, it is not used directly https://review.openstack.org/139146 | 17:53 |
| openstackgerrit | Matthew Farrellee proposed openstack/sahara: Update oslo-incubator lockutils https://review.openstack.org/139147 | 17:53 |
| *** tellesnobrega_ has joined #openstack-sahara | 17:54 | |
| *** tellesnobrega_ has quit IRC | 17:59 | |
| elmiko | tmckay: this is almost ready to go, mind taking a look? https://review.openstack.org/#/c/137211/ | 18:13 |
| tmckay | sure | 18:14 |
| elmiko | thanks =) | 18:15 |
| *** ViswaV has quit IRC | 18:16 | |
| *** ViswaV has joined #openstack-sahara | 18:31 | |
| tmckay | elmiko, for consistency, shouldn't there be a specs/saharaclient/kilo? Or are we waiting for spec to exist before we make the dir? | 18:32 |
| tmckay | to avoid empty dirs | 18:32 |
| tmckay | definitely will be specs for sahara in any release, but the client ... well maybe not | 18:33 |
| tmckay | If that's the case, we might want to state that explicitly | 18:33 |
| tmckay | in the README | 18:33 |
| tmckay | it sounds to me like we'll have specs floating around in specs/saharaclient, whereas for Sahara they will always be in specs/release | 18:35 |
| tmckay | structure could be parallel ... empty dirs aren't really a problem. Kind of self-documenting, "we didn't do anything for client in this release" | 18:36 |
| elmiko | tmckay: sec | 18:39 |
| tmckay | "If that's the case, we might want to state that explicitly" -- I mean the motivation for the different structure | 18:40 |
| *** tosky has quit IRC | 18:43 | |
| elmiko | tmckay: i think i left out the specs/saharaclient because of the low number of specs for it. but in general you're correct we should have release based subdirs | 18:44 |
| tmckay | elmiko, ok, that's fine. I might just add a sentence or two to the readme that says "low churn, structure differs to avoid empty subdirs" | 18:46 |
| tmckay | just so someone doesn't think it's a mistake :) | 18:46 |
| elmiko | cool, i think you're correct that we might add a sentence or something about that in the readme | 18:47 |
| tmckay | other than that, lgtm | 18:47 |
| elmiko | i actually had thought about that while doing the patch, i just couldn't justify making all these empty dirs. especially since i don't think i've seen a spec for saharaclient yet | 18:48 |
| tmckay | :) I could go either way. But if we state the motivation explicitly it will prevent jokers like me from saying "shouldn't this structure be parallel?" | 18:54 |
| * tmckay shakes fist at FrozenClassError, his old nemesis | 18:55 | |
| tmckay | guess I have to copy the darn thing after all | 18:55 |
| elmiko | tmckay: well that, and then the question is also should their be a specific backlog dir under saharaclient? | 18:59 |
| tmckay | yes, actually, I had that thought too | 19:01 |
| tmckay | if something gets backlogged for client, do we just count on folks to figure it out from context? | 19:01 |
| elmiko | tmckay: maybe i should spell it out clearly in the readme, and specify that the dirs should be created as needed? | 19:03 |
| tmckay | yeah, on demand. someday a saharaclient backlog probably appears and then remains, empty | 19:04 |
| tmckay | you could just create it now ... | 19:04 |
| * tmckay shrugs | 19:04 | |
| elmiko | hmm, let me think about it and look at the other repos i borrowed from | 19:05 |
| tmckay | yeah, why a male in great shape? Why not your average man on the street? | 19:07 |
| tmckay | oops | 19:08 |
| tmckay | wrong window, heh! | 19:08 |
| tmckay | that one will keep them wondering .... | 19:08 |
| elmiko | tmckay: ok, i'm thinking leave it empty for now. keystone has a similar layout and they don't have anything in there. plus i don't think git will let us check in empty dirs | 19:08 |
| crobertsrh | Ha! | 19:08 |
| elmiko | lol | 19:08 |
| * tmckay for those who are wondering, it's about the choice of models for ballistics dummies | 19:09 | |
| tmckay | side project | 19:09 |
| crobertsrh | Not sure that helps make it any less weird | 19:09 |
| tmckay | maybe not | 19:09 |
| elmiko | lol | 19:10 |
| tmckay | elmiko, sounds good on the empty dir | 19:11 |
| *** samuelms has quit IRC | 19:28 | |
| *** tellesnobrega has quit IRC | 19:29 | |
| *** tellesnobrega_ has joined #openstack-sahara | 20:43 | |
| *** hdd has quit IRC | 20:49 | |
| *** _mattf is now known as mattf | 21:10 | |
| *** hdd has joined #openstack-sahara | 21:28 | |
| openstackgerrit | Andrew Lazarev proposed openstack/sahara: Fixed scaling with new node group with auto sg https://review.openstack.org/139285 | 21:36 |
| *** tmckay has quit IRC | 21:42 | |
| *** Longgeek has quit IRC | 21:42 | |
| *** crobertsrh is now known as _crobertsrh | 21:50 | |
| *** Networkn3rd has quit IRC | 22:25 | |
| openstackgerrit | Andrew Lazarev proposed openstack/sahara: Fixed subprocess error reporting https://review.openstack.org/139742 | 22:40 |
| *** egafford has quit IRC | 23:01 | |
| *** mattf is now known as _mattf | 23:19 | |
| alazarev | oikawa, hi | 23:21 |
| alazarev | oikawa, please take a look at https://bugs.launchpad.net/sahara/+bug/1399822 | 23:22 |
| uvirtbot | Launchpad bug 1399822 in sahara "[Vanilla2] Failed to scale cluster because of hive config" [Undecided,New] | 23:22 |
| alazarev | oikawa, it looks the issue was introduced by your change | 23:22 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!