| *** bradd1 has joined #openstack-sahara | 01:41 | |
| *** bradd1 has quit IRC | 01:42 | |
| *** zhiyan_ is now known as zhiyan | 02:35 | |
| *** zhiyan is now known as zhiyan_ | 05:04 | |
| *** akuznetsov has joined #openstack-sahara | 05:06 | |
| *** akuznetsov has quit IRC | 05:53 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/97006 | 06:01 |
|---|---|---|
| *** zhiyan_ is now known as zhiyan | 06:08 | |
| *** akuznetsov has joined #openstack-sahara | 06:13 | |
| openstackgerrit | A change was merged to openstack/sahara: Change IRC channel name to #openstack-sahara https://review.openstack.org/97450 | 06:14 |
| *** akuznets_ has joined #openstack-sahara | 06:28 | |
| *** akuznetsov has quit IRC | 06:29 | |
| *** akuznetsov has joined #openstack-sahara | 06:34 | |
| *** akuznets_ has quit IRC | 06:37 | |
| SergeyLukjanov | aignatov, alazarev, tmckay please take a look on https://review.openstack.org/#/c/74688/ | 06:41 |
| SergeyLukjanov | it's very importnat to land it before the j1 | 06:41 |
| *** jpich has joined #openstack-sahara | 06:56 | |
| *** skolekonov has joined #openstack-sahara | 07:03 | |
| *** witlessb has joined #openstack-sahara | 07:16 | |
| *** akuznets_ has joined #openstack-sahara | 07:25 | |
| *** akuznetsov has quit IRC | 07:27 | |
| *** akuznets_ has quit IRC | 07:51 | |
| *** akuznetsov has joined #openstack-sahara | 07:51 | |
| *** akuznets_ has joined #openstack-sahara | 07:53 | |
| *** akuznetsov has quit IRC | 07:56 | |
| *** akuznets_ has quit IRC | 08:01 | |
| *** openstackgerrit has quit IRC | 08:02 | |
| *** vrovachev has joined #openstack-sahara | 08:03 | |
| *** openstackgerrit has joined #openstack-sahara | 08:06 | |
| *** anteaya has quit IRC | 08:10 | |
| *** julienvey has joined #openstack-sahara | 08:25 | |
| *** Timotey has joined #openstack-sahara | 08:53 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 09:02 | |
| *** akuznetsov has joined #openstack-sahara | 09:38 | |
| *** tosky has joined #openstack-sahara | 09:41 | |
| *** anteaya has joined #openstack-sahara | 10:11 | |
| *** zhiyan is now known as zhiyan_ | 10:17 | |
| openstackgerrit | A change was merged to openstack/sahara-specs: Initial setup of python project stuff https://review.openstack.org/98116 | 11:13 |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Add new jobs for sahara-image-elements https://review.openstack.org/98763 | 12:08 |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Add new jobs for sahara-image-elements https://review.openstack.org/98763 | 12:10 |
| tmckay | SergeyLukjanov, ack, I have to run an errand but I will look at it as soon as possible | 12:10 |
| SergeyLukjanov | tmckay, thank you! | 12:12 |
| *** jimbobhickville has joined #openstack-sahara | 12:31 | |
| openstackgerrit | A change was merged to openstack/sahara: [HDP] Integration tests for HDP 2.0.6 https://review.openstack.org/95739 | 12:47 |
| *** crobertsrh has joined #openstack-sahara | 12:59 | |
| *** miqui has joined #openstack-sahara | 13:00 | |
| *** elmiko has joined #openstack-sahara | 13:08 | |
| *** tmckay has quit IRC | 13:13 | |
| *** liyuezhou has joined #openstack-sahara | 13:14 | |
| *** liyuezhou has quit IRC | 13:14 | |
| *** palar has joined #openstack-sahara | 13:24 | |
| *** bradd1 has joined #openstack-sahara | 13:25 | |
| *** bradd1 has quit IRC | 13:27 | |
| *** palar has quit IRC | 13:29 | |
| *** palar has joined #openstack-sahara | 13:30 | |
| *** vrovachev has quit IRC | 13:39 | |
| openstackgerrit | A change was merged to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/97006 | 13:41 |
| openstackgerrit | A change was merged to openstack/sahara: Updated architecture diagram in docs https://review.openstack.org/97451 | 13:41 |
| openstackgerrit | A change was merged to openstack/sahara: Fix running EDP job on transient cluster https://review.openstack.org/97454 | 13:41 |
| openstackgerrit | A change was merged to openstack/sahara: Added validate_edp method to Plugin SPI doc https://review.openstack.org/97456 | 13:42 |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/sahara-ci-config: Add check jobs for python-saharaclient https://review.openstack.org/98794 | 13:42 |
| openstackgerrit | A change was merged to openstack/sahara: Fix hardcoded tenant name for job binaries https://review.openstack.org/97458 | 13:45 |
| openstackgerrit | A change was merged to openstack/sahara: Create trusts for admin user with correct tenant name https://review.openstack.org/97453 | 13:45 |
| openstackgerrit | A change was merged to openstack/sahara: Add Spark 1.0.0 to the version list https://review.openstack.org/98380 | 13:46 |
| *** bradd1 has joined #openstack-sahara | 13:48 | |
| *** mattf has joined #openstack-sahara | 13:52 | |
| *** mattf has quit IRC | 13:52 | |
| *** mattf has joined #openstack-sahara | 13:52 | |
| *** liyuezhou has joined #openstack-sahara | 13:59 | |
| *** bradd1 has quit IRC | 14:04 | |
| *** tmckay has joined #openstack-sahara | 14:08 | |
| *** liyuezhou has quit IRC | 14:08 | |
| *** bradd1 has joined #openstack-sahara | 14:13 | |
| tellesnobrega | tmckay: i'm trying to understand a little better how oozie works in sahara. When is the Oozie workflow created for a job? | 14:16 |
| tmckay | tellesnobrega, hi | 14:21 |
| tellesnobrega | tmckay: hi | 14:21 |
| tellesnobrega | i saw your spark email, seems like you are doing some really good progress, thats great | 14:22 |
| tmckay | The workflow is created when a job is executed. There is a method in job_manager.py called run_job() which creates the workflow based on job type, uploads files to hdfs, and submits the workflow to the Oozie server | 14:22 |
| tmckay | tellesnobrega, thanks, I hope to produce something today/tomorrow. The method by which to run spark may require a lot of discussion, but I think a draft refactoring the job manager will be going in the right direction in general. | 14:23 |
| tmckay | tellesnobrega, I'll try to get something up relatively quickly that may not be perfect but is good for a working model and discussion | 14:24 |
| tellesnobrega | tmckay: cool | 14:24 |
| tellesnobrega | i need to understando oozie integration better so i can think on how to make it pluggable so storm can be an option here | 14:24 |
| tmckay | tellesnobrega, this may help. The job manager has essentially 3 functions -- submit, poll status, terminate | 14:27 |
| tmckay | My hope is to push those operations behind a common layer | 14:27 |
| tmckay | Two other aspects of this -- 1) a link to a web console if possible (from cluster details) and 2) a plugin has to deploy the service | 14:28 |
| crobertsrh | Any chance this makes sense to anyone? I just grabbed the latest from master and for sahara-db-manage, I get.. http://fpaste.org/108379/32405814/ | 14:28 |
| tmckay | tellesnobrega, so, storm might fit as an Oozie action (shell, Java?) or it might be something altogether separate from Oozie. Not sure, I'm still only very loosely familiar with Storm | 14:29 |
| tmckay | crobertsrh, nope. You might try the old wipe out venv and let it build deps again | 14:30 |
| crobertsrh | Running service gives me this.....http://ur1.ca/hhl44 | 14:30 |
| crobertsrh | new config that isn't covered in the sample config? | 14:30 |
| tmckay | certainly looks like sqlalchemy update | 14:30 |
| tmckay | I may run into the same thing this morning | 14:31 |
| crobertsrh | don't forget call :) | 14:31 |
| crobertsrh | I did wipe .tox and reinstall....same problem. | 14:31 |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/sahara-ci-config: Add check jobs for python-saharaclient https://review.openstack.org/98794 | 14:32 |
| tmckay | crobertsrh, you're scaring me | 14:32 |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Add new jobs for sahara-image-elements https://review.openstack.org/98763 | 14:40 |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/sahara-ci-config: Add check jobs for python-saharaclient https://review.openstack.org/98794 | 14:42 |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/sahara-ci-config: Add check jobs for python-saharaclient https://review.openstack.org/98794 | 14:44 |
| tmckay | SergeyLukjanov, on https://review.openstack.org/#/c/74688/, do you expect that with this change (the apps wrapped in the standard app validator) cert tokens should work? | 15:02 |
| tmckay | Just set the configs, and magic happens? | 15:02 |
| tmckay | That would be great. Our internal stack is using cert tokens | 15:03 |
| *** jimbobhickville1 has joined #openstack-sahara | 15:05 | |
| *** jimbobhickville has quit IRC | 15:08 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/sahara-ci-config: Add check jobs for python-saharaclient https://review.openstack.org/98794 | 15:24 |
| *** jpich has quit IRC | 15:34 | |
| *** skolekonov has quit IRC | 15:36 | |
| *** ViswaV has joined #openstack-sahara | 15:53 | |
| crobertsrh | I'll try one more round of bother....can anyone explain this to me? http://ur1.ca/hhls0 (after I pulled latest, even rm'ed venv and rebuilt) | 15:54 |
| *** ViswaV_ has joined #openstack-sahara | 15:54 | |
| elmiko | weird... | 15:55 |
| elmiko | i haven't seen that one | 15:55 |
| *** ViswaV has quit IRC | 15:57 | |
| *** julienvey has quit IRC | 15:58 | |
| *** IvanBerezovskiy has left #openstack-sahara | 16:04 | |
| *** sballe has joined #openstack-sahara | 16:25 | |
| *** akuznetsov has quit IRC | 16:31 | |
| *** bandarji has joined #openstack-sahara | 16:36 | |
| dmitryme | crobertsrh: haven’t seen it yet. Let us know if you find the solution | 16:39 |
| *** akuznetsov has joined #openstack-sahara | 16:39 | |
| *** akuznetsov has quit IRC | 16:42 | |
| *** tellesnobrega has quit IRC | 16:51 | |
| *** tellesnobrega has joined #openstack-sahara | 17:09 | |
| openstackgerrit | A change was merged to openstack/sahara: Add Spark to overview and feature matrix https://review.openstack.org/97709 | 17:15 |
| openstackgerrit | A change was merged to openstack/python-saharaclient: Updated from global requirements https://review.openstack.org/96828 | 17:15 |
| crobertsrh | dmitryme: Thanks. Will do. | 17:19 |
| openstackgerrit | Yaroslav Lobankov proposed a change to stackforge/sahara-ci-config: Option "min_transient_cluster_active_time" was added https://review.openstack.org/98841 | 17:20 |
| crobertsrh | Looks like 53e7d93ae4207d6469960af0482b0c65ffc577fd is where I break (sync latest DB code from oslo-incubator). Everything runs fine for me until that commit. | 17:22 |
| openstackgerrit | A change was merged to openstack/sahara: Rework keystone auth_token middleware configs https://review.openstack.org/74688 | 17:24 |
| openstackgerrit | Yaroslav Lobankov proposed a change to stackforge/sahara-ci-config: Option "min_transient_cluster_active_time" was added https://review.openstack.org/98841 | 17:26 |
| *** jimbobhickville1 is now known as jimbobhickville | 17:29 | |
| *** tellesnobrega has quit IRC | 17:44 | |
| *** witlessb has quit IRC | 17:47 | |
| SergeyLukjanov | tmckay, it should | 17:57 |
| tmckay | thanks | 17:58 |
| SergeyLukjanov | tmckay, I've not tested it with certs, but all configs are now exposed | 17:58 |
| tmckay | okay, maybe I can try it on our stack. | 17:58 |
| SergeyLukjanov | NOTE: https://review.openstack.org/#/c/74688/ merged, so, folks, you should update your config file | 17:59 |
| *** tellesnobrega has joined #openstack-sahara | 18:00 | |
| *** mattf is now known as _mattf | 18:24 | |
| *** bandarji has quit IRC | 18:30 | |
| *** bandarji has joined #openstack-sahara | 18:31 | |
| *** bandarji has quit IRC | 18:31 | |
| *** bob_nettleton has joined #openstack-sahara | 18:33 | |
| bob_nettleton | Hi All, is anyone seeing issues when trying to run the DB setup for Sahara, when creating a new dev env? I'm seeing some exceptions from the "sahara-db-manage" command, and was wondering if others are seeing it as well. | 18:35 |
| bob_nettleton | the main error I'm seeing is "AttributeError: 'NoneType' object has no attribute 'get_dialect'". | 18:35 |
| elmiko | yes | 18:35 |
| bob_nettleton | looks like a dependency breakage, but wanted to see if others are seeing this as well | 18:35 |
| elmiko | crobertsrh was having that issue earlier | 18:35 |
| elmiko | bob_nettleton: 13:22] < crobertsrh> Looks like 53e7d93ae4207d6469960af0482b0c65ffc577fd is where I break (sync latest DB code from oslo-incubator). Everything runs fine for me until that commit. | 18:35 |
| elmiko | that was a little bit ago | 18:36 |
| crobertsrh | actually, I think I'm still broken there too | 18:36 |
| crobertsrh | I'm still poking around, but have not yet sorted it all out | 18:36 |
| crobertsrh | bob_nettleton: Does, http://ur1.ca/hhmrx look familiar to you? | 18:36 |
| crobertsrh | I'm attempting to use sqlite, in case that winds-up mattering | 18:37 |
| bob_nettleton | crobertsrh, yes, that exception is exactly what I'm seeing. | 18:37 |
| crobertsrh | welcome to the club :( | 18:38 |
| bob_nettleton | :) | 18:38 |
| crobertsrh | I've even tried blowing away my old venv and reinstalling....no dice | 18:38 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed a change to openstack/sahara: Adding get_quotas to utils/openstack/nova.py https://review.openstack.org/98852 | 18:39 |
| bob_nettleton | crobertsrh, elmiko, ok, well thanks for the info anyway. I may just try to use a slightly older version of the code for now. | 18:45 |
| elmiko | bob_nettleton: np, good luck | 18:46 |
| *** bob_nettleton has quit IRC | 18:46 | |
| crobertsrh | Ok. Good luck. I'll be sure to let you know when I get it figured out. | 18:46 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed a change to openstack/sahara: Adding get_quotas to utils/openstack/nova.py https://review.openstack.org/98852 | 18:59 |
| elmiko | crobertsrh: man... no love from the horizon folks on your merges :( | 19:01 |
| crobertsrh | Yeah, I thought things were ramping up about 2 weeks back. I got a few -1's on a few things and worked through them. | 19:02 |
| crobertsrh | I think we even had a 1 or 2 reviews from cores at one point. | 19:02 |
| elmiko | oh well, we'll keep trying | 19:02 |
| crobertsrh | tmckay: have you tried out the new keystone auth stuff from Sergey yet? | 19:17 |
| tmckay | crobertsrh, not yet | 19:17 |
| elmiko | crobertsrh: you've got me afraid to upgrade | 19:18 |
| crobertsrh | ah, ok. I might back off it for now. Looks like "defaults" will not work for my setup | 19:18 |
| crobertsrh | I got past my other problem (I'm not exactly sure *what* did it though) | 19:18 |
| *** palar_ has joined #openstack-sahara | 19:19 | |
| elmiko | the exceptions you were seeing earlier? | 19:19 |
| crobertsrh | Yeah | 19:20 |
| elmiko | huh, weird | 19:20 |
| elmiko | gotta love it when problems just "go away" | 19:20 |
| crobertsrh | I think I made the mistake of tweaking multiple things at once | 19:20 |
| crobertsrh | so now, I'm not entirely sure what wound-up working | 19:21 |
| crobertsrh | but, I am able to work, which is also important | 19:21 |
| elmiko | always an issue when in uber-fixit mode | 19:21 |
| *** jimbobhickville has quit IRC | 19:22 | |
| *** palar has quit IRC | 19:23 | |
| *** palar_ has quit IRC | 19:25 | |
| *** akuznetsov has joined #openstack-sahara | 19:25 | |
| *** akuznetsov has quit IRC | 19:25 | |
| *** akuznetsov has joined #openstack-sahara | 19:26 | |
| *** akuznetsov has quit IRC | 19:27 | |
| *** akuznetsov has joined #openstack-sahara | 19:28 | |
| *** witlessb has joined #openstack-sahara | 19:33 | |
| *** boris-42 has quit IRC | 20:08 | |
| *** boris-42 has joined #openstack-sahara | 20:11 | |
| *** akuznetsov has quit IRC | 20:18 | |
| tmckay | I found a bug! I found a bug! | 20:22 |
| tmckay | we don't set mapreduce.jobhistory.address in our hadoop 2 clusters | 20:23 |
| elmiko | lol, nice! | 20:24 |
| tmckay | There are apparently cases in Hadoop 2 where the jobhistory server is contacted instead of the application master, and all of our worker nodes think the history is local. | 20:24 |
| tmckay | <property> | 20:24 |
| tmckay | <name>mapreduce.jobhistory.address</name> | 20:24 |
| tmckay | <value>0.0.0.0:10020</value> | 20:24 |
| tmckay | <description>MapReduce JobHistory Server IPC host:port</description> | 20:24 |
| tmckay | From a worker .... | 20:24 |
| tmckay | That's in the default xml, and I don't believe we fix it up anywhere. I am getting occasional MR failures trying to talk to 0.0.0.0:10020 | 20:25 |
| tmckay | usually seems to be when the two jobs that make up an oozie Java action are on different nodes ... must change the timing or the flow | 20:26 |
| * tmckay tries to verify | 20:26 | |
| elmiko | huh, interesting | 20:29 |
| tellesnobrega | crobertsrh: hey, you just reviewed a patch that i submitted and i have a question for you | 20:29 |
| elmiko | did you catch that in the logs, or was it an exception or something? | 20:29 |
| tellesnobrega | the patch is part of a bug solution | 20:29 |
| tellesnobrega | but it doesnt solve it | 20:29 |
| crobertsrh | tellesnobrega: ok | 20:29 |
| tellesnobrega | how should i reference it? | 20:29 |
| crobertsrh | There is a "Partial-Implements:" tag that you can use for blueprints | 20:30 |
| tmckay | hmm, looks like maybe we're supposed to set it ... | 20:30 |
| crobertsrh | so if a blueprint will require several patches, each can "Partial-Implements: bp <name>" | 20:30 |
| tellesnobrega | this is only a bug, i was just trying to keep it simple, commit this small part first and then the rest | 20:31 |
| crobertsrh | You can also use "Partial-Bug", I think | 20:31 |
| tellesnobrega | i will try | 20:31 |
| tmckay | elmiko, I've been playing with the job manager, and in so doing I ran some stuff that got KILLED because the Oozie launcher failed trying to contact 0.0.0.0:10020 from a worker. That port is listended for on the master, turns out it's the job history server port | 20:31 |
| crobertsrh | https://wiki.openstack.org/wiki/GitCommitMessages | 20:31 |
| *** miqui has quit IRC | 20:32 | |
| tmckay | elmiko, I found a JIRA in the hadoop stuff about a test rig that fails on a race condition if that config is not set | 20:32 |
| elmiko | tmckay: cool, sounds like an interesting investigation | 20:32 |
| tmckay | elmiko, I found that 0.0.0.0:10020 is the default value in mapred-default.xml in some of the client stuff under /opt/hadoop-2.3.0 | 20:33 |
| tmckay | so I am making the leap that we aren't setting it .... | 20:33 |
| tmckay | but, I could be wrong. The log message could be screwy and actually mean something else | 20:33 |
| elmiko | well, good luck | 20:34 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed a change to openstack/sahara: Adding get_quotas to utils/openstack/nova.py https://review.openstack.org/98852 | 20:36 |
| tellesnobrega | crobertsrh: found one, can you review it again? | 20:36 |
| crobertsrh | Sure | 20:37 |
| tmckay | elmiko, hmm, looks like in hadoop2 we know it exists, but we leave it as a "hidden conf" with the default value | 20:38 |
| tmckay | I wonder if we can just throw it in with the namenode and datanode during job generation... | 20:40 |
| elmiko | tmckay: which plugin is that for? | 20:41 |
| tmckay | vanilla 2 | 20:42 |
| tmckay | going to look at hdp, they touch it too (and actually test for it) | 20:42 |
| elmiko | yea, the hwx guys told me their setup looks for missed node types and adds them if necessary | 20:43 |
| *** crobertsrh is now known as _crobertsrh | 20:43 | |
| tmckay | hmmm, yep, they check for history server and update the mapred-site configuration in a method called "finalize_configuration" | 20:43 |
| tmckay | methings vanillar needs to borrow that | 20:44 |
| * tmckay notice the 'r' on vanillar | 20:44 | |
| tmckay | Massachusetts | 20:44 |
| elmiko | lol, totally | 20:44 |
| tmckay | I would like to say my work here is done, and fly off into the sunset, but, no ... | 20:45 |
| *** tmckay has left #openstack-sahara | 21:00 | |
| *** ghenriks has quit IRC | 21:20 | |
| *** ghenriks has joined #openstack-sahara | 21:21 | |
| *** witlessb has quit IRC | 21:28 | |
| *** witlessb has joined #openstack-sahara | 21:28 | |
| *** witlessb has quit IRC | 21:39 | |
| *** ViswaV_ has quit IRC | 21:45 | |
| *** ViswaV has joined #openstack-sahara | 21:46 | |
| *** witlessb has joined #openstack-sahara | 21:47 | |
| *** witlessb has quit IRC | 22:13 | |
| *** tosky has quit IRC | 22:13 | |
| *** bandarji has joined #openstack-sahara | 22:16 | |
| *** bradd1 has quit IRC | 22:18 | |
| *** miqui has joined #openstack-sahara | 22:20 | |
| *** sballe_ has joined #openstack-sahara | 22:46 | |
| *** sballe has quit IRC | 22:47 | |
| *** bradd1 has joined #openstack-sahara | 22:56 | |
| *** bradd1 has quit IRC | 23:29 | |
| *** tmckay has joined #openstack-sahara | 23:53 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Use in-memory sqlite DB for unit tests https://review.openstack.org/98923 | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!