*** bandarji has quit IRC | 00:00 | |
*** sreshetn1 has joined #openstack-sahara | 00:16 | |
*** sreshetn1 has quit IRC | 00:21 | |
*** hdd_ has joined #openstack-sahara | 00:31 | |
*** tnovacik has quit IRC | 00:33 | |
*** sreshetn1 has joined #openstack-sahara | 02:05 | |
*** sreshetn1 has quit IRC | 02:10 | |
*** tellesnobrega_ has joined #openstack-sahara | 02:14 | |
*** tellesnobrega_ has quit IRC | 02:14 | |
*** Networkn3rd has joined #openstack-sahara | 02:21 | |
*** coolsvap has joined #openstack-sahara | 02:33 | |
*** tmckay is now known as _tmckay | 02:40 | |
*** coolsvap has quit IRC | 02:45 | |
*** coolsvap has joined #openstack-sahara | 02:46 | |
*** coolsvap has quit IRC | 02:46 | |
*** coolsvap has joined #openstack-sahara | 02:46 | |
*** hdd_ has quit IRC | 02:52 | |
*** chandankumar has joined #openstack-sahara | 02:57 | |
*** hdd_ has joined #openstack-sahara | 03:46 | |
*** sreshetn1 has joined #openstack-sahara | 03:54 | |
*** sreshetn1 has quit IRC | 03:58 | |
*** Longgeek has joined #openstack-sahara | 03:59 | |
*** syncroswitch has quit IRC | 04:11 | |
*** Poornima has joined #openstack-sahara | 05:06 | |
*** hdd_ has quit IRC | 05:08 | |
*** sreshetn1 has joined #openstack-sahara | 05:09 | |
*** sreshetn1 has quit IRC | 05:14 | |
*** hdd_ has joined #openstack-sahara | 05:38 | |
*** chandankumar has quit IRC | 05:38 | |
*** Nikolay_St has quit IRC | 05:40 | |
*** chandankumar has joined #openstack-sahara | 05:54 | |
*** Longgeek has quit IRC | 06:04 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara: Imported Translations from Transifex https://review.openstack.org/162880 | 06:14 |
---|---|---|
*** sreshetn1 has joined #openstack-sahara | 06:21 | |
*** Longgeek has joined #openstack-sahara | 06:23 | |
*** hdd_ has quit IRC | 06:38 | |
*** coolsvap is now known as coolsvap|afk | 06:47 | |
*** sgotliv has quit IRC | 06:53 | |
*** coolsvap|afk is now known as coolsvap | 06:58 | |
*** tnovacik has joined #openstack-sahara | 07:19 | |
*** sreshetn1 has quit IRC | 07:20 | |
*** chlong has quit IRC | 07:21 | |
*** openstackgerrit has quit IRC | 07:35 | |
*** openstackgerrit has joined #openstack-sahara | 07:35 | |
*** sreshetn1 has joined #openstack-sahara | 07:36 | |
*** sreshetn1 has quit IRC | 07:44 | |
*** sreshetn1 has joined #openstack-sahara | 07:45 | |
*** coolsvap is now known as coolsvap|afk | 07:53 | |
*** coolsvap|afk is now known as coolsvap | 08:00 | |
*** sreshetn1 has quit IRC | 08:13 | |
*** sgotliv has joined #openstack-sahara | 08:15 | |
*** sgotliv has quit IRC | 08:20 | |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [HDP] Add validation check for dfs.replication https://review.openstack.org/162118 | 08:20 |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [HDP] Add validation check for dfs.replication https://review.openstack.org/162118 | 08:22 |
*** Nikolay_St has joined #openstack-sahara | 08:27 | |
*** IBerezovskiy has joined #openstack-sahara | 08:28 | |
*** sgotliv has joined #openstack-sahara | 08:33 | |
*** _tmckay has quit IRC | 08:33 | |
openstackgerrit | lu huichun proposed openstack/sahara: Add a common HBase lib in hdfs on cluster start https://review.openstack.org/162657 | 08:38 |
*** tmckay has joined #openstack-sahara | 08:48 | |
*** akuznetsov has joined #openstack-sahara | 08:48 | |
*** witlessb has joined #openstack-sahara | 08:51 | |
*** akuznetsov has quit IRC | 09:12 | |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [HDP] Add validation check for dfs.replication https://review.openstack.org/162118 | 09:25 |
*** sreshetn1 has joined #openstack-sahara | 09:29 | |
*** Poornima has quit IRC | 09:42 | |
openstackgerrit | Nikolay Starodubtsev proposed openstack/sahara: Take back upstream checks for commit message https://review.openstack.org/162932 | 09:44 |
*** Poornima has joined #openstack-sahara | 09:45 | |
*** sgotliv has quit IRC | 10:07 | |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [HDP] Add validation check for dfs.replication https://review.openstack.org/162118 | 10:12 |
*** chlong has joined #openstack-sahara | 10:17 | |
*** sgotliv has joined #openstack-sahara | 10:20 | |
*** Poornima has quit IRC | 10:30 | |
*** coolsvap is now known as coolsvap|afk | 10:33 | |
*** tosky has joined #openstack-sahara | 10:34 | |
*** chandankumar has quit IRC | 10:37 | |
*** Poornima has joined #openstack-sahara | 10:42 | |
*** sreshetn1 has quit IRC | 10:50 | |
*** sreshetn1 has joined #openstack-sahara | 10:50 | |
*** chandankumar has joined #openstack-sahara | 10:50 | |
*** chandankumar has quit IRC | 10:50 | |
*** chandankumar has joined #openstack-sahara | 10:51 | |
*** chlong has quit IRC | 10:51 | |
openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: Skip job_execution tempest client test https://review.openstack.org/162960 | 11:00 |
*** akuznetsov has joined #openstack-sahara | 11:03 | |
*** Krast has quit IRC | 11:09 | |
*** sreshetn1 has quit IRC | 11:11 | |
*** Longgeek has quit IRC | 11:14 | |
openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Global refactoring of CI scripts https://review.openstack.org/162102 | 11:16 |
*** sreshetn1 has joined #openstack-sahara | 11:18 | |
openstackgerrit | Nikolay Starodubtsev proposed openstack/sahara: Take back upstream checks for commit message https://review.openstack.org/162932 | 11:26 |
*** sreshetn1 has quit IRC | 11:28 | |
*** Longgeek has joined #openstack-sahara | 11:36 | |
openstackgerrit | Merged openstack/sahara: Imported Translations from Transifex https://review.openstack.org/162880 | 11:36 |
*** Poornima has quit IRC | 11:45 | |
*** coolsvap|afk is now known as coolsvap | 11:54 | |
openstackgerrit | Andrey Pavlov proposed openstack/sahara: Adding cluster, instance, job_execution ids to logs https://review.openstack.org/143964 | 11:56 |
openstackgerrit | Andrey Pavlov proposed openstack/sahara: Fixing log messages to avoid information duplication https://review.openstack.org/147504 | 11:57 |
*** sgotliv has quit IRC | 12:16 | |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [HDP] Add validation check for dfs.replication https://review.openstack.org/162118 | 12:20 |
*** tmckay has quit IRC | 12:26 | |
*** sgotliv has joined #openstack-sahara | 12:29 | |
*** akuznetsov has quit IRC | 12:32 | |
tellesnobrega | elmiko, i'm trying to execute a java job from a curl but i always get {"error_message": "The method is not allowed for the requested URL.", "error": 405} | 12:32 |
tellesnobrega | I used the cli and worked. Ran with debug, got the post dict but it still doesnt work | 12:33 |
tellesnobrega | elmiko, this is the curl call http://paste.openstack.org/show/191267/ | 12:33 |
*** chandankumar has quit IRC | 12:35 | |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Move updating provision progress to conductor https://review.openstack.org/159421 | 12:37 |
*** sreshetn1 has joined #openstack-sahara | 12:39 | |
*** _crobertsrh is now known as crobertsrh | 12:55 | |
openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Add unit-tests for new integration tests https://review.openstack.org/155298 | 13:03 |
*** DenisEgorenko has left #openstack-sahara | 13:05 | |
*** DenisEgorenko has joined #openstack-sahara | 13:05 | |
*** coolsvap is now known as coolsvap|afk | 13:06 | |
DenisEgorenko | Hi! Can somebody tell me, why job is failed? https://review.fuel-infra.org/#/c/4455/1 http://osci-jenkins.srt.mirantis.net:8080/job/6.1.mos.install-rpm/1/console | 13:07 |
DenisEgorenko | sorry | 13:07 |
DenisEgorenko | wrong chat | 13:07 |
openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Add unit-tests for new integration tests https://review.openstack.org/155298 | 13:12 |
*** tnovacik has quit IRC | 13:17 | |
*** macjack has quit IRC | 13:17 | |
*** tmckay has joined #openstack-sahara | 13:21 | |
elmiko | tellesnobrega: i think you want to POST not PUT | 13:26 |
*** akuznetsov has joined #openstack-sahara | 13:27 | |
elmiko | anyone know if the CI issues have been fixed yet? | 13:30 |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add usages for step_type field https://review.openstack.org/162997 | 13:30 |
vgridnev | seems that there are one fix for tempest job and one skip for some another test of tempest tests | 13:35 |
elmiko | vgridnev: ok, cool. hopefully tempest is working now | 13:36 |
vgridnev | this changes is going to be merged only : ( | 13:36 |
vgridnev | see: https://review.openstack.org/#/c/162960/ | 13:37 |
elmiko | oh... =( | 13:37 |
*** chandankumar has joined #openstack-sahara | 13:43 | |
*** sreshetn1 has quit IRC | 13:46 | |
openstackgerrit | Merged openstack/sahara: Fix tempest tests for Sahara https://review.openstack.org/162568 | 13:47 |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Implement poll util and plugin poll util https://review.openstack.org/157392 | 13:50 |
openstackgerrit | Merged openstack/sahara: [EDP] Add Oozie Shell Job Type https://review.openstack.org/159920 | 13:50 |
openstackgerrit | Merged openstack/sahara: Implemented multi-worker solution for Sahara API https://review.openstack.org/146661 | 13:50 |
openstackgerrit | Artem Osadchiy proposed openstack/sahara: Add Spark support for MapR plugin https://review.openstack.org/163007 | 13:53 |
openstackgerrit | Merged openstack/sahara: Add validation for node group templates update https://review.openstack.org/160814 | 13:55 |
openstackgerrit | Merged openstack/sahara: Adding additional validation to node group template edit https://review.openstack.org/161267 | 13:55 |
openstackgerrit | Merged openstack/sahara: Add validation for cluster templates update https://review.openstack.org/160939 | 13:57 |
openstackgerrit | Andrey Pavlov proposed openstack/sahara: Added support of instance locality to engines https://review.openstack.org/162172 | 13:57 |
*** hdd_ has joined #openstack-sahara | 14:01 | |
*** egafford has joined #openstack-sahara | 14:06 | |
openstackgerrit | Merged openstack/sahara: Collect errors in new integration tests https://review.openstack.org/157842 | 14:09 |
openstackgerrit | Merged openstack/sahara: Add concurrency support in new integration tests https://review.openstack.org/151288 | 14:09 |
*** tnovacik has joined #openstack-sahara | 14:10 | |
*** sreshetn1 has joined #openstack-sahara | 14:20 | |
elmiko | crobertsrh: if i want to test the guided cluster creation stuff, should i install this patch https://review.openstack.org/#/c/163016/ ? | 14:21 |
crobertsrh | elmiko: That will give you the most authentic up-to-date everything | 14:22 |
elmiko | k, thanks. hopefully i can get this working =) | 14:22 |
crobertsrh | Actually, I might want to rebase it real quick | 14:23 |
crobertsrh | I had a couple changes get merged that are "nice to have" | 14:23 |
elmiko | yea | 14:23 |
crobertsrh | Ok, rebased | 14:23 |
elmiko | i'll hold off for a bit, i've got plenty more reviews.... | 14:23 |
elmiko | oh nice =) | 14:23 |
openstackgerrit | Nikolay Starodubtsev proposed openstack/sahara: Take back upstream checks for import order https://review.openstack.org/163021 | 14:31 |
openstackgerrit | Nikolay Starodubtsev proposed openstack/sahara: Rewrite log levels and messages https://review.openstack.org/154037 | 14:34 |
*** chandankumar has quit IRC | 14:39 | |
*** hdd_ has quit IRC | 14:42 | |
SergeyLukjanov | dhellmann, I've released the 0.8.0 client and updated CR to global requirements https://review.openstack.org/#/c/155428/ | 14:56 |
tmckay | anybody know how the "updated_at" fields in the database are supposed to work? I'm calling udpate on a node group template and the updated_at field is not being filled in. | 15:02 |
tmckay | "created_at" is there, but updated_at is empty (or None). New template. Update doesn't seem to fill it in | 15:03 |
* tmckay brb | 15:03 | |
tmckay | SergeyLukjanov, alazarev ^^ | 15:04 |
SergeyLukjanov | tmckay, hm, that's strange | 15:04 |
alazarev | tmckay, yeap, it's strange, it works for job executions | 15:04 |
elmiko | does it change updated_at on an update() or only on a save() ? | 15:07 |
*** tosky_ has joined #openstack-sahara | 15:11 | |
*** tosky is now known as Guest18369 | 15:11 | |
*** tosky_ is now known as tosky | 15:11 | |
tmckay | elmiko, not sure. | 15:12 |
tmckay | sigh. okay. | 15:13 |
*** ylobankov has joined #openstack-sahara | 15:13 | |
elmiko | tmckay: if you're curious how it works, https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/models.py#L117 is where it comes from | 15:13 |
elmiko | i wonder if it's possible that we are just calling save() on the model, when updated_at expects us to call update() or something? | 15:14 |
tmckay | SergeyLukjanov, elmiko, alazarev, while we're here (I'll have to figure out this update problem) substitution is maybe the biggest issue with default-templates. I'm not sure how to do it. The simplest thing I think is to allow replacement values to be specified in a JSON file. So, you might have "flavor_id": "%(small)s" in a node group template json file, and you define "small" (and other values) in another JSON file passed as an ar | 15:16 |
tmckay | make sense? | 15:16 |
tmckay | config options don't map well, because the list of things to replace has to be known up front | 15:16 |
tmckay | I thought of allowing key/value pairs on the command line, but then it has to parse different types, etc, which json.load() will already do | 15:16 |
*** chandankumar has joined #openstack-sahara | 15:17 | |
crobertsrh | tmckay: I'm a little fuzzy on why a config can't be used. | 15:17 |
tmckay | elmiko, I'll poke at it a little more | 15:17 |
elmiko | so the user would call the cli app to generate a template, and they need provide all these values but you don't like sending them on the command line? | 15:18 |
elmiko | crobertsrh++ | 15:18 |
tmckay | crobertsrh, well, configs will mean that we have to explicitly list every possible substitution | 15:18 |
tmckay | so, what if you want to set flavor differently for several different templates? | 15:18 |
tmckay | how are you going to do that? | 15:18 |
tmckay | "flavor_id:" .... what? | 15:19 |
SergeyLukjanov | tmckay, no good ideas on your question ;( | 15:19 |
crobertsrh | I was thinking that your config could be a map of some sort | 15:19 |
tmckay | crobertsrh, so essentially the same as a json dict defining values, but specified in a config file? | 15:19 |
elmiko | why wouldn't it make sense for the user to create a config file and use it to generate a template? | 15:19 |
crobertsrh | {"vanilla": {"1.2.1": {<set of specific configs>}}} | 15:19 |
crobertsrh | with a {"default": ....} section winning out if nothing specific was given | 15:20 |
crobertsrh | tmckay: I suppose it's pretty similar :) | 15:20 |
tmckay | trouble with configs is they have to be stored in a known value | 15:20 |
tmckay | so, we would need something like "template_values = {}" with entries for each plugin/version, maybe | 15:21 |
tmckay | and then you look in template_values for the right thing | 15:21 |
crobertsrh | I was thinking that a config would be easier for a user to find/edit should they need to since it would be in a "standard" place | 15:21 |
tosky | oh, it's the same problem I'm facing when I need to provide "good" working templates in tempest for node group/cluster template API tests | 15:22 |
tmckay | crobertsrh, could be. Not sure how powerful configs may be. | 15:22 |
tmckay | tosky, how are you doing it? | 15:22 |
crobertsrh | Oh good, tosky will solve the problem for us :) | 15:22 |
elmiko | lol | 15:22 |
crobertsrh | and then he will test it for us...thanks tosky! | 15:22 |
* crobertsrh goes out for a drink | 15:22 | |
tmckay | the trouble here is time | 15:22 |
tmckay | the clock is ticking | 15:22 |
tmckay | loudly | 15:22 |
tosky | but it's a bit different: I have at most one template for each supported version for each supported openstack releases | 15:22 |
crobertsrh | and we lost an hour!!! | 15:23 |
tosky | with some specific keys just to test cluster config | 15:23 |
tosky | not necessarily the cleanest/user friendly templates | 15:23 |
elmiko | tmckay: just so i understand, this is for getting the values in to the cli app that generates templates? | 15:23 |
tosky | but anyway, it's just a set of nested dicts/lists :) | 15:24 |
tmckay | elmiko, well, there is a CLI that reads json files representing node group templates and cluster templates. It reads them and adds them to the database. | 15:24 |
*** tnovacik has quit IRC | 15:25 | |
tmckay | elmiko, the sticking point is what if those json files need substitution? for instance, flavor_id is one that can be customized. Other things with uuid values need to be substituted. | 15:25 |
elmiko | tmckay: i guess i'm trying to get the workflow down in my head. the user would be running the cli app to create them? | 15:26 |
tmckay | elmiko, image_id for node_group templates (not used very much, I think) and default_image_id for cluster templates are other cases | 15:27 |
tmckay | no | 15:27 |
tmckay | we create the base set of files, user runs the CLI to add a template set to his database so that when he starts sahara the first time he can run, say, a vanilla cluster without adding anything. | 15:27 |
tmckay | Just launch | 15:27 |
tmckay | goal is to pre-populate sane template sets for the out of box experience | 15:28 |
tmckay | so, depends on verbage. user is running CLI app to create node group and cluster templates in the sahara db, from json files that we distribute with Sahara | 15:28 |
tmckay | with a few values that can be replaced | 15:28 |
elmiko | and those values should be replaced by the user running the app? | 15:29 |
tmckay | yes | 15:29 |
elmiko | hmm | 15:29 |
tmckay | elmiko, they don't *have* to be. I made the starting directory an option | 15:29 |
elmiko | why not allow the user to create a yaml file with the subst values they want and then name it as a parameter to the cli app? | 15:30 |
tmckay | elmiko, so, you could say "copy this and fill in the appropriate fields" and then run against that. But that seems less nice. | 15:30 |
tmckay | elmiko, why yaml instead of plain old json? | 15:30 |
elmiko | less verbose | 15:31 |
tmckay | I don't actually know a whole lot about yaml | 15:31 |
elmiko | yaml is equivalent to json in many ways | 15:31 |
elmiko | i think it's actually a superset | 15:31 |
*** sreshetn1 has quit IRC | 15:32 | |
elmiko | you could even do it with a plain old ini file | 15:32 |
elmiko | we just need key,value pairs from the user right? | 15:32 |
tmckay | elmiko, well, yes. There is the question of whether or not to scope a particular set of replacement values to a particular template set, but it can work as a flat set. Especially since you can limit processing to a single directory (or a single plugin/version) | 15:34 |
tmckay | elmiko, I thought of command line, but it gets more complex if you want to say make a json file where you allow node_processes to be replaced | 15:34 |
tmckay | now you have to pass a list in a key/value pair | 15:34 |
elmiko | yea, but if you want to replace node_processes aren't we going past the convenience of canned templates? | 15:35 |
crobertsrh | +1, I think node_process tweaking is out of scope for this (at least for now) | 15:35 |
elmiko | the more you talk about it tmckay, the more i think ini file might be appropriate | 15:36 |
tmckay | possibly. I could limit replacements to specific options. Anything that takes a uuid, plus flavor | 15:36 |
elmiko | the user could specify a [DEFAULT] section that does replacement for every template type | 15:36 |
elmiko | and then specify [TEMPLATE_NAME] sections for specific template replacements | 15:36 |
elmiko | could do something similar with yaml as well | 15:36 |
elmiko | and both formats have convenient python parsers | 15:37 |
tmckay | elmiko, what's a python parser for an ini file? | 15:37 |
elmiko | configparser in the main lib | 15:37 |
tmckay | oh, so a standard .conf file. I don't think of them as "ini" | 15:38 |
elmiko | hehe | 15:38 |
tmckay | elmiko, that will work as long as the replacement set is restricted to a specific set of fields | 15:38 |
tmckay | elmiko, we can even do [plugin.version] for section headers | 15:38 |
elmiko | right | 15:38 |
tmckay | very easy to match up | 15:38 |
elmiko | and you can specify global options as well | 15:39 |
tmckay | elmiko, only question at that point is flavor. We may want multiple values -- "small", "medium", "large", "xtra-large", etc. | 15:39 |
tmckay | I've never tried specifiying a dict in a config file | 15:40 |
elmiko | you wouldn't want to do that | 15:40 |
tmckay | well, then, perhaps you need [plugin.version.name] | 15:40 |
elmiko | why not just let the user specify "flavor=small" for some template? | 15:40 |
tmckay | so that flavor can very across multiple templates | 15:41 |
elmiko | sure, just have something like | 15:41 |
tmckay | elmiko, right, but we have sets of templates. A set is a whole. so for vanilla 1.2.1, you have a bunch of ng templates and one or more cluster templates | 15:41 |
elmiko | [template name] | 15:41 |
elmiko | flavor=small | 15:41 |
elmiko | [other template name] | 15:41 |
elmiko | flavor=medium | 15:41 |
elmiko | at some point we need to define the bounds of what a user can define for use with these templates | 15:42 |
elmiko | if they want to play with all the knobs and buttons, then they should probably just craft their own json | 15:42 |
elmiko | i think the whole point here is simplicity, once it gets complex we have gone too far | 15:43 |
tmckay | alright, so last opinion. Do we map config sections to name alone (hdp, vanilla 1, and vanilla 2 might all have an ng called "master") and have the user pass different config files for different plugin/version, or do we map config sections to [plugin.version.templatename] | 15:43 |
tmckay | elmiko, ack on complexity, but it's tough to balance | 15:44 |
elmiko | i like the latter, just make it 1:1 section:template name | 15:44 |
elmiko | tmckay: agreed | 15:44 |
elmiko | i think it's easy for a user to understand that if they call out a template name as a section, then any variable will be substituted | 15:45 |
tmckay | elmiko, just to clarify --- you like the idea of a section called [hdp-2.0.6-master] ? | 15:45 |
tmckay | ok, you meant "the former" | 15:46 |
elmiko | if that syncs up nicely to the template name, then yes | 15:46 |
elmiko | yea, i guess former | 15:46 |
elmiko | i think it should be easy for a user to see a template named "hdp-2.0.6-master.json" and know that they should use that name as the section | 15:48 |
*** hdd_ has joined #openstack-sahara | 15:49 | |
tmckay | elmiko, okay, I'll think about how best to do that. We've got plugin/version/name in the json file itself, and then we have the possibility of naming convention | 15:49 |
elmiko | cool | 15:49 |
tmckay | elmiko, thanks. I've got to figure out this update issue then come back to this. | 15:50 |
elmiko | no worries, look at the yaml stuff too. just in case you feel like the user will need more than a simple key,value config file | 15:50 |
tmckay | elmiko, I think first pass should be simple key value. If someone wants to/needs to they can copy the default template dir and edit to their hearts content as a fallback | 15:51 |
elmiko | +1 | 15:51 |
tmckay | crobertsrh, thanks to you too | 15:52 |
*** bandarji has joined #openstack-sahara | 15:56 | |
*** Nikolay_St has quit IRC | 15:57 | |
tmckay | SergeyLukjanov, alazarev, oops, updated_at was not changed because the field values were actually all the same ... | 16:11 |
tmckay | elmiko, ^^ | 16:11 |
*** plieb has joined #openstack-sahara | 16:11 | |
elmiko | lol, nice | 16:12 |
*** akuznetsov has quit IRC | 16:15 | |
SergeyLukjanov | tmckay, :) | 16:22 |
*** sreshetn1 has joined #openstack-sahara | 16:24 | |
*** tnovacik has joined #openstack-sahara | 16:29 | |
SergeyLukjanov | folks, I'm going to move all non-started specs to liberty and I will create a wiki page for FPF exceptions; if we'll have something to return back to Juno - we'll just revert corresponding changes | 16:31 |
SergeyLukjanov | tmckay, elmiko ^^ | 16:31 |
tmckay | SergeyLukjanov, okay. default-templates and jobs-endpoint should both be started | 16:32 |
elmiko | SergeyLukjanov: sounds good | 16:32 |
SergeyLukjanov | tmckay, are you expecting need for FF exceptions for them? | 16:36 |
*** crobertsrh has quit IRC | 16:36 | |
tmckay | SergeyLukjanov, looks like March 19th? I hope not. I am a little worried about not enough review time. | 16:37 |
SergeyLukjanov | tmckay, yeah | 16:37 |
tmckay | SergeyLukjanov, trying hard to get someting up for default-templates. A few issues left. | 16:37 |
SergeyLukjanov | tmckay, but anyway I think it'll be ok to provide FF exceptions for them | 16:37 |
tmckay | SergeyLukjanov, okay. job-types endpoint is mostly done (I need to check reviews) | 16:38 |
tmckay | SergeyLukjanov, core of default-template is done, just need to do substitution and code cleanup. And then, we will actually need to make defaults for the plugins :) | 16:38 |
SergeyLukjanov | tmckay, yeah, I'm in progress of reviewing things, not so fast unfortunately :( and there was a vacation yesterday in Russia | 16:38 |
tmckay | SergeyLukjanov, one question, the template CLI touches the conductor (needs to in order to get template magic between cluster/node group templates). So, a bunch of options are registered with oslo from imports which show up in the help. Do you know a way to deal with that? I am going to look for a way to "unregister" | 16:40 |
tmckay | oslo config, I mean | 16:40 |
SergeyLukjanov | tmckay, oh... I'm not sure that it's really possible, it's very tricky | 16:41 |
tmckay | okay, I'll see if I can find something. maybe I could intercept the help :) | 16:42 |
*** tnovacik has quit IRC | 16:42 | |
openstackgerrit | Merged openstack/sahara-image-elements: spark: small syntax hiccup in the vars check https://review.openstack.org/162583 | 16:42 |
*** Nikolay_St has joined #openstack-sahara | 16:52 | |
*** DenisEgorenko has left #openstack-sahara | 17:00 | |
openstackgerrit | lu huichun proposed openstack/sahara: Add a common HBase lib in hdfs on cluster start https://review.openstack.org/162657 | 17:01 |
*** Longgeek has quit IRC | 17:06 | |
plieb | is there an easy way to build the sahara docs in html from source with sphnix on a system that does not have sahara installed? | 17:24 |
*** hdd_ has quit IRC | 17:25 | |
elmiko | plieb: you can run `tox -edocs` from the root of the project if you have tox installed | 17:26 |
elmiko | that will create a virtualenv to make the docs | 17:26 |
*** hdd_ has joined #openstack-sahara | 17:28 | |
*** coolsvap|afk is now known as coolsvap | 17:32 | |
*** sreshetn1 has quit IRC | 17:34 | |
*** crobertsrh has joined #openstack-sahara | 17:34 | |
*** tnovacik has joined #openstack-sahara | 17:36 | |
*** Nikolay_St has quit IRC | 17:40 | |
*** Nikolay_St has joined #openstack-sahara | 17:41 | |
plieb | elmiko: cheers | 17:43 |
elmiko | ;) | 17:44 |
tosky | interesting, dfs.replication for every plugin BUT CDH, which uses dfs_replication | 17:45 |
tosky | I wonder: why? | 17:45 |
elmiko | huh, oversite maybe? | 17:47 |
tosky | maybe it depends on CDH itself | 17:48 |
*** IBerezovskiy has quit IRC | 17:54 | |
egafford | Hi crobertsrh. Is it useful to rebase the shell action patch now, or should I wait until more changes are merged? | 18:01 |
egafford | crobertsrh: You certainly have a better sense of the pulse of these things in Horizon. | 18:01 |
crobertsrh | Might want to wait, especially if it isn't ready yet (back end isn't ready) | 18:01 |
*** victorfeitosa has joined #openstack-sahara | 18:02 | |
crobertsrh | I don't think it will be a particularly tricky rebase | 18:02 |
crobertsrh | I think the patch looks pretty good though. I'll have to try it out. | 18:02 |
egafford | crobertsrh: Yeah, I was happy enough with it. The code was perfectly extensible. | 18:04 |
crobertsrh | The Horizon stuff is pretty good....it's even better for our "well behaved" pages. | 18:04 |
crobertsrh | Some day we'll make the launch/config stuff a better horizon citizen, until then it's tribal knowledge ftw. | 18:05 |
* elmiko starts beating the sahara drums | 18:05 | |
crobertsrh | Oooh...Openstack Summit drum circle! | 18:06 |
*** sgotliv has quit IRC | 18:06 | |
crobertsrh | It's not exactly the party at the carnival place, but it's something | 18:06 |
egafford | I am uncertain that I want my job to be the party at the carnival place. | 18:06 |
egafford | That sounds less productive than other options. | 18:07 |
crobertsrh | Oh, egafford...you would have loved it | 18:07 |
elmiko | agreed, highly unproductive, yet highly fun =) | 18:07 |
crobertsrh | I almost died on a vintage bicycle/carousel thingy that went about mach 5 | 18:07 |
crobertsrh | My shoelace came untied and wrapped around the pedal | 18:07 |
crobertsrh | I got quite a workout trying to get it freed and keep it free of the pedal (of which I had no control once all 30 people were making this contraption go) | 18:08 |
egafford | Wait, there was an actual party at an actual carnival place? | 18:09 |
tosky | crobertsrh: oh, I forgot/didn't notice that "incident" | 18:10 |
elmiko | yea | 18:10 |
elmiko | egafford: it was a vintage 18/19th century carnival museum | 18:10 |
crobertsrh | tosky: I tried to play it cool. | 18:10 |
tosky | more laaate 19th/early 20th, maybe | 18:10 |
crobertsrh | It was my "inside" foot...I had to hold it straight out for 90% of the ride | 18:10 |
crobertsrh | Best workout I've had in ages | 18:11 |
elmiko | tosky: yea, my bad. 19th/20th | 18:11 |
crobertsrh | Plus my options for my left foot were bad too....awkwardly keep "pedaling" with it or hold it out too | 18:11 |
elmiko | lol, 18th would have been a little too early | 18:11 |
crobertsrh | I tried both....both were horrible | 18:11 |
egafford | elmiko, crobertsrh: Wow. | 18:12 |
elmiko | tosky won all the games too, he had a huge pile or prizes | 18:12 |
elmiko | egafford: yea, we kept speculating about how much money hp laid out for that party. it was intense | 18:13 |
crobertsrh | Tosky probably had to declare nearly 2 euros worth of prizes | 18:13 |
elmiko | lol! | 18:13 |
elmiko | i thought no declarations inside EU ? | 18:13 |
tosky | almost, I missed one of the (cheap-fake-retro) robots :) | 18:13 |
elmiko | aww =( | 18:13 |
crobertsrh | Oh man...you could have taken all of mine | 18:13 |
tosky | yep, not taxes inside the EU \o/ | 18:13 |
tosky | (no _additional_ taxes) | 18:13 |
elmiko | yea, that's more like it | 18:13 |
elmiko | tosky: are the taxes different from country to country? | 18:14 |
crobertsrh | I think I stealthily just laid my prizes in a pile and went for another drink. | 18:14 |
tosky | elmiko: yes, the VAT rates changes, unfortunately | 18:14 |
elmiko | tosky: ahh, that could get pricey in some places i imagine | 18:14 |
elmiko | crobertsrh: lol | 18:14 |
crobertsrh | tosky, egafford, Any chance you'll be in Vancouver? | 18:14 |
tosky | mine are probably close to 0 | 18:14 |
elmiko | =( | 18:15 |
egafford | crobertsrh: It sounds like I probably will, though my Tokyo chances are close to 0. | 18:15 |
crobertsrh | Other plans for Tokyo timeframe? Or just hating on Asia? | 18:15 |
egafford | It'll be nice to meet everyone in the actual world in which we are human. | 18:15 |
elmiko | egafford++ | 18:15 |
elmiko | and it's known as "meat space" | 18:15 |
egafford | Nah, Asia's fabulous. | 18:16 |
crobertsrh | Yeah, it was nice to see that tosky (Luigi) does not wear a green plumber uniform (at least not all the time) | 18:16 |
egafford | elmiko: I'm not a fan of the whole brevity thing. | 18:16 |
elmiko | egafford: lol | 18:16 |
* tosky facepalms | 18:16 | |
elmiko | crobertsrh: so bad... | 18:16 |
crobertsrh | I can't help it, I'm a nintendo kid | 18:16 |
crobertsrh | ok, a nintendo adult | 18:16 |
elmiko | no, you are still a nintendo kid ;) | 18:16 |
crobertsrh | ha | 18:17 |
*** victorfeitosa has quit IRC | 18:17 | |
tmckay | elmiko, crobertsrh, did you know that you can register config sections and options with oslo at any point and it will fill in the values from files it has already read? | 18:17 |
crobertsrh | tmckay: I'd never tried it, but that sounds pretty neat | 18:18 |
tmckay | which means I can define config sections on the fly for reading as I go through the templates .... | 18:18 |
elmiko | tmckay: that was my understanding | 18:18 |
crobertsrh | It's either really neat, or a bug waiting to happen | 18:18 |
tmckay | didn't know that | 18:18 |
tmckay | no, the docs actually say it | 18:18 |
openstackgerrit | Artem Osadchiy proposed openstack/sahara: Add Spark support for MapR plugin https://review.openstack.org/163007 | 18:21 |
*** tnovacik has quit IRC | 18:27 | |
openstackgerrit | Nikolay Starodubtsev proposed openstack/sahara: Take back upstream checks for import order https://review.openstack.org/163021 | 18:31 |
openstackgerrit | Nikolay Starodubtsev proposed openstack/sahara: Take back upstream checks for commit message https://review.openstack.org/162932 | 18:31 |
openstackgerrit | Nikolay Starodubtsev proposed openstack/sahara: Rewrite malformed imports order https://review.openstack.org/163123 | 18:31 |
*** Nikolay_St has quit IRC | 18:31 | |
*** Nikolay_St has joined #openstack-sahara | 18:32 | |
*** syncroswitch has joined #openstack-sahara | 18:40 | |
*** hdd_ has quit IRC | 18:41 | |
*** plieb has quit IRC | 18:49 | |
*** hdd_ has joined #openstack-sahara | 18:56 | |
*** sreshetn1 has joined #openstack-sahara | 18:56 | |
*** Nikolay_St has quit IRC | 19:00 | |
*** sgotliv has joined #openstack-sahara | 19:00 | |
*** Nikolay_St has joined #openstack-sahara | 19:01 | |
*** tosky has quit IRC | 19:15 | |
*** coolsvap is now known as coolsvap|afk | 19:20 | |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add usages for step_type field https://review.openstack.org/162997 | 19:28 |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add usages for step_type field https://review.openstack.org/162997 | 19:30 |
*** Nikolay_St has quit IRC | 19:31 | |
*** Nikolay_St has joined #openstack-sahara | 19:31 | |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Move updating provision progress to conductor https://review.openstack.org/159421 | 19:34 |
*** syncroswitch has quit IRC | 19:48 | |
*** Nikolay_St has quit IRC | 19:50 | |
*** Nikolay_St has joined #openstack-sahara | 19:51 | |
openstackgerrit | Sergey Kolekonov proposed stackforge/sahara-ci-config: Pin Glance client version https://review.openstack.org/163158 | 19:54 |
tmckay | elmiko, crobertsrh, okay, I have some config section magic going. Any thoughts for how to indicate that a field should be replaced? | 19:54 |
crobertsrh | $REPLACEME is always a favorite | 19:55 |
crobertsrh | %REPLACEME% is kinda windowsy | 19:55 |
crobertsrh | $(REPLACEME) | 19:55 |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add usages for step_type field https://review.openstack.org/162997 | 19:56 |
elmiko | tmckay: did you see my comments on the review? | 19:56 |
crobertsrh | Looking to see if there are other examples in openstack | 19:57 |
tmckay | elmiko, oh, no, I've had my head down. | 19:57 |
elmiko | no worries | 19:57 |
elmiko | my vote is for format syntax, i.e. {REPLACEME} | 19:57 |
openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add usages for step_type field https://review.openstack.org/162997 | 19:57 |
tmckay | elmiko, I've got the section stuff going. you can have [plugin] or [plugin_version] or [plugin_version_templatename] | 19:57 |
elmiko | tmckay: nice! | 19:57 |
tmckay | it will prefer more specific | 19:58 |
crobertsrh | +1 tmckay! | 19:58 |
tmckay | elmiko, now I just need an indicator that means "replace me with the corresponding field" | 19:58 |
tmckay | so, I have something in the config say, and it's "flavor_id = 17" | 19:58 |
tmckay | what should the flavor_id in the json look like? | 19:58 |
* tmckay goes to read comments | 19:59 | |
elmiko | tmckay: as for the way it stands now, my vote is for "flavor_id = {FLAVOR_ID}" | 19:59 |
crobertsrh | {<replaceme>} works for me | 19:59 |
crobertsrh | does case matter? | 20:00 |
elmiko | nah | 20:00 |
*** tnovacik has joined #openstack-sahara | 20:00 | |
crobertsrh | Heh...as long as it's not [default] :) | 20:00 |
elmiko | it only matters for doing the name of the interpolation variable in the format call | 20:00 |
crobertsrh | right, tmckay? :) | 20:00 |
tmckay | blah. DEFAULT | 20:01 |
crobertsrh | We had a little voyage into oslo_config a couple weeks back....case insensitive.....except for default != DEFAULT | 20:01 |
elmiko | oh man, that hurts like stubbing a toe | 20:01 |
crobertsrh | It was kinda fun | 20:01 |
*** hdd_ has quit IRC | 20:02 | |
crobertsrh | well, as much fun as watching someone else stub their toe can be | 20:02 |
elmiko | shaudenfreude! | 20:02 |
crobertsrh | heh, you spelled it quicker than I could | 20:02 |
tmckay | elmiko, crobertsrh, okay, I was thinking along the same lines. repeating the field name as the value gives reader a clue what to look for in the config files, I think | 20:02 |
crobertsrh | absolutely, +1 | 20:02 |
elmiko | works for me | 20:02 |
tmckay | breathing hard ... hope I'm coming to the end of this. Until reviews. | 20:03 |
crobertsrh | or, just name all variables schadenfreude and watch the users squirm | 20:03 |
elmiko | crobertsrh: LOL, you are so cruel! | 20:03 |
crobertsrh | reminds me of the "marklar" episode of South Park | 20:04 |
elmiko | haha, nice | 20:05 |
elmiko | now if you replaced them all with marklar, that would be funny | 20:05 |
crobertsrh | I suspect there is a crazy amount of source code available somewhere with heavy use of marklar as a var name | 20:06 |
elmiko | maybe in the obscure C coding comeptition | 20:07 |
crobertsrh | or maybe it's just my code :) | 20:07 |
crobertsrh | and I wonder why nothing works | 20:07 |
elmiko | lol | 20:07 |
crobertsrh | var marklar = marklar(marklar); | 20:08 |
crobertsrh | of course, I only do that in javascript | 20:08 |
crobertsrh | sorry, I only do that in marklar | 20:08 |
elmiko | would be funny to make a language named marklar | 20:10 |
elmiko | all library functions named marklar | 20:10 |
elmiko | all types named marklar | 20:10 |
elmiko | everything inferred | 20:10 |
*** sreshetn1 has quit IRC | 20:11 | |
tmckay | elmiko, crobertsrh, if I expect the case to match, then I can just use "value".format(**dict(CONF[section])) | 20:17 |
tmckay | haven't tried this yet | 20:17 |
elmiko | looks about right though | 20:18 |
tmckay | and, I can use String.Formatter.parse(value) as a guess to see if Formatter thinks there is something in there it can replace. | 20:18 |
elmiko | is CONF[section] denoting what the user will create? | 20:18 |
tmckay | elmiko, yeah, at that point it dict(CONF[section]) will return a dict with user specified conf values, with None for those not specified | 20:19 |
tmckay | if you replace with a None, not my fault | 20:19 |
tmckay | the validator will give an error eventually | 20:19 |
elmiko | cool, sounds good to me | 20:19 |
tmckay | so you think it's okay to expect case match? | 20:20 |
elmiko | yea i think so | 20:20 |
elmiko | especially if we provide the .conf file for the user to fill in | 20:21 |
tmckay | elmiko, all caps? | 20:21 |
elmiko | for the keys or the sections, or ? | 20:21 |
*** hdd_ has joined #openstack-sahara | 20:22 | |
tmckay | oh, the keys. I *could* lower() both of them at the time of replacement. Then you can have whatever sugar you want. | 20:22 |
elmiko | yea | 20:22 |
elmiko | i don't have a preference either way for all caps or all lower. | 20:22 |
tmckay | hmm , maybe CONF lowers the key values for me already when it reads. I'll check, | 20:23 |
elmiko | i guess, all lower would be nice. then it's similar to the main config file. more consistency =) | 20:24 |
tmckay | ok | 20:24 |
tmckay | opts are apparently the case you register them in. So I can lower the format string, to be nice, if you want to yell in your JSON | 20:25 |
elmiko | lol, NICE! | 20:26 |
*** albertoffb has quit IRC | 20:27 | |
crobertsrh | that sounds great. +1 | 20:30 |
tmckay | I guess if we require that the lower() of the field in the format string matches the name of the config option, I don't even have to play games with the formatter. | 20:31 |
tmckay | Just try it | 20:31 |
*** tnovacik has quit IRC | 20:38 | |
*** hdd_ has quit IRC | 20:39 | |
*** hdd_ has joined #openstack-sahara | 20:41 | |
tmckay | crobertsrh, hmm, I don't think file modification time can work. You've got the json file time, but the config file that drives the values could be different. | 20:44 |
crobertsrh | Yeah...that is probably not likely to be a good condition | 20:44 |
tmckay | crobertsrh, sqlalchemy is smart enough not to set the update time if the values did not change. Or, I can compare values myself (I am fetching it anyway at this point to check the timestamp) | 20:45 |
crobertsrh | I think I was slightly divergent in my streams of thought in the spec. | 20:45 |
tmckay | maybe we just try to update, and let sqlalchemy work it out. | 20:45 |
crobertsrh | A manual check is probably fine, if anything | 20:45 |
crobertsrh | yeah, I was just thinking about leaving it to the db gods | 20:45 |
crobertsrh | I think my time stamp bit was thinking that this might be an automated check of some sort | 20:45 |
crobertsrh | ...rather than a CLI tool | 20:46 |
crobertsrh | If you run it, we'll try to update it | 20:46 |
tmckay | well, even automated -- it's the sythesis of the config file and the JSON. Not really a (simple) way to track that. Didn't think of that til now | 20:47 |
tmckay | Darn, I worked hard on that timestamp code | 20:47 |
crobertsrh | Yeah....some things sound better in the spec than in the code | 20:48 |
*** sreshetn1 has joined #openstack-sahara | 20:55 | |
tmckay | crobertsrh, hmm, even if you script this with a cron job, I think it's okay. Default templates can only be updated from the tool. If you change a file and/or a config section and re-run, and it produces a delta according to sqlalchemy, then you changed it, right? | 20:59 |
crobertsrh | Makes sense to me, sounds right. | 20:59 |
elmiko | tmckay: do they really need to change on every file change? | 21:00 |
tmckay | elmiko, no. If I was an admin, personally, I'd run manually as needed. Like I decided to modify my defaults. | 21:01 |
elmiko | tmckay: i'm +1 for that as the default modus operandi | 21:01 |
tmckay | elmiko, we were trying to use file timestamp to skip updates if the file was older than the object in the db | 21:02 |
crobertsrh | I think that the admin manually running this is the most likely scenario anyways | 21:02 |
elmiko | why bother though, if the user asks, just let it update. imo | 21:02 |
tmckay | but what we send is file + config, so file timestamp is not the whole picture. | 21:02 |
crobertsrh | I can't imagine needing to tweak the default templates very often (maybe ever) | 21:02 |
elmiko | yea, totally | 21:02 |
crobertsrh | Maybe a few tweaks when you're getting started, but then forget about it. | 21:03 |
elmiko | crobertsrh: and if you did, you'd want to make sure that running the app did the updating | 21:03 |
tmckay | you might decided that your flavor needs to be bigger, or you want to change the node count, or something. | 21:03 |
crobertsrh | Yeah | 21:03 |
tmckay | You could just as well copy the templates and make your own non-default modifications | 21:03 |
elmiko | i just don't like the implicit updating, or not, depending on outside factors | 21:03 |
crobertsrh | Yeah, I could see a flavor update, maybe a node count, but mostly static forever and ever. | 21:03 |
tmckay | I wonder if the cron thing is really really rare. As in, we don't make any effort in that direction | 21:04 |
elmiko | tmckay: +1 for no effort | 21:04 |
elmiko | make the update explicit through the tool | 21:04 |
crobertsrh | I think cron can go >/dev/null for now | 21:04 |
tmckay | alright, we'll let sqlalchemy work it out. I'll just make the calls and report. | 21:04 |
elmiko | yea, if the user bones it up. then they'll have to fix it. explicit > implicit | 21:05 |
tmckay | thanks guys, this is thornier than it seemed at first. | 21:05 |
crobertsrh | That seems plenty good. 99.8% of all cases | 21:05 |
elmiko | glad to help =) | 21:05 |
tmckay | elmiko, yeah, +1 for *copying* the sahara set before you muck with it. | 21:05 |
tmckay | I have also included a "delete default templates" operation | 21:05 |
crobertsrh | If people have huge problems, we can tweak for L | 21:05 |
tmckay | so you can wipe the slate and start over | 21:06 |
crobertsrh | Should be plenty | 21:06 |
elmiko | tmckay: it might be a nice convenience to have an option to instruct the app to copy the templates to a local dir, but i wouldn't worry about it now | 21:06 |
*** tnovacik has joined #openstack-sahara | 21:14 | |
*** crobertsrh has quit IRC | 21:20 | |
*** sreshetn1 has quit IRC | 21:23 | |
tmckay | elmiko, okay, if I'm not crazy I have this essentially done. I need some code cleanup, some comments, maybe tweak the spec, and hopefully upload something later tonight or as early as possible in the morning | 21:24 |
tmckay | I'm sure there are gaps that the reviewers will find | 21:24 |
elmiko | tmckay: cool! i'll clear my review plate ;) | 21:27 |
tmckay | elmiko, heh. I haven't looked too closely but it looks like maybe gate tests are still messed up? | 21:32 |
tmckay | oh, maybe not. At least one is a merge conflict | 21:33 |
openstackgerrit | Trevor McKay proposed openstack/sahara: Add an is_default field to cluster templates and node group templates https://review.openstack.org/160920 | 21:33 |
elmiko | tmckay: there have also been some patches merged by hand, i think | 21:35 |
elmiko | mine hasn't rerun yet | 21:36 |
tmckay | k, thanks. | 21:36 |
*** chlong has joined #openstack-sahara | 21:40 | |
*** chandankumar has quit IRC | 21:48 | |
*** coolsvap|afk has quit IRC | 22:03 | |
*** chlong has quit IRC | 22:07 | |
openstackgerrit | Trevor McKay proposed openstack/sahara: Move node group template schema definition to its own file https://review.openstack.org/162314 | 22:11 |
*** tmckay has quit IRC | 22:14 | |
*** egafford has quit IRC | 22:16 | |
*** hdd_ has quit IRC | 22:20 | |
*** hdd_ has joined #openstack-sahara | 22:20 | |
*** tnovacik has quit IRC | 23:15 | |
*** hdd_ has quit IRC | 23:22 | |
*** bandarji has quit IRC | 23:50 | |
*** chlong has joined #openstack-sahara | 23:54 | |
*** hdd_ has joined #openstack-sahara | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!