Friday, 2017-02-03

*** jovon has quit IRC00:11
*** valw has joined #craton00:56
*** valw has quit IRC01:02
*** VW has quit IRC01:59
*** VW has joined #craton02:32
*** Syed__ has quit IRC02:35
*** VW has quit IRC02:56
*** VW has joined #craton03:33
*** VW has quit IRC03:55
*** VW has joined #craton03:56
*** VW has quit IRC04:01
*** jcannava has quit IRC04:23
*** jcannava has joined #craton04:25
*** VW has joined #craton04:47
*** VW has quit IRC04:53
*** valw has joined #craton04:58
*** valw has quit IRC05:03
*** VW has joined #craton07:13
*** VW has quit IRC07:18
*** VW has joined #craton08:28
*** VW has quit IRC08:33
*** valw has joined #craton08:59
*** valw has quit IRC09:04
*** VW has joined #craton09:59
*** VW has quit IRC10:04
*** VW has joined #craton11:17
*** VW has quit IRC11:21
*** tojuvone has quit IRC11:26
thomasemo/11:33
*** tojuvone has joined #craton11:56
suloo/12:19
thomasemAlright, I think this guy's out of WIP and ready for final review: https://review.openstack.org/#/c/427777/1512:25
thomasemOop, lemme give better link https://review.openstack.org/#/c/42777712:25
thomasemDon't like it referencing the patch set #.12:25
*** VW has joined #craton12:32
*** VW has quit IRC12:37
*** VW has joined #craton13:40
*** VW has quit IRC13:41
*** VW has joined #craton13:41
*** VW has quit IRC13:45
*** Syed__ has joined #craton15:56
thomasemHmmm... I think we might be getting back inconsistent data types for project ID from the db api.16:00
thomasemsometimes it's UUID type, other times it's str16:00
sigmavirusthomasem: oh?16:01
sigmavirusIt should always be UUID type16:01
thomasemOkay, lemme confirm. Could be something I did wrong.16:01
thomasemThanks for the clarification.16:01
*** VW has joined #craton16:03
thomasemTEM create: <class 'str'>16:05
thomasemTEM by_id: <class 'uuid.UUID'>16:05
thomasemYeah, on create it returns a Project object with an str ID attribute, and on get it returns a Project with a UUID type id.16:05
jimbakerright, we like to keep it in the abstract type to the extent possible16:06
thomasemsigmavirus: ^^16:06
jimbakersame with ip addresses, etc16:06
jimbakeror ints for that matter :)16:06
thomasemSo.. you're saying that we want that behavior?16:06
thomasemEven though it's inconsistent Project objects from the dbapi16:06
jimbakerthomasem, we want consistent behavior16:06
thomasemOkay, so then this is a bug. :)16:06
jimbakerand we want to keep it in the closest possible type16:06
jimbakerso that gives you the acceptance criteria :)16:07
jimbakerfortunately, we are not writing this code in tcl16:07
thomasemCool. Thanks. I'll write up the bug and work around it in the tests with notes.16:07
thomasemOh?16:07
thomasemThanks, btw sigmavirus, jimbaker16:09
sigmavirusthomasem: you're always welcome =)16:09
thomasemjimbaker: Speaking of acceptance criteria, did I adequately lay out what needs to be done for the project vars bug? https://bugs.launchpad.net/craton/+bug/164862616:10
openstackLaunchpad bug 1648626 in craton "Projects should support variables, which other entities use in resolution" [Wishlist,In progress] - Assigned to Thomas Maddox (thomas-maddox)16:10
thomasemAlmost done with 1-4, and then going to do the CLI work16:10
thomasemRan into some extra fun adding project to the resolution order when it came to fixing the tests, which is how I found the UUID/str bug above.16:11
jimbakertcl is a formerly popular scripting language for writing web sites. but tcl treats everything as string16:11
jimbakerstrings16:11
thomasemOhhhh16:11
thomasemHahahaha16:11
jimbakerit results in even more fun coercions than javascript ;)16:11
thomasemYeah, no thank you.16:11
thomasemThat sounds like a headache16:11
jimbakervery much so16:11
jimbakerdynamic typing, good. loose typing, no16:12
thomasemHaha, yeah. Well, I'm gaining new respect for strong typing nowadays.16:12
jimbakermaybe this is no typing for tcl? i have never thought about, or properly investigated. but i'm not going to16:12
sigmavirus"strongly typed" isn't a real thing16:12
* sigmavirus puts on pedant hat16:12
thomasemI went from Java/C++ to Javascript to Python to Golang.16:12
sigmavirusthomasem: oh you mean you like highly intelligent compilers that take advantage of static typing16:13
sigmavirus;)16:13
jimbakerone could always write in pascal...16:13
sigmavirusJava -> C -> {Python,Perl} -> Clojure -> Python -> Rust for me16:13
sigmavirus(There were others along the way)16:13
jimbakeri'm not certain if any coercions were available. that goes way back16:14
thomasemsigmavirus: That's what I mean, yes. What did you think I meant? :)16:14
sigmavirusthomasem: I'm mostly teasing over the use of "strong" or "loose" w/r/t to typing16:14
jimbakeranyway, typing discussions involves lots of typing16:14
sigmavirusjimbaker: I wonder if we should add mypy to craton eventually and use typing16:15
jimbakerbut the usual distinction i would make is between static and dynamic; as further refined by gradual typing systems16:15
jimbakersuch as mypy16:15
thomasemjimbaker: I see what you did there.16:15
sigmavirusThat wouldn't have caught this for thomasem but16:15
thomasemHaha, I16:15
sigmavirusfor those of you using operating system editors (*cough* emacs) and IDEs, that would be useful ;)16:15
thomasemI'm sure we could go round-and-round on opinions of how different languages and compilers handle types.16:16
jimbakersigmavirus, i certainly would approve for us adopting mypy as a type checker for our code16:16
thomasemAlright, folks. Going back under water. Want to fix these tests. If I can get this finished before the end of the week I'll feel productive.16:17
jimbakerthat this can be done incrementally (gradual typing ftw!) is a huge plus16:17
jimbakerthomasem, +116:17
sigmavirusjimbaker: agreed. I've not had a project to really test out mypy on16:18
sigmavirusfigured we'd use craton as the guinea16:18
jimbakeryep, and if i have a chance to get back into jython (specifically 3.x), it will be useful background as well16:20
thomasemAlright, would really appreciate eyes on https://review.openstack.org/#/c/42777716:38
thomasemOh, project isn't added to the CLI yet.17:14
thomasemHmmm, so that'll block the project variables bug for a time until we shave that yak.17:14
sigmavirusheh17:17
thomasemhttps://bugs.launchpad.net/python-cratonclient/+bug/165923717:18
openstackLaunchpad bug 1659237 in Craton's Python Client "Client is missing Project commands" [Undecided,New]17:18
thomasemShall I pick that one up next, then?17:19
thomasemThen it'll probably follow https://review.openstack.org/#/c/427032/17:20
jimbakerthomasem, yes, that would be a natural fit17:25
jimbakeri still need to finish up https://review.openstack.org/#/c/427032/17:25
jimbakeralthough we got a lot of demo points out of it :)17:25
thomasemLOL yeah, I was wondering about that.17:25
thomasemBut, now I remember we were demo'ing in-review stuff17:25
thomasemIt's all good. I'll pick up adding project CLI commands, and then when that patch is done for cell/host/region vars, I'll follow-up with project vars support17:26
jimbakerthomasem, sounds like the right timing to me17:44
antonymhey guys, few questions when you have some time... have you thought about doing tenant -> *cloud* -> region -> cell -> host -> *container*?  are you going to support loading multiple vars at once?  and have you pieced together anything for like importing all variables from like ansible facts to import them as variables on a host so that i have a full picture of that inventory?17:50
*** cloudnull has joined #craton17:51
*** thomasem has quit IRC17:56
*** empanada has quit IRC17:56
*** d34dh0r53 has quit IRC17:56
*** d34dh0r53 has joined #craton17:56
*** anonymike has joined #craton17:57
*** thomasem has joined #craton17:59
jimbakerantonym, so we have discussed bulk import - sort of the opposite of the ansible inventory endpoint18:06
jimbakerright now our assumption has been that the client probably is fine, with whatever corner cases removed. but if we find otherwise, we can definitely do a bulk endpoint18:07
antonymi guess there's a few ways, the initial load of the environment to populate the db, and then the ongoing passes to collect and update data i guess18:07
jimbakerantonym, so for multiple vars, do you mean for different resources at the same time? or for the same resource. (if for multiple, that would be the bulk above... which probably could be done in a reasonable fashion)18:08
antonymyou probably have your openstack configs you want to track that configure the environment but also have a way to query information about the host itself, like bios revs, network configs, etc18:08
antonymso for instance i had a tool that worked with our galaxy that could grab all of the info of the server and in one call to galaxy update those attributes18:09
jimbakerantonym, it would be very straightforward to do. ideally it would not be ansible specific, but that massaging can be done in the tool calling the client18:09
jimbakerantonym, got it18:09
antonymi think i was just using the ansible uri method to push those values in18:10
jimbakerso let's say we had the following tool: it grabbed the various facts from one or more ansible files. it then produced a json that reversed the whole override stuff. then we feed that into a bulk endpoint18:11
antonymyeah, i think that would be useful for the configuration part18:11
jimbakerso the good thing would be, it wasn't dependent on ansible's rules18:11
jimbakerand could also work for importing in say puppet hiera files, etc18:12
antonymare openstack configuration variables essentially the same as server/container attributes?18:12
antonymor is the goal to keep those seperate18:12
jimbakeryou mean for OSA?18:12
antonymjust in general i guess... if i wanted to regeneate my configs from the stuff in inventory i probably wouldn't want the server attribute variables bundled in there18:13
jimbakerright now, we are assuming just a bag of attributes. but see this blueprint for possible implementation: https://blueprints.launchpad.net/craton/+spec/craton-namespaces18:13
jimbakeryeah, so i think the namespace stuff is exactly what we need here :)18:14
antonymyeah, kinda sounds like it18:14
jimbakerfwiw, i like the / because i don't think this is used in any key structure i'm aware of18:15
jimbakerso it will be easy to just do now as a convention, then start adding support18:15
jimbakerjust revisited https://docs.puppet.com/hiera/3.2/configuring.html#example-config-file to jog my memory on hiera files18:20
jimbakeransible restricts keys to being valid python identifiers from what i recall18:20
jimbakeryaml has no restriction other than being valid unicode, so that's effectively what our variables support for key/values now, http://yaml.org/spec/1.2/spec.html#id276404418:24
jimbakerantonym, anyway, i think it's pretty clear that we want namespace support to support your needs, plus others that come up. hopefully we don't have to spend much time on a separator character18:25
jimbakeri just don't want it to be backslash, like it is in php. that's just wrong18:25
antonymyeah, think we just want to be able to do a full collection of the hosts outside of the openstack configuration and store them so that they are easy to query18:27
jimbakerand that can be readily done with a namespace filter18:27
antonymso that we can build tools on top of it to drill down to things we're looking for18:27
jimbakerfor bulk, maybe something simple like18:27
*** jovon has joined #craton18:28
jimbakerthis payload [{resource_type: <type>, resource_id: <id>, [attribute: value]*, variables: <json>}, ...]18:29
jimbakeron the server side, this would build up the corresponding obj graph, which we would then serialize. usual concerns would presumably apply about maybe chunking into pieces for actual commits18:31
antonymyeah18:32
jimbakerthe ansible data collection script would use the grouping structure to put in at the appropriate level. maybe it could infer some additional grouping? as far as the bulk endpoint is concerned, it doesn't matter18:33
jimbakerantonym, back to the original point. i think the idea is that keystone domain like concepts would let us combine multiple projects for a given customer18:35
jimbakerand that can be part of how the rbac role assignments work. the specific integration with keystone (if desired) would still have be worked out18:36
jimbakeras for the other part18:36
jimbaker host -> *container*18:36
jimbakerthis is implemented, but not yet in the client/CLI18:36
jimbakersulo will be working on this18:36
jimbakerso project effectively equals cloud from our perspective18:37
antonymso your project is essentially the cloud level with a tenant id then?18:37
jimbakeryeah18:37
jimbakerthe assumption is that the project equals the level of isolation that we see in clouds18:37
antonymmakes sense18:38
jimbakerso if there are project vars, or workflows that are "shared" across projects, they do have to be copied18:38
jimbakerbut we will feel this is sort of why one isolates into a cloud to begin with18:38
jimbakerso yeah, sounds like we agree on this decision18:39
antonymgood to know about the containers too18:39
jimbakeryeah, this is super important for us18:39
jimbakeri usually call this when talking about it the "device tree"18:40
antonymone more question i guess... i think i saw a blueprint about this too, but being able to query for servers with specific attributes... like ?datacenter=dfw&os=redhat&rack=d12.dfw is that still in the works?18:40
jimbakerso that works today at the REST level. WIP for the CLI18:41
antonymone thing i was using for discovery of the server was using lldp to pick up the switch name and port on the network and using that to determine the node i was looking for18:41
jimbakerthe one thing it's broken on is that the var search does not support resolution18:42
jimbakerhttps://bugs.launchpad.net/craton/+bug/166122618:42
openstackLaunchpad bug 1661226 in craton "Filter by variables lacks resolved variables" [Medium,Confirmed]18:42
antonymok, i'll poke around at the rest level to see how that works18:42
antonymgot a dev env set up so was going to start trying to load it with data to see how it works18:43
jimbakerantonym, i suggest you look at this https://review.openstack.org/#/c/427717/18:44
jimbakeri should add this info to the bug that sulo posted in response to my review18:44
jimbaker:)18:44
antonymah, cool18:45
jimbakerthomasem, you mentioned acceptance criteria earlier18:45
jimbakerthe other thing i go for is reproducibility of bugs18:45
jimbakerso explicit code that i can use, always awesome18:45
jimbakerjovon, also highly handy for our docs18:45
jimbakerantonym, anyway, it can be a bit painful to figure out the exact REST incantation18:46
jovon+118:46
jimbakerwe will hopefully get better in documenting this18:46
jimbakerantonym, in the meantime, i will add the discussion of a bulk endpoint as a feature request, and we will go from there18:47
antonymyeah, sounds good, just thinking doing one var per call might be a bit much for the amount of data we'd probably want to pull in18:48
jimbakerantonym, https://bugs.launchpad.net/craton/+bug/166171418:58
openstackLaunchpad bug 1661714 in craton "Bulk endpoint for working with resources" [Undecided,New]18:58
antonymnice, thanks18:59
jimbakernp19:00
*** VW has quit IRC19:42
thomasemjimbaker: agreed. I will start adding that to my bug reports. Apologies.19:43
thomasemjimbaker: is there a particular one you're looking at that I could help by getting some reproducing code for?19:44
cloudnullhey all, look to see if craton can do something like so http://cdn.pasteraw.com/90abpeqkkw8sa0zxaa1vj3jm03ld5rl ?20:02
cloudnull's/look/looking/g'20:04
sulocloudnull: most of it yes, except there is no specific type for container, we just separate by "device_type"20:08
sulobut would like to know if thats something thats needed20:08
suloi've had a few request on container type .. just not sure what properties we want to track20:08
cloudnullits a crufty drawing but mocks out some of the env types I've run up against.20:09
cloudnullesentially a single environment owner w/ many disconnected clouds under it20:10
cloudnulllooking into the API is it possible for a host to exist in >1 cell/region ?20:10
sulono20:10
cloudnullthinking of regions as different isles in the same DC20:10
suloeverythin is assumed to be under one region20:10
cloudnullwith an interconnect20:10
cloudnullok20:11
sulois that a real use case though ?20:11
cloudnullcells?20:11
cloudnullwe have that in the osic today w/ ironic in one region and vms in the other20:11
cloudnullboth within the same dc20:11
cloudnullsharing parts of the control plane20:11
sulothats really good to know .. all our assumptions so far has been customer(tenant) -> n x region -> each region -> n x cells and so on20:13
cloudnullwhich also covers networking gear20:13
cloudnullthough I've not dug into that part of the craton scheme20:13
cloudnullok20:13
cloudnullidk if tracking the containers specifically will be of benifit, i wrote it down because we have in OSA. we might be able to achieve the same thing by simply allowing unstructured and nested data to be added to a specific host.20:15
cloudnullthough the containers are consumers of IP space so idk if craton wants to track that ?20:16
suloyeah, right now, we just allow creating generic hosts with "device-type" field20:16
suloso device type could be server or container20:16
antonymyeah, i was kinda on the fence with containers, but it might be nice to store all of that ip data... yes20:16
suloassuming they have the same properties20:16
sulobut if we need differnt things then its fairly easy to add it20:16
cloudnullwithin osa the containers will assume the properties of the host and then allow specifics vars to be added to a given container as needed.20:17
suloso that use case is covered as each device can have its own vars20:17
cloudnullso in the craton scheme it'd be similar to the profile of a host. however containers are considered ephemera20:17
cloudnullok.20:18
cloudnulland a device can be nested under a host?20:18
sulocloudnull: yes, parent -> child20:18
cloudnullok20:18
sulowe are in the process of exposing the tree from the api .. but we can get th parent_id right anow and draw it out on client side if needed20:19
cloudnulland can a device exist in two regions ? or is that bound to the constraint mentioned before?20:19
suloantonym: so did the current structure not work for you ?20:20
cloudnullIE a switch in two cells, regions ?20:20
sulocloudnull: right now it cant, unless the same device is done twice20:20
cloudnullok20:20
sulocloudnull: we can work on this use case20:20
cloudnullok20:20
sulocloudnull: so you'd need both cells and devices to be in more than one region ?20:21
suloi guess just devices20:21
cloudnulldevices and hosts.20:22
sulook20:22
suloantonym: if you'd need something specific on containers tracked that is not currently there do let know20:23
sulotoan has asked more than once that we support continers specifically20:23
suloi dont have the exact req's though .. as to what to track vs whats available in hosts right now20:24
antonymi could see it'd be nice to have the container generated name and ips that were being used tracked in the db20:24
antonymmainly for querying which container is on what host if there was an issue from the logs20:24
suloah gotcha20:25
suloso we can do that now in a way .. cratea host with device_type ="container" .. it can have ips etc .. then partent_id = the host where it resides20:25
suloso when we draw out the tree .. we will see that host x has containers a b c etc20:26
antonymyeah, that seems like that would track it well20:26
suloantonym: do you think we'd need a separate containers specific endpoint20:26
sulojust thinking if that makes things more easy to use20:27
antonymendpoint?20:28
antonymoh for querying?20:28
suloi mean rather than doing host GET  type=container .. just doing GET container20:28
suloyeah for query20:28
suloi guess it only makes sense if there were container specific things to track20:29
sulothat different from what we have in hosts20:29
antonymyeah, it might, i'm just wondering if there's any attributes more that we'd need20:29
suloyeah same20:30
antonymwonder if we'd get to the point where we capture processes running in the container20:30
antonymor maybe define the roles of the container20:30
*** VW has joined #craton20:41
Syed__sulo: uploaded the patch for project and user update, kindly look20:49
*** toan has joined #craton21:04
*** valw has joined #craton21:05
*** valw has quit IRC21:09
*** VW has quit IRC21:32
*** VW has joined #craton21:32
jimbakerthomasem, i was mentioning this with respect to https://bugs.launchpad.net/craton/+bug/1661226 - not specifically one of your bugs you filed recently21:35
openstackLaunchpad bug 1661226 in craton "Filter by variables lacks resolved variables" [Medium,Confirmed]21:35
thomasemjimbaker: ohh gotcha, okay. Good to know!21:36
jimbakerbut in general, if we have a bug that shows the failing REST API call, this is very good. so this would ideally be: 1) specific request; 2) response; 3) relevant traceback in the log file of the API serfver21:36
thomasemgot https://review.openstack.org/#/c/428996 up and working now21:36
jimbakersimilar specificity in bug reporting for say using the python client, CLI, etc21:37
thomasemAgreed.21:37
thomasemThat review is for basic Project support in the CLI ^^21:37
jimbakerthomasem, nice21:38
thomasemHoping we can dry out a lot of that in the future.21:39
thomasemSince all of the *_shell.py have a lot of repeated stuff21:40
thomasemBut, not a big deal... it's really not much code.21:40
thomasemBut code that doesn't exist need not be debugged.21:40
thomasemGonna test that patch against a master craton-api21:47
thomasemrealized I was testing against my patch. Want to be sure it works that way, too.21:48
thomasemYep, works.21:51
jimbakerso cloudnull, sulo - our assumption is that a project = cloud22:00
jimbakera device can be in only one region, and optionally a cell. this is based on our understanding of standard DC layout22:01
jimbakerbut a device can  have any number of labels; which can overlap with other devices as desired. so n-to-m relationship between a device and a label22:02
jimbakerthis allows for logical grouping22:02
jimbakerthomasem, lots of repeated code in the CLI22:03
jimbakertake a look at some stuff i did to address this in my WIP branch22:03
jimbakerhttps://review.openstack.org/#/c/427032/3/cratonclient/shell/v1/hosts_shell.py22:04
jimbakercan be further improved, and i'm sure i will when i get that fixed up22:04
jimbakerbut just as a general implementation principle22:05
jimbakerwhen i see the same error handling code again and again - well, it's just a bit much for me22:05
jimbakerfortunately, decorators22:05
jimbaker:)22:05
thomasemHaha, agreed22:06
thomasemThanks for sharing.22:06
thomasemSo, in past projects I've been on there's been sort of a standard of trying to keep changes relevant to the referenced bug/task, where refactorings midway through can tend to derail or bloat code reviews a bit. In those projects we would set up a separate task to specifically attack refactorings separately. I'm assuming that's not a thing on this project?22:08
thomasemUltimately I want to avoid derailing code reviews with unrelated changes, so want to know what's generally accepted behavior on that front.22:10
thomasemAnywho, I need to get running to prepare for tonight. Hope everyone has a lovely weekend! Thanks for being so welcoming and helping me get started, everyone.22:11
cloudnulljimbaker: how fesable would it be to get it so that a single project could have many clouds?22:17
cloudnullI updated my crufty drawing but this is more what my enviornments looks like http://cdn.pasteraw.com/mclm54g53jmi3k4w56ucgcplbocd2q322:17
cloudnullprimary concerns for me would be a host in >1 cell/region and a "device" (network or otherwise) in >1 cell/regio n22:19
cloudnullwhen we say standard dc layout are we talking about rax public cloud? asking because standard dc layouts in ! rax public cloud would be flat and outside of openstack there's not concept of cells (maybe we mean cabinets?)22:22
cloudnull**there's no22:25
*** VW_ has joined #craton22:35
*** VW_ has quit IRC22:35
*** VW has quit IRC22:38
jimbakerthomasem, generally best to do refactorings separately23:00
jimbakeroccasionally i will do what is going to be a refactoring in a bug fix, with the intent of doing it later more pervasively23:00
jimbakercloudnull, it would require some work to had this additional layer. but again the whole intent in our design was that a project = a cloud23:00
jimbakercloudnull, for cells: they are optional in our model23:01
jimbakeroutside of openstack, one example that is reasonably close is an availability zone in AWS23:02
jimbakeras for cabinets, we capture this with the idea of a general device tree23:03
jimbakerone priority change we agreed on yesterday, based on toan's request: sulo is adding support for the device tree, which is generally a cabinet with everything in it, down to any containers23:04
jimbakerthis is supported in the dbapi, but needs support pushed out to the client/CLI23:04
jimbakerthe specific thing we agreed on is to make it to get all the info about a cabinet, without requiring additional navigation. mostly because this seems like such a common thing to do23:05
jimbakerhope this makes sense!23:05
*** VW has joined #craton23:10
*** VW has quit IRC23:15
cloudnullmakes sense. if there's a meeting that I can attend and/or a spec that you think I should write to see about getting another layer in between there for a single project w/ many sub clouds I'd be happy to try and get that submitted.23:31
cloudnullmost of my environments(projects) have several disconnected clouds.23:31
cloudnulland i'd love to see about trying to use this in a more meaningful way if at all possible23:32

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!