Wednesday, 2018-09-26

dmsimardwkid: hey did you end up figuring it out ? I'm curious :D02:08
openstackgerritVieri proposed openstack/ara-server master: fix tox python3 overrides  https://review.openstack.org/60531603:03
*** apollo13 has joined #ara08:40
apollo13hi there, is there any way to run ara without direct access to a database? like an append only rest api?08:40
apollo13mhm, nope the code seems to be rather clear :D08:41
apollo13I am getting this warning: https://dpaste.de/1FqL/raw and unlogged entries. is this worth reporting or is 3.6 unsupported completly? it only happens for some tasks…09:04
dmsimardapollo13: (sorry just woke up)11:42
dmsimardThis looks like a module-specific issue11:42
apollo13dmsimard: no worries, it is not like I am paying you for anything!11:42
apollo13the module is django_manage11:43
dmsimardIt returns data structured in a way that's not expected11:43
dmsimardOr at least that's what it looks like11:43
apollo13https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/web_infrastructure/django_manage.py it is; if you tell me what I need to debug I'll happily do that11:44
apollo13maybe tuple on https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/web_infrastructure/django_manage.py#L304 works?11:44
apollo13or rather 2 lines above call tuple instead of list11:44
dmsimardChanged is supposed to be a boolean ?11:45
apollo13is it?11:46
dmsimardI can look later after properly waking up11:46
apollo13I'll "bool()" changed for now and see if that fixes it11:46
apollo13sure, take your time. I'll lurk around11:46
dmsimardhttps://github.com/openstack/ara/blob/master/ara/models.py#L37711:47
apollo13well yes, in ara; was wondering about ansible11:47
apollo13but looks to be the idea11:48
apollo13I'll propose a bug fix to ansible11:48
dmsimardWell the variable is supposed to be a bool, but I guess Ansible doesn't enforce it11:48
apollo13the question is where to fix it11:49
apollo13if ansible doesn't enfore it, it would make sense for ara to bool() it11:49
apollo13or change ansible to ensure it is bool11:49
apollo13or just fix the django module11:49
apollo13let me ask in #ansible-devel11:49
dmsimardFrom ARA's perspective, we can coerce it to bool I guess, but it's not exactly accurate11:50
apollo13well if the field is required to be bool in the database, it seems like a sensible "defensive coding" change11:50
dmsimardLike that module would always return changed trur11:50
dmsimardtrue*11:51
dmsimardDespite what actually happened11:51
apollo13why would it always return true?11:51
apollo13https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/web_infrastructure/django_manage.py#L297 sets changed to false initially11:51
dmsimardIn python as soon as there is a string, it's true11:51
apollo13bool([]) would be false11:51
dmsimardAh, I didn't check what happened before11:51
apollo13well I guess it depends on what exactly ends up in filtered_output in the end11:52
dmsimardAnyway, django_manage needs to properly return a bool there and the results in a msg field or something like that11:52
dmsimardYou can carry a patched version of the module in your role/etc to test11:53
apollo13yeah, I know how to do that, just wanted to so if that is something you wanted to fix either way11:53
apollo13https://github.com/ansible/ansible/pull/4615712:09
apollo13dmsimard: lets see, ansible will probably accept my PR. that said; what would you fine folks think about adding a rest API to ara for the plugins? I rather not expose my database directly :/12:19
dmsimardapollo13: you mean something like this? https://demo.getara.org/api/v1/ :p12:41
apollo13dmsimard: and a callback that submits to that, yes12:41
dmsimardapollo13: it exists12:42
dmsimardapollo13: it's called ara 1.012:42
apollo13ah cool12:42
dmsimardit's not quite ready yet, but it's getting there12:42
dmsimardhttps://github.com/openstack/ara-server12:43
dmsimardhttps://github.com/openstack/ara-clients12:43
dmsimardhttps://github.com/openstack/ara-plugins12:43
apollo13sorry for the noise then :)12:43
dmsimardnp, depending on what you are working on, it's probably ready to be used in some capacity but consider it beta12:44
dmsimardthere is no frontend with feature parity yet12:44
dmsimardit's essentially a complete rewrite12:44
apollo13mhm, I'd like to work on a django project again12:45
apollo13I'll dig into the code if I find time12:46
dmsimardapollo13: neat, always excited to get some help, let me know if you have questions12:55
apollo13dmsimard: well most importantly, which timezone are you in so I know when I can sync up with you. I'll probably will have question about the main ideas etc… (I am a django core dev, so I'll be fine with the code; but I do not know what your ideas/goals are)13:02
dmsimardeastern time zone in north america13:03
dmsimardi.e, it was 4:40AM when you started writing here :p13:03
apollo13good, so in about 3 hours when I am at home begins to start being a good time for you :þ13:04
dmsimardapollo13: your skills in django would be quite useful, I am but a sysadmin/ops person who tries his best :)13:04
dmsimardwell I am usually available right about now, before that I'm either waking up or sending kids to school etc13:04
apollo13perfect13:05
dmsimardapollo13: if you happen to be at ansiblefest next week, we can chat13:07
dmsimardI'll bring ara stickers too. :D13:07
apollo13I am in austria :/13:07
*** sshnaidm is now known as sshnaidm|mtg13:07
apollo13and america is currently a no-go area for me :/13:07
dmsimardouch13:08
dmsimardWell I'm from Canada and even for me it's almost risky to go with the current political climate ? :(13:08
apollo13dmsimard: mhm, lets say I am interested; how much breakage can you deal with before you kick me out? :D13:15
apollo13as in update it to what I would expect django code to look like13:15
dmsimardapollo13: I'm by no means an expert at django, I would defer to gvincent for reviews13:16
apollo13ok, I'll hack around and open a PR :)13:16
dmsimardapollo13: 1.0 is not released yet and we are allowing ourselves to break backwards compatibility -- there's no one using it yet13:16
gvincent:D13:16
dmsimardif we want to break things, it's now13:16
apollo13not backwards compat, just code overhaul :)13:17
dmsimardapollo13: no but I mean, ara 0.x and ara 1.0 are vastly different13:17
apollo13yeah, got that13:17
apollo13ok, I'll start hacking away13:17
apollo13btw what are your thoughts on using black?13:17
dmsimardapollo13: openstack projects don't use github pull requests though, I can walk you through the gerrit workflow -- it takes a few minutes to jump through some hoops once and then you're good to go13:18
dmsimardblack ? :/13:18
apollo13mhm, I "know" gerrit from qt; but yeah a short walk through would certainly help13:18
apollo13https://github.com/ambv/black13:19
dmsimardthis is the first I've heard of it, it looks interesting13:19
apollo13it might be especially useful for a new project13:20
dmsimardI would prefer an approach where we run an integration job with black instead of pre-commit hooks13:21
dmsimardbut otherwise happy to give it a try13:22
apollo13oh yes, forget the pre-commit hooks, I am also using it as part of CI13:22
dmsimardapollo13: https://ara.readthedocs.io/en/stable/contributing.html should be a good starting point for submitting patches -- tl;dr, 1) Have a launchpad account ( https://login.launchpad.net/+login ) 2) Sign in to gerrit (top right https://review.openstack.org/ ) 3) Set up your public ssh key in gerrit 4) hack your patch 5) send it with "git review" (there are distro packages for it or it can be installed with pip)13:27
dmsimardthankfully steps 1 through 3 only need to be done once, after that it's pretty straightforward13:28
dmsimardand then a patch looks like this in practice https://review.openstack.org/#/c/602049/13:28
dmsimardwe're able to manage our own unit and integration jobs and the jobs themselves even have ara reports :D http://logs.openstack.org/49/602049/1/check/ara-server-ansible-integration/f54302d/ara-report/13:29
dmsimardSince the CI system, Zuul, actually has you write your jobs as ansible roles and playbooks and has ara enabled13:29
apollo13nice13:30
apollo13are there docs for zuul?13:30
apollo13I do not really understand the jobs.yaml and project.yaml yet :)13:30
apollo13https://zuul-ci.org/ this one?13:31
dmsimardyeah13:31
apollo13ok, let me RTFM13:31
dmsimardok, see you in a month13:31
dmsimardhaha13:31
apollo13Invalid SSH Key, I so hate gerrit13:31
apollo13lets try rsa *shrug*13:32
dmsimardyou might need to set something in your .ssh/config13:32
dmsimardbecause if your local ssh user is "apollo13" and your gerrit user is "foo", it's going to try and authenticate as appolo13@review.openstack.org instead of foo13:32
apollo13dmsimard: no worries, I will manage :)13:32
dmsimard++13:33
dmsimardI'll clean some things up that might not be up to date in the readme/docs/etc13:35
dmsimardapollo13: there's some great talks about zuul if you're into that13:39
apollo13mhm what is my gerrit username?13:40
apollo13ah I have to set one :D13:41
dmsimardapollo13: once you're logged in to gerrit, click on your name at the top right and then settings13:41
dmsimardit's the first field13:41
openstackgerritFlorian Apolloner proposed openstack/ara-server master: Small cleanups for django best practices.  https://review.openstack.org/60542113:44
dmsimardwell there you go13:44
dmsimardapollo13: FWIW authentication/permissions/administration/rbac/acl are things that have so far been an afterthought13:45
dmsimardapollo13: any implementation of such things that are currently in the code is likely because it was copy/paste from gvincent's template13:46
apollo13ok, should there be a link to the CI run somewhere?13:47
dmsimardBy that, I mean that the only thing that django is used for currently is for the API (django-rest-framework)13:47
dmsimardand that the API is currently 100% open13:47
dmsimardapollo13: it only posts the link once it's finished running, you can look at queued or in-progress jobs at http://zuul.openstack.org/ if need be13:47
dmsimardyou can write ara-server in the filter box13:48
dmsimardhttps://i.imgur.com/sBwqjJe.png13:49
dmsimardzuul has a component called nodepool which creates and destroys single-purpose VMs for every job -- when something is queued, it means the job is waiting for a node to be allocated to it13:49
apollo13okay, so I understand project.yaml now; how does it select the node? or what do the nodes run? ubuntu, something?13:50
apollo13or is that specified by parent: tox13:50
dmsimardin https://github.com/openstack/ara/blob/master/zuul.d/jobs.yaml you'll see nodesetsz13:50
dmsimardnodesets*13:50
dmsimardapollo13: https://review.openstack.org/#/c/557933 is an example of setting things up from scratch13:51
dmsimardthose jobs are failing but hopefully it gives you a good idea of how it works13:51
dmsimardThat project, ansible-role-ara, doesn't actually have any jobs set up right now13:52
dmsimardI sent a patch to add jobs, they're not merged, but zuul loaded the config from the patch and ran things speculatively13:52
dmsimardso it allows to experiment and iterate without needing to merge things13:52
apollo13ok but how does ara-server currently select the nodes to run on?13:52
apollo13I see no nodeset or anything13:53
dmsimardoh, sorry, I linked ara13:53
apollo13nor a tox parent13:53
apollo13ah13:53
apollo13my bad13:53
dmsimardright13:53
dmsimardara-server doesn't specify any nodeset13:53
dmsimardso it takes whatever is the nodeset inherited by the parent jobs13:53
apollo13and where is this tox parent specified?13:54
dmsimardin this case it'd be ubuntu13:54
dmsimardsec13:55
*** themurph has joined #ara13:59
dmsimardtristanC: do you know when the zuul UI refactor will land ?14:00
dmsimardtristanC: I don't know if the job graph was part of it ? i.e, how to tell where a job was coming from14:00
dmsimardapollo13: there's no great UI for it (yet) but another thing that zuul does is that the configuration is actually dynamically aggregated from all the projects it manages14:01
dmsimardapollo13: the idea is that if a project defines a job, I can use it too without having to duplicate it14:02
dmsimardtox-py35 and tox-pep8 come from https://github.com/openstack-infra/zuul-jobs/blob/master/zuul.yaml14:02
dmsimardI don't see nodeset definitions there, so it means it's actually picking the nodeset up to the base job which is inherited by default14:02
apollo13ok, that gives me stuff to read into, maybe I can integrate flake8/isort/black :)14:03
dmsimardand the base job is here https://github.com/openstack-infra/project-config/blob/master/zuul.d/jobs.yaml#L2614:03
dmsimardapollo13: are you also familiar with django-rest-framework btw ?14:04
apollo13partially14:05
apollo13mainly with the plain ApiViews14:05
apollo13but I can manage and know whom to ask ;)14:05
dmsimardok!14:06
dmsimardToday I'll push some updates to readme/docs and try to get some sort of roadmap doc going14:07
apollo13neat14:08
*** tbielawa has joined #ara14:24
openstackgerritFlorian Apolloner proposed openstack/ara-server master: Added CI for black/isort.  https://review.openstack.org/60543914:33
apollo13I didn't run isort/black over the codebase yet; but this should show you the differences14:33
openstackgerritFlorian Apolloner proposed openstack/ara-server master: Fixed tox warning.  https://review.openstack.org/60544314:39
apollo13that's it for now, gotta go home14:40
tristanCdmsimard: i don't know when the UI refactor will land, #zuul said soon14:45
tristanCdmsimard: the job graph hasn't been rebased on the refactor14:48
dmsimardok, np, I was just struggling myself to find the chain of inheritance for a job :p14:48
dmsimardbecause it was spread over different repos14:48
dmsimardtristanC: I guess https://review.openstack.org/#/c/591604/ doesn't pass python unit test jobs though ? unless that's a flap14:52
gvincentapollo13, dmsimard for https://review.openstack.org/#/c/605443/ why can we just put py3 instead of py35 to remove the warning?15:19
apollo13depends on what you wanna test and support I guess15:19
apollo13py3 would be 3.7 on my machine15:19
gvincentbecause basepython = python3.5 for user to have python 3.5 installed15:19
dmsimardthe problem with that is that if we put python3.5 as the basepython15:19
dmsimardyeah15:19
dmsimardwhat gvincent said15:20
gvincentdmsimard, are you ok to update only the env name?15:20
dmsimardI haven't had a chance to look, there's someone else who sent a patch that changes the tox environments https://review.openstack.org/#/c/605316/15:21
apollo13I am fine with either, I just thought you wanted to test against 3.515:21
dmsimardI do not have an opinion right now, would need to look15:21
dmsimardthere's even 3.7 now heh15:22
dmsimardit's tricky, 3.7 even broke some things15:22
apollo13every python release breaks things :þ15:23
dmsimardIt sort of sucks, I guess it means we need to start testing with different versions of python15:23
dmsimardmakes for a complicated matrix15:23
dmsimarddifferent OS, different version of python, different version of ansible15:23
apollo13sure, in the end that is what you will end up with as soon as ara-server leaves pre-beta :þ15:24
openstackgerritGuillaume Vincent proposed openstack/ara-server master: Remove tox basepython warning  https://review.openstack.org/60545715:24
dmsimardapollo13: if you look here you'll see an (outdated) table of what we test with ara 0.x https://github.com/openstack/ara#contributing-testing-issues-and-bugs15:25
dmsimard"py35" is really just "python3" in reality15:26
apollo13not if you remove basepython, then py35 defaults axctually to python3.515:26
apollo13according to the tox docs15:27
dmsimarddepends on which distro15:27
dmsimardfor example, centos doesn't have py315:27
apollo13right, so I would expect that to fail on centos15:28
apollo13according to tox: tox allows setting the python version for an environment via the basepython setting. If that’s not set tox can set a default value from the environment name ( e.g. py37 implies Python 3.7). Matching up the python version with the environment name has became expected at this point, leading to surprises when some configs don’t do so.15:28
*** sshnaidm|mtg is now known as sshnaidm15:29
apollo13unless you installed python3 from epel15:30
dmsimardright15:31
gvincentdepending on with py3 we use we will have some trouble with django 2.X also no?15:31
dmsimardthe thing is I'm not sure yet how we might want to test with different versions of python15:31
dmsimardif we need to put "one" version of python in the tox file15:31
apollo13gvincent: yes, django tries to support newest python versions but it might not always be up2date15:33
apollo13dmsimard: you don't need to put one version of python into the tox file15:33
apollo13every testenv can have it's own15:34
apollo13and if not it inherits from the default testenv15:34
apollo13so you can add testenv:py36 with basepython=python3.6 and voila15:34
dmsimardapollo13: I think you would probably have an interesting opinion on a "hack" that I'm using with django16:14
dmsimardtl;dr, ara needs to stay as simple as possible and there shouldn't be much needed other than "pip install ara", export the ansible callback config and run your playbook as usual16:15
apollo13makes sense16:16
dmsimardI am determined at keeping things this way and so it could not be a requirement for people to stand up an API server16:17
dmsimardThere *is* a http client which is a simple wrapper around python requests16:17
dmsimardbut there is also an "offline" client which doesn't require standing up an API server16:17
dmsimardthe hack is that it uses the django test client -- the one that is usually only used in unit tests16:18
dmsimardIt works very well, actually. No http, no traffic.16:18
dmsimardand then, the idea is that these two clients have the exact same interface (post/get/patch/delete/etc) so that the callback is able to use one or the other, depending on the configuration16:19
apollo13interesting16:19
dmsimardhttps://github.com/openstack/ara-clients/blob/master/ara/clients/offline.py16:19
dmsimardhttps://github.com/openstack/ara-clients/blob/master/ara/clients/http.py16:19
apollo13mhm, to be honest I'd probably manually resolve the urls and call the view directly with a fake request obect16:21
dmsimardsure, that's why I said you'd probably have an opinion on it :D16:21
dmsimardthe current implementation works but I'm open to doing something cleaner16:22
apollo13I like the test approach, but it is really ment for testing16:22
apollo13ie if it breaks you get to keep the pieces16:22
dmsimardyup, hence my description of it being a hack16:22
dmsimardthe test client takes care of a lot of boilerplate that I didn't need to write so it was a convenient choice16:23
apollo13I will see what I can come up with; we might loose a few things like middlewares, but eg access control etc probably doesn't make much sense for a offline client anyways16:23
apollo13or CORS and whatnot ;)16:24
dmsimardyeah, it's not a priority for the time being16:24
dmsimardapollo13: so what timezone are you in then ?16:24
apollo13as of now Europe/Vienna, ie UTC + 2 till summer time is over which will get me back to +116:25
apollo13zuul seems rather overloaded, all jobs still queued?16:25
dmsimardYeah, there's a bit of a backlog right now. It's not usually this busy but there were some cascading issues not necessarily related to Zuul.16:28
dmsimardThe nodepool component that takes care of provisioning the VMs that I told you about -- it relies on our different cloud providers to work and there was an outage in one of them16:29
dmsimardWe have graphs to keep track of things -- http://grafana.openstack.org/d/rZtIH5Imz/nodepool?orgId=1 and http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=116:31
*** tbielawa is now known as tbielawa|mtg16:39
apollo13I see16:41
openstackgerritFlorian Apolloner proposed openstack/ara-server master: Small cleanups for django best practices.  https://review.openstack.org/60542116:56
apollo13is "__import__('pkg_resources').declare_namespace(__name__)" there for a reason? do you really want ara/api also be a namespace package?16:58
dmsimardapollo13: it's a bit messy right now because of ara 0.x and ara 1.0 are vastly different and "pip install ara" currently installs 0.x17:05
dmsimardthe end goal is probably that "ara" will more or less become a metapackage, taking care of installing ara-server, ara-plugins, ara-clients and ara-web17:06
dmsimardright now -- server, clients, plugins are all namespaced packages of ara17:06
dmsimardin practice, what it translates to is that they end up being installed in the same path17:06
dmsimardsite-packages/ara/api, site-packages/ara/clients, site-packages/ara/plugins, etc... so then you can do "from ara.clients import ..." instead of something like "from ara_clients.clients import ..."17:07
apollo13dmsimard: yes I understand that, this is allowed in python3 by default17:07
apollo13but ara/api/__init__.py has declare namespace17:08
apollo13which makes ara/api itself namespaced and allows for instance to let another package install into ara/api/blabal17:08
apollo13if I understand it correctly at least17:08
dmsimardhmmmm17:08
apollo13by not having __init__.py in ara/ itself you implicitelly make ara a namespaced package in py317:09
apollo13declare_namespace was the old setuptools or whatever stuff17:09
dmsimardbut there's things in ara itself right now https://github.com/openstack/ara/tree/master/ara17:10
apollo13well you do not expect this ara to work with ara-server together, or?17:11
dmsimardI'm admittedly not super knowledgeable around these parts -- I looked things up, I tried things and it works so I left it at that17:11
dmsimardara 0.x will have to be retired at some point17:11
apollo13right17:11
apollo13I am pretty sure (tm) that removing declare_namespace from ara/api and ara/server will not hurt17:11
apollo13anyways, that's for another day17:11
dmsimardgo for it17:12
apollo13later, I am looking into viewsets currently17:13
dmsimardI love what I'm seeing in https://review.openstack.org/#/c/605421/ already, thanks a lot for that :)17:14
*** tbielawa|mtg is now known as tbielawa17:17
*** tbielawa is now known as tbielawa|lunch17:17
*** tbielawa|lunch is now known as tbielawa17:22
gvincent+1 dmsimard thank you apollo1317:22
apollo13Oh I am not done yet :þ17:23
gvincentahah17:23
*** tbielawa is now known as tbielawa|lunch17:28
openstackgerritFlorian Apolloner proposed openstack/ara-server master: Small cleanups for django best practices.  https://review.openstack.org/60542117:32
apollo13ok, time to eat something, will see if I can clean up the remaining Todos in the code later17:32
apollo13dmsimard: interesting: error: Namespace package problem: ara.api is a namespace package, but its  __init__.py does not call declare_namespace()! Please fix it. -- gotta read python docs; though I wonder if that is a pbr issue17:33
apollo13guess you were right :)17:33
dmsimardapollo13: we can figure this out later17:33
dmsimardit's a detail :p17:34
openstackgerritFlorian Apolloner proposed openstack/ara-server master: Small cleanups for django best practices.  https://review.openstack.org/60542118:35
*** tbielawa|lunch is now known as tbielawa18:35
openstackgerritFlorian Apolloner proposed openstack/ara-server master: Only ara is a namespace package, subpackages are not.  https://review.openstack.org/60550318:37
apollo13ok, I am done for today; let me know what you think :)18:38
apollo13if you have any questions, please don't hesitate to ask18:40
apollo13if tests fail I'll fix tomorrow or so18:40
openstackgerritFlorian Apolloner proposed openstack/ara-server master: Small cleanups for django best practices.  https://review.openstack.org/60542118:51
apollo13last update, I promise :)18:51
apollo13but the code got simpler imo18:51
*** evrardjp has joined #ara20:11
*** tbielawa has quit IRC20:29
*** tbielawa has joined #ara20:29
*** tbielawa is now known as tbielawa|g0n320:30
openstackgerritDavid Moreau Simard proposed openstack/ara-server master: Update and improve README to be more accurate  https://review.openstack.org/60552820:42
openstackgerritDavid Moreau Simard proposed openstack/ara-server master: Bootstrap documentation and publish it to readthedocs  https://review.openstack.org/60552920:42
openstackgerritDavid Moreau Simard proposed openstack/ara-server master: Bootstrap documentation and publish it to readthedocs  https://review.openstack.org/60552920:48
*** themurph has quit IRC21:25
*** mgariepy has quit IRC21:59
*** mgariepy has joined #ara22:13
*** openstackgerrit has quit IRC23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!