16:00:31 #startmeeting API-SIG 16:00:32 Meeting started Thu Dec 5 16:00:31 2019 UTC and is due to finish in 60 minutes. The chair is dtantsur. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:33 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:35 The meeting name has been set to 'api_sig' 16:00:41 heya! 16:00:49 dtantsur, cool, if you can chair me too please :) 16:01:00 today we're a bit hijacked by Ansible SIG and the ongoing effort of bringing OpenStack ansible modules under our wing 16:01:03 #chair sshnaidm elmiko 16:01:04 Current chairs: dtantsur elmiko sshnaidm 16:01:28 ok, so thanks to SDK folks for the channel time :) 16:01:32 a big part of it will be re-designing the modules for consistency, hence our (API SIG) involvement 16:01:54 sshnaidm: take it from here :) 16:02:03 dtantsur, right 16:02:05 please use the etherpad: https://etherpad.openstack.org/p/openstack-ansible-modules 16:02:18 #link https://etherpad.openstack.org/p/openstack-ansible-modules 16:02:18 we have update from gundalow from Ansible team 16:02:27 * gundalow waves 16:02:32 it's written in the bottom of etherpad 16:02:43 and I think we have a few decisions to make 16:03:00 1. what is namespace name of openstack collections 16:03:01 * gundalow is available if there are any questions now (or later) 16:03:11 gundalow, great 16:03:17 i just want to state for the record, that it is currently not within the goals of the api-sig to define guidance for these type of apis. /but/ with that said, it is within our goals to facilitate and promote these types of discussions. 16:03:58 elmiko, thanks 16:04:09 well, our *actual* goals is to spend some time joking on each other 16:04:19 soooo... :D 16:04:19 haha, too true 16:04:33 * cloudnull reengages with the meeting 16:05:10 so, can we use "openstack." namespace? 16:05:15 So goal wise, all (will nearly all) modules/plugins will be removed from ansible/ansible before Ansible 2.10 ships. Getting a head of the curve and thinking about this now is good think 16:05:46 is it official target of Ansible to remove modules from core? 16:05:49 sshnaidm: my only concern is the fact that the collection won't be strictly "official" 16:05:50 gundalow, so no more community modules in 2.10? 16:06:23 What about modules that won't be able to find a new home? 16:06:26 dtantsur, yeah, these licenses differences, that's why raised this 16:06:27 ansible/ansible:stable-2.10 will only contain https://github.com/ansible-community/collection_migration/tree/master/scenarios/minimal 16:06:29 * jistr waves 16:07:04 ekultails: for "homeless modules" that are currently in ansible/ansible they may go in the "catch-all" (aka dumping ground) repo & Collection 16:07:22 ansible's stackforge :) 16:07:53 cool 16:07:57 so we're on time.. 16:08:04 sshnaidm: +1 for using openstack. namespace for the collection, since it's supposed to be "the official thing" 16:08:09 gundalow: Okay, good to know, thanks. 16:08:19 and what about collection name? 16:08:30 openstack.openstack? :) 16:08:36 too weird 16:08:37 https://github.com/theforeman/foreman-ansible-modules/pull/591 might have some useful background 16:08:40 openstack.? 16:08:45 otherwise why do we need both? 16:08:50 gundalow, well, I think nobody should take "openstack" namespace except Openstack.. 16:08:50 Not openstack.ansible, that could get confusing with OpenStack-Ansible. :-P 16:09:00 having both is a requirement by Ansible galaxy IIUC 16:09:01 #link https://github.com/theforeman/foreman-ansible-modules/pull/591 some helpful background 16:09:06 for my org I ended up with "namespace.core" - for the core modules 16:09:11 What else could go under `openstack.`? 16:09:21 gundalow: we can have a collection per service 16:09:26 (forgive me, I'm not familiar with all the parts of OpenStack) 16:09:29 although it will probably complicate everything 16:09:44 yea i also thing that would complicate it... 16:09:50 gtema: we no longer have an official concept of core modules 16:09:58 I don not think collection per service make sense, more a one collection with "core" modules, and perhaps additional ones for some "extra-weird" stuff 16:10:00 we could have openstack.openstack and openstack.unofficial 16:10:18 what is "core" modules? 16:10:25 let's avoid the term "core", it's a different thing for everyone 16:10:26 gtema, dtantsur: +1 i think that scenario might come up perhaps 16:10:27 well, everything what we have now 16:10:51 and i.e. for some weird provisioning of triple-o infra - there might be openstack.tripleo 16:10:56 everything we have is kinda casual.. 16:11:06 it's just what people could made quickly 16:11:40 gtema: +1. So for the ones that are getting imported from Ansible, i'd go openstack.openstack or openstack.core or something like that 16:11:40 and I think it will look very different in the end 16:11:57 yeah, but I think openstack.core.server looks better than openstack.openstack.server or openstack.compute.server 16:12:09 I'd vote for openstack.[service/project] names 16:12:11 o/ 16:12:18 I think for many people 'core' may be nova+cinder+neutron+glance+keystone 16:12:23 not things like ironic 16:12:30 gtema, openstack.compute.servers is actually as in SDK 16:12:42 and we may align with SDK here 16:12:46 sshnaidm: hmm that does look good indeed 16:12:51 I do not want to end up needing to do "ansible-galaxy collection install nova" 16:12:58 and neutron and ... 16:13:07 we'd force people to fetch multiple collections though... maybe not much of a problem 16:13:12 but i wonder about code sharing 16:13:36 Would there be a way to have a meta collection that installs all relevant collections? 16:13:43 I would recommend openstack.openstack as the collection then you call openstack.openstack.os_nova, or maybe even community.openstack.os_nova 16:13:44 will openstack. pattern complicate *writing* those collections in case we want to share code? 16:13:51 we could avoid the problem with code sharing by putting EVERYTHING in openstacksdk 16:13:56 jistr, as we were talking previously there should be definitely common part, like auth for example 16:14:23 opentack.openstacksdk.os_nova would work too 16:14:47 are we keeping the os_ prefix? 16:14:50 openstack.compute, openstack.network, openstack.tripleo - can it work? 16:14:51 but, i think you still want os_nova / os_neutron, otherwie, you will break a ton of playbooks 16:14:53 FYI you can use https://github.com/ansible-community/collection_migration with a scenario file to pull the contents out of ansible/ansible into the correct structure 16:14:54 I guess it was only useful before namespacing? 16:15:09 I dont' think need os_ now, it's a dup 16:15:11 while "openstacksdk" forever, what should it stand for? 16:15:17 for ansible/ansible migration, they should all be migrated to single repo 16:15:18 and namespace 16:15:23 os_ should be definitely dropped 16:15:33 If you change the module names, you may break backwards compatibility between 2.9 and 2.10 unless you put in aliasing (symlink) 16:15:38 gtema: that is a huge breaking change 16:15:39 pabelanger: we're considering some breaking changes anyway, like unifying authentication 16:16:04 if you do, openstack.sdk.os_nova, you could still reference os_nova in your playbook 16:16:04 gundalow: would plain os_nova (without openstack.) work after the migration? 16:16:13 yes 16:16:15 mmm, I see 16:16:19 he? is the migration from ansible to collection not a breaking change by itself? 16:16:30 it sounds like it's not THAT breaking 16:16:36 you anyway need to change your playbooks 16:16:38 or rather: it doesn't break the playbooks 16:16:43 I'd say no, you can keep same playbooks for most part 16:16:43 it does 16:16:57 gtema: There is the idea of a "Community version of Ansible" that will ship with all the collections that make up what's in Ansible 2.9 16:17:04 you need to refer to modules from collection or import collection 16:17:23 I'd seriously regret if we couldn't unify the auth though 16:17:35 (and s/os_ironic/os_baremetal/ ) 16:17:38 gundalow, but you anyway need to modify playbook 16:17:38 maybe we can keep migrated ones and develop their replacement aside with new names..? 16:17:46 IMO, this goal for migration out of ansible/ansible, would be to use migration tool from ansible-core, then keep naming schema of modules the same 16:17:56 or "community version" will do some extra magic? 16:18:01 then do deprecation process on os_nova, os_neutron 16:18:07 We may add some magic during Ansible 2.10 development that means if you are using the Ansible Community Distribution that you wouldn't need to change your playbooks - Though the exact details of this haven't been worked out uet 16:18:09 yet* 16:18:13 sshnaidm: yah, that would be fine 16:18:31 realistically, we should probably do what sshnaidm proposes 16:18:37 +1 16:18:46 I'm not sure we're going to have enough cycles to simultaneously refactor things 16:19:00 so maybe keep os_ prefix to designate old modules 16:19:02 we'll just advertise new names for a few versions and then will remove old ones I think 16:19:03 and drop it for new ones 16:19:14 dtantsur, ya 16:19:15 os_ironic_node is old, baremetal_node is new 16:19:18 +1 16:19:22 perhaps better name them properly now and make alias 16:19:30 FYI Ansible's deprecation cycle has been 4 major versions, so two years 16:19:41 gtema: not everything will be a mere rename 16:19:57 gundalow: is it the same for collections? 16:19:58 gtema: separating will allow developing new ones in a common way without worrying about breaking existing use 16:20:15 I know we are talking about more than simple rename 16:20:15 gundalow: to generalize my question: is there an expected release cadence? 16:20:18 dtantsur: unsure 16:20:22 but if we do rename and alias 16:20:36 we can develop some standard, follow it 16:20:38 dtantsur: at least twice a year (so new Ansible Engine releases can pick up the new contents) 16:20:49 I'd hope that over time we can get quicker at doing releases 16:20:51 and be consistent in the new modules for those 4 cycles 16:20:57 okay, so synchronized with ansible releases? 16:20:59 I would go as far, until code is deleted in ansible/ansible (which date is up in the air), we continusoly run migration sync nightly, and make new collection repo read-only, then once ansible/ansible code is deleted, we unfreeze openstack collection 16:21:06 otherwise we have a mix of different patterns 16:21:08 this is our plan for ansible network team 16:21:32 pabelanger: I'd prefer to freeze only the existing modules, but allow new development 16:21:37 So, for a moment ignoring refactoring. I don't think you want to break Playbooks that work in Ansible 2.9 16:21:46 pabelanger, +1 to dtantsur 16:22:11 pabelanger, we'll have old ones for usage, but the main goal is new ones.. 16:22:22 sshnaidm: dtantsur: that is fine, but keep in mind, if a nightly sync happens, like proposed bot, we want to avoid conflicts 16:22:36 right, so make sure not to modify the same files 16:22:39 for me, we need to keep old ones for 2 years 16:22:43 even in a collection 16:22:52 like, set up gerrit to outright reject changes to os_*.py until we unfreeze 16:22:57 s/gerrit/zuul jobs/ 16:23:01 Once the collection is up and running we can `git rm` the files from ansible/ansible 16:23:08 pabelanger, I think we're good with that 16:23:30 yah, once migration happens, if everything is sync'd we can push on git rm on ansible/ansible 16:23:33 then freeze is over 16:23:42 but, that sync should be a zuul jobs 16:23:57 #agreed Import the existing modules as they are and freeze them. Keep syncing until ansible removes them from tree. 16:23:59 right? 16:24:01 I'm happy to share job for that, as we do it 16:24:09 (ansible-network) 16:24:31 #action to collaborate with pabelanger to create sync zuul jobs 16:24:40 for how long do we expect to have this freeze? 16:24:42 pabelanger, thanks, it'd be great 16:24:55 gtema, to keep old modules you mean? 16:24:58 sshnaidm: #actions requires 16:25:07 #action sshnaidm to collaborate with pabelanger to create sync zuul jobs 16:25:20 gtema: freeze should be as long as we all agree, final sync looks good and we can rm -rf ansible/ansible 16:25:32 we should be able to drive delete date 16:25:36 gtema: until they are removed from ansible/ansible. then it's bugfix-only mode. 16:25:52 that means, getting content uploaded into galaxy and some sort of pre-release versioning 16:26:00 ok then. I just wanted to avoid freeze, which will keep us ourselves with tied hands 16:26:33 FYI for bug fixes for ansible/ansible:stable-2.9, once you've gitrm'd from devel you can create a PR directly against that branch, rather than `git cherry-pick -x` from devel into stable-2.9 16:26:43 and these bugfixes should be done in openstack repos since moving, right? 16:26:47 Which may mean one fewer things to keep moving 16:27:36 Once the collection is done, I'd expect bug fixes do be done in the collection first. Then optionally made to ansible/ansible:stable-2.9. Realistically I expect fewer backports to ansible/ansible once Collections are up and running 16:27:56 gundalow, lemme understand, 2.9 backports via ansible github, all new - via our repos, right? 16:28:00 hmm, that's the opposite direction 16:28:25 backwards? 16:28:40 "main dev on our side" 16:28:54 and "port" fixes back to ansible-2.9 16:29:06 that seem correct to me 16:29:23 gtema, I think we shouldn't port there, maybe only some critical fixes 16:29:43 that's exactly about critical things, but I do not expect those honestly 16:29:56 it raises the question of maybe stable backports team? 16:30:15 which can handle sync to github 16:30:30 I don't think that just for 2.9 it make sense to have this overhead 16:30:35 Current backport docs: https://docs.ansible.com/ansible/latest/community/development_process.html#backporting-merged-prs 16:30:35 Before Collections: 1) PR to `devel`, 2) wait for that to be merged. 3) For bug fixes only Create backport PR to `stable-2.x` 16:30:37 I don't think we should backport anything to 2.9 at all, it looks working as is, and no much activity was there 16:30:46 Can I get some reviews on https://github.com/ansible/ansible/pull/65339 and https://github.com/ansible/ansible/pull/65292 ? (new qos_policy modules for Ansible) 16:31:07 sshnaidm: we can't say that however, as ansible/ansible still maintans that 16:31:18 Unless I see a critical bug, I'm not going to be backporting anything to stable-2.9 after we've gitrm'd from devel 16:31:21 dasp - wrong channel. We are deciding to drop that at all ;-) 16:31:29 pabelanger, I think we agree it should be something exceptional 16:31:36 +1 16:31:46 okay, so do we even need a constant sync then? 16:31:52 gtema, :D 16:32:01 and "critical bug" is subjective, maybe you'd use that to your advantage :P 16:32:04 or just copy everything once and apply some policies by humans? 16:32:20 dtantsur, I'd go for that for simplifying things 16:32:26 Maybe I could give an example of what I'm doing with Grafana 16:32:34 sshnaidm: ++ KEEP THINGS SINGLE 16:32:38 SIMPLE 16:32:43 I think: import, github freeze, collection release, github rm 16:32:48 single and simple :) 16:33:09 github = ansible upstream 16:33:14 so no need to have sync jobs as well? 16:33:37 pabelanger, ^ 16:33:38 if we handle import and first release in few days - I would say no 16:33:56 well, need some import process to gerrit 16:34:00 which, usually isn't manual 16:34:04 so we need something automated 16:34:11 or ask infra what they want to do 16:34:17 well, we did a manual import for ironic-tempest-plugin 16:34:21 it was a bit.. interesting 16:34:28 well, we have already changes in gerrit, so just more-or-less merge and release it 16:34:28 For grafana (own repe & collection). 16:34:29 I've create a new repo and the maintainers are doing there work there (CI is up, they are merging stuff) 16:34:29 Once we've got publishing to Galaxy done and a final check we will see if there have been any other changes to grafana code in ansible/ansible, copy those across 16:34:29 then gitrm & add `migrated_to` in BOTMETA.yml 16:34:29 Then do $something with open grafana issues & PRs 16:34:32 right, not out of the question but we can ask infra 16:34:37 we extracted a git subtree from ironic, then asked infra adminds to push it :) 16:34:44 right 16:35:06 gundalow: sounds right and simple 16:35:22 I'd avoid having to keep too many things in progres, and even fewer things in sync 16:35:48 So maybe create a test collection, play with it, check it works. then do the move for real 16:36:07 therefore if we here (mostly the only ones approving changes in ansible) agree not to review anything else during releasing collection - we are fine 16:36:07 (and I think we have some job that testing modules, it runs on github changes) 16:36:36 gtema, I think we can agree here 16:36:44 There is an OpenStack specific Zuul job that runs against OpenStack PRs in ansible/ansible 16:37:08 gundalow, yeah, exactly, we can import it for collections repo 16:37:15 as a start 16:37:15 +1 16:37:15 now with tests: from what I have experienced as of now with collections - you MUST modify plybooks 16:37:31 #agreed No periodic sync, just one-time migration with temporary freezes on both sides (ending with `git rm` on the ansible side) 16:37:32 right? 16:37:44 gtema: ansible-test (in 2.9) has been updated so it should work without FQCN (Fully Qualified Collection Name) 16:37:46 dtantsur++ 16:38:01 gundalow, ah ok. Nice to know 16:38:13 dtantsur: Please add + add `migrated_to: openstack.FOO` in `.github/BOTMETA.yml` 16:38:18 In the same PR 16:38:20 #undo 16:38:21 Removing item from minutes: #agreed No periodic sync, just one-time migration with temporary freezes on both sides (ending with `git rm` on the ansible side) 16:38:35 but then we need to change our jobs to use "ansible-test" instead of "ansible-playbook". Right? 16:38:40 #agreed No periodic sync, just one-time migration with temporary freezes on both sides (ending with `git rm` and `migrated_to: openstack.FOO` on the ansible side) 16:38:49 gundalow: like ^^? 16:39:06 gtema, ansible-test is just a different tool afaiu 16:39:16 with sanity checks etc 16:39:39 yeah, looks like that 16:39:43 maybe let's set a time for the moving? 16:39:50 when can we do it? 16:40:01 gundalow, any restrictions..? 16:40:11 dtantsur: perfect, thank you 16:40:21 gundalow, what do I need to do not to use FQCN? 16:40:30 probably not right before winter holidays :) 16:40:36 or are we in time pressure? 16:40:44 dtantsur, yeah, before xmass of after :) 16:40:52 I'd suggest you continue to use `ansible-test sanity` that will help spot functional problems 16:41:00 I'm here next week, then out till January 16:41:27 realistically I'd say play now (do a trial move, get CI working, define merging/+1, etc), move in Jan 16:41:33 gundalow, our tests run "ansible-playbook ..." 16:41:38 if no time pressure, let's do after holidays 16:41:45 Jan is fine for me 16:42:11 any specific dates? 16:42:16 right the week of 6th or? 16:42:41 dtantsur, we'll need to sync maybe 16:42:45 I'm back on the 13th Jan 16:43:08 pabelanger, are you available in Jan? 16:43:49 dtantsur, I think we can start at 13 then 16:44:00 let's put it this way 16:44:07 dtantsur, or at least to decide to postpone it then 16:44:10 #agreed Do the migration in the middle of January 2020 16:44:35 gundalow, any change to "survive" with "ansible-playbook our_tests.yml"? 16:44:49 gtema: So continue to use `ansible-playbook` directly for integration tests if that's how you currently do things 2) Please look at using `ansible-test sanity` in your new repo (currently this is run via Shippable on ansible/ansible PRs), this *will* catch valid issues that your integration tests may not 16:45:10 ok, let's get back to agenda then 16:45:19 are we keeping them GPL? 16:45:29 gtema: unsure, if you speak to mattclay in #ansible-devel he might be able to give some ideas 16:45:30 but I mean ansible-playbook will fail when there is "git rm" 16:45:30 can we practically re-license them? 16:45:37 or we start develop new in Apache2? 16:45:57 then we'll need clean-room development (without looking at old ones) 16:46:07 gundalow, ok, will ask in #ansible-devel, but this is important to verify the whole migration 16:46:36 dtantsur: without looking will be very hard 16:46:52 exactly, but that's the condition of re-licensing via rewriting 16:46:55 and we repeat all the issues we were fixing for ages 16:47:12 otherwise we need an agreement from every contributor whose code is still there 16:47:13 https://bit.ly/2OUz1Xa 16:47:16 I know, and this is not what I would like to do 16:47:46 I'd keep shared bits permissive if possible, but the modules themselves probably have to be GPL 16:47:50 this is terrible, even if one has changed a typo 16:48:00 are there licenses experts here? :) 16:48:17 sshnaidm: yes 16:48:24 * dtantsur wow 16:48:34 can we write new modules in different licenses? 16:48:41 I mean not in GPL 16:48:54 the goal is to deprecate old ones 16:49:06 git log --pretty=format:"%an%x09" lib/ansible/modules/cloud/openstack | sort | uniq | wc -l ----> 157 16:49:25 (may be some duplicates, though that's a lot of people to get sign off from) 16:50:05 wow 16:50:17 should we ask all 155 people to agree? 16:50:41 IANAL, though I if one person doesn't respond you can't do it 16:50:42 I don't think it's doable at all 16:51:02 gundalow, but it's for re-licensing, I'm talking about new modules develop 16:51:20 I think it's kinda gray area here.. 16:51:44 it's grey indeed. how to distinguish new development from copy-pasting in case of very similar code? 16:51:59 no idea, need license experts opinion :) 16:52:24 although I hope it won't be copy paste 16:52:48 IANAL, only relaying what I've previously heard 16:52:50 we don't need to copy paste, we just have them available.. 16:53:26 mnaser mentioned some license-... maillist, maybe worth to ask there 16:53:38 legal-discuss :) 16:53:42 ^^^ 16:53:44 mnaser, yes! :) 16:54:04 and yes i think you need to get all 155 people to agree AFAIK. but im not a license expert 16:54:21 the problem as I understand it, if you're familiar with old code, you won't be able to claim you wrote the new code without taking from it. 16:54:28 ok, but modules that don't have a parallel I think we can start with Apache2 16:54:29 I mean, legally claim, we all here believe you :) 16:54:49 probably? I'd really bring it to legal-discuss 16:54:54 ok, time is running out. Summary? 16:55:03 gtema: we had a few #agreed 16:55:09 I would also vote for raising 16:55:17 in legal-discuss 16:55:21 #action sshnaidm To raise the licensing topic on legal-discuss 16:55:23 #action to figure out which license to use for new developed modules in openstack collection, 16:55:23 correct? :) 16:55:32 dtantsur, ack 16:55:34 you forgot 16:55:35 #undo 16:55:36 Removing item from minutes: #action to figure out which license to use for new developed modules in openstack collection, 16:55:44 okay, now we have one item 16:55:59 anything else? 16:55:59 great 16:56:03 Merged openstack/openstacksdk master: tox: Keeping going with docs https://review.opendev.org/692503 16:56:14 wanted to ask about versioning of modules 16:56:27 I don't think we need branches as openstack releases, right? 16:56:57 should we just keep them compatible as much as possible? in case we have differences in openstacksdk for example 16:57:33 any strong opinions about it? 16:57:50 I guess we need to match ansible? 16:57:56 I think current version checking in modules is sufficient. So no need for openstack releases from my POV 16:58:01 dtantsur, why? 16:58:24 to allow combining them as gundalow mentioned? 16:58:42 SDK has stable branches, OSC too (?) 16:58:46 dtantsur, sorry, I missed it, to combine? 16:58:48 it feels like the modules could 16:58:50 at least "API" for the collections structure need to be kept in sync with Ansible 16:58:59 gtema, yeah, that's right 16:59:04 sshnaidm: a distribution of ansible with all collections 16:59:08 right 16:59:44 yeah, I think we agree on keeping them compatible with ansible 16:59:55 otherwise it won't work :) 17:00:01 exactly :) 17:00:14 I guess we need to decide about stable branches 17:00:16 probably next time 17:00:17 #agreed to keep modules compatible with Ansible collections API 17:00:26 dtantsur, yep 17:00:34 ok, thanks all for participation! 17:00:42 next Thu same time same place 17:00:44 thanks, and thanks elmiko for tolerating us! 17:00:56 #endmeeting