14:00:26 <mwhahaha> #startmeeting tripleo
14:00:26 <mwhahaha> #topic agenda
14:00:26 <mwhahaha> * Review past action items
14:00:26 <mwhahaha> * One off agenda items
14:00:26 <mwhahaha> * Squad status
14:00:27 <mwhahaha> * Bugs & Blueprints
14:00:27 <mwhahaha> * Projects releases or stable backports
14:00:27 <openstack> Meeting started Tue Sep  3 14:00:26 2019 UTC and is due to finish in 60 minutes.  The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:28 <mwhahaha> * Specs
14:00:28 <mwhahaha> * open discussion
14:00:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:29 <mwhahaha> Anyone can use the #link, #action and #info commands, not just the moderatorǃ
14:00:29 <mwhahaha> Hi everyone! who is around today?
14:00:30 <openstack> The meeting name has been set to 'tripleo'
14:00:36 <cloudnull> o/
14:00:42 <zbr> mwhahaha: i am curious about one thing: are you against adding the zuul-job or even about ading the molecule scenario to the repo? having that scenario would be handy for local testing/development afaik
14:00:49 <zbr> o/
14:00:53 <tbarron> Hi
14:00:58 <dpeacock> o/
14:01:02 <marios> o/
14:01:03 <panda|rover> o/
14:01:05 <Tengu> \o/
14:01:05 <ksambor> o/
14:01:06 <mwhahaha> zbr: setting up molecule is not handy for local testing/development IMHO
14:01:17 <mwhahaha> zbr: that's your workflow which may not be useful for everyone
14:01:46 <mwhahaha> it might be for other project, but not that one
14:02:38 <weshay> 0/
14:04:21 <ykarel> o/
14:04:24 <cloudnull> mwhahaha +1 - I use molecule a lot, but never on my local workstation.
14:04:51 <mwhahaha> alright let's get on with it
14:04:58 <mwhahaha> #topic review past action items
14:04:58 <mwhahaha> None
14:05:11 <mwhahaha> need more action items
14:05:26 <fultonj> o/
14:05:31 <mwhahaha> #topic one off agenda items
14:05:31 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:05:39 <mwhahaha> (Tengu + holser) podman ghost containers https://bugzilla.redhat.com/show_bug.cgi?id=1747885
14:05:41 <openstack> bugzilla.redhat.com bug 1747885 in rhosp-director "[UPDATE] undercloud update failed with error "ERROR configuring mysql"" [Urgent,Assigned] - Assigned to cjeanner
14:05:51 <mwhahaha> oh noes
14:06:02 <holser> mwhahaha sorry
14:06:02 <Tengu> :)
14:06:08 <holser> :)
14:06:12 <Tengu> I know you like podman issues, mwhahaha
14:06:21 <Tengu> holser: care to do the talk?
14:06:30 <holser> sure
14:07:40 <jfrancoa> o/
14:09:43 <EmilienM> so we have a potential solution
14:10:00 <EmilienM> but tl;dr we need to add --storage in podman rm in THT/container-puppet.py
14:10:14 <EmilienM> and do a downstream backport of the version of podman. It's a safe mitigation according to podman folks
14:10:21 <EmilienM> until they root cause the actual issue
14:10:36 <rfolco> o/
14:10:41 <chandankumar> \o/
14:13:02 <holser> o/
14:13:20 <mwhahaha> sooo anything else about this issue?
14:13:26 <mwhahaha> we just need to workaround a podman problem?
14:13:27 <holser> EmilienM to add --storage we need newer podman
14:13:37 <holser> podman 1.0.3 doesn't have such option
14:13:44 <EmilienM> holser: that's what I wrote if you read me
14:13:51 <EmilienM> "do a downstream backport of the version of podman"
14:13:58 <mwhahaha> i don't think we can do that
14:14:01 * holser hides
14:14:03 <mwhahaha> i thought the containers team needs to
14:14:07 <EmilienM> it's already done
14:14:10 <mwhahaha> ok
14:14:11 <EmilienM> and I'm working on the import now
14:14:17 <EmilienM> can we move on and take it offline?
14:14:25 <mwhahaha> fair enough
14:14:31 <holser> I am fine
14:14:47 <mwhahaha> ssbarnea: Use case: How to testing python code locally, isolated and on multiple-platforms?
14:15:08 <mwhahaha> zbr please explain
14:16:34 <weshay> zbr ^
14:16:40 <zbr> use case: i develop code on a centos-7 host and I want to check how it behaves on another platform, like rhel-8
14:17:24 <zbr> now I cannot run anything *local* to do this, I need to provision these platforms on my own, even if I want to do only a "cat /etc/os-release".
14:18:20 <rh-jelabarre> funny that I'm looking at an article that might do just that for you...   https://developers.redhat.com/blog/2019/08/23/run-red-hat-enterprise-linux-8-in-a-container-on-rhel-7/?sc_cid=701f2000000RtqCAAS
14:18:39 <zbr> even when I do succed doing that I often re-use these them between tests and they become dirty, so when I test again i may get different results.
14:18:48 <openstackgerrit> Merged openstack/tripleo-docs master: Fix trunk.rdoproject.org url  https://review.opendev.org/679621
14:19:06 <weshay> my concern is running rhel-8 upstream, as it's not an approved platform
14:19:16 <weshay> even if it's just a container
14:19:18 <zbr> if I would be able to test locally what-happens-on-platform-x, before even creating a CR, it means faster feedback-loop, better CR from strart, less load on zuul.
14:19:41 <weshay> zbr have you run that idea by the infra guys?
14:19:43 <mwhahaha> IMHO we should not be reliant on deployment tests for pure python projects such as tripleo-repos.  In the tripleo-repos case we should be able to write a unit test to cover any specific cases that we expect to support.  Then the actual integration tests should likely be covered by our normal CI jobs.
14:20:07 <zbr> weshay: we have being running fedora jobs for 9+ months, even if is not approved platform ;)
14:20:12 <mwhahaha> mocking out the cat /etc/os-release should be trivial
14:20:15 <weshay> fedora is approved
14:20:22 <mwhahaha> We've done multi-os testing via unit tests in puppet for years so I don't see why we can't perform similar tests in python.
14:20:27 <weshay> ok.. cool
14:20:47 <mwhahaha> The whole point of tripleo-repos was to replace the ansible specific code that existed in quickstart which didn't have any form of unit testing and would have required something like molecule to actually check.  We were repeatedly bitten by changes to those roles.
14:20:51 <zbr> also, i care less about what if we run them or not, more about for me being able to run them locally. -- is practical.
14:21:18 <weshay> zbr mwhahaha so.. do we need to mock rhel-8 or can we use a container?
14:21:38 <mwhahaha> so my way is being more testy driven development
14:21:42 <mwhahaha> s/testy/test
14:21:47 <mwhahaha> vs deployment driven development
14:21:59 <mwhahaha> we've missed a ton of issues by relying on deployments
14:22:01 <mwhahaha> see tripleo-ci
14:22:20 <mwhahaha> so i would prefer that we not push reliance on actual deployments unless absolutely necessary
14:22:29 <mwhahaha> in this case, for tripleo-repos I do not believe it's is necessary
14:22:51 <weshay> so .. just making sure I understand.. you would like to see us help push more molecule tests across ansible-roles, and unit tests across python
14:22:55 <zbr> mwhahaha -- i agree here, still. molecule is clearly not deployment, is more closer to functional testing, i would say.
14:23:26 <mwhahaha> right that's fine, I just don't think that's the platform we should be pushing for python projects
14:23:29 <mwhahaha> ansible stuff, sure
14:23:31 <mwhahaha> python no
14:23:40 <mwhahaha> it's not what the other upstream projects are doing
14:23:52 <mwhahaha> if you want to test, use an actual zuul node as that's the current pattern
14:24:10 <mwhahaha> if molecule gets adopted for other openstack python projects, then we can revisit
14:24:18 <mwhahaha> but i believe we should limit it to ansible related items at this time
14:24:23 <mwhahaha> similar to the beaker tests we have in puppet
14:24:31 <mwhahaha> we don't push beaker for python even if we could reuse it
14:24:44 <mwhahaha> it has it's place but it should be specifically defined when/where to be used
14:26:05 <zbr> mainly I seen molecule here as a way of allowing us some kind of testing which was impossible (or inconvenient) outside zuul, by using local virtualization.
14:26:17 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: container-puppet: run podman rm with --storage  https://review.opendev.org/679793
14:26:56 <zbr> i confess, it is bit outside its original scope, but it works. not going to push more for it, just wanted to explain why I found it useful.
14:28:00 <mwhahaha> i see it as a specific method of doing functional testing which may work for you but may not for others. there are other options (vagrant, beaker, etc)
14:28:38 <mwhahaha> I'm just not sold on this specific case
14:29:46 <weshay> mwhahaha  so to summarize, limit molecule use for ansible only
14:30:06 <weshay> in other use cases, mock out what is required
14:30:11 <mwhahaha> yes i believe that should be the current scope
14:30:27 <mwhahaha> we can revisit later, but i think that's where we should draw the line today
14:30:33 <weshay> zbr comments?
14:30:37 <marios> zbr: can't we say, start using it as much as possible for ansible. once we start to have some decent coverage, revisit the 'use it for python ' too?
14:30:58 <marios> zbr: was just on a call with panda, he was saying there are good use case for molecule/pyuthon (e.g. different distros)
14:31:26 <zbr> no more comments
14:31:32 <mwhahaha> i also think this should be something we raise to the larger community as possible replacements for maybe existing testing
14:31:35 <marios> zbr: so maybe once we start using it more/enough for ansible, folks might like to revisit, and we can also see what the wider community does wrt python/moluecule
14:31:47 <mwhahaha> if we get broader adoptions then it makes it easier to say we should be using it too
14:31:56 <mwhahaha> but currently the upstream doesn't do this type of testing
14:32:35 <panda|rover> yep, I had similar concerns, when you're testing multiple platforms on a node, you're adding a provisioning layer on top of zuul that is left to the developer
14:33:00 <panda|rover> and that's something we definitely want to do with support from infra
14:33:17 <panda|rover> but the use case is clear
14:33:31 <zbr> upstream testing was mainly focused around use of zuul exclusively, every time i mentioned something related to local testing they were not really interested, zuul was considered enough.
14:33:56 <zbr> but i am still working to slowly introduce them to molecule for zuul roles, i will see what a success rate I will have.
14:34:19 <panda|rover> zull is grat, but nothing beats running local/isolated tests, the iteration on the changes is really much faster
14:34:50 <openstackgerrit> Harald Jensås proposed openstack/tripleo-heat-templates master: Don't add IpList for disabled networks  https://review.opendev.org/679354
14:35:03 <openstackgerrit> Jose Luis Franco proposed openstack/tripleo-heat-templates stable/stein: Split upgrade_steps_playbook into different plays.  https://review.opendev.org/679737
14:35:05 <zbr> mainly "create new zuul jobs" was the defacto answer for any new test-case, but we all know what happens when your test matrix explodes ;)
14:35:21 <panda|rover> but if can have coverage even for multiple platforms with just unit test then it's ok for me
14:35:45 <marios> zbr: one problem with unit tests is we currently have some
14:35:50 <panda|rover> I just hope it can be achieved easily.
14:35:50 <marios> zbr: i.e. we'd need to convert those first
14:36:00 <marios> zbr: vs net new additions for the molecule/ansible case
14:36:09 <mwhahaha> i definitely don't think unit tests should be replaced by molefule items
14:36:39 <weshay> augmented?
14:36:50 <panda|rover> not replaced, moleucle would probably just run tox on a different distribution
14:37:11 <mwhahaha> there shouldn't be a need to run tox a different distro
14:37:17 <mwhahaha> like that doesn't make sense really
14:37:40 <mwhahaha> molecule should be actually validating system changes not for use to rerun unit tests
14:37:48 <mwhahaha> unit tests shouldn't care what distro it's run on
14:37:55 <mwhahaha> that's the whole point of python
14:40:03 <ykarel> so what if python code touches distro things,
14:40:15 <zbr> that depends a lot on what your python code is doing, the code mayb be portable, but if it does os.system("python") and python exec does not exist on newer platforms, you will fail to find a platform specific bug.
14:40:23 <ykarel> like https://github.com/openstack/tripleo-repos/blob/master/tripleo_repos/main.py#L79-L106
14:40:23 <mwhahaha> so in the case of tripleo-repos, installing tripleo-repos and executing it would be the molecule tests
14:40:46 <panda|rover> yes
14:40:51 <panda|rover> I imagine
14:40:52 <mwhahaha> in the case of that, we really should be doing a pyton file open and parsing, not relying on sourcing os-release
14:41:03 <mwhahaha> that bit of code is not really proper python
14:41:07 <mwhahaha> it's a bit more of a shell hack
14:41:38 <mwhahaha> however, it can easily be mocked to return different valuyes
14:41:51 <mwhahaha> so mocking _get_distro to return various info is trivial
14:42:22 <mwhahaha> anyway i think we've discussed this a bit much, let's take further discussions about specifics offline
14:42:35 <ykarel> +1
14:42:48 <mwhahaha> alright moving on
14:42:48 <mwhahaha> fultonj: Split Control Plane Unified Ansible Inventory
14:42:54 <fultonj> ack
14:43:02 <fultonj> i have an open blueprint
14:43:11 <fultonj> i'm raising my hand and saying i plan to have it done before m3
14:43:24 <mwhahaha> note m3 is this week
14:43:27 <fultonj> ack
14:43:31 <mwhahaha> so we need to merge it like today/tomorrow
14:43:34 <mwhahaha> for a chance to make m3
14:43:36 <fultonj> yes
14:43:44 <fultonj> two pathces
14:43:51 <fultonj> one is ready for review. it depends on a second
14:43:59 <fultonj> the second just needs to unit test i'm writing
14:44:06 <fultonj> https://review.opendev.org/#/c/664065
14:44:07 <fultonj> https://review.opendev.org/#/c/679520
14:44:15 <mwhahaha> currently they are zuul -1, is that not related? /me has to check
14:44:25 <ykarel> isn't M3 not next week?
14:44:32 <fultonj> sept 9
14:44:33 <fultonj> 6 days
14:44:42 <fultonj> monday
14:44:44 <ykarel> yes Sep 09 - Sep 13
14:45:04 <mwhahaha> oh right
14:45:23 <mwhahaha> ok so next week, needs to land this week then
14:45:30 <mwhahaha> (ideally)
14:45:32 <fultonj> mwhahaha: yes, i think this is possible
14:46:02 <fultonj> wanted to state this. since it's an open blueprint and m3 is coming.
14:46:07 <fultonj> that's all unless anyone has comments
14:46:14 <fultonj> owalsh: ^ fyi
14:47:34 <mwhahaha> k i'll keep an eye on them this week and review
14:47:41 <mwhahaha> anything else about these patches?
14:48:06 <fultonj> the inventory generation code is kept the same if there's one heat stack
14:48:13 <fultonj> we're just adding a multistack option
14:48:27 <fultonj> so i don't think it's dangerous
14:49:05 <mwhahaha> k
14:49:12 <mwhahaha> shall we move on?
14:49:17 <fultonj> works for me. thanks
14:49:22 <mwhahaha> Chandankumar: Gating https://github.com/containers/libpod repo using ooo CI jobs
14:49:23 <owalsh> fultonj: ack
14:49:33 <chandankumar> yes it's me
14:49:40 <mwhahaha> soooo shouldn't podman stuff be gated in like delorean deps?
14:49:49 <mwhahaha> or are we getting it straight from centos repos
14:50:03 * fultonj will remove dnm on https://review.opendev.org/#/c/679520 today when unit tests added
14:50:05 <ykarel> it's already gated in RDO
14:50:26 <chandankumar> mwhahaha: we can gate any project hosted on github using software factory github app
14:50:38 <ykarel> like https://review.rdoproject.org/r/#/c/21994/
14:50:41 <chandankumar> there is some work also done by jpena to test ceph-ansible
14:50:49 <weshay> chandankumar in theory, we haven't done it yet
14:51:02 <weshay> we have a to-do item from gfidente to do it for ceph
14:51:08 <weshay> and now podman
14:51:10 <gfidente> weshay++
14:51:16 <gfidente> ceph-ansibke
14:51:19 <weshay> aye
14:51:28 <chandankumar> Currently In RDO, we update the version then test it
14:51:38 <fultonj> isn't shade doing it already; so it's a working example?
14:51:41 <chandankumar> if it does not passes all the jobs then we roll back
14:51:48 <mwhahaha> seems more like a questions for the containers team on viability of doing such things.
14:52:00 <chandankumar> now we support two distro both having different podman versions
14:52:03 <openstackgerrit> Sorin Sbarnea proposed openstack/tripleo-ci master: Removed fedora28-master from molecule testing  https://review.opendev.org/679800
14:52:55 <chandankumar> It is basically about gating early once a release lands or on each commit where we have a feeling that it is going to break the tripleo stuff
14:52:57 <weshay> chandankumar either way.. we should probably tackle this first w/ ceph-ansible
14:53:02 <weshay> then look at podman
14:53:16 <weshay> the ceph-ansible folks have been pretty patient
14:53:36 <chandankumar> weshay: ok, I will keep this in the tripleo-ci backlog
14:53:49 <ykarel> i assume most of the podman contributor/maintainers are not that aware of TripleO, so just running jobs there might not help
14:53:50 <chandankumar> But one question how we want to gate?
14:53:50 <weshay> chandankumar ya.. put it right under ceph-ansible
14:54:03 <ykarel> would need some monitoring and tracking also
14:54:04 <chandankumar> per commit or per tag
14:54:16 <weshay> ykarel chandankumar there are some background conversations going on about podman atm
14:54:37 <weshay> regarding this 3rd party checking
14:55:10 <weshay> it may be premature for this level of detail in this meeting
14:55:14 <ykarel> weshay, where ?
14:55:15 <chandankumar> weshay: ack
14:55:17 <ykarel> weshay, ack
14:55:25 <mwhahaha> alright we only have 5 minutes left and we have the rest of the agenda to tackle. let's take the rest of this offline
14:55:30 <chandankumar> we have https://github.com/packit-service/packit provided by fedora can be also used
14:55:32 <weshay> I'll be in touch w/ you guys offline
14:55:34 <mwhahaha> sounds like there's some other discussions around this
14:55:42 <mwhahaha> moving on real quick
14:55:46 * mwhahaha goes turbo
14:55:53 <weshay> look at em go
14:55:54 <mwhahaha> #topic Squad status
14:55:54 <mwhahaha> ci
14:55:54 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting
14:55:54 <mwhahaha> upgrade
14:55:54 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status
14:55:55 <mwhahaha> edge
14:55:55 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status
14:55:56 <mwhahaha> integration
14:55:56 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status
14:55:57 <mwhahaha> validations
14:55:57 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status
14:55:58 <mwhahaha> networking
14:55:58 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status
14:55:59 <mwhahaha> transformation
14:55:59 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ansible-agenda
14:56:10 <mwhahaha> call for any important status items?
14:56:28 <panda|rover> just a note on the current situation in CI. There was an outage this weekend in the dlrn repo instances with cascading effect also on the AFS mirrors. Right now every other job is unable to correctly download delorean repos for a specific hash. jpena has the specifics, but apparently DNS propagation is giving some problems.
14:56:47 <mwhahaha> :(
14:57:26 <weshay> the pass rate upstream is quite terrible atm
14:57:32 <mwhahaha> sounds like maybe hold off on some rechecks until resolved
14:57:39 <mwhahaha> can someone send a note to the ML
14:57:49 <mwhahaha> with details and an update when it's resolved
14:58:11 <panda|rover> I'll make sure it's done
14:58:15 <mwhahaha> thanks
14:58:18 <mwhahaha> #topic bugs & blueprints
14:58:18 <mwhahaha> #link https://launchpad.net/tripleo/+milestone/train-3
14:58:18 <mwhahaha> For Train we currently have 27 blueprints and 506 (+1) open Launchpad Bugs. 505 train-3, 1 train-rc1.  167 (-3) open Storyboard bugs.
14:58:18 <mwhahaha> #link https://storyboard.openstack.org/#!/project_group/76
14:58:28 <mwhahaha> fyi i created the future milestones now that we have a name for U
14:58:31 <mwhahaha> and created an rc1
14:58:34 <mwhahaha> m3 is next week
14:58:42 <mwhahaha> #topic projects releases or stable backports
14:58:52 <mwhahaha> m3 is next week, so land your patches (when ci is fixed)
14:59:04 <mwhahaha> #topic specs
14:59:04 <mwhahaha> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:59:30 <mwhahaha> plz update your specs for the ussuri cycle
14:59:35 <mwhahaha> i'll lift my -2s today
14:59:41 <mwhahaha> #topic open discussion
14:59:48 <mwhahaha> anything else (with a minute to go)?
15:00:24 <mwhahaha> thanks everyone
15:00:25 <mwhahaha> #endmeeting