19:02:46 <fungi> #startmeeting infra
19:02:47 <openstack> Meeting started Tue Apr 25 19:02:46 2017 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:47 <Shrews> presents?
19:02:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:51 <openstack> The meeting name has been set to 'infra'
19:02:55 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:57 <fungi> #topic Announcements
19:03:04 <fungi> #info Let fungi know if you hope to attend the PTG in Denver this September so he can get a rough head count
19:03:12 <fungi> as always, feel free to hit me up with announcements you want included in future meetings
19:03:19 <fungi> #topic Actions from last meeting
19:03:26 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-04-18-19.02.html Minutes from last meeting
19:03:34 <fungi> #action fungi put forth proposal to flatten git namespaces
19:03:37 <fungi> still drafting a spec locally based on ttx's brainstorming etherpad
19:03:40 <fungi> #link https://etherpad.openstack.org/p/repo-name-shortening-impact Impact of shortening repository names
19:03:42 <fungi> this has finally bubbled to the top of my to-do backlog at least
19:03:50 <fungi> #action pabelanger Open an Ubuntu SRU for bug 1251495
19:03:52 <openstack> bug 1251495 in mailman (Ubuntu Trusty) "Lists with topics enabled can throw unexpected keyword argument 'Delete' exception." [High,Triaged] https://launchpad.net/bugs/1251495
19:03:52 <fungi> i checked the bug a few minutes ago so assuming this is still outstanding
19:04:03 <pabelanger> still ongoing, sadly
19:04:10 <fungi> #action clarkb to add citycloud to nodepool
19:04:13 <fungi> #link https://review.openstack.org/458621 Update citycloud to non demo accounts
19:04:15 <fungi> #link https://review.openstack.org/458622 Add citycloud to nodepool
19:04:18 <fungi> these are presumably wip until we know what regions we'll be using and confirm appropriate quota
19:04:50 <clarkb> yup
19:04:54 <fungi> cool
19:05:00 <clarkb> I don't want to bug them but maybe I should
19:05:02 <fungi> #topic Specs approval
19:05:07 <fungi> we don't seem to have anything new up this week
19:05:12 <fungi> #topic Priority Efforts
19:05:19 <fungi> nothing called out specifically here
19:05:28 <fungi> #topic New propose job for OpenStack Ansible (hwoarang, odyssey4me)
19:05:32 <fungi> #link https://review.openstack.org/453130 Add 'propose' job for openstack-ansible-tests
19:05:36 <fungi> per the agenda, "Need to either agree to do it or reject it and proceed with manual syncing instead"
19:05:39 <fungi> concerns raised are the usual outlined in our reviewing documentation
19:05:45 <fungi> #link http://git.openstack.org/cgit/openstack-infra/project-config/tree/REVIEWING.rst?id=291daea#n51 Reviewing project-config: Proposal jobs
19:05:51 <fungi> the stated use case seems similar to this change which we approved a year ago:
19:05:55 <fungi> #link https://review.openstack.org/301375 Implement periodic job to update Puppet OpenStack constraints
19:06:35 <pabelanger> It might be worth reaching out to openstack puppet team, since they last did this and quickly revert it
19:06:51 <fungi> well, they reverted the delorean proposal job, right?
19:06:54 <pabelanger> IIRC, they got tired of +3 automated patches
19:07:14 <pabelanger> not delorean, but job promotion things
19:07:29 <fungi> ahh, i assumed "dlrn" was delorean
19:07:37 <fungi> something else then, i guess
19:08:04 <pabelanger> But, no objection from me, if they want to give it a go
19:08:25 <fungi> the propose-puppet-openstack-constraints job seems to still be in place anyway
19:08:56 <fungi> yeah, still in the periodic pipeline for openstack/puppet-openstack-integration
19:09:23 <pabelanger> odd, maybe I'm thinking of some other one.
19:09:30 <pabelanger> I'll dive into in more later
19:09:39 <fungi> they removed their dlrn promotion proposal job
19:09:49 <fungi> saw that in our git history when researching this current request
19:09:50 <pabelanger> k, that must have been it
19:10:48 <fungi> ran successfully today for example:
19:10:49 <fungi> #link http://logs.openstack.org/periodic/propose-puppet-openstack-constraints/0595ba5/console.html
19:12:19 <fungi> hwoarang and odyssey4me don't seem to be present to discuss this, but does anyone disagree with the basic idea of 453130 anyway?
19:12:31 <hwoarang> I am around
19:12:35 <fungi> ahh, cool
19:13:33 <hwoarang> I am happy either way. A job is convenient but we can also do it with a script
19:13:54 <clarkb> could we maybe make them a python module and consume them that way?
19:14:26 <fungi> some things like the shared bindep.txt file may be hard to cover that way
19:15:22 <fungi> i guess it's a question of how much of the things which change frequently can be covered by a shared dependency, and whether what's left changes often enough to benefit from automated consistency
19:15:47 <hwoarang> They don't change often to be fair
19:15:57 <hwoarang> Since they are only used for manual testings
19:16:07 <hwoarang> Not for the gates
19:16:17 <hwoarang> Apart from bindep.txt
19:16:32 <ianw> personally, we've had the devstack plugin doc proposal job for a while ... workflow seems to be fine, even though yes it could be done other ways.  the bindep bit seems most convincing to me
19:16:39 <hwoarang> Which should change once in a blue moon
19:16:51 <fungi> it's always possible to make a different builder macro to use a shared bindep.txt from some central repository you maintain too
19:17:22 <fungi> and if zuul-cloner's checking that out, you can reliably use depends-on relationships when changing it too
19:17:49 <ianw> fungi: i feel like the bindep in the source tree should be the one ... otherwise aren't we making it quite hard for externals?
19:17:49 <hwoarang> But we also need to consider local testing
19:17:51 <AJaeger> cd
19:18:13 <hwoarang> We need to have a current bindep in place without depending on another repo
19:18:14 <fungi> we had a similar discussion about the ~90 puppet module repos the infra team maintains, and decided to just stick with a shared gem as a dependency for the things we could cover that way and manually handle everything else on the rare occasion we need to change it
19:19:35 <clarkb> right thats why I'm wondering if something like pypi package would be good
19:19:38 <ianw> gem, or a requirements.txt addition seems a bit different, since that's already a "go and grab it" type situation
19:19:56 <ianw> maybe if bindep had a "include from http" method ...
19:20:36 <fungi> i'm not opposed to bindep including remote references, though that is a fair amount of added complexity and error handling
19:20:48 <hwoarang> True
19:21:33 <pabelanger> ansible should be learning about bindep shortly: https://github.com/ansible/ansible/pull/22159
19:21:47 <pabelanger> so, it wouldn't be hard to write a playbook to fetch bindep.txt file first
19:22:15 <fungi> so from my perspective, i'm not opposed to the requested job... but it is very easy to rely on automated code duplication for things where having actual shared dependencies is more appropriate, so have to make sure to avoid that temptation
19:23:02 <fungi> it doesn't seem like it would be proposing new changes particularly often, so likely don't need to worry about the reviewer fatigue downside anyway
19:24:29 <hwoarang> Yes i want to make sure all changes are in before the job is in place so we know only have one big round of reviews and that's it
19:25:03 <hwoarang> Sorry typing from phone
19:26:39 <fungi> current list of files it's syncing are run_tests.sh, bindep.txt and Vagrantfile
19:26:49 <hwoarang> Yes
19:26:55 <fungi> which presumably are all similarly hard to cover with a python dep
19:27:47 <clarkb> ya, they don't slot in directly
19:28:06 <fungi> i'm inclined to approve it once the issue AJaeger has identified is corrected
19:28:22 <fungi> AJaeger: any opinion on this since you've been the most engaged reviewer on that change?
19:28:28 <hwoarang> so what we want to solve here is the vagrant bit
19:28:48 <hwoarang> we expect users and devs to simply do 'vagrant up' and start the testsuite
19:28:50 <AJaeger> fungi: I'm fine with taking it
19:29:10 <ianw> i am also happy, per comment; modulo AJaeger's review
19:29:13 <fungi> okay, cool. i feel like we've done some thorough diligence discussing it
19:29:15 <hwoarang> for what we need Vagrantfile and run_tests.sh in the repo. Otherwise the user will have to fetch these files from omewhere else
19:29:23 <hwoarang> *that
19:30:18 <fungi> #agreed The use case for the "propose-openstack-ansible-tests-scripts" job put forth in change 453130 is sufficiently justified
19:30:42 <fungi> thanks for bringing this one to the meeting hwoarang!
19:31:02 <fungi> #topic What to do about debs for vhd-util and bubblewrap (mordred)
19:31:04 <hwoarang> thank you for all the discussion around it.
19:31:08 <fungi> per the agenda, "PPAs on Launchpad seem to be sad, as does the backports process currently"
19:31:29 <mordred> yah.
19:31:50 <mordred> so - according to SpamapS there is a dearth of humans processing the backports queue
19:31:53 <jeblair> i don't know that i have what it takes to make them happy.
19:32:05 * jeblair tries balloon animals
19:32:07 <mordred> this is the normal way we'd consume bubblewrap on xenial
19:32:26 <fungi> there's of course the filthy option of building these and sticking them in a directory on tarballs.o.o by hand. not that i'm advocating for such a thing
19:32:48 <fungi> but there's probably an acceptable middle ground between ubuntu ppa and that
19:33:01 <clarkb> right so my idea for middle ground was to use nix
19:33:18 <clarkb> we might have to make a nix package but then it would theoretically be useable on centos/fedora/ubuntu/gentoo/whatever
19:33:41 <ianw> fungi: not sure that's all that filthy ... since all the building is logged and repeatable.  afs kernel modules for example
19:34:04 <SpamapS> mordred: I have gotten some feedback on backports
19:34:13 <jeblair> other options include hosting our own .deb repo (apparently we're closer to be able to do that now after all the deb packaging work?).  and another option is switch to another os.
19:34:20 <fungi> ianw: well, ultimately i think we want some build automation to exist for anything we serve from tarballs.o.o
19:34:20 <SpamapS> Apparently there's more or less 1 person doing backports, and he went on vacation last week.
19:34:55 <clarkb> using nix also potentially gives us a good answer for when people want $reallynewthing like libvirt or ovs in $job
19:35:26 <SpamapS> Iain Lane has also basically suggested that the process is too cumbersome and they'd like to make backports more developer-self-service
19:36:14 <fungi> it's not as if rerunning apt-ftparchive every time we upload a package into the directory is all that hard to implement if the desire is to host a deb package repository
19:36:29 <SpamapS> clarkb: help me out. nix packages are source only? Or they build in chroots for all abi's?
19:36:59 <SpamapS> fungi: I agree.. hosting a small collection of no-change backports is quite simple and many many many shops do it.
19:37:37 <clarkb> SpamapS: aiui its both?
19:38:03 <clarkb> SpamapS: you can bootstrap locally by compiling from the ground up or you if you can use published packages
19:38:13 <pabelanger> building on deb package work will require some work, since it was debian-jessie based.  However, not the end of the world
19:38:28 <clarkb> and since nix installs are self contained if you grab published packages it just grabs all the published deps too and puts them in the install root
19:38:42 <fungi> SpamapS: yeah, my script for it looks like this (makes secure-apt perfectly happy too wen my openpgp key is installed): http://paste.openstack.org/show/607916/
19:38:55 <clarkb> and in theory this will all just work except for the kernel because well that is harder to replace on the fly
19:39:14 <SpamapS> clarkb: yeah so, it is basically adoptiong nixos :)
19:39:23 <clarkb> SpamapS: no
19:39:26 <SpamapS> just not the kernel :)
19:39:28 <clarkb> no
19:39:41 <clarkb> everything nix would be in its own curated corner that you'd opt into as the user
19:39:53 <clarkb> adopting nixos implies a lot more changes
19:39:58 <fungi> in /opt/into/nix!
19:40:01 <clarkb> especially around image building and config managment
19:40:07 <SpamapS> Yeah, you're adopting nixos, for bubblewrap. :)
19:40:16 <SpamapS> I like it
19:40:19 <SpamapS> don't get me wrong at all!
19:40:22 <clarkb> yes but not necessarily for zuul if that makes sense
19:40:26 <SpamapS> It's a container approach really. :)
19:40:34 <clarkb> yes its similar to snaps
19:40:38 <clarkb> or the flatpak stuff
19:40:47 <SpamapS> which uses bubblewrap ;)
19:40:50 <clarkb> but its been around longer and seems to be much richer
19:40:54 <fungi> or rpath/conary, or...
19:40:55 * SpamapS just went circular
19:41:00 <clarkb> right because its not actually doing process isolation
19:41:10 <clarkb> its just telling your linker and path and all that where to load things into memory
19:41:14 <clarkb> once thats done its up to you
19:42:16 <clarkb> its just a fancy way of putting bits on the filesystem so that when added to your path they just work
19:42:30 <ianw> like software collections?
19:43:00 <clarkb> ianw: ya except its on a per user basis and it dedups and does fancy things
19:44:20 <clarkb> anyways, I was just thinking that maybe instead of building $package x distro x releases we could make packages in a system that we'd set up once
19:44:55 <clarkb> snaps and flatpaks are other options here and they come with different garuntees and drawbacks
19:46:07 <fungi> so i guess we need short-term and long-term solutions... the vhd-util package is something we need a means of updating on our own and may not be able to wait for extensive package build automation to be implemented (be that nix or whatever)
19:46:31 <fungi> and getting a bubblewrap package is a blocker for zuul v3 rollout
19:46:36 <jeblair> what's the status of our .deb build automation?
19:47:36 <SpamapS> Whatever does the apt mirror building could just as easily process uploads from an incoming dir, yes?
19:47:56 <SpamapS> (not into the same mirror, but into the same afs) ?
19:47:58 <ianw> i mean, i have no experience with any of those tools in production/dev-ops situations ... so not sure i'd have anything useful to add.  TBH, the vhd-util stuff, a small periodic build and push to a tarballs repo seems fairly simple and well tested
19:47:58 <jeblair> (i heard things happened for the openstack debian work, but i don't know what)
19:48:13 <SpamapS> oh right there is already some deb build automation
19:48:14 <pabelanger> yes, openstack debian is still a thing, but currently debian-jessie based
19:48:17 * SpamapS forgot about that
19:48:26 <fungi> looks like it was working at least as of a few weeks ago
19:48:26 <pabelanger> so, we can add ubuntu-xenial if needed
19:48:30 <clarkb> pabelanger: iirc that shouldn't matter because you set the chroot to xenial and its fine?
19:48:30 <fungi> #link http://tarballs.openstack.org/packaging-deb/deb-nova/uploads/2081d05921eafc724ccb99a7b02c2e58f42ddf26/
19:48:34 <pabelanger> then create a repo in reprepro
19:48:35 <SpamapS> should be relatively straight forward.
19:48:40 <clarkb> pabelanger: which is what we tried to get zigo to do in reverse when setting things up?
19:48:48 <pabelanger> clarkb: right, we'd need to write the build scripts for that
19:49:16 <clarkb> oh i see the builds themselves are hardcoded to be debian-jessie
19:49:21 <mordred> sorry - the sprinkler guy arrived right as this topic started
19:49:22 <clarkb> they also run on debian-jessie instances
19:49:28 <pabelanger> right
19:49:41 <pabelanger> so, it would take a little work to move this to ubuntu, but I can help do that
19:49:48 * mordred can help pabelanger
19:49:53 <clarkb> the running on debian-jessie is fine, we just need to make the builds target xenial
19:50:02 <pabelanger> yes
19:50:13 <fungi> i wouldn't be surprised if there's a fair amount of the pkg-deb team's workflow assumptions baked into those jobs right now (as far as the repo being a fork of what's being packaged, specific branch naming, et cetera)
19:50:28 <pabelanger> likely need to make some changes to our JJB publishers, so we don't upload to openstack debian repos
19:50:41 <fungi> that too
19:51:55 <fungi> right now we just need bubblewrap packaged for whatever distro/release we're going to be running zuul executors on, right?
19:52:45 <pabelanger> centos / fedora are okay today
19:53:10 <fungi> and vhd-util for wherever we want to run nodepool builders?
19:53:15 <mordred> fungi: yup
19:53:26 <clarkb> and vhd-util doesn't have any distros that work since its a super local patch right?
19:53:30 <mordred> yes
19:53:46 <mordred> there is no work that I'm aware of for getting that patched vhd-util into a distro in a reasonable manner
19:53:56 <fungi> i gather it's not anything any sane distro would ever carry directly due to being a fork of xen or something
19:54:03 <mordred> I've also heard rumors that modern qemu-img can make working vhd images
19:54:15 <pabelanger> but some rummors of rackspace using qcow2? or qemu-img?
19:54:17 <mordred> but as of yet that has not been verified by anyone here
19:54:32 <mordred> pabelanger: no- just that qemu-img can convert raw to vhd that work on rax
19:54:41 <pabelanger> ack
19:55:09 <mordred> fungi: and then yes, getting bubblewrap for xenial is, I think, the other thing
19:55:20 <fungi> also, on the bubblewrap front, debian does have packages (and backported to jessie too). we wouldn't necessary have to _run_ debian to use those, we could consider just using those packages on trusty or xenial since their dependencies can be met there with no trouble
19:55:21 <clarkb> so sounds like we need something regardless for vhd util
19:55:29 <clarkb> then we can decide if we want to piggy back bubblewrap on that
19:55:41 <clarkb> or use $otherpackagesource/distro/whatever
19:55:50 <mordred> well - we can try to get qemu-img working for vhd and maybe drop our need for vhd-util
19:56:06 <pabelanger> I'll do an upload today
19:56:12 <mordred> but yes, if that still doens't work (and yuck long iteration cycle) - we need a place to put vhd-util
19:56:15 <jeblair> that might be a good first question to answer
19:56:23 <fungi> curous if anyone has tried running the bubblewrap package from jessie/backports or from stretch
19:56:24 <jeblair> since if we need it for vhd-util, may as well do the same for bwrap.
19:56:32 <jeblair> but otherwise, lots more interesting options.
19:57:35 <fungi> mordred: do you mind if we punt your other topic to next week? or to the ml (feedback from current dib contributors would be nice to get first anyway)
19:59:09 <fungi> to sum this up, if newer qemu-img and cross-distro bubblewrap solve these issues, then i wouldn't worry about $topic for now. otherwise we probably want a spec
19:59:46 <fungi> well, s/$topic/automating package builds and hosting our own/
20:00:04 <fungi> and that's all we have time for today
20:00:14 <fungi> thanks everyone!
20:00:16 <fungi> #endmeeting