19:01:15 <clarkb> #startmeeting infra
19:01:16 <openstack> Meeting started Tue Jul 17 19:01:15 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:20 <openstack> The meeting name has been set to 'infra'
19:01:31 <mordred> p/
19:01:34 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:44 <clarkb> #topic Announcements
19:01:56 <clarkb> I've been reminded to remind you all to submit to the summit CFP
19:02:00 <clarkb> I think you have just under 12 hours now
19:02:12 <clarkb> in any case it closes soon
19:02:51 <clarkb> yup 11:59 PST today which is 11 hours and 57 minutes from now
19:03:16 <clarkb> #topic Actions from last meeting
19:03:27 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-07-10-19.01.txt minutes from last meeting
19:03:56 <clarkb> ianw and fungi took an action to begin a letsencrypt POC. I don't expect this is done yet particularly with people traveling and on PTO
19:04:17 <fungi> yeah, i'm more focused on the mm3 poc first
19:04:17 <clarkb> ianw: fungi: anything worth discussin on this item yet?
19:04:31 <ianw> not yet, but yeah, i was on pto all last week
19:04:35 <fungi> and about to disappear for ~two weeks starting tomorrow
19:04:53 <fungi> but i can pitch in on whatever ianw starts when i get back, if he gets time
19:05:25 <clarkb> #action ianw fungi Set up letsencrypt POC
19:05:34 <clarkb> #topic Specs approval
19:05:43 <clarkb> #link https://review.openstack.org/565550 Containerized infra
19:05:56 <clarkb> two weeks ago I asked that we do our best to review this spec so that it can be put up for approval this week
19:06:27 <clarkb> A number of us have reviewed it and there are a few +1s. Any objections to formally putting it up for approval? I'd likely get around to doing that thursday afternoon local time
19:07:08 <fungi> no objection from me
19:07:21 <Shrews> i reread that today. lgtm
19:08:37 <corvus> i think it's ready
19:08:40 <clarkb> alright not hearing any objections. I will look forward to your final review prior to hopefully approving it on thursday
19:08:55 <mordred> woot
19:09:33 <mordred> also, relatedly, we're very close to having jobs building containers of zuul and nodepool - so some progress on parts of it have been made
19:09:54 <clarkb> cmurphy has also been pushing on the getting us out of the puppet hole tasks
19:10:13 <cmurphy> ya
19:10:50 <clarkb> continuing to review topic:puppet-4 would be helpful (I try to dig into that periodically myself)
19:10:50 <mordred> yay puppet hole
19:11:03 <cmurphy> yes please
19:11:04 <clarkb> I'm expecting we are close to being able to pick a service for using puppet 4
19:11:34 <cmurphy> yes, i proposed https://review.openstack.org/582765 to try to turn on the future parser for review-dev.o.o
19:11:55 <clarkb> #link https://review.openstack.org/582765 turn on future parser (puppet 4 parser) for review-dev.o.o
19:11:59 <cmurphy> most of the other patches are just adding tests or are xenial fixes
19:12:30 <cmurphy> there are some scary iptables changes that should probably be babysat
19:13:16 <cmurphy> I found that the drush puppet module we're using was never updated for xenial + php7 so groups.o.o might be tough to update
19:13:31 <clarkb> cmurphy: we can probably loop in smarcet to help with that
19:13:49 <cmurphy> the the refstack module is kind of broken, i'm hoping the refstack people can help update https://review.openstack.org/#/c/557539/
19:13:51 <fungi> or mrmartin if he's interested
19:14:07 <fungi> (on the drush/groups front)
19:14:33 <cmurphy> but mostly i think we should start taking the plunge and turning on the future parser as much as possible
19:15:00 <clarkb> as far as spec implementation planning goes, do we think there are a specific set of tasks that we want done before the ptg so that we can focus on other tasks face to face?
19:15:12 <clarkb> mordred: ^ you may have this paged in better than anyone
19:15:19 <clarkb> cmurphy: ++
19:15:27 <cmurphy> i think we can get all the puppet 4 stuff done before the ptg
19:16:24 <mordred> clarkb: I think we should at least have split the run_all into individual playbooks - and it would probably be good to have an ansible version of openstack_project::server
19:16:48 <mordred> but honestly, once we have openstack_project::server the rest of things are pretty parallelizable
19:17:42 <mordred> clarkb: if I can ever get the pypi mirror working again, I'll start in on the base ansible things
19:17:46 <mrmartin> I’m in for xenial / php7.
19:17:48 <clarkb> to summarize then we can get the puppet 4 work done so that we can limp along while we then parallelize ansible container conversions after having a version of openstack_project::server in ansible
19:18:17 <clarkb> order is roughly puppet 4 and openstack_project::server in ansible in parallel, then we can convert services to containers after ansible is done
19:18:52 <mordred> ++
19:18:55 <mordred> I agree with those words
19:19:07 <clarkb> I want to say we are in good shape for the puppet stuff as long as we can get reviews (thank you cmurphy). Does anyone have the base server playbook on their radar?
19:19:37 <mordred> oh, that's what I was going to work on after this container job
19:19:38 <corvus> clarkb: maybe that warrants an #info ? :)
19:19:44 <clarkb> corvus: ++
19:20:22 <corvus> i'd like to pitch in a little on the base server, but i'm not prepared to bootstrap it
19:20:33 <clarkb> #info Puppet 4 work and openstack_project::server conversion to ansible can happen in parallel and are our prereqs. Once the ansible base playbook work is done we can parallelize per service container conversions
19:20:53 <mordred> I can get it started, then will ping corvus when there's something there to rage-code against
19:21:02 <clarkb> #action mordred to begin work on openstack_project::server conversion to ansible with corvus helping
19:21:04 <corvus> cool
19:21:31 <clarkb> #info review topic:puppet-4 to help get the puppet work completed
19:22:27 <clarkb> We sort of blended together the config mgmt priority effort and the spec approval topic. If there isn't anything else I'll jump to storyboard next
19:23:29 <clarkb> #topic Priority Efforts
19:23:38 <clarkb> #topic Storyboard
19:24:04 <clarkb> I've seen patches from Fatema go by in #storyboard. If you are interested in javascripty things reviews on those are likely helpful
19:24:21 <clarkb> We are also seeing interest from starlingx users
19:24:30 <fungi> and airship too
19:24:45 <fungi> but yesterday was starlingx questions, yes
19:25:42 <mordred> there's an ML thread about openstack/requirements and needing to track tasks across storyboard and launchpad projects
19:25:48 <clarkb> Will be interesting to see what sort of feedback they come back with as very new users without storyboard history background
19:25:49 <mordred> I almost responded to it- but didn't
19:26:45 <clarkb> More a heads up that there is stuff happening there than anything else. Fatema's changes in particular are aimed at improving search in storyboard which si something our users have wanted
19:26:52 <fungi> yeah, i keep meaning to follow up on that one because the vmt has some similar patterns
19:27:04 <fungi> the requirements sb thread i mean
19:27:18 <mordred> clarkb: ++
19:27:37 <fungi> the obvious answer is what the vmt went with (live with having two different places you're copying stuff when it affects projects across different trackers)
19:28:05 <mordred> fungi: yah - my answer was shaping into "we didn't really intend for the 'integrated release' projects to span both places for a long time" - but I realize that wasn't super helpful advice
19:28:13 <mordred> I think fungi's VMT answer is likely the better answer
19:28:22 <fungi> it's not a great solution, but it's less work than trying to engineer a complex solution for what should hopefully be a transitional situation
19:28:29 <mordred> since it's at least based on what someone is actually doing
19:28:45 <fungi> we didn't originally plan for integrated release projects to span both places
19:28:46 <clarkb> lp does know how to track external bugs, possible we could hack on top of that?
19:28:50 <clarkb> at least until we stop using lp
19:29:24 <fungi> but the situation we ended up with is that different integrated release projects have different development workflows, use their defect and task tracking solutions in different ways, and are ready to transition at different times
19:29:44 <fungi> so telling some of them that they can't have storyboard until all of them can have storyboard was a worse situation
19:30:49 <clarkb> ya we compromised in the opposite direction of making this particular problem easy
19:30:59 <fungi> yes, lp has a feature where you can tell it your bug tracker is at some other url. i wasn't able to get that to actually work but the lp devs assured me in the last forum session we had that it does something so should find time to try again
19:31:23 <clarkb> fungi: I've used it when submitting python package bugs against ubuntu and for the python bug tracker it seemed to wrok
19:32:05 <clarkb> Any other storyboard related items before moving on?
19:32:16 <fungi> yeah, i think when i tried a year-ish ago it was built around the idea that they needed to program a driver to figure out how to find bug reports in your particular tracker and they'd implemented it for some popular ones like bugzilla and debbugs
19:32:39 <fungi> but maybe that's changed
19:33:00 <mordred> that is the last state of it that I was aware of
19:34:38 <clarkb> #topic General Topics
19:35:07 <clarkb> Really quickly before we get to corvus' topic, we haven't forgotten about winterscale. I'm told jonathan will be attempting a followup today
19:35:46 <clarkb> I've also got a service breakdown proposal going at https://etherpad.openstack.org/p/winterscale-service-branding
19:35:58 <clarkb> your input on that is much apprecitated
19:36:03 <clarkb> (also I can't type)
19:36:07 * Shrews proposes "winterscale is coming" as a motto
19:36:18 <cmurphy> ++
19:36:39 <clarkb> alright corvus: swift and logs
19:37:34 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2018-July/006020.html thread on moving zuul job logs into swift
19:38:36 <corvus> there's been some good discussion in email; mostly wanted to see if folks had feedback or questions in irc
19:38:36 * mordred welcomes our new logs-in-swift overlords
19:38:39 <clarkb> I'll try to summarize. Zuulv3 makes it a bit easier for us to talk to swift from zuul through the use of encrypted secrets. We've also found the ara middleware can crash our logs server in some cases due to use of memory leading to swapping
19:38:56 <clarkb> This leads to rebooting which potentially will require many hour long fscks
19:39:25 <corvus> well, on friday, ext4 did its thing well and no fsck was needed
19:40:06 <corvus> but it's a possibility, and we've still got potential iscsi failures looming over us (which always require a fsck)
19:40:15 <corvus> i never want to do another of those
19:40:32 <clarkb> as for feedback I think we should probably identify which of our clouds provide an acceptable swift deployment and work with one of them to start. Then we can test things like how slow is uploading 30k small files for example
19:40:46 <corvus> clarkb: is 30k typical or max?
19:41:10 <clarkb> corvus: I think that is more like the max, but we can grab a few ara db files and generate the files to check
19:41:20 <clarkb> to get an idea of what it is like for our current jobs
19:41:31 <corvus> good idea.  maybe for some typical devstack jobs or something.
19:41:36 <clarkb> ++
19:41:45 <ssbarnea1> i support the move to swift idea, in the past I did something similar jenkins logs to S3 instead of jenkins, to offload the master. obviously that this also depends on how scalable is our swift, maybe experiment with parallel usage first.
19:41:51 <clarkb> and multinode as that may potentially create a file count multiplier
19:41:59 <fungi> seems like the massive ones were tripleo and openstack-ansible which were including their own ara content?
19:42:08 <clarkb> fungi: ya
19:42:23 <corvus> did they change to using middleware too?
19:42:30 <clarkb> I believe tripleo did
19:42:37 <clarkb> not sure about osa
19:42:43 <corvus> k.  we should test that in particular then.
19:43:12 <corvus> tbh, i'd really like us to favor the idea that we should get out of the middleware game.
19:44:19 <ssbarnea1> corvus++
19:44:24 <mordred> ++
19:45:01 <fungi> i suppose an ara javascript front-end which retrieves an sqlite file and operates locally on it is a no-go ;)
19:45:03 <clarkb> I'm in favor as long as it doesn't create a big impact particularly for already very short jobs
19:45:16 <ssbarnea1> regarding ara, i think it may be changed to lower its footprint in both total size and number of files. i was planning to investigate that but didn't hat time.
19:45:30 <clarkb> I'm less concerned if 3 hour tripleo job needs 10 minutes to upload files than 3 minute pep8 job needing 10 minutes to upload files
19:46:31 <corvus> fungi: i'm not aware of js parsing sqlite libraries.  but if there were a way to store data in json or something js can deal with, maybe that'd be a possibility.  or maybe ssbarnea1's idea.  :)
19:46:36 <ssbarnea1> at leas in theory javascript can work directly with sqlite files, meaning that instead of tons of json files on disk, ara could just download the sqlite file and get data from it.
19:46:58 <corvus> ssbarnea1: oh that's awesome!  that may be something worth looking into
19:47:21 <mordred> fungi, corvus: https://github.com/kripken/sql.js/
19:47:27 <corvus> i don't know what the ara middleware does right now; but i imagine a solution like that would look like implementing whatever that is in js...
19:47:45 <fungi> anyway, bit of a rabbit hole
19:47:48 <fungi> sorry
19:48:05 <corvus> it's worth talking about.  it's definitely the weak point in the plan.
19:48:24 <mordred> ++
19:48:29 <corvus> my argument is basically we should accept the cost as long as it's reasonable, because it's a net benefit.  but if we can minimize the cost, or even reverse it, great!
19:49:04 <clarkb> fwiw similar discussion has come up re kata using zuul
19:49:19 <clarkb> they wanted to know how to make certain things htmly and the suggestion mnaser and I had was to write html that gets uploaded
19:49:44 <clarkb> I also think outside of general zuul reporting the people best prepared to present their logs are the ones producing them
19:50:11 <corvus> i think that's a good suggestion.  it gives folks local control and the system overall flexibility.
19:50:14 <clarkb> what we've done with os loganalyze is sort of a "here is a best effort thing that adds some color based on a guess"
19:51:25 <corvus> in my message to the zuul list, i do, however, suggest that we can js-ify osla and make it generic.  i think that might be a nice thing to do that would help out people in general, and probably handle a lot of the typical use cases.
19:51:48 <mordred> yah. html-ifying log files with timestamps seems doable and friendly
19:52:01 <clarkb> corvus: I do like that for very consistent and specific outputs like journald logs
19:52:07 <mordred> ++
19:52:21 <clarkb> with python and openstack in particular we have never had that kind of consistency though
19:53:34 <clarkb> Running out of time. Lets talk project renames really quickly
19:53:39 <clarkb> #topic Project Renames
19:54:34 <clarkb> I sort of punted on this a couple weeks ago to see where we would be at today. I think at least one of us is on vacation over the next couple weeks. Rocky release schedule makes it look like July 30- August 3 is our best bet
19:55:00 <clarkb> I expect I will be around taht week (parents are flying in night of the 2nd though)
19:55:39 <clarkb> Anyone else expect to be around on say August 3 at 1600UTC?
19:55:58 <clarkb> if so I can run it by the release team and see if that works
19:56:10 <fungi> i'll be around
19:56:17 <fungi> count me in
19:57:12 <clarkb> #info Project renames tentatively scheduled for August 3 at 1600UTC.
19:57:18 <clarkb> #topic Open Discussion
19:58:44 <cmurphy> do we want to set up a time to babysit https://review.openstack.org/582765 and maybe a few others and start making headway on that?
19:58:50 <ssbarnea1> i have one subject, reviving few dormant projects
19:59:18 <clarkb> cmurphy: I can help with that on Friday probably
19:59:24 <clarkb> (I try to keep friday's clear for that sort of thing)
19:59:26 <fungi> as mentioned earlier, i have a mailman3 proof of concept going on. notes at:
19:59:27 <clarkb> ssbarnea1: go for it
19:59:28 <fungi> #link https://etherpad.openstack.org/p/mm3poc
19:59:39 <cmurphy> clarkb: friday's not good for me :/
19:59:46 <fungi> if anybody's interested in helping with it, i've got it most of the way there
19:59:49 <clarkb> cmurphy: would thursday work instead? I can be flexible
19:59:53 <cmurphy> thursday works
19:59:53 <fungi> but am disappearing for the next two weeks
19:59:54 <ssbarnea1> examples: gerritlib, git-review, bashate.  I would like to help with them and I thin that electrofelix also wants the same.
19:59:56 <clarkb> but today and tomorrow probably need time to review more
20:00:15 <fungi> all infra-root ssh keys should be accepted by lists-dev01.o.o
20:00:27 <ssbarnea1> at least one of them has a review that is 36 months old!
20:00:34 <clarkb> ssbarnea1: yes there was the mailing list thread for git-review. I think that helped set expectations at least for that project?
20:01:39 <ssbarnea1> indeed, i need to recheck the thread as I do not remember what was the conclusion.
20:01:52 <clarkb> bashate isn't an infra project iirc it is a qa project. Gerritlib is definitely a case of it does what we need ti to do and otherwise doesn't get much attention but probably should. There is also an alternative implementation on pypi that maybe we should consider moving too? Unsure if that would work for zuul
20:02:10 <clarkb> corvus: ^ zuul may have interest in gerritlib in particular as a dep?
20:02:11 <ianw> ssbarnea1: yes, for bashate, i merged a few things and will do a release
20:03:09 <clarkb> Not to kill the conversation, but I do have a meeting in under half an hour I need to prep for. I'll stop the meeting here, but feel free to continue discussion in #openstack-infra or on the mailing list
20:03:13 <clarkb> thank you everyone
20:03:15 <clarkb> #endmeeting