Friday, 2019-04-12

*** ianychoi has joined #openstack-infra00:01
*** owalsh_ has joined #openstack-infra00:01
*** harlowja has joined #openstack-infra00:01
*** owalsh has quit IRC00:02
*** owalsh has joined #openstack-infra00:05
*** owalsh_ has quit IRC00:06
*** igordc has quit IRC00:07
*** owalsh_ has joined #openstack-infra00:09
openstackgerritMerged openstack-infra/zuul master: Handle github exceptions better when trying to get a PR
*** owalsh has quit IRC00:10
openstackgerritArtom Lifshitz proposed openstack-infra/project-config master: Make whitebox use Storyboard
*** owalsh has joined #openstack-infra00:13
*** hamzy has joined #openstack-infra00:14
*** owalsh_ has quit IRC00:14
*** ijw has quit IRC00:15
*** ijw has joined #openstack-infra00:15
*** gyee has quit IRC00:15
*** owalsh_ has joined #openstack-infra00:18
*** owalsh has quit IRC00:19
*** ijw has quit IRC00:20
*** owalsh has joined #openstack-infra00:21
*** Lucas_Gray has quit IRC00:22
*** owalsh_ has quit IRC00:23
*** sthussey has quit IRC00:24
*** owalsh_ has joined #openstack-infra00:26
*** owalsh has quit IRC00:27
*** owalsh has joined #openstack-infra00:31
*** owalsh_ has quit IRC00:32
*** owalsh_ has joined #openstack-infra00:35
*** ijw has joined #openstack-infra00:36
*** owalsh has quit IRC00:36
*** owalsh has joined #openstack-infra00:39
*** rh-jelabarre has quit IRC00:39
openstackgerritMerged openstack-infra/system-config master: Pass SSL certs to
*** owalsh_ has quit IRC00:40
*** ijw has quit IRC00:40
*** owalsh_ has joined #openstack-infra00:43
*** rh-jelabarre has joined #openstack-infra00:43
*** owalsh has quit IRC00:44
*** owalsh has joined #openstack-infra00:47
*** owalsh_ has quit IRC00:48
*** owalsh_ has joined #openstack-infra00:51
*** owalsh has quit IRC00:52
*** mriedem is now known as mriedem_afk00:57
*** owalsh has joined #openstack-infra00:57
*** owalsh_ has quit IRC00:58
*** ricolin has joined #openstack-infra01:05
*** hongbin has joined #openstack-infra01:08
*** whoami-rajat has joined #openstack-infra01:09
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Implement max-servers for AWS driver
*** owalsh_ has joined #openstack-infra01:29
*** owalsh has quit IRC01:30
openstackgerritIan Wienand proposed openstack-infra/puppet-graphite master: Fix ssl lookups
*** mriedem_afk is now known as mriedem01:42
*** bhavikdbavishi has joined #openstack-infra01:47
*** apetrich has quit IRC01:57
*** yamamoto has joined #openstack-infra01:58
*** diablo_rojo has quit IRC02:00
*** markvoelker has joined #openstack-infra02:00
*** gregoryo has joined #openstack-infra02:10
*** yamamoto has quit IRC02:21
*** yamamoto has joined #openstack-infra02:23
*** mriedem has quit IRC02:29
*** markvoelker has quit IRC02:35
*** rlandy|ruck|bbl is now known as rlandy|ruck02:42
*** rlandy|ruck has quit IRC02:49
*** irclogbot_0 has quit IRC03:01
*** irclogbot_0 has joined #openstack-infra03:01
*** bhavikdbavishi has quit IRC03:03
openstackgerritMerged openstack-infra/puppet-graphite master: Fix ssl lookups
*** rcernin has quit IRC03:15
*** psachin has joined #openstack-infra03:19
*** ykarel has joined #openstack-infra03:24
*** raukadah is now known as chandankumar03:25
*** nicolasbock has quit IRC03:25
openstackgerritTrinh Nguyen proposed openstack-infra/project-config master: Move Telemetry to Storyboard
*** rcernin has joined #openstack-infra03:27
*** markvoelker has joined #openstack-infra03:32
*** owalsh has joined #openstack-infra03:40
*** owalsh_ has quit IRC03:40
*** efried has quit IRC03:43
*** efried has joined #openstack-infra03:44
openstackgerritTrinh Nguyen proposed openstack-infra/project-config master: Move Telemetry to Storyboard
*** owalsh_ has joined #openstack-infra03:48
*** owalsh has quit IRC03:49
openstackgerritTrinh Nguyen proposed openstack-infra/project-config master: Move Telemetry to Storyboard
*** yamamoto has quit IRC03:51
*** ykarel_ has joined #openstack-infra03:51
*** yamamoto has joined #openstack-infra03:52
*** ykarel has quit IRC03:53
*** ykarel has joined #openstack-infra03:54
*** ykarel_ has quit IRC03:56
*** yamamoto has quit IRC03:57
*** rcernin has quit IRC03:57
*** lseki has quit IRC03:59
*** bhavikdbavishi has joined #openstack-infra03:59
*** hwoarang has quit IRC03:59
*** hwoarang has joined #openstack-infra04:00
*** rcernin has joined #openstack-infra04:01
*** yamamoto has joined #openstack-infra04:02
*** imacdonn_ has quit IRC04:04
*** yamamoto has quit IRC04:04
*** markvoelker has quit IRC04:04
*** imacdonn_ has joined #openstack-infra04:04
*** yamamoto has joined #openstack-infra04:08
*** ramishra has joined #openstack-infra04:11
*** yamamoto has quit IRC04:13
*** udesale has joined #openstack-infra04:13
*** hongbin has quit IRC04:16
*** owalsh has joined #openstack-infra04:21
*** owalsh_ has quit IRC04:22
*** _d34dh0r53_ has joined #openstack-infra04:24
*** cloudkiller has joined #openstack-infra04:25
clarkbwe still need to get the webserver restart in right? but we have 90 days for that?04:29
ianwyep ... i'll look at your work soon :)04:29
clarkbthe hardest bit will be testing it04:31
clarkbbut maybe we hand wave around that for a bit04:31
ianwclarkb: that said; we're going to probably need to get some httpd deployment into ansible for these arm64 mirrors ...04:32
clarkbalso the gitea tests can probably test it too04:33
clarkbrestart gitea instead of apache but same general idea04:33
*** roman_g has quit IRC04:35
ianwalso looking at accounts, the only thing it really gives us is we register an email with the cert, so it send "you need to renew" reminders.  not sure if not getting those is a feature or a bug04:37
clarkbour certchecker works with sni now so I think we ha e that covered04:38
clarkbI ran it in the foreground and checked it had the right values with git.*04:39
clarkbshould add the new graphite cert to that checklist :)04:39
*** Lucas_Gray has joined #openstack-infra04:40
*** tkajinam has quit IRC05:00
*** markvoelker has joined #openstack-infra05:02
*** tkajinam has joined #openstack-infra05:07
*** ykarel has quit IRC05:16
*** ykarel has joined #openstack-infra05:29
*** ykarel_ has joined #openstack-infra05:34
*** markvoelker has quit IRC05:34
*** zhurong has quit IRC05:35
*** ykarel has quit IRC05:36
*** ricolin has quit IRC05:39
*** jtomasek has joined #openstack-infra05:39
*** Lucas_Gray has quit IRC05:40
*** zhurong has joined #openstack-infra05:41
*** yamamoto has joined #openstack-infra05:45
*** ykarel__ has joined #openstack-infra05:46
*** ykarel_ has quit IRC05:48
*** psachin has quit IRC05:49
*** quiquell has joined #openstack-infra05:50
*** quiquell is now known as quiquell|rover05:50
*** armax has quit IRC05:53
*** yamamoto has quit IRC05:58
*** lpetrut has joined #openstack-infra06:02
*** gregoryo has quit IRC06:03
*** psachin has joined #openstack-infra06:04
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml
*** pgaxatte has joined #openstack-infra06:20
AJaegeryeah, proposal job works again. mordred, thanks for your help ^06:26
*** markvoelker has joined #openstack-infra06:31
*** dangtrinhnt has joined #openstack-infra06:31
dangtrinhntHi fungi, I would like to migrate Telemetry projects to storyboard. #link
dangtrinhntIf you need any information to proceed that, please let me know. Thank for helping.06:33
openstackgerritMerged openstack-infra/project-config master: Normalize projects.yaml
AJaegerdangtrinhnt: fungi is US based, best wait 7 hours and then ask again. I hope he reads backscroll and replies...06:36
*** pcaruana has joined #openstack-infra06:41
openstackgerritMerged openstack-infra/project-config master: Add airship-governance project
openstackgerritMerged openstack-infra/project-config master: Add vexxhost/ansible-role-docker-distribution
*** quiquell|rover is now known as quique|rover|brb06:44
*** e0ne has joined #openstack-infra06:45
*** ykarel_ has joined #openstack-infra06:49
*** ykarel__ has quit IRC06:51
*** slaweq has joined #openstack-infra06:59
*** hamzy has quit IRC07:05
*** markvoelker has quit IRC07:05
*** hamzy has joined #openstack-infra07:05
*** whoami-rajat has quit IRC07:05
*** whoami-rajat has joined #openstack-infra07:06
*** mnaser has quit IRC07:06
*** Dobroslaw_afk has quit IRC07:06
*** kambiz has quit IRC07:06
*** HenryG has quit IRC07:06
*** mnaser has joined #openstack-infra07:06
*** egonzalez has quit IRC07:07
*** kambiz has joined #openstack-infra07:07
*** egonzalez has joined #openstack-infra07:08
*** rpittau|afk is now known as rpittau07:08
*** quique|rover|brb is now known as quiquell|rover07:08
openstackgerritBrendan proposed openstack-infra/zuul master: gerrit: Add support for 'oldValue' comment-added field
*** ykarel_ is now known as ykarel07:16
*** yamamoto has joined #openstack-infra07:21
*** e0ne has quit IRC07:21
*** iurygregory has joined #openstack-infra07:22
*** ginopc has joined #openstack-infra07:25
*** jpich has joined #openstack-infra07:32
*** e0ne has joined #openstack-infra07:36
*** apetrich has joined #openstack-infra07:42
*** tosky has joined #openstack-infra07:43
*** jpena|off is now known as jpena07:47
*** kjackal has joined #openstack-infra07:49
*** e0ne has quit IRC07:52
*** ykarel is now known as ykarel|lunch07:52
*** ccamacho has joined #openstack-infra07:53
*** kjackal has quit IRC07:56
*** kjackal has joined #openstack-infra07:56
*** Lucas_Gray has joined #openstack-infra07:57
*** e0ne has joined #openstack-infra07:59
*** ricolin has joined #openstack-infra08:01
*** dtantsur|afk is now known as dtantsur08:01
*** markvoelker has joined #openstack-infra08:02
*** Lucas_Gray has quit IRC08:08
*** Lucas_Gray has joined #openstack-infra08:09
*** lucasagomes has joined #openstack-infra08:10
*** calbers has quit IRC08:14
*** calbers has joined #openstack-infra08:27
*** markvoelker has quit IRC08:34
*** derekh has joined #openstack-infra08:39
*** electrofelix has joined #openstack-infra08:39
openstackgerritMerged openstack-infra/irc-meetings master: Updates Octavia meeting time to 1600 UTC
*** e0ne has quit IRC08:55
*** dtantsur is now known as dtantsur|brb08:59
*** e0ne has joined #openstack-infra09:00
*** bhavikdbavishi1 has joined #openstack-infra09:07
*** bhavikdbavishi has quit IRC09:09
*** bhavikdbavishi1 is now known as bhavikdbavishi09:09
*** janki has joined #openstack-infra09:15
*** ginux has joined #openstack-infra09:15
*** ginopc has quit IRC09:17
*** ginux is now known as ginopc09:17
*** ykarel|lunch is now known as ykarel09:21
*** rcernin has quit IRC09:30
*** roman_g has joined #openstack-infra09:30
*** markvoelker has joined #openstack-infra09:31
*** ricolin has quit IRC09:33
*** gfidente has joined #openstack-infra09:36
*** bhavikdbavishi has quit IRC09:43
*** Lucas_Gray has quit IRC09:56
*** e0ne has quit IRC09:58
*** e0ne has joined #openstack-infra10:03
*** markvoelker has quit IRC10:04
*** kjackal has quit IRC10:10
*** Lucas_Gray has joined #openstack-infra10:18
*** Lucas_Gray has quit IRC10:20
*** Wryhder has joined #openstack-infra10:20
*** Wryhder is now known as Lucas_Gray10:21
fricklerinfra-root: the hacker has posted a list of issues that all helped in the coup at
*** e0ne has quit IRC10:28
frickleralthough at a glance they don't seem to affect us directly, I think we should take the time to maybe go through them in a bit more detail10:28
fricklerin particular looking at setting up 2FA for ssh access would seem to be a sensible improvement of security. I've never done that myself before, though, but I'll try to do a bit of testing over the weekend10:29
*** pcaruana has quit IRC10:31
*** nicolasbock has joined #openstack-infra10:43
*** kjackal has joined #openstack-infra10:44
*** gfidente has quit IRC10:49
*** ociuhandu has joined #openstack-infra10:53
*** ociuhandu has quit IRC10:53
*** tbachman has quit IRC10:53
*** gfidente has joined #openstack-infra10:56
*** gfidente has quit IRC11:01
*** markvoelker has joined #openstack-infra11:02
*** bhavikdbavishi has joined #openstack-infra11:05
*** udesale has quit IRC11:05
aspiersfungi, corvus, clarkb:
openstackgerritM V P Nitesh proposed openstack/diskimage-builder master: Adding 'oel' as a new dib element
*** jpena is now known as jpena|lunch11:27
*** jaosorior has quit IRC11:33
*** markvoelker has quit IRC11:35
*** pcaruana has joined #openstack-infra11:36
*** e0ne has joined #openstack-infra11:37
*** yamamoto has quit IRC12:03
*** tbachman has joined #openstack-infra12:05
*** EmilienM is now known as EvilienM12:05
*** yamamoto has joined #openstack-infra12:06
pabelangerfrickler: jenkins strikes again12:08
pabelangerfrickler: some good info in issues12:08
zbrclarkb: just to let you know that we abandoned the gate-check jobs. we still need to do some internal house cleaning but the jobs are no longer run.12:09
*** psachin has quit IRC12:11
*** rlandy has joined #openstack-infra12:16
*** rlandy is now known as rlandy|ruck12:16
*** ricolin has joined #openstack-infra12:25
*** e0ne has quit IRC12:27
*** kgiusti has joined #openstack-infra12:28
*** jpena|lunch is now known as jpena12:28
*** ricolin has quit IRC12:29
*** e0ne has joined #openstack-infra12:41
mordredfrickler: nicely enough - we already don't use agent forwarding - so taht's good12:41
mordredI disagree with 36012:42
mordred359 is ... yeah12:42
*** yamamoto has quit IRC12:43
*** jamesmcarthur has joined #openstack-infra12:43
mordredfrickler: yeah- those are all interesting12:45
*** fdegir has quit IRC12:45
aspiershmm, couple of the mails I sent to -discuss yesterday are still missing from the archives :-/12:45
*** fdegir has joined #openstack-infra12:46
*** timburke has quit IRC12:46
*** cjloader has quit IRC12:46
*** gfidente has joined #openstack-infra12:46
*** timburke has joined #openstack-infra12:48
mordredaspiers: that's because I'm secretly deleting them just to frustrate you12:48
aspiersmordred: I suspected as much12:49
aspiersfrickler: urgh, nickserv passwords need changing now then12:50
* aspiers changes his12:50
*** dtantsur|brb is now known as dtantsur12:51
fungiianw: one thing we can likely also do in the course of switching to letsencrypt is to add caa records in dns like 'graphite IN CAA 0 issue ""' and 'graphite IN CAA 0 iodef ""'12:55
*** e0ne has quit IRC12:55
*** mriedem has joined #openstack-infra12:55
openstackgerritFatih Degirmenci proposed openstack/diskimage-builder master: Constraint networkx to 2.2
fungiianw: and since we unconditionally redirect http to https we could also consider turning on hsts in the apache config like 'Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"' (i should have caught that when reviewing the vhost config)12:58
frickleraspiers: yes, there was a freenode global advice about that earlier12:58
aspiersoh yeah12:58
*** e0ne has joined #openstack-infra12:59
fungidangtrinhnt: the launchpad to storyboard migration process we usually follow is documented at
fungii can test importing all the telemetry projects on storyboard-dev today and assuming we don't find any problems (these days there usually aren't any because we don't find many new things to fix in the migration script) we can figure out when a good time is for your team to cut over and disable bug reporting in launchpad13:01
*** gfidente has quit IRC13:01
fungifrickler: i've done otp configurations for opensshd using opie before, though i haven't tried setting that up as a second factor of authentication13:02
*** jcoufal has joined #openstack-infra13:02
mordredaspiers: thanks for mentioning the nickserv thing though - I missed the freenode announcement - but have now changed my password13:03
fungiaspiers: did you get any moderation notices for those? i usually check the moderation queue at least once a day to purge all the spam and see if any legit messages landed in moderation for some reason13:03
*** gfidente has joined #openstack-infra13:03
mordredfungi: speaking of dns things - would it be useful to publish SSHFP records for our servers now that we've got our happy new dns system? or would it be more pain than it's worth?13:03
*** altlogbot_2 has joined #openstack-infra13:04
aspiersfungi: I got a moderation notice for an unrelated email to -discuss, due to a large recipient list13:04
fungiaspiers: ahh, yep, "Reason: Too many recipients to the message"13:05
fungiwill approve those now13:05
aspiersfungi: I've posted 4 times to this thread
aspiersfungi: but only two of my posts are shown in the archives13:05
fungimordred: yes, sshfp for sure, though openssh and other clients usually need it explicitly enabled13:05
fungimordred: also you can't easily add sshfp records in advance of launching the server so that would need to be followup anyway13:06
*** dave-mccowan has joined #openstack-infra13:06
fungimordred: maybe doable at the same time we add address records though if we get the launch script to look at the host key and spit out appropriate sshfp records for it?13:07
*** gfidente has quit IRC13:09
*** gfidente has joined #openstack-infra13:10
mordredfungi: yeah - that's not a bad idea13:12
*** aaronsheffield has joined #openstack-infra13:13
*** yamamoto has joined #openstack-infra13:13
fungiaspiers: luckily, in an effort to troubleshoot some dkim signature challenges we've been logging copies of all messages to openstack-discuss for a couple months now13:14
fungiaspiers: unfortunately, i only see two messages from you arriving at the mta with that subject13:15
fungiDate: Thu, 11 Apr 2019 15:10:57 +010013:15
aspiersand Fri, 12 Apr 2019 13:57:31 +010013:15
fungiDate: Fri, 12 Apr 2019 13:57:31 +010013:15
fungiyeah, those two13:15
aspiersyeah, the two showing in the archives13:15
aspiersfungi: can I try resending the others right now?13:16
fungiso whatever happened to the others, they haven't reached the server (yet anyway, maybe hung in a deferral queue at the sending side?)13:16
fungifeel free, you just may end up with duplicates if the others are eventually delivered to us13:16
*** dave-mccowan has quit IRC13:17
*** betherly has joined #openstack-infra13:20
*** Wryhder has joined #openstack-infra13:22
*** Lucas_Gray has quit IRC13:23
fungiaspiers: normally if messages are stuck in a deferral queue, the mta holding them will send "temporarily undeliverable" bounces to the sender to let them know, but with the sorry state of smtp standards these days i no longer count on it13:23
*** Wryhder is now known as Lucas_Gray13:23
aspiersohhh, I found one of those13:23
aspiersResponse: 501 Authentication Failure13:24
fungiaspiers: it looks like these transit a couple of mtas at, if the received headers on the ones which made it through are any indication13:24
aspiersor similar13:25
fungiand smtp.nue.novell.com13:25
aspiersthey might be crappy Groupwise servers13:25
*** tkajinam_ has joined #openstack-infra13:25
fungithis i rather expect, yes13:25
aspiersnow that we're independent from Micro Focus we'll be getting rid of those finally13:25
fungiwouldn't it be awesome if the driving factor in suse's independence was that it wanted to finally be able to have reliable mailservers?13:26
aspierswell, that's one of the things the employees are most excited about ;-)13:27
aspiersthat, and a new expenses system13:27
dangtrinhntThanks, fungi :)13:27
*** gfidente has quit IRC13:27
*** tkajinam has quit IRC13:27
fungias opposed to red hat, who recently moved all their employee e-mail off free software and onto a proprietary mail service13:27
aspiersOh, I didn't say we'd be getting a free replacement :-/13:28
aspiersWe won't AFAIK, and I have complained loudly about that many times13:28
mordredyeah. nothing like the free software companies being shining beacons for giving up13:28
aspiersI've used words like "hypocrisy" more than once13:28
fungithat is certainly a good word for it13:28
mordredthe flame war here about that particular instance of us giving up happened before I arrived, so I did not participate13:29
aspiersDon't even get me started on Jira ...13:29
mordredyeah. or ghe13:29
mordredor AWS13:29
mordredbasically - literally nobody cares anymore13:29
* fungi is nobody, for sure13:30
funginobodies of the free software world, unite!13:30
mordredI mean - statistically. luckily there is a nice collection of the remaining humans who care in this channel13:30
aspiersAlso had a big "told you so" moment recently when Trello broke their promise13:30
fungithe last, best hope for free tools13:30
fungiopendev 513:30
mordredaspiers: oh? I didn't know about that ... I'd love another piece of ammunition to mock people with13:31
aspiersShould we rename the channel to #rebel-alliance?13:31
mordredaspiers: what did they do?13:31
aspiersmordred: still searching for the link13:31
*** altlogbot_2 has quit IRC13:32
*** altlogbot_1 has joined #openstack-infra13:33
* persia mumbles that the problem with identifying as a rebellion is that it forces one to prop up the evil empire in order to be confident one has something against which to rebel, else one ends up sacrificing one's identity if one succeeds13:33
*** eglute has quit IRC13:33
*** cloudnull has quit IRC13:34
*** cloudkiller is now known as cloudnull13:34
*** d34dh0r53 has quit IRC13:35
mordredpersia: ++13:35
aspiersthey are forcing heavier team users to upgrade13:35
mordredpersia: also - I seem to remember one of the key members of the referred-to rebellion having the title of "princess" which certainly does indicate an aristrocracy13:35
mordredaspiers: whoopsie!13:37
aspiersmordred: to be fair, the UK royal family couldn't do nearly as bad a job of running the country than the current shower of "democratically elected" morons13:37
*** quiquell|rover is now known as quiquell|off13:37
mordredaspiers: ++13:37
persiamordred: Any good modern empire needs to immediately execute the aristocracy when started, else there is a chance that someone might remember history.13:37
mordredaspiers: I'm currently holding out mild hope for QE2 to just put her foot down somehow13:37
aspiersI'm not remotely a monarchist, but our queen is orders of magnitude more competent than 99.99% of our politicians13:37
mordredsame here13:38
*** bhavikdbavishi has quit IRC13:38
aspiersmordred: one of my colleagues had a nice idea how she could do that13:38
persiamordred: Amusingly, she explicitly gave up power to put her foot down the April before referendum.  She might un-give-it-up, but it's a two-step process.13:38
*** altlogbot_1 has quit IRC13:38
aspiersIf I remember correctly, his suggestion went something like this13:38
aspiersAll British passports are property of the Crown13:38
aspiersThe Queen is actually descended from Germans13:39
*** altlogbot_0 has joined #openstack-infra13:39
aspiersIf she claimed German citizenship, then all British passports would be the property of a German13:39
*** gfidente has joined #openstack-infra13:40
aspiersUnfortunately I think she lost her claim to German citizenship a long time ago13:40
aspiersbut it was a cute idea13:40
aspiersmordred: here's another one
aspiersAnother reason for projects to move to IMHO13:41
mordredor to stop trying to manage themselves like they are auto manufacturers13:41
aspiershaha :)13:42
*** eharney has joined #openstack-infra13:42
mordredperpetuating the false impression that we are like factory workers and/or cogs in a large machine that are interchangeable13:42
*** e0ne has quit IRC13:42
persiaTo be fair, some actual auto manufacturers have moved beyond that style of management./13:42
mordredrather than what we should think of ourselves like - an industry of creative types13:42
mordredpersia: ++13:42
aspiersYou mean kanban, or something wider than that?13:42
* aspiers recalls that Kanban came from factory-like environments in Japan13:43
aspierstaiga is much more than kanban, fortunately13:44
mordredyup. it's a system derived from toyota13:44
mordredas is LEAN13:44
mordredaspiers: yeah - unfortunately for me, I also don't subscribe to the usefulness (or agility) of agile13:45
aspiersI'm not religious about it either13:45
mordredso a tool described as "platform for agile developers" leaves me quite cold13:45
aspiersbut I do like having somewhere to track issues and collaborate effectively :)13:45
mordredoh - sure. that can be useful13:46
aspiersnot forgetting StoryBoard of course13:46
mordredbut as soon as I have to start using catch-phrases and jargon like "epic" or "sprint" I'm pretty much going to be done with whatever the thing is13:46
aspiersI can't stand the legacy email-heavy approach taken by Debian and the kernel / git communities13:47
persiaaspiers: Both specific practices used by a specific automaker that have been misused out of context for a while, and the general attitude (which was never held at the originating auto firm) that creative activity such as coding has any relation to repetitive process execution (for which things like actual kanban (which isn't implemented by any of the tools that are labeled "kanban") are incredibly useful).13:47
mordredpersia: ++13:47
persiaI've used the email workflows in the past to great success, but it depends on using specific email clients with specific plugins in specific ways, and is incompatible with how most people have used email for most of the 21st century, sadly.13:48
mordredaspiers: unfortunately, I have reached the grumpy place where none of the available tracking solutions provide more value to me than the time it takes for me to use them13:48
mordredI really wish this were not true - I would LOVE to keep track of things more better collaboratively13:49
aspiersThat's surprising to hear, I find even GitHub's dumb issue tracker pretty useful13:51
aspiersAlthough their projects UI is pretty horrible13:51
*** sthussey has joined #openstack-infra13:52
mordredaspiers: I find github's issue tracker very unpleasant - but I generally find github's UI to be categorically very hard to use13:52
persiaFor my current role, there cannot exist any issue tracker that helps me, and certainly not one that anyone else can see.13:52
aspiersmordred: what are the main problems for you?13:52
persiaFor the few projects where I'm responsible for collaboration, I find most participants have enough to do with private issue tracking systems that public ones are considered a dumping ground.13:53
aspierspersia: not sure I follow13:53
mordredaspiers: too much clicking. too hard for me to find a list of "these are thigns I should care about""13:54
persiaaspiers: So, someone wants something, and hires people to do it.  They then typically use some mechanism to tell those people what to do and how.  If the things to be done influence external parties, it's usually impolite to publish them, as nobody wants to be respoinsible for that sort of forward-looking statement when the situation changes tomorrow.13:55
aspiersmordred: you mean globally across repos?13:55
persiaThis means that people doing things end up writing down the "what" twice if thy have to put it someone external.13:55
mordredaspiers: yeah. the use case of "I only care about this one repo" is never a thing that I need13:55
aspiersmordred: is missing the filters or sorting you need?13:56
* persia gets a 404 from that URI13:57
aspiersI'm very far from a GitHub fanboy, but I find the search pretty powerful13:57
aspierspersia: not logged in?13:57
mordredaspiers: I don't think I've ever seen that13:58
aspiersmordred: it's linked from the top of every page ;-)13:59
* dirk is the github fanboy if you need anyone13:59
aspiersit's pretty easy to pick your favourite search and bookmark it13:59
*** slaweq has quit IRC14:00
aspiersdirk is lying, in case you couldn't guess X-D X-D14:00
mordredaspiers: it's possible that I don't actually use github unless forced to by some other projects use of it - so usually to file drive-by issues and then leave14:00
*** altlogbot_0 has quit IRC14:00
aspiersmordred: fair enough14:00
mordredso - you know - my opinions might not be the most accurate14:00
mordredbut I find most of my interactions with gh to be quite painful - similar to interactions with mac osx, which I find very difficult to use14:01
aspiersissues with gh are tracked here
*** slaweq has joined #openstack-infra14:02
mordredI am very unlikley to use that :)14:02
clarkbfungi: ok my day is starting but not yet at akeyboard so your help starting the snapshot is much appreviated14:03
mordredclarkb: are you using your telepathy interface to IRC again?14:03
clarkbre 2fa one of the struggles K have with t hat is the relianve on third party services to implemwnt it14:04
clarkbmordred softkeyboards are meh14:04
clarkbso if google shuts down the service or yubico has an outage we lose14:04
mordredclarkb: ++ to the 2fa thing14:04
clarkbthat said I do have a pair of yubikeys so could test their pam module if we want14:05
aspiersso, back to the proprietary vs. open thing ... has anyone looked at matomo for analytics?14:05
aspiers(formerly known as Piwik)14:05
aspiersI just did yesterday and was pretty blown away by how much they offer compared to Google and competitors14:05
mordredaspiers: I wanna say yes - I think investigation was done related to analytics for the zuul website? fungi would probably remember14:06
fungiaspiers: i can't remember if i looked at that one. i rules out any which required web bugs or cookies for tracking users, even if only local to the site14:07
fungier, ruled out. but i'll take a look at it14:07
aspiersfungi: erm, how come has google analytics then? ;-)14:07
fungiaspiers: i don't run that site14:08
clarkbaspiers: we dont run that14:08
clarkbpart of the motivation for opendev on our sideis to remove that confusion14:08
fungiand i disagree with the choices made by the folks who do, but in their case they have decided that market intelligence is more important than avoiding handing visitor tracking information over to third party services14:08
aspierswell that's what's so awesome about matomo - no tracking data gets handed over to 3rd parties14:09
aspiersit's GPL and you own the data14:09
aspiersassuming you self-host, of course14:09
fungii've also ruled out a couple which didn't have any convenient means to publish properly anonymized reports. they seemed to want to include far too much sensitive information. my goal with our (opendev) web analytics is to be able to publish fully open reports anyone can see14:09
aspiers(they have a SaaS offering too)14:09
fungibut yeah, looks like they rely on more than just web log analysis14:10
clarkbfungi is snapshot started?14:10
aspiersfungi: you think that's necessarily bad?14:10
fungiclarkb: getting there. sorry, too many conversations ;)14:10
aspiersI should stop distracting people now (including myself)14:10
fungiaspiers: yes, as it relies on cookies and/or web bugs injected into site content14:10
aspiersNot sure I see the issue, as long as appropriate permission is requested14:11
*** lpetrut has quit IRC14:11
fungiyes, one of the things i would love to avoid is adding a "prominent cookie notice" to every site we host so that we don't violate eu laws14:12
aspiersIIUC you don't need a cookie notice unless you track *personal* data (which includes things like IPs)14:12
fungicookies used in the implementation of features of a service i'm reasonably okay with, cookies used for tracking site visitors i'm not14:13
aspiersweb logs also track site visitors, but you're OK with that?14:13
aspierscurious where you are drawing the line14:14
fungiweb logs are an expected feature of webservers14:14
aspiersnot to non-technical folk ;-)14:14
fungithough if we start aggregating ... hold on i have actual work to do14:14
aspiersyeah, let's do this over a beer in Denver! it's an interesting topic ;-)14:14
aspiersI have actual work too :-/14:15
aspierssome weekend reading in case you want it
*** jamesmcarthur has quit IRC14:16
fungiclarkb: image in progress14:16
fungiaspiers: but if we start aggregating data and reporting on statistical trends, we can actually reduce our access log retention as well14:16
*** jamesmcarthur has joined #openstack-infra14:17
fungii'd very much like to effectively replace our access logs with public reports of anonymized visitor stats14:17
aspiersthat could be done with matomo pretty easily, I suspect14:18
aspiersif you self-host, you have the entire data set and can do whatever you want with it14:18
aspiersbut now I'm wondering which sites infra does actually manage14:19
aspierseven has google analytics14:19
fungiat least for sites where users allow their browser to set cookies. but maybe i'm basing concerns of complexity on my own browsing habits where i stick to text-based browsers where possible, block cookeis i haven't personally vetted, block loading third-party images, and so on14:19
clarkbwe run the webserver for docs but not the content14:19
clarkbI believe openstack docs add google analytics themselves14:20
fungiaspiers: in the case of we didn't traditionally manage it, but when we offered to give them a place to host we didn't tell them what content they were allowed to include14:20
aspiersmakes sense14:20
clarkband that goes back to annegentle iirc14:21
clarkbits possible current team doesnteven know it is there14:21
aspierswell, if we care about avoiding handing that data to google then probably the only way to get equivalent analytics is to replace the GA tag with a self-hosted matomo equivalent14:21
clarkbI tend to be massively ignorant because my firewall, dns server, and browser all filter this stuff oit for me14:22
fungiyes, back when she was managing it in rackspace's non-openstack mass web hosting service14:22
aspiersit also raises the question, how many GA accounts does OpenStack use, and who has access to them?14:22
fungii believe the osf web team handles that14:22
aspiersignoring the data concerns, ideally they should be managed by Google Tag Manager14:22
clarkb(and you should set up your firewall, dns server and browser to do the same :) )14:23
*** dpawlik has quit IRC14:24
fungiyeah, part of my assumption that direct web log analysis would be more effective is that i know the more popular "modern" methods wouldn't effectively capture my activity anyway14:25
*** altlogbot_3 has joined #openstack-infra14:26
fungiand also i'd rather not track and trend individual visitors because i find that creepy14:26
aspierssurely only creepy if you retain personal data?14:27
*** gfidente has quit IRC14:27
fungiand i don't want to hand over sensitive visitor information to anyone, not even my employer (osf) so want to limit this to aggregate statistics which are safe to publish for all to see14:27
aspierseven vanilla web logs contain IPs which is arguably more invasive than what GA claims to capture in a standard configuration14:27
aspierssorry, s/capture/retain, or at least expose to the user/14:28
fungiyep, which is why i'd like to truncate our access log retention where possible, but also all service logs (not just web server logs) are by their nature potentially contain some sensitive information. if they're used for troubleshooting and not for marketing intelligence i'm less concerned14:29
aspiersaggregating sessions by IP isn't much different to aggregating by session id persisted in a cookie14:29
*** altlogbot_3 has quit IRC14:29
fungii wouldn't want to store statistics on a per-ip-address granularity14:29
aspiersmodulo NAT, proxies etc.14:29
aspiersbut that's what standard web logs do14:29
*** altlogbot_2 has joined #openstack-infra14:30
fungiyep, which is why i wouldn't want to publish arbitrary server logs and would also like to reduce our web access log retention to protect us from, say, being compelled to hand over that information to authorities14:30
*** gfidente has joined #openstack-infra14:31
fungithe best way to avoid disclosing private information is to delete as much of it as possible or never even save it to begin with (the latter is hard if you need to be able to troubleshoot those services though)14:31
*** e0ne has joined #openstack-infra14:31
aspiersconsuming analytics based on logs or cookies, and then rotating aggressively plus only publishing anonymised reports is a far cry from publishing raw data14:31
aspiersI'm not sure anyone would advocate the latter14:32
aspiersand if you are doing the former, I don't really see a huge problem with cookies14:32
fungiso in answering the question as to where i draw the line, i'm okay with tracking access to aid in troubleshooting service problems, i'm not okay with doing so to provide market intelligence14:32
aspiersI entirely sympathise with that point of view; the devil is in the details though14:33
*** altlogbot_2 has quit IRC14:33
aspierse.g. define "market intelligence"14:33
aspiers"number of page views" and "number of unique visitors" are both "market intelligence" to me14:33
fungiyes, and those numbers don't need individual user ip addresses14:34
aspiershow would you track unique visitors without IPs or cookies?14:34
aspierseven with both it's still almost impossible14:34
aspiers(due to multiple devices per user)14:35
clarkbthis is why google forces you to login on chrome :/14:35
*** ykarel is now known as ykarel|afk14:35
clarkbor rather doesnt forget who you were on logout14:35
funginumber of unique visitors doesn't require long-term tracking of ip addresses beyond whatever granularity you're averaging across14:36
aspiersclarkb: chrome doesn't sync cookies14:36
fungiso number of unique visitors each hour only needs an hour's worth of logs14:36
clarkbaspiers: its not cookies aiui14:36
clarkbaspiers: the browser itself is keeping track14:36
*** gfidente has quit IRC14:36
clarkbthere was some big controversy about it a few weeks back14:37
fungiyes, with chrome the browser reports your browsing activity directly to google. that's why they're so keen to provide it to you for free14:37
aspiersthis one?
clarkbI thought there was a more recent one14:39
*** lseki has joined #openstack-infra14:39
*** gfidente has joined #openstack-infra14:39
aspiersmaybe there is - either way I missed it. gonna have to look more closely into this stuff14:39
aspiersbut right now I really need to work14:39
aspierslooking forward to some good beer discussions in Denver14:39
aspiersis it even worth scheduling a session on analytics?14:39
*** gfidente has quit IRC14:39
fungianyway, the only reason i'm looking into web statistic reporting *at all* is that osf has requested numbers to determine how "popular" different pages are over time. i really don't have any desire for that data myself and wouldn't even be looking into implementing something if it weren't for that14:40
aspiersI think that kind of data can be very useful14:40
aspiersto avoid misallocating effort14:40
*** gfidente has joined #openstack-infra14:40
fungii think the modern obsession with collecting data about what other people do or don't do is caustic14:40
aspiersI don't disagree. Would love to hear more in Denver14:41
clarkbaspiers: chrome 73 shuffled around settings previously under "privacy" to another locatio  including those that created problems in v69 as per the linked article14:41
clarkbso all related to that I guess14:41
aspiersclarkb: interesting, thanks14:41
aspiersI really should switch to Brave14:41
fungii don't personally appreciate the idea that others are collecting data about my activities, and where possible do my best to thwart it, so wouldn't want to be the one doing that to others14:42
clarkbI' rather thave a pihole install session or compare pfblockerng rulesets :)14:42
fungitwo sessions. one on how to collect data on people and one on how to protect your activities from being unduly tracked. they could be scheduled to run at the same time since the people who care about one generally don't care about the other14:43
*** Lucas_Gray has quit IRC14:44
*** gfidente has quit IRC14:46
*** altlogbot_3 has joined #openstack-infra14:48
dmsimardSo, just to confirm, can I go ahead and +W replication changes to the non-retired ara repos and proceed with the move ? I can volounteer to be the guinea pig for the github org transfer script :p14:48
dmsimardThe patch to modify the replication config is
*** zhurong has quit IRC14:49
fungidmsimard: did you confirm regular expressions work there? you're replacing a globbnig pattern with a regex14:50
clarkbdmsimard: hrm does gerrit's regex engine do negative lookaheads?14:50
dmsimardI've tested using the same regex pattern on review-dev14:50
fungigood enough for me14:50
dmsimardsomething like gtest-org/(?!gtest).*14:51
fungiand you confirmed it didn't stop other repos from replicating i take it?14:51
*** altlogbot_3 has quit IRC14:51
fungi(because if it doesn't support regexes, that simply could have disabled all replication)14:51
dmsimardoh, we've used that kind of patterns before in RDO's gerrit14:51
dmsimardI'm not worried about whether or not it works14:52
clarkbya docs say negative lookahead will work14:52
clarkbso should be fine to approve now14:52
fungicool, approved14:52
*** altlogbot_3 has joined #openstack-infra14:52
dmsimardfungi: the "tricky" part of the regex is making sure to match $14:53
fungiotherwise you match everything14:53
zbrdo we have any reason why tox jobs do often take longer to start than other heavier jobs?14:53
*** stephenfin is now known as finucannot14:53
fungizbr: "longer to start" in what way?14:53
*** efried is now known as fried_rice14:53
dmsimardfungi: I've matched sahara unintentionally when searching for ara things before :p14:53
fungizbr: as in time between node allocation and when the log starts to stream?14:54
zbrfungi: longer to start, means seeing "QUEUED" for longer than others.14:54
fungizbr: okay, so longer to get a node allocated14:54
openstackgerritFatih Degirmenci proposed openstack/diskimage-builder master: Constraint networkx to <2.3
clarkbnodepool node allocatio  is effectively random14:54
zbrso there is nothing we can do.14:55
clarkbeach job males a node request. If that job's request is handled by a slower cloud it will take longer14:55
*** armax has joined #openstack-infra14:55
fungialso how long it takes to boot the nodes for different images can play into that14:55
zbri was curious because most of time tox jobs are light, and likely that could run in containers.14:55
*** altlogbot_3 has quit IRC14:55
clarkbone suggestion I have made to the zuul team is that we handle the job side requests as a proper fifo amd only have the variance behind the scenes14:55
dmsimardone of the things that we can do is to tweak the nodepool min-server for labels if they are used more than others14:55
pabelangerdmsimard: min-ready doesn't really work any more, since node requests go to random clouds14:56
clarkbso that the nextup job that needa type foo gets ehatever the next inetance of type foo14:56
fungialso min-ready only ever helped if we weren't already running at full load14:56
pabelangerdmsimard: IMO, all should be set to 1, and just wait for cloud to come onlie14:56
fungiif we're maxing out our quota, every node allocation request has to wait in line for available quota14:57
zbrare we going to support running jobs in containers? this could lower the resource usage for simple ones.14:57
*** Goneri has joined #openstack-infra14:58
clarkbzbr: we technically already do, but it doesnt really lower resource consumpyion since the container hosts have to always be on14:58
mordredwe could also support having smaller nodes14:58
clarkbat least vexxhost has indicated that the VM approach is better for them because we shut the VMs off14:58
*** altlogbot_2 has joined #openstack-infra14:59
clarkbwhereas the k8s cluster is always on14:59
mordredlike - define soe labels that are for 1G or 2G nodes ... that can be helpful on some of our providers14:59
*** jcoufal has quit IRC14:59
mordredbut on other providers the limiting factor is ip addresses, so it still won't help much14:59
pabelangerwe also talked about smaller vms, eg: 1gb for tox say, in an effort to squeeze more VMs running at once, however the limit then becomes IPv4 addresses14:59
pabelangerbut native ipv6 shouldn't be an issue15:00
*** ijw has joined #openstack-infra15:00
zbris true that a k8s cluster is always on, so it has a permanent load, mainly we can think of it as another cloud-provider :) but at scale it does payoff.15:00
*** ykarel|afk is now known as ykarel15:00
fungipart of the challenge with doing something like a nodepool kubernetes in each provider is that's yet another persistent piece of infrastructure we have to run there, and build when we bring a new provider on line (even just building more mirror hosts is a not-insignificant amount of effort for us)15:00
mordredalthough it's worth pointing out that tox is _not_ lightweight in some of our larger projects15:00
zbrfor runing 1-2 jobs as containers is not a good idea, but if you run lots of them, you easily see the benefits.15:00
mordredzbr: our cloud providers have indicated the opposite - it would be more costly for them if we switched aggressively to containers15:01
pabelangerit helps more when you have to pay for cloud resources, 8 x 1vcpu == 1 x 8vpcu at the end of the day :) that's what we have setup in currently15:01
mordredbut seriously - tox -epy35 on nova is NOT a small or quick job15:02
*** gfidente has joined #openstack-infra15:02
zbrmordred: i don't think we can talk about massive switch. but I do see it as a mix.15:02
aspiers-efunctional is even slower15:02
clarkband it wants 8cpus :)15:02
* clarkb somewhat halpy aboit that as pre testr nova unittests took even longer15:02
fungizbr: just to confirm, you're observing lengthy node allocation times when we're not running at capacity? because if we're running at capacity then container or vm doesn't matter much. at best it shaves something like 1-2% utilization due to not having boot and delete overhead (you have to consider the average runtimes across all our jobs, which is far higher than those of most tox jobs) so15:02
mordredtox -epy36 on nova takes 16 minutes to run15:02
fungieffectively just means a 1-2% boost in available quota15:02
clarkband I cant type15:02
pabelangerHmm, is not loading properly for anybody else?15:04
*** igordc has joined #openstack-infra15:04
pabelangerthe datasource I mean15:04
fungizbr: if "use containers" is slang for running some jobs on smaller amounts of server resources (not dedicating 8gb ram, 80gb disk and 4-8vcpus to every build), note that's not really a property unique to container solutions, we could do that with virtual machines too as indicated15:04
*** ijw has quit IRC15:04
zbrok, so shortly benefits would likely so small that it would not payoff for the complexity of having to deal with both vm and containers.15:04
mordredzbr: there's definitely some interesting possibilities with containers - but from the investigation we've done so far they do not actually end up being an immediate/quick/easy win in our case, due to overall load, the source of our compute resources, and ...15:04
mordredzbr: yeah - what you said15:05
zbrprobably this is what made travis stop using containers in december15:05
fungii wouldn't be surprised15:06
mordredyeah. containers at scale aren't necessarily the win people think they are15:06
zbrorchestrating them is not an easy job15:06
mordredthey're great for many things, of course15:06
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Update datasource to https for graphite.o.o
mordredand I've been enjoying our adoption of them15:06
clarkbpabelanger: possible fallout from httpsing graphite?15:06
pabelangerclarkb: fungi: mordred: ianw: ^I think that will fix the datasource issue15:06
pabelangerdid we drop http support?15:06 redirects http to https15:07
fungiso maybe the redirect isn't being followed?15:07
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Update datasource to https for graphite.o.o
pabelangerOh, it is now15:07
pabelangerlet me use that15:08
clarkbya since we  ant sanely LE openstack domains15:08
zbrpabelanger: yep, wrong certificate. -1 to that.15:08
fungii bet we missed redirecting to graphite.opendev.org15:08
clarkbfungi: we didnt15:08
clarkbbut it only works when you usehttp15:08
clarkbbecausethe cert doesnt have an altname for openstack.org15:08
fungibut it was using http15:08
clarkbya I think it must not follow redirects15:09
*** pgaxatte has quit IRC15:09
fungiis the http to https redirecthappening before the openstack to opendev redirect?15:09
clarkband expects 20X15:09
pabelangerhttp redirects to https://graphite01.o.o15:09
fungifor me it's redirecting from openstack to opendev and then http to https15:09
clarkbfungi: I dont think so as the redirect works in my browser15:09
fungialso, as noted, it's apparently redirecting to fqdn not the cname, so that needs fixing i think15:10
fungianyway, i need to disappear for an early lunch so i'm not mia during the lists.o.o maintenance15:10
fungiback soon15:10
*** tosky has quit IRC15:11
clarkb worls as expected15:12
clarkbso updating the grafana source to that is correct15:12
clarkbthen we can fix the graphite01.opendev redirect target separately15:12
fungilists.o.o image creation is still in progress, btw (not surprising)15:12
*** tosky has joined #openstack-infra15:12
*** eharney has quit IRC15:13
clarkbpabelanger: were you going to push a new ps for
*** ginopc has quit IRC15:15
pabelangerclarkb: yah, doing now15:16
pabelangerclarkb: actually15:16
pabelangerI think we might need to do a migration on grafana side15:16
pabelangerlet me create a 2nd datasource15:16
pabelangerthen grep / replace openstack15:16
pabelangerno, I am wrong15:17
pabelangerI think we can update it in 1 patch15:17
clarkbthe keys didnt change15:17
clarkbjust the target webserver15:17
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Switch grafana datasource to
*** gfidente has quit IRC15:19
*** ccamacho has quit IRC15:20
*** electrofelix has quit IRC15:23
*** e0ne has quit IRC15:24
*** jamesmcarthur has quit IRC15:27
*** tbachman has quit IRC15:27
dmsimardhmmm, the namespace move is bound to break more than one implementation of "src/<project>" in different zuul jobs15:30
mnaseris there a way to 'test' the redirects in advance?15:31
clarkbbyt not just the namespace move. will not bevalid there anymore either15:31
dmsimardsymlinks ?15:31
clarkbmnaser: yes see the emails :) fungi set up the redirect rules on a webserver allowing anyone to test them15:32
dmsimardclarkb: I guess it's going to be src/<namespace>/<project>15:32
mnaserI worry that potentially openstack-ansible releases might be broken with the move .. so maybe accounting for that before we change things to test them out15:32
*** roman_g has quit IRC15:32
clarkbdmsimard: correct15:32
dmsimardclarkb: the http redirects are fine, the problem is about jobs in zuul running with the assumption that things are in src/git.o.o15:32
clarkbdmsimard: I know, was responding to mnaser15:33
clarkbdmsimard: and it will be broken in zuul even if we dont change namespaces15:33
clarkbSo now is best opportunity to change namespaces15:33
*** HuaChangWang has joined #openstack-infra15:33
dmsimardyeah I realize there is going to be fallout for sure15:33
dmsimardis there a plan for this particular bit though ?15:33
*** roman_g has joined #openstack-infra15:34
clarkbdmsimard: the plan is to push changes with the new names aiui15:34
mnaserso as I understand it, 'git' access will stop working at some point?15:34
clarkbmnaser: git:// stops when we switch15:34
mnaserwell that breaks *every* single release of OSA ever15:35
mnaserI just thought of that15:35
dmsimardmnaser: not using http for git operations ?15:35
mnaserif we point to via DNS.. wouldn't git:// just magically start working again transparently?15:35
clarkbianw has pushed changes up to replace git:// with https://15:35
clarkbmnaser: no becayse gitea hasno support for git protocol and it cannot be vhosted15:35
mnaserlooks like we were using http://15:36
mnasersorry for the noise15:36
*** chandankumar is now known as raukadah15:36
* mnaser is reliefed15:36
cloudnull - mnaser i started looking at that the other day15:36
*** slaweq has quit IRC15:37
*** slaweq_ has joined #openstack-infra15:37
mnasercloudnull: ah okay.15:37
mnaserit would be nice if infra can give us a date where we can consider 'live' even if the cutover hasn't been changed15:38
mnaserbut that might already be in place15:38
*** jcoufal has joined #openstack-infra15:38
clarkbmnaser: I think that us the cutover day?15:38
mnaserclarkb: well cutover day = our stuff breaks, having a bit of time in advance to transition is helpful15:38
clarkbwhy does it break?15:38
mnaseresp in the OSA case where there are so many references + we have like another 30 repos to update15:39
clarkbwe are redirecting http(s) so old urls will continue to work15:39
clarkbyou should be able to safely update docs after the transition15:39
mordredyeah. the only thing that should break day-of is git:// - but that's safe to change now15:40
mnaseralright ill try to check our usage of that (If any)15:41
clarkbon the zuul job sideit might be bumpy but that shouldnt be user noticeable and is dev specific15:41
clarkbbecause zuul uses url based file structure15:42
*** slaweq__ has joined #openstack-infra15:44
clarkbmnaser: can you review ?15:46
*** slaweq_ has quit IRC15:46
mnaserclarkb: +w15:47
clarkbmnaser: but ya long story short we expect the public consumption of this to mostly just work via our redirects and that can be tested today15:47
mnaserclarkb: yeah, that seems like an ok approach15:47
mnaserI've had to deal worth.. like post-github acquisition15:48
clarkbthere are some notable expceptions. git:// will stop working and you have to use http(s). And in zuul jobs we refer to projects by their canonical name which will be changing15:48
clarkbso CI may have a few bumps while we update those names15:48
mnaserwhen a bunch of folks moved their repos off of github15:48
mnaserstuff like spice-html5 decided it didn't want to live in GitHub and rm'd it there15:48
clarkbI intend on being around the entire week after our transition to help people debug (and possibly even force merge fixes for) the zuul canonical name fallout15:49
clarkbthough aiui our plan is to generate many of those changes automatically15:49
clarkbso more a matter of merging stuff than debugging and figuring out fixes I hope15:49
*** yamamoto has quit IRC15:50
dmsimardIt looks like puppet is considering the EOL deprecation of py27 warning as a failure?
*** yamamoto has joined #openstack-infra15:50
dmsimardunrelated failure when trying to land the gerrit replication patch for ara15:50
*** slaweq__ has quit IRC15:51
clarkbdmsimard: the error is one line below that15:52
dmsimardI was suspecting that might be the case15:52
clarkbI'm guessing that 'lockfile' just made a release?15:52
clarkbso now the old version we had installed wants to updated?15:52
clarkbpypi says no that isn't hte case15:53
dmsimardlockfile was last updated in 2015 :D15:53
clarkbnot sure why that seems to have just happened now. But it is the lockfile issue not the deprecation warning that caused the failure15:53
*** yamamoto has quit IRC15:53
dmsimardI can try a recheck, this failure is from a recheck from an unrelated gate failure :p15:53
*** yamamoto has joined #openstack-infra15:54
*** derekh has quit IRC15:58
*** jamesmcarthur has joined #openstack-infra15:58
clarkbya a recheck showing if it is consistent or not would probably be worthwhile15:58
pabelangerclarkb: +3 on
* clarkb finds breakfast before the list server upgrade16:00
pabelangerlooking at system-config one now16:00
clarkbpabelanger: for the system-config one we may want to give it extra thought considering matrix current struggles16:01
clarkbbut 651389 should be totally safe :)16:01
*** psachin has joined #openstack-infra16:01
pabelangerclarkb: yah, we should first limit zuul_console on firewall to zuul-executors, not to leak that port16:02
*** jamesmcarthur has quit IRC16:05
openstackgerritMerged openstack-infra/project-config master: Switch grafana datasource to
*** janki has quit IRC16:05
*** finucannot is now known as stephenfin16:08
openstackgerritFabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver -
openstackgerritMerged openstack-infra/project-config master: Remove zuul_reconfigure CD job
*** lucasagomes has quit IRC16:08
pabelangerclarkb: I've been thinking for ansible side, maybe adding extra layer of email when somebody logins in via SSH, like we do with sudo failures.  However, that would likely get chatty with ansible sshing into servers ever hour16:10
*** iurygregory has quit IRC16:10
pabelangermaybe better to expose to statsd16:10
pabelangerand grafana dashboard16:10
*** rlandy|ruck is now known as rlandy|ruck|mtg16:12
*** dtantsur is now known as dtantsur|afk16:12
*** HuaChangWang has quit IRC16:13
*** eharney has joined #openstack-infra16:15
*** gyee has joined #openstack-infra16:15
clarkbI'm putting lists.o.o in the ansible emergency file list now16:15
*** ykarel is now known as ykarel|away16:16
fungiokay, i'm back. lists.o.o snapshot is still in progress16:16
*** jpich has quit IRC16:18
clarkbbah I planned list upgrade during kids soccer practice so I would have peace and quiet, well soccer just cancelled on us :)16:18
clarkbfwiw after the lists upgrade I think I'm gonna go haev a stein in honor of stein :) I put that off ealier in the weke because I knew we had stuff to do after release16:20
fungiand yeah, i'm currently working on the script to "fix" hostnames and namespaces in git review and zuul configuration in every repository we host, which we'll be pushing directly into those repositories on gerrit's filesystem while it's offline. not only are those changes bypassing zuul, they're bypassing gerrit too (though i'll be testing the script at scale and we'll all be very closely reviewing16:20
fungithe script and results over the course of next week)16:20
dmsimardclarkb: dadops hug :p16:20
anteayaclarkb: sorry I got working on other things and didn't get back to irc yesterday16:21
*** rh-jelabarre has quit IRC16:21
anteayaclarkb: thanks for posting that update16:21
clarkbanteaya: no worries16:21
anteayaclarkb: I will say that I was looking for something with the subject line "gerrit will be offline April 19th'16:22
anteayafor the folks that don't pay attention to anything less than that16:22
anteayamind if I post that kind of email and point to your update?16:22
anteayaI can draft an email for your approval16:22
clarkbanteaya: ah. I was just going to suggest a response with that clarifcation16:23
clarkbfeel free to do so16:23
anteayaI will start an etherpad16:23
*** rpittau is now known as rpittau|afk16:24
*** ykarel|away has quit IRC16:24
*** jamesmcarthur has joined #openstack-infra16:28
anteayaand anyone else who wants to offer their view16:29
*** jpena has quit IRC16:29
anteayaI don't know that we had set a start time for the outage16:29
clarkbthinking about specific outage windows. I expect we'll narrow that down during our meeting next week16:29
anteayaso don't refer to a start time for now?16:30
anteayaI will adjust that bit16:30
clarkbmaybe for now just say the Gerrit service will be offline at times on the 19th. Specific outage window to be posted as we get closer to the event.16:30
clarkbsomething like that16:30
anteayaI can do that16:30
fungiyeah, it's entirely possible we have multiple outages throughout the course of the 19th16:31
fungiespecially if we need to roll something back or spot something else which needs gerrit offline to fix16:31
clarkbwe strategically selected a day that is a holiday in many parts of the world16:31
fungithis is most definitely the largest and most sweeping change we have made to our git repositories in the history of openstack, so i'm not assuming it will be quick nor seamless16:32
clarkbfungi: I think the snapshot is completed, if you want to confirm16:32
anteayahow's that?16:32
fungiclarkb: indeed, it seems to have just changed in the last minute16:32
fungiso i think we're clear on that front16:32
anteayaI'm trying to be both specific and vague16:33
*** rlandy|ruck|mtg is now known as rlandy|ruck16:33
fungithat seems great to me, thanks anteaya!16:33
clarkbanteaya: I made two minor edits but other than that lgtm16:33
anteayathank you16:33
anteayawill the comma after the link break the link?16:33
fungii have a feeling we'll do a status alert too and tell people to find us in here16:33
anteayaI agree the comma is correct grammer16:34
fungianteaya: if you want to be sure it doesn't, add an ugly space just before the , or wrap the url in <>16:34
anteayaI just never put a character after a link, just leave a whitespace16:34
anteayaI'll whitespace16:34
anteayaclarkb: that okay with you?16:35
anteayayeah, I figured you'd be statusing as things proceed16:35
*** rh-jelabarre has joined #openstack-infra16:35
anteayaanyone else want to take a peek at the draft email?
clarkbfungi: for lists I guess my plan then will be to stop services at 1700UTC then start running through the upgrade steps in a root screen16:35
fungiclarkb: sounds great. i'll attach once you indicate the session is initiated16:36
fungior maybe i'll start the screen session now and you can attach16:36
anteayaI'll post to the email list16:36
clarkbfungi: that way your terminal is correctly sized16:37
fungiand then you get to be subjected to my teensy terminal geometry ;)16:37
clarkbthat ++ was for both things16:37
clarkbthe email and the terminal size16:37
anteayaa + for each of us16:37
fungiclarkb: screen session running under root on lists.o.o now, it's the only session according to screen -list16:37
clarkbfungi: ok16:38
*** tosky has quit IRC16:38
anteayaoh subject line is good?16:38
anteayaI purposely did not use any tags on it16:39
anteayais that the best way to reach all the people?16:39
clarkbanteaya: [all] is the suggested global tag now I think16:39
anteayaI can toss one of those in front16:39
anteayaI'll wait for a few minutes and then press send16:40
clarkbbefore I forget16:41
*** bgmccollum has quit IRC16:41
clarkb#success Infra team has deployed its first letsencrypt cert to production16:42
openstackstatusclarkb: Added success to Success page (
mordredclarkb: \o/16:42
fungii'll try to get the caa and hsts changes for grafana up today, lest i forget16:42
*** jamesmcarthur has quit IRC16:44
clarkbI'm pretty happy with the setup we ended up with. We largely avoid spofs and can create new instances without worrying about dns A/AAAA records beforehand and its all automated. Thank you ianw for putting that together16:44
clarkbstill some minor details to sort out like caa records and restarting apache when certs update but I expect we'll get all of that sorted shortly16:45
anteayathank you, sending16:47
* mordred hands ianw this pie he found16:47
*** Lucas_Gray has joined #openstack-infra16:48
openstackgerritMerged openstack-infra/system-config master: Disable gerrit replication to GitHub for ara/ara-infra/ara-web
*** bgmccollum has joined #openstack-infra16:51
*** jamesmcarthur has joined #openstack-infra16:51
*** cjloader has joined #openstack-infra16:53
openstackgerritJeremy Freudberg proposed openstack-infra/project-config master: Minor/useful tweaks to Sahara repo descriptions
*** tbachman has joined #openstack-infra16:57
clarkb#status log Pre snapshot completed and is named
*** ykarel has joined #openstack-infra16:57
openstackstatusclarkb: finished logging16:57
clarkbfungi: ready? I'll stop mailman and exim services if so17:00
fungiyep, let's get this party started17:00
*** ramishra has quit IRC17:00
clarkbok proceeding17:00
fungithe way i survive on a tiny terminal is by filtering out puppet deprecation notices ;)17:01
*** jcoufal has quit IRC17:02
*** e0ne has joined #openstack-infra17:03
*** e0ne has quit IRC17:03
fungithe service command needs a --wait option17:04
fungisad that do-release-upgrade clears the terminal17:04
*** jamesmcarthur has quit IRC17:05
fungithen again, the existence of that command is very ubuntu to begin with17:05
*** bgmccollum has quit IRC17:05
clarkbfungi: I've added the iptables rule and reconnected via 1022 in another shell if you want to do that too17:05
fungiwe're already at line 66 of by the way17:07
* mordred cheers fungi and clarkb on17:08
fungiwow. we could stand to clean up some kernels on there at some point17:08
clarkbfungi: ya running apt-get autoremove in a cron might be a good idea17:09
fungii think it's an option for unattended-upgrades17:09
clarkboh even better17:09
fungii don't think we need the old sysstat/sar files, no17:09
*** bobh has joined #openstack-infra17:10
clarkbI've done this a few times at this point and those go away17:10
fungii take it including puppet would cause it to get started (or fail to start and then abort the upgrade)17:13
openstackgerritTobias Henkel proposed openstack-infra/zuul master: WIP: Add support for smart reconfigurations
clarkbfungi: that was my concern17:13
clarkbfungi: I actually never tested it when it was listed17:14
fungisince it's not running it doesn't need a restart, so either way that's fine17:14
fungiwe're waiting between lines 73 and 74 now i thin17:16
clarkbI removed it out of caution and doing so made the upgrade work17:16
clarkboh I notice I had a "we should check exim doesn't need an update" comment in there and I did check :)17:16
clarkbthe file updates were to macros not our actual configs17:16
clarkband on my test server thinsg worked without a puppet run17:16
fungiokay, we're being prompted to replace configs now17:21
fungiand more configs17:23
*** e0ne has joined #openstack-infra17:23
fungii expect the upcoming question about removing obsolete packages will clear out all those ancient kernels17:23
clarkbfungi: fwiw the process didn't differ much between lstis and afs so will likely apply to wiki as well17:24
clarkbif you end up planning to inplace wiki17:24
fungiat line 79 already17:24
fungiand yeah, i expect so but need to get puppet-mediawiki working with it first17:24
fungiat least for wiki-dev17:24
*** adriancz has quit IRC17:28
clarkbremoving obsolete packages now17:28
fungiyep, almost to line 9117:28
fungier, 8117:28
*** e0ne has quit IRC17:29
*** jamesmcarthur has joined #openstack-infra17:29
fungitime to reboot!17:30
clarkbready? I'm going for it17:30
clarkbfungi: when it comes back do you want to start another screen for me to attach to to do the packaging cleanups?17:31
fungiwill do17:31
clarkbI'm in17:32
fungiscreen ready, attach at will17:32
fungilooks like exim and mailman started17:33
*** _erlon_ has joined #openstack-infra17:34
fungiremoval set lgtm17:36
clarkbk was just going to ask you to confirm17:37
clarkbok that concludes the documented upgrade process17:39
clarkbfungi: we should probably reboot again to make sure the new kernels all come up as expected (we had trouble with that on some afs servers)17:40
clarkbbut after that I think we reenable ansible/puppet and monitor the service17:40
fungii would definitely reboot again17:40
clarkbfungi: ^ that sound good? if so I will issue a reboot command17:40
clarkbok rebooting now17:40
fungii'll start a new screen session as soon as it's back up17:40
clarkbits up17:41
fungiand screen is ready17:42
clarkbexim and mailman are running I checked that before attaching to screen. kernel looks correct17:42
clarkbready to enable puppet?17:42
fungiyeah, go for it17:42
clarkbok removed from the emergency file17:43
clarkbsomeone should send an email to a list :)17:43
clarkbI think puppet is about 30 minutes away17:43
fungiodds are a number of people have sent e-mail to some lists and they're waiting for the server to be up17:44
clarkbdo we want to wait or run
clarkbI suppose we should just and keep moving17:44
fungii'm fine waiting/testing in the interim too. either way17:44
clarkb/opt/system-config/tools/ look right to you?17:45
clarkbI'm going to run that now17:45
fungiyeah, that looks right17:45
fungii'll see if there are any messages to approve in the moderation queues for a couple of lists17:46
clarkbI think it updated snmpd.conf so that should be a keep on future in place upgrades it also updated the sources.list17:47
clarkbotherwise looking good so far17:47
fungii have one legit message waiting in the moderator queue for openstack-discuss17:49
clarkbApr 12 17:48:56 lists puppet-user[14704]: (/Stage[main]/Mailman/File[/etc/mailman/]/mode) mode changed '0644' to '0444'17:49
fungigoing to catch up my e-mail for that list and then approve it17:49
clarkbthat is the only change from running puppet17:49
clarkbfungi: I think your grep wasn't working for some reason. I grepped just puppet-user from syslog and saw puppet ran17:49
clarkbI think we are good on the puppet side17:49
fungiyeah, that's strange17:49
clarkbfungi: well too late now with the -f17:50
clarkbin any case I think its happy17:50
clarkbnow to double check functionality17:50
fungistrange that `|grep -v deprec` filtered out other lines too17:51
clarkbI can browse the archives of openstack-discuss17:51
clarkbI'll send an email to the infra list17:51
*** jamesmcarthur has quit IRC17:51
fungiwaiting for the message i just approved out of moderation to get delivered17:52
*** Lucas_Gray has quit IRC17:52
clarkbmy test email hasn't shown up yet either :/17:54
clarkbI sent it via my mua and not nc though17:54
clarkbso maybe my smtp servers noticed it was offlien earlier and are queuing?17:54
fungithe one i approved just got delivered to me17:54
clarkbfungi: ok I see that one17:55
clarkbso maybe fastmail did notice we were offline and is queuing17:55
clarkbI shall strive to practice more patience17:55
fungiyeah, if someone else from there tried to send a message to one of our lists while it was offline, their mta may be holding all deliveries for the same subdomain to retry together17:55
*** diablo_rojo has joined #openstack-infra17:56
clarkbI think your approval email getting through confirms mailman is generally working as is exim outbound17:56
clarkbwe just need to confirm exim inbound ?17:56
anteayafungi: the email you approved, what list was it for?17:58
anteayawas it a post to a placement ptg thread?17:58
clarkbI think it was the telemetry core one17:59
anteayaa nominating thread?17:59
clarkb is my test email17:59
clarkbwhihc I've not yet received back to myself yet18:00
clarkbanteaya: ya18:00
clarkb"Asking core privileges on Telemetry projects"18:00
anteayaI have this subject line [telemetry] Nominating Rong Zhu for Telemetry core18:01
anteayaI don't have 'Asking core privvileges on Telemetry projects'18:01
anteayanor to I have clarkb's test post18:01
clarkbI see files in the mailman queues for openstack vhost18:04
clarkbso it is doing some processing of things at least18:04
anteayathis is the most recent email I have:
*** dave-mccowan has joined #openstack-infra18:05
clarkbthe errors file for that vhost doesn't show any errors we didn't already have from prior to the upgrade18:06
anteayathe server recieved but thus far I have not18:07
fungithe one i approved was Subject: Re: [tc][telemetry] Asking core privileges on Telemetry projects18:07
fungiFrom: Trinh Nguyen18:07
fungiDate: Fri, 12 Apr 2019 22:28:50 +090018:07
clarkbfungi: any ideas on why the email I sent hasn't gone out the out queue?18:07
*** imacdonn_ is now known as imacdonn18:08
fungimost common cause is if the mailman qrunner for it hasn't started18:08
fungibut it looks like it's been running since reboot18:08
anteayaI have this one in my email client:
*** irclogbot_0 has quit IRC18:08
clarkbwe have 5 outgoing qrunners which is the number I expect18:09
clarkb/srv/mailman/openstack/qfiles/out/1555091557.751896+3f57e87a69100bb36b456e8836d21b1f7e17e526.pck is the email I sent to test18:10
clarkbso it is sitting in the out queue18:10
*** raissa has joined #openstack-infra18:10
*** irclogbot_0 has joined #openstack-infra18:10
fungii sent one to openstack-discuss a few minutes ago too18:10
clarkbsmtp-failures doesn't show any failures we didn't have already (someone is subscribed with a no longer valid email addr looks like(18:11
clarkbstrace implies it is trying to get the mailman lock file18:12
clarkbfungi: `strace -fp 1345` if you want to see that18:12
fungii wonder if we have a stale lockfile18:12
fungiwe could stop all the runners and then see if any lockfiles are left behind18:13
anteayathe server recieved fungi's email but not clarkb's and I have not recieved fungi's yet:
clarkbanteaya: the server got mine too18:13
clarkbfungi: ya I'll stop mailman-openstack now18:14
anteayaI don't see it in
fungihe sent his to the infra ml18:14
anteayaah sorry18:14
anteayayes, there it is:
clarkbnow patiently waiting for the qrunner to actually stop18:15
anteayayou had shared the link with me earlier and I congitaively disassociated18:15
clarkbit is waiting on the outgoing qrunner to stop which is the one that wasn't able to get the locks (I think that is what it was doing)18:16
clarkbwith the other processes stopped maybe that does point to a stale lock18:16
clarkbfungi: is the lockfile suffix the pid?18:18
clarkbbeacuse 1537 doesn't exist as a pid18:18
clarkbbut appears to be the current lock18:18
fungiyes, we might find it in old logs18:18
*** tosky has joined #openstack-infra18:18
*** bgmccollum has joined #openstack-infra18:18
clarkbApr 12 17:32:33 2019 (1537) IncomingRunner qrunner started.18:19
fungithere you go18:19
fungiso that was from an earlier reboot18:19
clarkbya I think so18:19
fungithe one right at the end of do-release-upgrade18:19
clarkbso now how do we safely remove that lock? do we clear it from the mailman.lock pointer file or rm it on disk?18:19
fungithe time matches18:19
fungiremove the file from disk should be fine18:20
clarkbok I'm doing that now18:20
clarkbI don't see a change18:21
clarkbI think we might have to edit mailman.lock?18:21
fungiworth a shot18:22
clarkbor do I remove mailman.lock18:22
clarkbzuul doesn't have a mailman.lock18:22
anteayawhat about rename mailman.lock?18:22
clarkbya can move it aside18:22
clarkbfungi: ^ you good with that?18:24
fungiyep, it should be fine to remove the mailman.lock file when the qrunner isn't running18:24
clarkbwell it is running18:24
clarkboh but not the one that had the lock18:24
clarkbgot it18:24
clarkbok it stopped the service after that. I am starting it again18:25
fungiit was likely in a busywait for the file to disappear18:25
clarkbout/ contents disappearing18:25
clarkbwhich implies we should be getting emails18:25
clarkbyup I got my email18:25
clarkbok I'll sanity check the other 4 vhosts don't have a similar situation18:26
anteayaclarkb: got yours18:26
*** roman_g has quit IRC18:26
clarkbthe other vhosts do not appear to have the same problem18:27
clarkbso I think that may have been it and email seems to be flowing from input through mailman out exim again18:27
clarkbshould I respond to my maintenance notice emails now with a note that we are done and expect things to be working?18:27
clarkbthat will test all of the vhosts18:27
anteayasounds reasonable18:28
anteayaI still don't have cdent's email nor fungi's from discuss18:28
clarkbI got them18:28
clarkbanteaya: so may just be a matter of exim getting around to sending them to you18:28
anteayahowever my client seems to be occupied with qq spam18:28
anteayaso likely just me18:28
fungithey can take time to show up. openstack-discuss is around 1k subscribers now18:28
clarkbfungi: ^ anythign else you think we should check before I send my "we're done and think it is working" emails?18:29
anteayayeah, so I'm likely to get them soon18:29
funginope, i think we're good. everything's exercised now18:29
clarkbalright working on that now18:29
fungithanks for driving!18:29
anteayanice work18:29
anteayaand there is fungi's and cdent's email18:31
anteayathank you18:31
clarkbit'll have a bit more backlog than usual due to things queuing up while the runner was sad18:32
clarkbbut should chew through it and be quicker soon enough18:32
*** armax_ has joined #openstack-infra18:33
*** armax has quit IRC18:33
*** armax_ is now known as armax18:33
anteayaclarkb: sent a reply to your -infra list email18:34
anteayawhich hasn't been picked up yet:
clarkbanteaya: ya the incoming runner is running very quickly right now18:36
clarkbbut it has the lock and strace shows it reading and writing so I think it is working just trying to get through its backlog18:36
clarkbwe should keep monitoring it to see that it does reach a normal steady state18:36
anteayavery good18:36
clarkbfungi: ^ fyi18:37
clarkbok in/ count is falling now18:38
clarkbwent from 360 to 300 so it seems to be processing18:38
anteayagoing in the correct direction, anyhow18:38
clarkbI'm sure a good chunk of that is spam18:39
clarkbwhich is annoying18:39
anteayagood old qq18:39
clarkbnow down to 26818:39
clarkband 22418:39
anteayaI which I could just block everything coming from qq18:39
clarkbso ya lets watch it but cautiously saying it still looks fine18:40
anteayaI wonder what email options folks have in china18:41
anteayaI'm curious how it would go over if we considered blocking everything coming from qq to our lists18:41
clarkbI think we haven't done it yet because it is a popular service with peopel and not just spammers18:42
anteayaI wonder if the people have other options?18:42
clarkbfungi: ^ or do we auto reject those now?18:42
anteayalists with better filtering18:42
fungii looked into it a while back and counted some number of legitimate posts to our lists from users with addresses18:42
anteayasorry email services with better filtering18:42
anteayayeah there are some18:42
clarkbconsidering that things look ok other than the backlog I think I'm going to step out for a bit and then check on the backlog after a break18:43
anteayaI'm just wondering what the email landscape is for those humans18:43
fungialso, yes, the spam isn't actually coming from, it's spoofed to use addresses so that people are *less likely* to block it18:43
anteayaclarkb: enjoy some fresh air18:43
clarkbfungi: I'm just looking at ls -l | wc -l output in the in/ and out/ qfile dirs18:43
*** kjackal has quit IRC18:43
anteayafungi: oh, I didn't know that18:43
*** kjackal has joined #openstack-infra18:43
clarkbif people are wondering how I was trending that. And now taking a short break18:44
fungiqq is a very popular messaging service in mainland china18:44
clarkbthanks for the help everyone18:44
anteayaoh it is messaging not just email18:44
clarkbI learned a lot about mailman doing this too18:44
anteayaclarkb: thanks for shouldering the load18:44
anteayanow the expertise is growing :)18:44
fungiyeah, qq is an integrated messaging platform which spans a number of protocols18:45
anteayaah, I didn't know that18:45
fungiand acts as a freemail service provider18:45
fungikinda like yahoo! or gmail18:45
anteayaall I know of it is that the addresses filling my spam folder are majority qq18:45
anteayaI wish they did a better job on spam filtering on their end18:46
fungiwell, as i said, it's not actually coming from there18:46
fungiit's spoofed to look like it's coming from there18:46
*** psachin has quit IRC18:46
anteayado we have any way of viewing the spoof and blocking it on our lists?18:47
anteayaand my reply to clarkb18:48
anteaya's email has shown up:
fungichecking against spf records would be one way, though we'd need to implement a more full-featured spam filtering mechansim on the server (i have one in mind, but haven't had time to write a spec)18:49
clarkbthis is why dkim/dmarc is so popular18:49
clarkbwell one reason18:49
*** jamesmcarthur has joined #openstack-infra18:50
anteayaoh okay so possible and includs substantial work18:50
clarkbdmarc/dkim create other problems for mailing lists18:50
fungisubstantial enough i may or may not get around to it, right18:50
anteayaso I have seen, yes18:50
anteayaokay fair enough, I certainly can't take anything off your plate, so won't feel right trying to add to it18:51
anteayathe qq spam does get redirected to my spam folder, there is just so much of it18:52
fungii mean, i have a similar setup on my mailservers, using exim filters to score and reject messages over a certain threshold rather than queuing them, but extensive discussion would be required to determine if such a solution would be considered suitable for our listserv18:52
anteayaah, we could get lost in the extensive discussion18:53
anteayaand never crawl out of the hole18:53
fungii already am18:53
anteayaha ha ha18:53
*** apetrich has quit IRC19:00
*** bobh has quit IRC19:02
dmsimardmay I add myself to the openstack github org to initate the ara repository transfers ?19:08
*** whoami-rajat has quit IRC19:09
dmsimardthought I was in it, I guess not19:09
fungidmsimard: it may be better to test adding our shared account temporarily to the ara org, since that's how we're going to ask other folks who aren't infra sysadmins to do it?19:10
dmsimardsure, what organization is it ?19:10
dmsimarder, account*19:10
fungiit's documented in our credentials list19:11
fungialong with how to operate the one-time password generator to log into it19:11
dmsimardack, will look19:11
*** raissa has quit IRC19:14
clarkbopenstack qfiles dirs are all empty19:15
clarkbI think that means we are caught up19:15
clarkband I see at least one email since I left so still looking good19:15
*** raissa has joined #openstack-infra19:15
*** raissa has quit IRC19:15
*** raissa has joined #openstack-infra19:16
clarkband now I can go have celebratory lunch for stein release :) looks like it will be margaritas instead of helles.19:17
mordredclarkb: margaritas are always the right choice19:17
clarkb#status log Upgraded from trusty to xenial19:17
openstackstatusclarkb: finished logging19:17
*** raissa has joined #openstack-infra19:18
*** raissa has quit IRC19:19
clarkbfungi: I've hopped off that screen session if you want to close it feel free19:19
*** ijw has joined #openstack-infra19:19
clarkbmordred: "Margarita Factory" opened up down the street from us last week. Its a vancouver restaurant expanding from washington into oregon I guess. Gonna go full burbs mode margaritas19:19
*** raissa has joined #openstack-infra19:19
*** raissa has quit IRC19:19
fungithanks, closed down the screen session now19:20
fungiclarkb: a stein full of margaritas would work19:20
*** eernst has joined #openstack-infra19:21
fungithough i had helles on wednesday. the local brewery down the road made a nice (if strong) attempt at a helles (well, maibock)19:22
* cmurphy misses good helles19:23
*** eernst has quit IRC19:23
fungithere's a bit of a revival of american microbreweries doing their takes on traditional german recipes19:24
clarkbninkasi has one that is ok19:24
*** eernst has joined #openstack-infra19:27
*** bgmccollum has quit IRC19:30
*** bgmccollum has joined #openstack-infra19:31
*** eernst has quit IRC19:32
*** kjackal has quit IRC19:32
*** kjackal has joined #openstack-infra19:33
*** eernst has joined #openstack-infra19:34
*** eernst has quit IRC19:38
*** eernst has joined #openstack-infra19:40
*** jamesmcarthur has quit IRC19:42
*** eernst has quit IRC19:45
*** eernst has joined #openstack-infra19:46
*** eernst has quit IRC19:50
*** eernst has joined #openstack-infra19:53
*** ykarel is now known as ykarel|away19:54
*** eernst has quit IRC19:57
*** eernst has joined #openstack-infra19:59
dmsimardfungi, clarkb: tried the shared account for the transfer but the authentication requires a token when using 2fa:
dmsimardshould I create a token for the transfers and delete the token ? should we have a single token ?20:02
*** eernst has quit IRC20:02
clarkbso to auth for the transfer itself you need a new api token?20:03
dmsimardthe token becomes the password20:03
fungidmsimard: did you read the instructions in the credentials list? it tells you how to use the otp generator on bridge.o.o20:04
fungii've used those instructions (thanks ianw!) a few times already and they've been working for me20:04
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Remove apport from servers
dmsimardfungi: yes, I am able to authenticate on the web UI but when running a dry-run with the script, it bails out:
*** eernst has joined #openstack-infra20:05
fungiso to use a 2fa-enabled account you need to generate a code and add that to the api call, i guess?20:06
dmsimardoh, actually, in the docs I linked, there is a way to use OTP during the authentication20:06
dmsimarde.g, curl --request POST --url --header 'authorization: Basic PASSWORD' --header 'content-type: application/json' --header 'x-github-otp: OTP' --data '{"scopes": ["public_repo"], "note": "test"}'20:07
dmsimardI'm not sure how that works in practice since the otp changes over time ?20:07
*** diablo_rojo has quit IRC20:08
mordredfungi, clarkb, dmsimard, Shrews: ^^ patch a few lines ago - Shrews and I have been trying to figure out why zuul-preview is hanging ...20:08
dmsimardfungi: when you generate an API token, you authenticate against the API with the username and the token in-lieu of the password20:09
mordredin so doing have discovered that core files are configured to be piped in to apport20:09
openstackgerritMerged openstack-infra/project-config master: Retire openstack-ansible unused roles
mordredwhich seems like a thing that provides negative value20:09
mordredI'm not 100% sure what is setting the /proc setting - but I'm sort of hoping uninstalling apport will set the pattern back to "core"20:09
*** eernst has quit IRC20:09
mordreddoes anyone have an issue with me uninstalling apport by hand on zp01 to verify if removing the package will revert the proc setting?20:10
Shrewsmordred: it's not doing what it's supposed to anyway, i say go for it20:10
mordredof course, doing that makes it also want to remove ubuntu-server package20:11
Shrewsof course20:11
dmsimardand removing ubuntu-server does what ? removes everything ? :p20:12
mordredk. uninstalling isn't needed - we can just stop the service and it causes the proc entry to properly revert20:12
mordredpatch change coming20:12
*** tjgresha_nope has quit IRC20:12
*** tjgresha has joined #openstack-infra20:13
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Disable apport on servers
mordreddmsimard, Shrews, fungi: ^^20:14
dmsimardfungi, clarkb: the shared account is member of openstack but not owner and so doesn't have the necessary privileges
dmsimardI'll fix it20:15
fungiyeah, ever since gh upended their organization rbac model i haven't been able to make heads or tails of it20:16
*** eernst has joined #openstack-infra20:16
*** ykarel|away has quit IRC20:17
*** ijw has quit IRC20:18
*** ijw has joined #openstack-infra20:19
*** eernst has quit IRC20:22
*** eernst has joined #openstack-infra20:23
*** ijw has quit IRC20:23
*** eernst has quit IRC20:25
openstackgerritDavid Moreau Simard proposed openstack-infra/system-config master: Add script to automate GitHub organization transfers
*** pcaruana has quit IRC20:26
dmsimardtransfers are done, there was a small bug that I fixed in ^20:26
dmsimardI did it using a personal access token (which I've deleted already)20:26
dmsimardI'll add a note in the passwords file20:28
* dmsimard googles how to save and exit emacs20:31
clarkbctrl x ctrl c iirc20:36
*** zbr has quit IRC20:37
*** ijw has joined #openstack-infra20:39
*** ijw has quit IRC20:44
dmsimardyeah, I'm just a vim person haha20:44
openstackgerritMonty Taylor proposed openstack-infra/zuul-preview master: Use splice instead of erase/push_front
*** fried_rice is now known as efried_schoolrun20:58
*** kgiusti has left #openstack-infra20:59
fungigonna go find some dinner, but back in a little while21:04
clarkbI'm back from celebrating the stein release :)21:12
clarkbfeeling extremely useless. I suppose I should review mordred and dmsimard's changes21:12
clarkbI still see email going through mailman so that looks good21:13
*** jtomasek has quit IRC21:14
*** ijw has joined #openstack-infra21:16
*** jamesmcarthur has joined #openstack-infra21:19
clarkbdmsimard: on what did we end up goign with for the auth? a new token? or somethign else/ I don't see us documenting whatever the choice was and am thinking that would be good21:23
*** jamesmcarthur has quit IRC21:23
mordredclarkb: oh wow - the testing really doesn't like that apport change21:27
clarkbmordred: it failed centos too though21:28
mordredwell, it CERTAINLY shouldn't have failed centos21:28
clarkbthat is why it failed oncentos and on our images I think21:28
clarkbif the service isn't there (like on our minimal dib images) it fails21:28
*** efried_schoolrun is now known as efried21:29
*** diablo_rojo has joined #openstack-infra21:29
*** betherly has quit IRC21:45
*** rosmaita has left #openstack-infra21:46
*** rlandy|ruck has quit IRC21:49
dmsimardclarkb: I added a note in the passwords file but I can add it in the script too21:50
dmsimardI ended up creating abd t21:50
dmsimardand deleting a token21:50
clarkbgot it21:51
*** calebb has joined #openstack-infra22:00
openstackgerritDavid Moreau Simard proposed openstack-infra/system-config master: Add script to automate GitHub organization transfers
*** EvilienM is now known as EmilienM22:13
openstackgerritClark Boylan proposed openstack-infra/project-config master: Double checking pre merge behavior on trusted repo
clarkbmordred: pabelanger ^ sanity check on our end22:14
*** Lucas_Gray has joined #openstack-infra22:15
pabelangerclarkb: if you remove all, and leave noop, is that different (i wouldn't expect it)22:15
clarkbseems like it isn't running tox-pep8 in that change22:16
openstackgerritClark Boylan proposed openstack-infra/project-config master: Double checking pre merge behavior on trusted repo
clarkbpabelanger: ^ checking just noop in that ps22:16
clarkbstill running openstack-zuul-jobs-linters22:17
clarkbso I don't think this is a general problem. Perhaps config or local code related22:18
pabelangerclarkb: I think I have a theory, can I PM?22:18
*** calebb has quit IRC22:20
*** ijw has quit IRC22:28
*** ijw has joined #openstack-infra22:29
*** ijw has quit IRC22:34
*** ijw has joined #openstack-infra22:49
*** dave-mccowan has quit IRC23:00
*** tosky has quit IRC23:11
*** _erlon_ has quit IRC23:11
*** diablo_rojo has quit IRC23:24
*** nicolasbock has quit IRC23:30
fungithe suspense is killing me23:35
*** igordc has quit IRC23:38
*** aaronsheffield has quit IRC23:40
*** sthussey has quit IRC23:47

Generated by 2.15.3 by Marius Gedminas - find it at!