| *** ianychoi has joined #openstack-infra | 00:01 | |
| *** owalsh_ has joined #openstack-infra | 00:01 | |
| *** harlowja has joined #openstack-infra | 00:01 | |
| *** owalsh has quit IRC | 00:02 | |
| *** owalsh has joined #openstack-infra | 00:05 | |
| *** owalsh_ has quit IRC | 00:06 | |
| *** igordc has quit IRC | 00:07 | |
| *** owalsh_ has joined #openstack-infra | 00:09 | |
| openstackgerrit | Merged openstack-infra/zuul master: Handle github exceptions better when trying to get a PR https://review.openstack.org/651609 | 00:09 |
|---|---|---|
| *** owalsh has quit IRC | 00:10 | |
| openstackgerrit | Artom Lifshitz proposed openstack-infra/project-config master: Make whitebox use Storyboard https://review.openstack.org/651942 | 00:11 |
| *** owalsh has joined #openstack-infra | 00:13 | |
| *** hamzy has joined #openstack-infra | 00:14 | |
| *** owalsh_ has quit IRC | 00:14 | |
| *** ijw has quit IRC | 00:15 | |
| *** ijw has joined #openstack-infra | 00:15 | |
| *** gyee has quit IRC | 00:15 | |
| *** owalsh_ has joined #openstack-infra | 00:18 | |
| *** owalsh has quit IRC | 00:19 | |
| *** ijw has quit IRC | 00:20 | |
| *** owalsh has joined #openstack-infra | 00:21 | |
| *** Lucas_Gray has quit IRC | 00:22 | |
| *** owalsh_ has quit IRC | 00:23 | |
| *** sthussey has quit IRC | 00:24 | |
| *** owalsh_ has joined #openstack-infra | 00:26 | |
| *** owalsh has quit IRC | 00:27 | |
| *** owalsh has joined #openstack-infra | 00:31 | |
| *** owalsh_ has quit IRC | 00:32 | |
| *** owalsh_ has joined #openstack-infra | 00:35 | |
| *** ijw has joined #openstack-infra | 00:36 | |
| *** owalsh has quit IRC | 00:36 | |
| *** owalsh has joined #openstack-infra | 00:39 | |
| *** rh-jelabarre has quit IRC | 00:39 | |
| openstackgerrit | Merged openstack-infra/system-config master: Pass SSL certs to graphite.opendev.org https://review.openstack.org/651687 | 00:40 |
| *** owalsh_ has quit IRC | 00:40 | |
| *** ijw has quit IRC | 00:40 | |
| *** owalsh_ has joined #openstack-infra | 00:43 | |
| *** rh-jelabarre has joined #openstack-infra | 00:43 | |
| *** owalsh has quit IRC | 00:44 | |
| *** owalsh has joined #openstack-infra | 00:47 | |
| *** owalsh_ has quit IRC | 00:48 | |
| *** owalsh_ has joined #openstack-infra | 00:51 | |
| *** owalsh has quit IRC | 00:52 | |
| *** mriedem is now known as mriedem_afk | 00:57 | |
| *** owalsh has joined #openstack-infra | 00:57 | |
| *** owalsh_ has quit IRC | 00:58 | |
| *** ricolin has joined #openstack-infra | 01:05 | |
| *** hongbin has joined #openstack-infra | 01:08 | |
| *** whoami-rajat has joined #openstack-infra | 01:09 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Implement max-servers for AWS driver https://review.openstack.org/649474 | 01:12 |
| *** owalsh_ has joined #openstack-infra | 01:29 | |
| *** owalsh has quit IRC | 01:30 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-graphite master: Fix ssl lookups https://review.openstack.org/651950 | 01:38 |
| *** mriedem_afk is now known as mriedem | 01:42 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:47 | |
| *** apetrich has quit IRC | 01:57 | |
| *** yamamoto has joined #openstack-infra | 01:58 | |
| *** diablo_rojo has quit IRC | 02:00 | |
| *** markvoelker has joined #openstack-infra | 02:00 | |
| *** gregoryo has joined #openstack-infra | 02:10 | |
| *** yamamoto has quit IRC | 02:21 | |
| *** yamamoto has joined #openstack-infra | 02:23 | |
| *** mriedem has quit IRC | 02:29 | |
| *** markvoelker has quit IRC | 02:35 | |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 02:42 | |
| *** rlandy|ruck has quit IRC | 02:49 | |
| *** irclogbot_0 has quit IRC | 03:01 | |
| *** irclogbot_0 has joined #openstack-infra | 03:01 | |
| *** bhavikdbavishi has quit IRC | 03:03 | |
| openstackgerrit | Merged openstack-infra/puppet-graphite master: Fix ssl lookups https://review.openstack.org/651950 | 03:10 |
| *** rcernin has quit IRC | 03:15 | |
| *** psachin has joined #openstack-infra | 03:19 | |
| *** ykarel has joined #openstack-infra | 03:24 | |
| *** raukadah is now known as chandankumar | 03:25 | |
| *** nicolasbock has quit IRC | 03:25 | |
| openstackgerrit | Trinh Nguyen proposed openstack-infra/project-config master: Move Telemetry to Storyboard https://review.openstack.org/651963 | 03:27 |
| *** rcernin has joined #openstack-infra | 03:27 | |
| *** markvoelker has joined #openstack-infra | 03:32 | |
| *** owalsh has joined #openstack-infra | 03:40 | |
| *** owalsh_ has quit IRC | 03:40 | |
| *** efried has quit IRC | 03:43 | |
| *** efried has joined #openstack-infra | 03:44 | |
| openstackgerrit | Trinh Nguyen proposed openstack-infra/project-config master: Move Telemetry to Storyboard https://review.openstack.org/651963 | 03:44 |
| *** owalsh_ has joined #openstack-infra | 03:48 | |
| *** owalsh has quit IRC | 03:49 | |
| openstackgerrit | Trinh Nguyen proposed openstack-infra/project-config master: Move Telemetry to Storyboard https://review.openstack.org/651963 | 03:51 |
| *** yamamoto has quit IRC | 03:51 | |
| *** ykarel_ has joined #openstack-infra | 03:51 | |
| *** yamamoto has joined #openstack-infra | 03:52 | |
| *** ykarel has quit IRC | 03:53 | |
| *** ykarel has joined #openstack-infra | 03:54 | |
| *** ykarel_ has quit IRC | 03:56 | |
| *** yamamoto has quit IRC | 03:57 | |
| *** rcernin has quit IRC | 03:57 | |
| *** lseki has quit IRC | 03:59 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:59 | |
| *** hwoarang has quit IRC | 03:59 | |
| *** hwoarang has joined #openstack-infra | 04:00 | |
| *** rcernin has joined #openstack-infra | 04:01 | |
| *** yamamoto has joined #openstack-infra | 04:02 | |
| *** imacdonn_ has quit IRC | 04:04 | |
| *** yamamoto has quit IRC | 04:04 | |
| *** markvoelker has quit IRC | 04:04 | |
| *** imacdonn_ has joined #openstack-infra | 04:04 | |
| *** yamamoto has joined #openstack-infra | 04:08 | |
| *** ramishra has joined #openstack-infra | 04:11 | |
| ianw | https://www.ssllabs.com/ssltest/analyze.html?d=graphite.opendev.org&s=162.209.77.51&latest | 04:12 |
| *** yamamoto has quit IRC | 04:13 | |
| *** udesale has joined #openstack-infra | 04:13 | |
| *** hongbin has quit IRC | 04:16 | |
| *** owalsh has joined #openstack-infra | 04:21 | |
| *** owalsh_ has quit IRC | 04:22 | |
| *** _d34dh0r53_ has joined #openstack-infra | 04:24 | |
| *** cloudkiller has joined #openstack-infra | 04:25 | |
| clarkb | success! | 04:28 |
| clarkb | we still need to get the webserver restart in right? but we have 90 days for that? | 04:29 |
| ianw | yep ... i'll look at your work soon :) | 04:29 |
| clarkb | the hardest bit will be testing it | 04:31 |
| clarkb | but maybe we hand wave around that for a bit | 04:31 |
| ianw | clarkb: that said; we're going to probably need to get some httpd deployment into ansible for these arm64 mirrors ... | 04:32 |
| clarkb | ya | 04:32 |
| clarkb | also the gitea tests can probably test it too | 04:33 |
| clarkb | restart gitea instead of apache but same general idea | 04:33 |
| *** roman_g has quit IRC | 04:35 | |
| ianw | also looking at accounts, the only thing it really gives us is we register an email with the cert, so it send "you need to renew" reminders. not sure if not getting those is a feature or a bug | 04:37 |
| clarkb | our certchecker works with sni now so I think we ha e that covered | 04:38 |
| clarkb | I ran it in the foreground and checked it had the right values with git.* | 04:39 |
| clarkb | should add the new graphite cert to that checklist :) | 04:39 |
| *** Lucas_Gray has joined #openstack-infra | 04:40 | |
| *** tkajinam has quit IRC | 05:00 | |
| *** markvoelker has joined #openstack-infra | 05:02 | |
| *** tkajinam has joined #openstack-infra | 05:07 | |
| *** ykarel has quit IRC | 05:16 | |
| *** ykarel has joined #openstack-infra | 05:29 | |
| *** ykarel_ has joined #openstack-infra | 05:34 | |
| *** markvoelker has quit IRC | 05:34 | |
| *** zhurong has quit IRC | 05:35 | |
| *** ykarel has quit IRC | 05:36 | |
| *** ricolin has quit IRC | 05:39 | |
| *** jtomasek has joined #openstack-infra | 05:39 | |
| *** Lucas_Gray has quit IRC | 05:40 | |
| *** zhurong has joined #openstack-infra | 05:41 | |
| *** yamamoto has joined #openstack-infra | 05:45 | |
| *** ykarel__ has joined #openstack-infra | 05:46 | |
| *** ykarel_ has quit IRC | 05:48 | |
| *** psachin has quit IRC | 05:49 | |
| *** quiquell has joined #openstack-infra | 05:50 | |
| *** quiquell is now known as quiquell|rover | 05:50 | |
| *** armax has quit IRC | 05:53 | |
| *** yamamoto has quit IRC | 05:58 | |
| *** lpetrut has joined #openstack-infra | 06:02 | |
| *** gregoryo has quit IRC | 06:03 | |
| *** psachin has joined #openstack-infra | 06:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/651973 | 06:06 |
| *** pgaxatte has joined #openstack-infra | 06:20 | |
| AJaeger | yeah, proposal job works again. mordred, thanks for your help ^ | 06:26 |
| *** markvoelker has joined #openstack-infra | 06:31 | |
| *** dangtrinhnt has joined #openstack-infra | 06:31 | |
| dangtrinhnt | Hi fungi, I would like to migrate Telemetry projects to storyboard. #link https://review.openstack.org/#/c/651963/3 | 06:32 |
| dangtrinhnt | If you need any information to proceed that, please let me know. Thank for helping. | 06:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/651973 | 06:35 |
| AJaeger | dangtrinhnt: fungi is US based, best wait 7 hours and then ask again. I hope he reads backscroll and replies... | 06:36 |
| *** pcaruana has joined #openstack-infra | 06:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add airship-governance project https://review.openstack.org/651342 | 06:41 |
| openstackgerrit | Merged openstack-infra/project-config master: Add vexxhost/ansible-role-docker-distribution https://review.openstack.org/651879 | 06:42 |
| *** quiquell|rover is now known as quique|rover|brb | 06:44 | |
| *** e0ne has joined #openstack-infra | 06:45 | |
| *** ykarel_ has joined #openstack-infra | 06:49 | |
| *** ykarel__ has quit IRC | 06:51 | |
| *** slaweq has joined #openstack-infra | 06:59 | |
| *** hamzy has quit IRC | 07:05 | |
| *** markvoelker has quit IRC | 07:05 | |
| *** hamzy has joined #openstack-infra | 07:05 | |
| *** whoami-rajat has quit IRC | 07:05 | |
| *** whoami-rajat has joined #openstack-infra | 07:06 | |
| *** mnaser has quit IRC | 07:06 | |
| *** Dobroslaw_afk has quit IRC | 07:06 | |
| *** kambiz has quit IRC | 07:06 | |
| *** HenryG has quit IRC | 07:06 | |
| *** mnaser has joined #openstack-infra | 07:06 | |
| *** egonzalez has quit IRC | 07:07 | |
| *** kambiz has joined #openstack-infra | 07:07 | |
| *** egonzalez has joined #openstack-infra | 07:08 | |
| *** rpittau|afk is now known as rpittau | 07:08 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:08 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add support for 'oldValue' comment-added field https://review.openstack.org/649900 | 07:12 |
| *** ykarel_ is now known as ykarel | 07:16 | |
| *** yamamoto has joined #openstack-infra | 07:21 | |
| *** e0ne has quit IRC | 07:21 | |
| *** iurygregory has joined #openstack-infra | 07:22 | |
| *** ginopc has joined #openstack-infra | 07:25 | |
| *** jpich has joined #openstack-infra | 07:32 | |
| *** e0ne has joined #openstack-infra | 07:36 | |
| *** apetrich has joined #openstack-infra | 07:42 | |
| *** tosky has joined #openstack-infra | 07:43 | |
| *** jpena|off is now known as jpena | 07:47 | |
| *** kjackal has joined #openstack-infra | 07:49 | |
| *** e0ne has quit IRC | 07:52 | |
| *** ykarel is now known as ykarel|lunch | 07:52 | |
| *** ccamacho has joined #openstack-infra | 07:53 | |
| *** kjackal has quit IRC | 07:56 | |
| *** kjackal has joined #openstack-infra | 07:56 | |
| *** Lucas_Gray has joined #openstack-infra | 07:57 | |
| *** e0ne has joined #openstack-infra | 07:59 | |
| *** ricolin has joined #openstack-infra | 08:01 | |
| *** dtantsur|afk is now known as dtantsur | 08:01 | |
| *** markvoelker has joined #openstack-infra | 08:02 | |
| *** Lucas_Gray has quit IRC | 08:08 | |
| *** Lucas_Gray has joined #openstack-infra | 08:09 | |
| *** lucasagomes has joined #openstack-infra | 08:10 | |
| *** calbers has quit IRC | 08:14 | |
| *** calbers has joined #openstack-infra | 08:27 | |
| *** markvoelker has quit IRC | 08:34 | |
| *** derekh has joined #openstack-infra | 08:39 | |
| *** electrofelix has joined #openstack-infra | 08:39 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Updates Octavia meeting time to 1600 UTC https://review.openstack.org/651929 | 08:49 |
| *** e0ne has quit IRC | 08:55 | |
| *** dtantsur is now known as dtantsur|brb | 08:59 | |
| *** e0ne has joined #openstack-infra | 09:00 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 09:07 | |
| *** bhavikdbavishi has quit IRC | 09:09 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 09:09 | |
| *** janki has joined #openstack-infra | 09:15 | |
| *** ginux has joined #openstack-infra | 09:15 | |
| *** ginopc has quit IRC | 09:17 | |
| *** ginux is now known as ginopc | 09:17 | |
| *** ykarel|lunch is now known as ykarel | 09:21 | |
| *** rcernin has quit IRC | 09:30 | |
| *** roman_g has joined #openstack-infra | 09:30 | |
| *** markvoelker has joined #openstack-infra | 09:31 | |
| *** ricolin has quit IRC | 09:33 | |
| *** gfidente has joined #openstack-infra | 09:36 | |
| *** bhavikdbavishi has quit IRC | 09:43 | |
| *** Lucas_Gray has quit IRC | 09:56 | |
| *** e0ne has quit IRC | 09:58 | |
| *** e0ne has joined #openstack-infra | 10:03 | |
| *** markvoelker has quit IRC | 10:04 | |
| *** kjackal has quit IRC | 10:10 | |
| *** Lucas_Gray has joined #openstack-infra | 10:18 | |
| *** Lucas_Gray has quit IRC | 10:20 | |
| *** Wryhder has joined #openstack-infra | 10:20 | |
| *** Wryhder is now known as Lucas_Gray | 10:21 | |
| frickler | infra-root: the matrix.org hacker has posted a list of issues that all helped in the coup at https://github.com/matrix-org/matrix.org/issues | 10:27 |
| *** e0ne has quit IRC | 10:28 | |
| frickler | although at a glance they don't seem to affect us directly, I think we should take the time to maybe go through them in a bit more detail | 10:28 |
| frickler | in particular looking at setting up 2FA for ssh access would seem to be a sensible improvement of security. I've never done that myself before, though, but I'll try to do a bit of testing over the weekend | 10:29 |
| *** pcaruana has quit IRC | 10:31 | |
| *** nicolasbock has joined #openstack-infra | 10:43 | |
| *** kjackal has joined #openstack-infra | 10:44 | |
| *** gfidente has quit IRC | 10:49 | |
| *** ociuhandu has joined #openstack-infra | 10:53 | |
| *** ociuhandu has quit IRC | 10:53 | |
| *** tbachman has quit IRC | 10:53 | |
| *** gfidente has joined #openstack-infra | 10:56 | |
| *** gfidente has quit IRC | 11:01 | |
| *** markvoelker has joined #openstack-infra | 11:02 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:05 | |
| *** udesale has quit IRC | 11:05 | |
| aspiers | fungi, corvus, clarkb: https://www.eventbrite.com/e/gerrit-user-summit-2019-tickets-58884641548 | 11:21 |
| openstackgerrit | M V P Nitesh proposed openstack/diskimage-builder master: Adding 'oel' as a new dib element https://review.openstack.org/625501 | 11:23 |
| *** jpena is now known as jpena|lunch | 11:27 | |
| *** jaosorior has quit IRC | 11:33 | |
| *** markvoelker has quit IRC | 11:35 | |
| *** pcaruana has joined #openstack-infra | 11:36 | |
| *** e0ne has joined #openstack-infra | 11:37 | |
| *** yamamoto has quit IRC | 12:03 | |
| *** tbachman has joined #openstack-infra | 12:05 | |
| *** EmilienM is now known as EvilienM | 12:05 | |
| *** yamamoto has joined #openstack-infra | 12:06 | |
| pabelanger | frickler: jenkins strikes again | 12:08 |
| pabelanger | frickler: some good info in issues | 12:08 |
| zbr | clarkb: just to let you know that we abandoned the gate-check jobs. we still need to do some internal house cleaning but the jobs are no longer run. | 12:09 |
| *** psachin has quit IRC | 12:11 | |
| *** rlandy has joined #openstack-infra | 12:16 | |
| *** rlandy is now known as rlandy|ruck | 12:16 | |
| *** ricolin has joined #openstack-infra | 12:25 | |
| *** e0ne has quit IRC | 12:27 | |
| *** kgiusti has joined #openstack-infra | 12:28 | |
| *** jpena|lunch is now known as jpena | 12:28 | |
| *** ricolin has quit IRC | 12:29 | |
| *** e0ne has joined #openstack-infra | 12:41 | |
| mordred | frickler: nicely enough - we already don't use agent forwarding - so taht's good | 12:41 |
| mordred | I disagree with 360 | 12:42 |
| mordred | 359 is ... yeah | 12:42 |
| *** yamamoto has quit IRC | 12:43 | |
| *** jamesmcarthur has joined #openstack-infra | 12:43 | |
| mordred | frickler: yeah- those are all interesting | 12:45 |
| *** fdegir has quit IRC | 12:45 | |
| aspiers | hmm, couple of the mails I sent to -discuss yesterday are still missing from the archives :-/ | 12:45 |
| *** fdegir has joined #openstack-infra | 12:46 | |
| *** timburke has quit IRC | 12:46 | |
| *** cjloader has quit IRC | 12:46 | |
| *** gfidente has joined #openstack-infra | 12:46 | |
| *** timburke has joined #openstack-infra | 12:48 | |
| mordred | aspiers: that's because I'm secretly deleting them just to frustrate you | 12:48 |
| aspiers | mordred: I suspected as much | 12:49 |
| aspiers | frickler: urgh, nickserv passwords need changing now then | 12:50 |
| * aspiers changes his | 12:50 | |
| *** dtantsur|brb is now known as dtantsur | 12:51 | |
| fungi | ianw: one thing we can likely also do in the course of switching to letsencrypt is to add caa records in dns like 'graphite IN CAA 0 issue "letsencrypt.org"' and 'graphite IN CAA 0 iodef "mailto:infra-root@openstack.org"' | 12:55 |
| *** e0ne has quit IRC | 12:55 | |
| *** mriedem has joined #openstack-infra | 12:55 | |
| openstackgerrit | Fatih Degirmenci proposed openstack/diskimage-builder master: Constraint networkx to 2.2 https://review.openstack.org/652063 | 12:56 |
| fungi | ianw: and since we unconditionally redirect http to https we could also consider turning on hsts in the apache config like 'Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"' (i should have caught that when reviewing the vhost config) | 12:58 |
| frickler | aspiers: yes, there was a freenode global advice about that earlier | 12:58 |
| aspiers | oh yeah | 12:58 |
| *** e0ne has joined #openstack-infra | 12:59 | |
| fungi | dangtrinhnt: the launchpad to storyboard migration process we usually follow is documented at https://docs.openstack.org/infra/storyboard/migration.html | 12:59 |
| fungi | i can test importing all the telemetry projects on storyboard-dev today and assuming we don't find any problems (these days there usually aren't any because we don't find many new things to fix in the migration script) we can figure out when a good time is for your team to cut over and disable bug reporting in launchpad | 13:01 |
| *** gfidente has quit IRC | 13:01 | |
| fungi | frickler: i've done otp configurations for opensshd using opie before, though i haven't tried setting that up as a second factor of authentication | 13:02 |
| *** jcoufal has joined #openstack-infra | 13:02 | |
| mordred | aspiers: thanks for mentioning the nickserv thing though - I missed the freenode announcement - but have now changed my password | 13:03 |
| fungi | aspiers: did you get any moderation notices for those? i usually check the moderation queue at least once a day to purge all the spam and see if any legit messages landed in moderation for some reason | 13:03 |
| *** gfidente has joined #openstack-infra | 13:03 | |
| mordred | fungi: speaking of dns things - would it be useful to publish SSHFP records for our servers now that we've got our happy new dns system? or would it be more pain than it's worth? | 13:03 |
| *** altlogbot_2 has joined #openstack-infra | 13:04 | |
| aspiers | fungi: I got a moderation notice for an unrelated email to -discuss, due to a large recipient list | 13:04 |
| fungi | aspiers: ahh, yep, "Reason: Too many recipients to the message" | 13:05 |
| fungi | will approve those now | 13:05 |
| aspiers | fungi: I've posted 4 times to this thread http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004779.html | 13:05 |
| aspiers | fungi: but only two of my posts are shown in the archives | 13:05 |
| fungi | mordred: yes, sshfp for sure, though openssh and other clients usually need it explicitly enabled | 13:05 |
| fungi | mordred: also you can't easily add sshfp records in advance of launching the server so that would need to be followup anyway | 13:06 |
| *** dave-mccowan has joined #openstack-infra | 13:06 | |
| fungi | mordred: maybe doable at the same time we add address records though if we get the launch script to look at the host key and spit out appropriate sshfp records for it? | 13:07 |
| *** gfidente has quit IRC | 13:09 | |
| *** gfidente has joined #openstack-infra | 13:10 | |
| mordred | fungi: yeah - that's not a bad idea | 13:12 |
| *** aaronsheffield has joined #openstack-infra | 13:13 | |
| *** yamamoto has joined #openstack-infra | 13:13 | |
| fungi | aspiers: luckily, in an effort to troubleshoot some dkim signature challenges we've been logging copies of all messages to openstack-discuss for a couple months now | 13:14 |
| fungi | aspiers: unfortunately, i only see two messages from you arriving at the mta with that subject | 13:15 |
| aspiers | hrmph | 13:15 |
| fungi | Date: Thu, 11 Apr 2019 15:10:57 +0100 | 13:15 |
| aspiers | and Fri, 12 Apr 2019 13:57:31 +0100 | 13:15 |
| fungi | Date: Fri, 12 Apr 2019 13:57:31 +0100 | 13:15 |
| fungi | yeah, those two | 13:15 |
| aspiers | yeah, the two showing in the archives | 13:15 |
| aspiers | fungi: can I try resending the others right now? | 13:16 |
| fungi | so whatever happened to the others, they haven't reached the server (yet anyway, maybe hung in a deferral queue at the sending side?) | 13:16 |
| aspiers | possibly | 13:16 |
| fungi | feel free, you just may end up with duplicates if the others are eventually delivered to us | 13:16 |
| *** dave-mccowan has quit IRC | 13:17 | |
| *** betherly has joined #openstack-infra | 13:20 | |
| *** Wryhder has joined #openstack-infra | 13:22 | |
| aspiers | hrm | 13:22 |
| *** Lucas_Gray has quit IRC | 13:23 | |
| fungi | aspiers: normally if messages are stuck in a deferral queue, the mta holding them will send "temporarily undeliverable" bounces to the sender to let them know, but with the sorry state of smtp standards these days i no longer count on it | 13:23 |
| *** Wryhder is now known as Lucas_Gray | 13:23 | |
| aspiers | ohhh, I found one of those | 13:23 |
| aspiers | Response: 501 Authentication Failure | 13:24 |
| fungi | aspiers: it looks like these transit a couple of mtas at novell.com, if the received headers on the ones which made it through are any indication | 13:24 |
| aspiers | emea4-mta.ukb.novell.com? | 13:25 |
| aspiers | or similar | 13:25 |
| fungi | and smtp.nue.novell.com | 13:25 |
| aspiers | they might be crappy Groupwise servers | 13:25 |
| *** tkajinam_ has joined #openstack-infra | 13:25 | |
| fungi | this i rather expect, yes | 13:25 |
| aspiers | now that we're independent from Micro Focus we'll be getting rid of those finally | 13:25 |
| fungi | wouldn't it be awesome if the driving factor in suse's independence was that it wanted to finally be able to have reliable mailservers? | 13:26 |
| aspiers | well, that's one of the things the employees are most excited about ;-) | 13:27 |
| aspiers | that, and a new expenses system | 13:27 |
| dangtrinhnt | Thanks, fungi :) | 13:27 |
| *** gfidente has quit IRC | 13:27 | |
| *** tkajinam has quit IRC | 13:27 | |
| fungi | as opposed to red hat, who recently moved all their employee e-mail off free software and onto a proprietary mail service | 13:27 |
| aspiers | Oh, I didn't say we'd be getting a free replacement :-/ | 13:28 |
| mordred | \o/ | 13:28 |
| mordred | :( | 13:28 |
| aspiers | We won't AFAIK, and I have complained loudly about that many times | 13:28 |
| mordred | yeah. nothing like the free software companies being shining beacons for giving up | 13:28 |
| aspiers | Right | 13:28 |
| fungi | :( | 13:28 |
| aspiers | I've used words like "hypocrisy" more than once | 13:28 |
| fungi | that is certainly a good word for it | 13:28 |
| mordred | the flame war here about that particular instance of us giving up happened before I arrived, so I did not participate | 13:29 |
| aspiers | Don't even get me started on Jira ... | 13:29 |
| mordred | yeah. or ghe | 13:29 |
| mordred | or AWS | 13:29 |
| mordred | basically - literally nobody cares anymore | 13:29 |
| * fungi is nobody, for sure | 13:30 | |
| fungi | nobodies of the free software world, unite! | 13:30 |
| mordred | I mean - statistically. luckily there is a nice collection of the remaining humans who care in this channel | 13:30 |
| aspiers | Also had a big "told you so" moment recently when Trello broke their promise | 13:30 |
| fungi | the last, best hope for free tools | 13:30 |
| fungi | opendev 5 | 13:30 |
| mordred | aspiers: oh? I didn't know about that ... I'd love another piece of ammunition to mock people with | 13:31 |
| aspiers | Should we rename the channel to #rebel-alliance? | 13:31 |
| mordred | aspiers: what did they do? | 13:31 |
| aspiers | mordred: still searching for the link | 13:31 |
| *** altlogbot_2 has quit IRC | 13:32 | |
| *** altlogbot_1 has joined #openstack-infra | 13:33 | |
| * persia mumbles that the problem with identifying as a rebellion is that it forces one to prop up the evil empire in order to be confident one has something against which to rebel, else one ends up sacrificing one's identity if one succeeds | 13:33 | |
| *** eglute has quit IRC | 13:33 | |
| *** cloudnull has quit IRC | 13:34 | |
| *** cloudkiller is now known as cloudnull | 13:34 | |
| aspiers | mordred: https://help.trello.com/article/1177-personal-vs-team-boards | 13:34 |
| *** d34dh0r53 has quit IRC | 13:35 | |
| mordred | persia: ++ | 13:35 |
| aspiers | they are forcing heavier team users to upgrade | 13:35 |
| aspiers | https://blog.trello.com/trello-teams-at-work | 13:35 |
| mordred | persia: also - I seem to remember one of the key members of the referred-to rebellion having the title of "princess" which certainly does indicate an aristrocracy | 13:35 |
| mordred | aspiers: whoopsie! | 13:37 |
| aspiers | mordred: to be fair, the UK royal family couldn't do nearly as bad a job of running the country than the current shower of "democratically elected" morons | 13:37 |
| *** quiquell|rover is now known as quiquell|off | 13:37 | |
| mordred | aspiers: ++ | 13:37 |
| persia | mordred: Any good modern empire needs to immediately execute the aristocracy when started, else there is a chance that someone might remember history. | 13:37 |
| mordred | aspiers: I'm currently holding out mild hope for QE2 to just put her foot down somehow | 13:37 |
| aspiers | I'm not remotely a monarchist, but our queen is orders of magnitude more competent than 99.99% of our politicians | 13:37 |
| mordred | yup | 13:37 |
| mordred | same here | 13:38 |
| *** bhavikdbavishi has quit IRC | 13:38 | |
| aspiers | mordred: one of my colleagues had a nice idea how she could do that | 13:38 |
| persia | mordred: Amusingly, she explicitly gave up power to put her foot down the April before referendum. She might un-give-it-up, but it's a two-step process. | 13:38 |
| *** altlogbot_1 has quit IRC | 13:38 | |
| aspiers | If I remember correctly, his suggestion went something like this | 13:38 |
| aspiers | All British passports are property of the Crown | 13:38 |
| aspiers | The Queen is actually descended from Germans | 13:39 |
| *** altlogbot_0 has joined #openstack-infra | 13:39 | |
| aspiers | If she claimed German citizenship, then all British passports would be the property of a German | 13:39 |
| aspiers | :) | 13:39 |
| *** gfidente has joined #openstack-infra | 13:40 | |
| aspiers | Unfortunately I think she lost her claim to German citizenship a long time ago | 13:40 |
| aspiers | but it was a cute idea | 13:40 |
| aspiers | mordred: here's another one https://techcrunch.com/2019/03/19/trello-aims-for-the-enterprise/?guccounter=1&guce_referrer_us=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_cs=wlkI7TFbzn5kwycywRrSWQ | 13:41 |
| aspiers | Another reason for projects to move to https://taiga.io/ IMHO | 13:41 |
| mordred | or to stop trying to manage themselves like they are auto manufacturers | 13:41 |
| aspiers | haha :) | 13:42 |
| *** eharney has joined #openstack-infra | 13:42 | |
| mordred | perpetuating the false impression that we are like factory workers and/or cogs in a large machine that are interchangeable | 13:42 |
| *** e0ne has quit IRC | 13:42 | |
| persia | To be fair, some actual auto manufacturers have moved beyond that style of management./ | 13:42 |
| mordred | rather than what we should think of ourselves like - an industry of creative types | 13:42 |
| mordred | persia: ++ | 13:42 |
| aspiers | You mean kanban, or something wider than that? | 13:42 |
| * aspiers recalls that Kanban came from factory-like environments in Japan | 13:43 | |
| aspiers | taiga is much more than kanban, fortunately | 13:44 |
| mordred | yup. it's a system derived from toyota | 13:44 |
| mordred | as is LEAN | 13:44 |
| mordred | aspiers: yeah - unfortunately for me, I also don't subscribe to the usefulness (or agility) of agile | 13:45 |
| aspiers | I'm not religious about it either | 13:45 |
| mordred | so a tool described as "platform for agile developers" leaves me quite cold | 13:45 |
| aspiers | but I do like having somewhere to track issues and collaborate effectively :) | 13:45 |
| mordred | oh - sure. that can be useful | 13:46 |
| aspiers | not forgetting StoryBoard of course | 13:46 |
| mordred | but as soon as I have to start using catch-phrases and jargon like "epic" or "sprint" I'm pretty much going to be done with whatever the thing is | 13:46 |
| aspiers | I can't stand the legacy email-heavy approach taken by Debian and the kernel / git communities | 13:47 |
| persia | aspiers: Both specific practices used by a specific automaker that have been misused out of context for a while, and the general attitude (which was never held at the originating auto firm) that creative activity such as coding has any relation to repetitive process execution (for which things like actual kanban (which isn't implemented by any of the tools that are labeled "kanban") are incredibly useful). | 13:47 |
| mordred | persia: ++ | 13:47 |
| aspiers | +1 | 13:48 |
| persia | I've used the email workflows in the past to great success, but it depends on using specific email clients with specific plugins in specific ways, and is incompatible with how most people have used email for most of the 21st century, sadly. | 13:48 |
| mordred | aspiers: unfortunately, I have reached the grumpy place where none of the available tracking solutions provide more value to me than the time it takes for me to use them | 13:48 |
| mordred | I really wish this were not true - I would LOVE to keep track of things more better collaboratively | 13:49 |
| aspiers | That's surprising to hear, I find even GitHub's dumb issue tracker pretty useful | 13:51 |
| aspiers | Although their projects UI is pretty horrible | 13:51 |
| *** sthussey has joined #openstack-infra | 13:52 | |
| mordred | aspiers: I find github's issue tracker very unpleasant - but I generally find github's UI to be categorically very hard to use | 13:52 |
| persia | For my current role, there cannot exist any issue tracker that helps me, and certainly not one that anyone else can see. | 13:52 |
| aspiers | mordred: what are the main problems for you? | 13:52 |
| persia | For the few projects where I'm responsible for collaboration, I find most participants have enough to do with private issue tracking systems that public ones are considered a dumping ground. | 13:53 |
| aspiers | persia: not sure I follow | 13:53 |
| mordred | aspiers: too much clicking. too hard for me to find a list of "these are thigns I should care about"" | 13:54 |
| persia | aspiers: So, someone wants something, and hires people to do it. They then typically use some mechanism to tell those people what to do and how. If the things to be done influence external parties, it's usually impolite to publish them, as nobody wants to be respoinsible for that sort of forward-looking statement when the situation changes tomorrow. | 13:55 |
| aspiers | mordred: you mean globally across repos? | 13:55 |
| persia | This means that people doing things end up writing down the "what" twice if thy have to put it someone external. | 13:55 |
| mordred | aspiers: yeah. the use case of "I only care about this one repo" is never a thing that I need | 13:55 |
| aspiers | mordred: https://github.com/issues is missing the filters or sorting you need? | 13:56 |
| * persia gets a 404 from that URI | 13:57 | |
| aspiers | I'm very far from a GitHub fanboy, but I find the search pretty powerful | 13:57 |
| aspiers | persia: not logged in? | 13:57 |
| mordred | aspiers: I don't think I've ever seen that | 13:58 |
| aspiers | mordred: it's linked from the top of every page ;-) | 13:59 |
| * dirk is the github fanboy if you need anyone | 13:59 | |
| aspiers | it's pretty easy to pick your favourite search and bookmark it | 13:59 |
| *** slaweq has quit IRC | 14:00 | |
| aspiers | dirk is lying, in case you couldn't guess X-D X-D | 14:00 |
| mordred | aspiers: it's possible that I don't actually use github unless forced to by some other projects use of it - so usually to file drive-by issues and then leave | 14:00 |
| *** altlogbot_0 has quit IRC | 14:00 | |
| aspiers | mordred: fair enough | 14:00 |
| mordred | so - you know - my opinions might not be the most accurate | 14:00 |
| aspiers | :) | 14:01 |
| mordred | but I find most of my interactions with gh to be quite painful - similar to interactions with mac osx, which I find very difficult to use | 14:01 |
| aspiers | issues with gh are tracked here https://github.com/isaacs/github/issues | 14:02 |
| *** slaweq has joined #openstack-infra | 14:02 | |
| mordred | I am very unlikley to use that :) | 14:02 |
| clarkb | fungi: ok my day is starting but not yet at akeyboard so your help starting the snapshot is much appreviated | 14:03 |
| mordred | clarkb: are you using your telepathy interface to IRC again? | 14:03 |
| clarkb | re 2fa one of the struggles K have with t hat is the relianve on third party services to implemwnt it | 14:04 |
| clarkb | mordred softkeyboards are meh | 14:04 |
| clarkb | so if google shuts down the service or yubico has an outage we lose | 14:04 |
| mordred | clarkb: ++ to the 2fa thing | 14:04 |
| clarkb | that said I do have a pair of yubikeys so could test their pam module if we want | 14:05 |
| aspiers | so, back to the proprietary vs. open thing ... has anyone looked at matomo for analytics? | 14:05 |
| aspiers | (formerly known as Piwik) | 14:05 |
| aspiers | I just did yesterday and was pretty blown away by how much they offer compared to Google and competitors | 14:05 |
| mordred | aspiers: I wanna say yes - I think investigation was done related to analytics for the zuul website? fungi would probably remember | 14:06 |
| fungi | aspiers: i can't remember if i looked at that one. i rules out any which required web bugs or cookies for tracking users, even if only local to the site | 14:07 |
| fungi | er, ruled out. but i'll take a look at it | 14:07 |
| aspiers | fungi: erm, how come openstack.org has google analytics then? ;-) | 14:07 |
| fungi | aspiers: i don't run that site | 14:08 |
| clarkb | aspiers: we dont run that | 14:08 |
| aspiers | ah | 14:08 |
| clarkb | part of the motivation for opendev on our sideis to remove that confusion | 14:08 |
| aspiers | gotcha | 14:08 |
| fungi | and i disagree with the choices made by the folks who do, but in their case they have decided that market intelligence is more important than avoiding handing visitor tracking information over to third party services | 14:08 |
| aspiers | well that's what's so awesome about matomo - no tracking data gets handed over to 3rd parties | 14:09 |
| aspiers | it's GPL and you own the data | 14:09 |
| aspiers | assuming you self-host, of course | 14:09 |
| fungi | i've also ruled out a couple which didn't have any convenient means to publish properly anonymized reports. they seemed to want to include far too much sensitive information. my goal with our (opendev) web analytics is to be able to publish fully open reports anyone can see | 14:09 |
| aspiers | (they have a SaaS offering too) | 14:09 |
| fungi | but yeah, looks like they rely on more than just web log analysis | 14:10 |
| clarkb | fungi is snapshot started? | 14:10 |
| aspiers | fungi: you think that's necessarily bad? | 14:10 |
| fungi | clarkb: getting there. sorry, too many conversations ;) | 14:10 |
| aspiers | I should stop distracting people now (including myself) | 14:10 |
| fungi | aspiers: yes, as it relies on cookies and/or web bugs injected into site content | 14:10 |
| aspiers | Not sure I see the issue, as long as appropriate permission is requested | 14:11 |
| *** lpetrut has quit IRC | 14:11 | |
| fungi | yes, one of the things i would love to avoid is adding a "prominent cookie notice" to every site we host so that we don't violate eu laws | 14:12 |
| aspiers | IIUC you don't need a cookie notice unless you track *personal* data (which includes things like IPs) | 14:12 |
| fungi | cookies used in the implementation of features of a service i'm reasonably okay with, cookies used for tracking site visitors i'm not | 14:13 |
| aspiers | web logs also track site visitors, but you're OK with that? | 14:13 |
| aspiers | curious where you are drawing the line | 14:14 |
| fungi | web logs are an expected feature of webservers | 14:14 |
| aspiers | not to non-technical folk ;-) | 14:14 |
| fungi | though if we start aggregating ... hold on i have actual work to do | 14:14 |
| aspiers | yeah, let's do this over a beer in Denver! it's an interesting topic ;-) | 14:14 |
| aspiers | I have actual work too :-/ | 14:15 |
| aspiers | some weekend reading in case you want it https://www.jeffalytics.com/gdpr-compliance/ | 14:15 |
| *** jamesmcarthur has quit IRC | 14:16 | |
| fungi | clarkb: image in progress | 14:16 |
| fungi | aspiers: but if we start aggregating data and reporting on statistical trends, we can actually reduce our access log retention as well | 14:16 |
| *** jamesmcarthur has joined #openstack-infra | 14:17 | |
| fungi | i'd very much like to effectively replace our access logs with public reports of anonymized visitor stats | 14:17 |
| aspiers | that could be done with matomo pretty easily, I suspect | 14:18 |
| aspiers | if you self-host, you have the entire data set and can do whatever you want with it | 14:18 |
| aspiers | but now I'm wondering which sites infra does actually manage | 14:19 |
| aspiers | even https://docs.openstack.org/ has google analytics | 14:19 |
| fungi | at least for sites where users allow their browser to set cookies. but maybe i'm basing concerns of complexity on my own browsing habits where i stick to text-based browsers where possible, block cookeis i haven't personally vetted, block loading third-party images, and so on | 14:19 |
| clarkb | we run the webserver for docs but not the content | 14:19 |
| clarkb | I believe openstack docs add google analytics themselves | 14:20 |
| fungi | aspiers: in the case of docs.openstack.org we didn't traditionally manage it, but when we offered to give them a place to host we didn't tell them what content they were allowed to include | 14:20 |
| aspiers | makes sense | 14:20 |
| clarkb | and that goes back to annegentle iirc | 14:21 |
| clarkb | its possible current team doesnteven know it is there | 14:21 |
| aspiers | well, if we care about avoiding handing that data to google then probably the only way to get equivalent analytics is to replace the GA tag with a self-hosted matomo equivalent | 14:21 |
| clarkb | I tend to be massively ignorant because my firewall, dns server, and browser all filter this stuff oit for me | 14:22 |
| fungi | yes, back when she was managing it in rackspace's non-openstack mass web hosting service | 14:22 |
| aspiers | it also raises the question, how many GA accounts does OpenStack use, and who has access to them? | 14:22 |
| fungi | i believe the osf web team handles that | 14:22 |
| aspiers | ignoring the data concerns, ideally they should be managed by Google Tag Manager | 14:22 |
| clarkb | (and you should set up your firewall, dns server and browser to do the same :) ) | 14:23 |
| *** dpawlik has quit IRC | 14:24 | |
| fungi | yeah, part of my assumption that direct web log analysis would be more effective is that i know the more popular "modern" methods wouldn't effectively capture my activity anyway | 14:25 |
| *** altlogbot_3 has joined #openstack-infra | 14:26 | |
| fungi | and also i'd rather not track and trend individual visitors because i find that creepy | 14:26 |
| aspiers | surely only creepy if you retain personal data? | 14:27 |
| *** gfidente has quit IRC | 14:27 | |
| fungi | and i don't want to hand over sensitive visitor information to anyone, not even my employer (osf) so want to limit this to aggregate statistics which are safe to publish for all to see | 14:27 |
| aspiers | even vanilla web logs contain IPs which is arguably more invasive than what GA claims to capture in a standard configuration | 14:27 |
| aspiers | sorry, s/capture/retain, or at least expose to the user/ | 14:28 |
| fungi | yep, which is why i'd like to truncate our access log retention where possible, but also all service logs (not just web server logs) are by their nature potentially contain some sensitive information. if they're used for troubleshooting and not for marketing intelligence i'm less concerned | 14:29 |
| aspiers | aggregating sessions by IP isn't much different to aggregating by session id persisted in a cookie | 14:29 |
| *** altlogbot_3 has quit IRC | 14:29 | |
| fungi | i wouldn't want to store statistics on a per-ip-address granularity | 14:29 |
| aspiers | modulo NAT, proxies etc. | 14:29 |
| aspiers | but that's what standard web logs do | 14:29 |
| *** altlogbot_2 has joined #openstack-infra | 14:30 | |
| fungi | yep, which is why i wouldn't want to publish arbitrary server logs and would also like to reduce our web access log retention to protect us from, say, being compelled to hand over that information to authorities | 14:30 |
| aspiers | sure | 14:30 |
| *** gfidente has joined #openstack-infra | 14:31 | |
| fungi | the best way to avoid disclosing private information is to delete as much of it as possible or never even save it to begin with (the latter is hard if you need to be able to troubleshoot those services though) | 14:31 |
| *** e0ne has joined #openstack-infra | 14:31 | |
| aspiers | consuming analytics based on logs or cookies, and then rotating aggressively plus only publishing anonymised reports is a far cry from publishing raw data | 14:31 |
| aspiers | I'm not sure anyone would advocate the latter | 14:32 |
| aspiers | and if you are doing the former, I don't really see a huge problem with cookies | 14:32 |
| fungi | so in answering the question as to where i draw the line, i'm okay with tracking access to aid in troubleshooting service problems, i'm not okay with doing so to provide market intelligence | 14:32 |
| aspiers | I entirely sympathise with that point of view; the devil is in the details though | 14:33 |
| *** altlogbot_2 has quit IRC | 14:33 | |
| aspiers | e.g. define "market intelligence" | 14:33 |
| aspiers | "number of page views" and "number of unique visitors" are both "market intelligence" to me | 14:33 |
| fungi | yes, and those numbers don't need individual user ip addresses | 14:34 |
| aspiers | how would you track unique visitors without IPs or cookies? | 14:34 |
| aspiers | even with both it's still almost impossible | 14:34 |
| aspiers | (due to multiple devices per user) | 14:35 |
| clarkb | this is why google forces you to login on chrome :/ | 14:35 |
| *** ykarel is now known as ykarel|afk | 14:35 | |
| clarkb | or rather doesnt forget who you were on logout | 14:35 |
| fungi | number of unique visitors doesn't require long-term tracking of ip addresses beyond whatever granularity you're averaging across | 14:36 |
| aspiers | clarkb: chrome doesn't sync cookies | 14:36 |
| fungi | so number of unique visitors each hour only needs an hour's worth of logs | 14:36 |
| aspiers | AFAIK | 14:36 |
| clarkb | aspiers: its not cookies aiui | 14:36 |
| clarkb | aspiers: the browser itself is keeping track | 14:36 |
| *** gfidente has quit IRC | 14:36 | |
| clarkb | there was some big controversy about it a few weeks back | 14:37 |
| fungi | yes, with chrome the browser reports your browsing activity directly to google. that's why they're so keen to provide it to you for free | 14:37 |
| aspiers | this one? https://www.extremetech.com/internet/277609-chrome-69-is-a-full-fledged-assault-on-user-privacy | 14:37 |
| clarkb | I thought there was a more recent one | 14:39 |
| *** lseki has joined #openstack-infra | 14:39 | |
| *** gfidente has joined #openstack-infra | 14:39 | |
| aspiers | maybe there is - either way I missed it. gonna have to look more closely into this stuff | 14:39 |
| aspiers | but right now I really need to work | 14:39 |
| aspiers | looking forward to some good beer discussions in Denver | 14:39 |
| aspiers | is it even worth scheduling a session on analytics? | 14:39 |
| *** gfidente has quit IRC | 14:39 | |
| fungi | anyway, the only reason i'm looking into web statistic reporting *at all* is that osf has requested numbers to determine how "popular" different pages are over time. i really don't have any desire for that data myself and wouldn't even be looking into implementing something if it weren't for that | 14:40 |
| aspiers | I think that kind of data can be very useful | 14:40 |
| aspiers | to avoid misallocating effort | 14:40 |
| *** gfidente has joined #openstack-infra | 14:40 | |
| fungi | i think the modern obsession with collecting data about what other people do or don't do is caustic | 14:40 |
| aspiers | :) | 14:40 |
| aspiers | I don't disagree. Would love to hear more in Denver | 14:41 |
| clarkb | aspiers: chrome 73 shuffled around settings previously under "privacy" to another locatio including those that created problems in v69 as per the linked article | 14:41 |
| clarkb | so all related to that I guess | 14:41 |
| aspiers | clarkb: interesting, thanks | 14:41 |
| aspiers | I really should switch to Brave | 14:41 |
| fungi | i don't personally appreciate the idea that others are collecting data about my activities, and where possible do my best to thwart it, so wouldn't want to be the one doing that to others | 14:42 |
| clarkb | I' rather thave a pihole install session or compare pfblockerng rulesets :) | 14:42 |
| fungi | two sessions. one on how to collect data on people and one on how to protect your activities from being unduly tracked. they could be scheduled to run at the same time since the people who care about one generally don't care about the other | 14:43 |
| *** Lucas_Gray has quit IRC | 14:44 | |
| aspiers | LOL | 14:46 |
| *** gfidente has quit IRC | 14:46 | |
| *** altlogbot_3 has joined #openstack-infra | 14:48 | |
| dmsimard | So, just to confirm, can I go ahead and +W replication changes to the non-retired ara repos and proceed with the move ? I can volounteer to be the guinea pig for the github org transfer script :p | 14:48 |
| dmsimard | The patch to modify the replication config is https://review.openstack.org/#/c/650914/ | 14:49 |
| *** zhurong has quit IRC | 14:49 | |
| fungi | dmsimard: did you confirm regular expressions work there? you're replacing a globbnig pattern with a regex | 14:50 |
| clarkb | dmsimard: hrm does gerrit's regex engine do negative lookaheads? | 14:50 |
| dmsimard | I've tested using the same regex pattern on review-dev | 14:50 |
| fungi | good enough for me | 14:50 |
| dmsimard | something like gtest-org/(?!gtest).* | 14:51 |
| fungi | and you confirmed it didn't stop other repos from replicating i take it? | 14:51 |
| *** altlogbot_3 has quit IRC | 14:51 | |
| fungi | (because if it doesn't support regexes, that simply could have disabled all replication) | 14:51 |
| dmsimard | oh, we've used that kind of patterns before in RDO's gerrit | 14:51 |
| dmsimard | I'm not worried about whether or not it works | 14:52 |
| clarkb | ya docs say negative lookahead will work | 14:52 |
| clarkb | so should be fine to approve now | 14:52 |
| fungi | cool, approved | 14:52 |
| *** altlogbot_3 has joined #openstack-infra | 14:52 | |
| dmsimard | fungi: the "tricky" part of the regex is making sure to match $ | 14:53 |
| fungi | indeed | 14:53 |
| fungi | otherwise you match everything | 14:53 |
| zbr | do we have any reason why tox jobs do often take longer to start than other heavier jobs? | 14:53 |
| *** stephenfin is now known as finucannot | 14:53 | |
| fungi | zbr: "longer to start" in what way? | 14:53 |
| *** efried is now known as fried_rice | 14:53 | |
| dmsimard | fungi: I've matched sahara unintentionally when searching for ara things before :p | 14:53 |
| fungi | zbr: as in time between node allocation and when the log starts to stream? | 14:54 |
| zbr | fungi: longer to start, means seeing "QUEUED" for longer than others. | 14:54 |
| fungi | zbr: okay, so longer to get a node allocated | 14:54 |
| openstackgerrit | Fatih Degirmenci proposed openstack/diskimage-builder master: Constraint networkx to <2.3 https://review.openstack.org/652063 | 14:54 |
| clarkb | nodepool node allocatio is effectively random | 14:54 |
| zbr | so there is nothing we can do. | 14:55 |
| clarkb | each job males a node request. If that job's request is handled by a slower cloud it will take longer | 14:55 |
| *** armax has joined #openstack-infra | 14:55 | |
| fungi | also how long it takes to boot the nodes for different images can play into that | 14:55 |
| zbr | i was curious because most of time tox jobs are light, and likely that could run in containers. | 14:55 |
| *** altlogbot_3 has quit IRC | 14:55 | |
| clarkb | one suggestion I have made to the zuul team is that we handle the job side requests as a proper fifo amd only have the variance behind the scenes | 14:55 |
| dmsimard | one of the things that we can do is to tweak the nodepool min-server for labels if they are used more than others | 14:55 |
| pabelanger | dmsimard: min-ready doesn't really work any more, since node requests go to random clouds | 14:56 |
| clarkb | so that the nextup job that needa type foo gets ehatever the next inetance of type foo | 14:56 |
| fungi | also min-ready only ever helped if we weren't already running at full load | 14:56 |
| pabelanger | dmsimard: IMO, all should be set to 1, and just wait for cloud to come onlie | 14:56 |
| pabelanger | online* | 14:56 |
| fungi | if we're maxing out our quota, every node allocation request has to wait in line for available quota | 14:57 |
| pabelanger | Yah | 14:57 |
| zbr | are we going to support running jobs in containers? this could lower the resource usage for simple ones. | 14:57 |
| *** Goneri has joined #openstack-infra | 14:58 | |
| clarkb | zbr: we technically already do, but it doesnt really lower resource consumpyion since the container hosts have to always be on | 14:58 |
| mordred | yup | 14:58 |
| mordred | we could also support having smaller nodes | 14:58 |
| clarkb | at least vexxhost has indicated that the VM approach is better for them because we shut the VMs off | 14:58 |
| *** altlogbot_2 has joined #openstack-infra | 14:59 | |
| clarkb | whereas the k8s cluster is always on | 14:59 |
| mordred | like - define soe labels that are for 1G or 2G nodes ... that can be helpful on some of our providers | 14:59 |
| *** jcoufal has quit IRC | 14:59 | |
| mordred | but on other providers the limiting factor is ip addresses, so it still won't help much | 14:59 |
| pabelanger | we also talked about smaller vms, eg: 1gb for tox say, in an effort to squeeze more VMs running at once, however the limit then becomes IPv4 addresses | 14:59 |
| pabelanger | but native ipv6 shouldn't be an issue | 15:00 |
| *** ijw has joined #openstack-infra | 15:00 | |
| mordred | yeah | 15:00 |
| zbr | is true that a k8s cluster is always on, so it has a permanent load, mainly we can think of it as another cloud-provider :) but at scale it does payoff. | 15:00 |
| *** ykarel|afk is now known as ykarel | 15:00 | |
| fungi | part of the challenge with doing something like a nodepool kubernetes in each provider is that's yet another persistent piece of infrastructure we have to run there, and build when we bring a new provider on line (even just building more mirror hosts is a not-insignificant amount of effort for us) | 15:00 |
| mordred | although it's worth pointing out that tox is _not_ lightweight in some of our larger projects | 15:00 |
| pabelanger | +1 | 15:00 |
| zbr | for runing 1-2 jobs as containers is not a good idea, but if you run lots of them, you easily see the benefits. | 15:00 |
| mordred | zbr: our cloud providers have indicated the opposite - it would be more costly for them if we switched aggressively to containers | 15:01 |
| pabelanger | it helps more when you have to pay for cloud resources, 8 x 1vcpu == 1 x 8vpcu at the end of the day :) that's what we have setup in zuul.ansible.com currently | 15:01 |
| mordred | but seriously - tox -epy35 on nova is NOT a small or quick job | 15:02 |
| *** gfidente has joined #openstack-infra | 15:02 | |
| zbr | mordred: i don't think we can talk about massive switch. but I do see it as a mix. | 15:02 |
| aspiers | -efunctional is even slower | 15:02 |
| clarkb | and it wants 8cpus :) | 15:02 |
| * clarkb somewhat halpy aboit that as pre testr nova unittests took even longer | 15:02 | |
| fungi | zbr: just to confirm, you're observing lengthy node allocation times when we're not running at capacity? because if we're running at capacity then container or vm doesn't matter much. at best it shaves something like 1-2% utilization due to not having boot and delete overhead (you have to consider the average runtimes across all our jobs, which is far higher than those of most tox jobs) so | 15:02 |
| mordred | tox -epy36 on nova takes 16 minutes to run | 15:02 |
| fungi | effectively just means a 1-2% boost in available quota | 15:02 |
| clarkb | and I cant type | 15:02 |
| pabelanger | Hmm, is http://grafana.openstack.org/dashboard/db/zuul-status not loading properly for anybody else? | 15:04 |
| *** igordc has joined #openstack-infra | 15:04 | |
| pabelanger | the datasource I mean | 15:04 |
| fungi | zbr: if "use containers" is slang for running some jobs on smaller amounts of server resources (not dedicating 8gb ram, 80gb disk and 4-8vcpus to every build), note that's not really a property unique to container solutions, we could do that with virtual machines too as indicated | 15:04 |
| *** ijw has quit IRC | 15:04 | |
| zbr | ok, so shortly benefits would likely so small that it would not payoff for the complexity of having to deal with both vm and containers. | 15:04 |
| mordred | zbr: there's definitely some interesting possibilities with containers - but from the investigation we've done so far they do not actually end up being an immediate/quick/easy win in our case, due to overall load, the source of our compute resources, and ... | 15:04 |
| mordred | zbr: yeah - what you said | 15:05 |
| zbr | probably this is what made travis stop using containers in december | 15:05 |
| fungi | i wouldn't be surprised | 15:06 |
| mordred | yeah. containers at scale aren't necessarily the win people think they are | 15:06 |
| zbr | orchestrating them is not an easy job | 15:06 |
| mordred | they're great for many things, of course | 15:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Update datasource to https for graphite.o.o https://review.openstack.org/652097 | 15:06 |
| mordred | and I've been enjoying our adoption of them | 15:06 |
| clarkb | pabelanger: possible fallout from httpsing graphite? | 15:06 |
| pabelanger | clarkb: fungi: mordred: ianw: ^I think that will fix the datasource issue | 15:06 |
| pabelanger | did we drop http support? | 15:06 |
| fungi | graphite.opendev.org redirects http to https | 15:07 |
| fungi | so maybe the redirect isn't being followed? | 15:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Update datasource to https for graphite.o.o https://review.openstack.org/652097 | 15:07 |
| pabelanger | Oh, it is graphite.opendev.org now | 15:07 |
| pabelanger | let me use that | 15:08 |
| clarkb | ya since we ant sanely LE openstack domains | 15:08 |
| zbr | pabelanger: yep, wrong certificate. -1 to that. | 15:08 |
| fungi | i bet we missed redirecting graphite.openstack.org to graphite.opendev.org | 15:08 |
| clarkb | fungi: we didnt | 15:08 |
| clarkb | but it only works when you usehttp | 15:08 |
| clarkb | becausethe cert doesnt have an altname for openstack.org | 15:08 |
| fungi | but it was using http | 15:08 |
| clarkb | ya I think it must not follow redirects | 15:09 |
| *** pgaxatte has quit IRC | 15:09 | |
| fungi | is the http to https redirecthappening before the openstack to opendev redirect? | 15:09 |
| clarkb | and expects 20X | 15:09 |
| pabelanger | oh | 15:09 |
| pabelanger | http redirects to https://graphite01.o.o | 15:09 |
| fungi | for me it's redirecting from openstack to opendev and then http to https | 15:09 |
| clarkb | fungi: I dont think so as the redirect works in my browser | 15:09 |
| fungi | also, as noted, it's apparently redirecting to fqdn not the cname, so that needs fixing i think | 15:10 |
| fungi | anyway, i need to disappear for an early lunch so i'm not mia during the lists.o.o maintenance | 15:10 |
| fungi | back soon | 15:10 |
| *** tosky has quit IRC | 15:11 | |
| clarkb | https://graphite.opendev.org worls as expected | 15:12 |
| clarkb | so updating the grafana source to that is correct | 15:12 |
| clarkb | then we can fix the graphite01.opendev redirect target separately | 15:12 |
| fungi | lists.o.o image creation is still in progress, btw (not surprising) | 15:12 |
| *** tosky has joined #openstack-infra | 15:12 | |
| *** eharney has quit IRC | 15:13 | |
| clarkb | pabelanger: were you going to push a new ps for opendev.org? | 15:15 |
| *** ginopc has quit IRC | 15:15 | |
| pabelanger | clarkb: yah, doing now | 15:16 |
| pabelanger | clarkb: actually | 15:16 |
| pabelanger | I think we might need to do a migration on grafana side | 15:16 |
| pabelanger | let me create a 2nd datasource | 15:16 |
| pabelanger | then grep / replace openstack | 15:16 |
| pabelanger | no, I am wrong | 15:17 |
| pabelanger | I think we can update it in 1 patch | 15:17 |
| clarkb | the keys didnt change | 15:17 |
| clarkb | just the target webserver | 15:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Switch grafana datasource to https://graphite.opendev.org https://review.openstack.org/652097 | 15:18 |
| *** gfidente has quit IRC | 15:19 | |
| *** ccamacho has quit IRC | 15:20 | |
| *** electrofelix has quit IRC | 15:23 | |
| *** e0ne has quit IRC | 15:24 | |
| *** jamesmcarthur has quit IRC | 15:27 | |
| *** tbachman has quit IRC | 15:27 | |
| dmsimard | hmmm, the namespace move is bound to break more than one implementation of "src/git.openstack.org/openstack/<project>" in different zuul jobs | 15:30 |
| clarkb | yes | 15:31 |
| mnaser | is there a way to 'test' the redirects in advance? | 15:31 |
| clarkb | byt not just the namespace move. git.openstack.org will not bevalid there anymore either | 15:31 |
| dmsimard | symlinks ? | 15:31 |
| clarkb | mnaser: yes see the emails :) fungi set up the redirect rules on a webserver allowing anyone to test them | 15:32 |
| dmsimard | clarkb: I guess it's going to be src/opendev.org/<namespace>/<project> | 15:32 |
| mnaser | I worry that potentially openstack-ansible releases might be broken with the move .. so maybe accounting for that before we change things to test them out | 15:32 |
| *** roman_g has quit IRC | 15:32 | |
| clarkb | dmsimard: correct | 15:32 |
| dmsimard | clarkb: the http redirects are fine, the problem is about jobs in zuul running with the assumption that things are in src/git.o.o | 15:32 |
| clarkb | dmsimard: I know, was responding to mnaser | 15:33 |
| dmsimard | http://codesearch.openstack.org/?q=src%2Fgit.openstack.org%2F&i=nope&files=&repos= | 15:33 |
| clarkb | dmsimard: and it will be broken in zuul even if we dont change namespaces | 15:33 |
| clarkb | So now is best opportunity to change namespaces | 15:33 |
| *** HuaChangWang has joined #openstack-infra | 15:33 | |
| dmsimard | yeah I realize there is going to be fallout for sure | 15:33 |
| dmsimard | is there a plan for this particular bit though ? | 15:33 |
| *** roman_g has joined #openstack-infra | 15:34 | |
| clarkb | dmsimard: the plan is to push changes with the new names aiui | 15:34 |
| mnaser | so as I understand it, 'git' access will stop working at some point? | 15:34 |
| clarkb | mnaser: git:// stops when we switch | 15:34 |
| mnaser | well that breaks *every* single release of OSA ever | 15:35 |
| mnaser | I just thought of that | 15:35 |
| dmsimard | mnaser: not using http for git operations ? | 15:35 |
| mnaser | if we point git.openstack.org to opendev.org via DNS.. wouldn't git:// just magically start working again transparently? | 15:35 |
| clarkb | ianw has pushed changes up to replace git:// with https:// | 15:35 |
| clarkb | mnaser: no becayse gitea hasno support for git protocol and it cannot be vhosted | 15:35 |
| mnaser | oh | 15:36 |
| mnaser | looks like we were using http:// | 15:36 |
| mnaser | sorry for the noise | 15:36 |
| *** chandankumar is now known as raukadah | 15:36 | |
| * mnaser is reliefed | 15:36 | |
| cloudnull | https://review.openstack.org/#/c/642049/ - mnaser i started looking at that the other day | 15:36 |
| *** slaweq has quit IRC | 15:37 | |
| *** slaweq_ has joined #openstack-infra | 15:37 | |
| mnaser | cloudnull: ah okay. | 15:37 |
| mnaser | it would be nice if infra can give us a date where we can consider opendev.org 'live' even if the cutover hasn't been changed | 15:38 |
| mnaser | but that might already be in place | 15:38 |
| *** jcoufal has joined #openstack-infra | 15:38 | |
| clarkb | mnaser: I think that us the cutover day? | 15:38 |
| mnaser | clarkb: well cutover day = our stuff breaks, having a bit of time in advance to transition is helpful | 15:38 |
| clarkb | why does it break? | 15:38 |
| mnaser | esp in the OSA case where there are so many references + we have like another 30 repos to update | 15:39 |
| clarkb | we are redirecting http(s) so old urls will continue to work | 15:39 |
| clarkb | you should be able to safely update docs after the transition | 15:39 |
| mordred | yeah. the only thing that should break day-of is git:// - but that's safe to change now | 15:40 |
| mnaser | alright ill try to check our usage of that (If any) | 15:41 |
| clarkb | on the zuul job sideit might be bumpy but that shouldnt be user noticeable and is dev specific | 15:41 |
| clarkb | because zuul uses url based file structure | 15:42 |
| *** slaweq__ has joined #openstack-infra | 15:44 | |
| clarkb | mnaser: can you review https://review.openstack.org/#/c/652097/3 ? | 15:46 |
| *** slaweq_ has quit IRC | 15:46 | |
| mnaser | clarkb: +w | 15:47 |
| clarkb | mnaser: but ya long story short we expect the public consumption of this to mostly just work via our redirects and that can be tested today | 15:47 |
| mnaser | clarkb: yeah, that seems like an ok approach | 15:47 |
| mnaser | I've had to deal worth.. like post-github acquisition | 15:48 |
| clarkb | there are some notable expceptions. git:// will stop working and you have to use http(s). And in zuul jobs we refer to projects by their canonical name which will be changing | 15:48 |
| clarkb | so CI may have a few bumps while we update those names | 15:48 |
| mnaser | when a bunch of folks moved their repos off of github | 15:48 |
| mnaser | stuff like spice-html5 decided it didn't want to live in GitHub and rm'd it there | 15:48 |
| clarkb | I intend on being around the entire week after our transition to help people debug (and possibly even force merge fixes for) the zuul canonical name fallout | 15:49 |
| clarkb | though aiui our plan is to generate many of those changes automatically | 15:49 |
| clarkb | so more a matter of merging stuff than debugging and figuring out fixes I hope | 15:49 |
| *** yamamoto has quit IRC | 15:50 | |
| dmsimard | It looks like puppet is considering the EOL deprecation of py27 warning as a failure? http://logs.openstack.org/14/650914/1/check/legacy-logstash-filters-ubuntu-trusty/d515d48/job-output.txt.gz#_2019-04-12_15_30_03_623711 | 15:50 |
| *** yamamoto has joined #openstack-infra | 15:50 | |
| dmsimard | unrelated failure when trying to land the gerrit replication patch for ara | 15:50 |
| *** slaweq__ has quit IRC | 15:51 | |
| clarkb | dmsimard: the error is one line below that | 15:52 |
| dmsimard | I was suspecting that might be the case | 15:52 |
| clarkb | I'm guessing that 'lockfile' just made a release? | 15:52 |
| clarkb | so now the old version we had installed wants to updated? | 15:52 |
| clarkb | pypi says no that isn't hte case | 15:53 |
| dmsimard | lockfile was last updated in 2015 :D | 15:53 |
| clarkb | not sure why that seems to have just happened now. But it is the lockfile issue not the deprecation warning that caused the failure | 15:53 |
| *** yamamoto has quit IRC | 15:53 | |
| dmsimard | I can try a recheck, this failure is from a recheck from an unrelated gate failure :p | 15:53 |
| *** yamamoto has joined #openstack-infra | 15:54 | |
| *** derekh has quit IRC | 15:58 | |
| *** jamesmcarthur has joined #openstack-infra | 15:58 | |
| clarkb | ya a recheck showing if it is consistent or not would probably be worthwhile | 15:58 |
| pabelanger | clarkb: +3 on https://review.openstack.org/651389 | 16:00 |
| * clarkb finds breakfast before the list server upgrade | 16:00 | |
| pabelanger | looking at system-config one now | 16:00 |
| clarkb | pabelanger: for the system-config one we may want to give it extra thought considering matrix current struggles | 16:01 |
| clarkb | but 651389 should be totally safe :) | 16:01 |
| *** psachin has joined #openstack-infra | 16:01 | |
| pabelanger | clarkb: yah, we should first limit zuul_console on firewall to zuul-executors, not to leak that port | 16:02 |
| *** jamesmcarthur has quit IRC | 16:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch grafana datasource to https://graphite.opendev.org https://review.openstack.org/652097 | 16:05 |
| *** janki has quit IRC | 16:05 | |
| *** finucannot is now known as stephenfin | 16:08 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/ https://review.openstack.org/604404 | 16:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove zuul_reconfigure CD job https://review.openstack.org/651389 | 16:08 |
| *** lucasagomes has quit IRC | 16:08 | |
| pabelanger | clarkb: I've been thinking for ansible side, maybe adding extra layer of email when somebody logins in via SSH, like we do with sudo failures. However, that would likely get chatty with ansible sshing into servers ever hour | 16:10 |
| *** iurygregory has quit IRC | 16:10 | |
| pabelanger | maybe better to expose to statsd | 16:10 |
| pabelanger | and grafana dashboard | 16:10 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 16:12 | |
| *** dtantsur is now known as dtantsur|afk | 16:12 | |
| *** HuaChangWang has quit IRC | 16:13 | |
| *** eharney has joined #openstack-infra | 16:15 | |
| *** gyee has joined #openstack-infra | 16:15 | |
| clarkb | I'm putting lists.o.o in the ansible emergency file list now | 16:15 |
| *** ykarel is now known as ykarel|away | 16:16 | |
| fungi | okay, i'm back. lists.o.o snapshot is still in progress | 16:16 |
| *** jpich has quit IRC | 16:18 | |
| clarkb | bah I planned list upgrade during kids soccer practice so I would have peace and quiet, well soccer just cancelled on us :) | 16:18 |
| clarkb | fwiw after the lists upgrade I think I'm gonna go haev a stein in honor of stein :) I put that off ealier in the weke because I knew we had stuff to do after release | 16:20 |
| fungi | and yeah, i'm currently working on the script to "fix" hostnames and namespaces in git review and zuul configuration in every repository we host, which we'll be pushing directly into those repositories on gerrit's filesystem while it's offline. not only are those changes bypassing zuul, they're bypassing gerrit too (though i'll be testing the script at scale and we'll all be very closely reviewing | 16:20 |
| fungi | the script and results over the course of next week) | 16:20 |
| dmsimard | clarkb: dadops hug :p | 16:20 |
| anteaya | clarkb: sorry I got working on other things and didn't get back to irc yesterday | 16:21 |
| *** rh-jelabarre has quit IRC | 16:21 | |
| anteaya | clarkb: thanks for posting that update | 16:21 |
| clarkb | anteaya: no worries | 16:21 |
| anteaya | clarkb: I will say that I was looking for something with the subject line "gerrit will be offline April 19th' | 16:22 |
| anteaya | for the folks that don't pay attention to anything less than that | 16:22 |
| anteaya | mind if I post that kind of email and point to your update? | 16:22 |
| anteaya | I can draft an email for your approval | 16:22 |
| clarkb | anteaya: ah. I was just going to suggest a response with that clarifcation | 16:23 |
| clarkb | feel free to do so | 16:23 |
| anteaya | I will start an etherpad | 16:23 |
| *** rpittau is now known as rpittau|afk | 16:24 | |
| *** ykarel|away has quit IRC | 16:24 | |
| anteaya | clarkb: https://etherpad.openstack.org/p/RfRCLEMT3J | 16:28 |
| *** jamesmcarthur has joined #openstack-infra | 16:28 | |
| anteaya | and anyone else who wants to offer their view | 16:29 |
| *** jpena has quit IRC | 16:29 | |
| anteaya | I don't know that we had set a start time for the outage | 16:29 |
| clarkb | thinking about specific outage windows. I expect we'll narrow that down during our meeting next week | 16:29 |
| anteaya | so don't refer to a start time for now? | 16:30 |
| anteaya | I will adjust that bit | 16:30 |
| clarkb | maybe for now just say the Gerrit service will be offline at times on the 19th. Specific outage window to be posted as we get closer to the event. | 16:30 |
| clarkb | something like that | 16:30 |
| anteaya | I can do that | 16:30 |
| fungi | yeah, it's entirely possible we have multiple outages throughout the course of the 19th | 16:31 |
| fungi | especially if we need to roll something back or spot something else which needs gerrit offline to fix | 16:31 |
| clarkb | we strategically selected a day that is a holiday in many parts of the world | 16:31 |
| fungi | this is most definitely the largest and most sweeping change we have made to our git repositories in the history of openstack, so i'm not assuming it will be quick nor seamless | 16:32 |
| clarkb | fungi: I think the snapshot is completed, if you want to confirm | 16:32 |
| anteaya | how's that? | 16:32 |
| fungi | clarkb: indeed, it seems to have just changed in the last minute | 16:32 |
| fungi | so i think we're clear on that front | 16:32 |
| anteaya | I'm trying to be both specific and vague | 16:33 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 16:33 | |
| fungi | that seems great to me, thanks anteaya! | 16:33 |
| clarkb | anteaya: I made two minor edits but other than that lgtm | 16:33 |
| anteaya | thank you | 16:33 |
| anteaya | will the comma after the link break the link? | 16:33 |
| fungi | i have a feeling we'll do a status alert too and tell people to find us in here | 16:33 |
| anteaya | I agree the comma is correct grammer | 16:34 |
| fungi | anteaya: if you want to be sure it doesn't, add an ugly space just before the , or wrap the url in <> | 16:34 |
| anteaya | I just never put a character after a link, just leave a whitespace | 16:34 |
| anteaya | I'll whitespace | 16:34 |
| anteaya | clarkb: that okay with you? | 16:35 |
| clarkb | yup | 16:35 |
| anteaya | thanks | 16:35 |
| anteaya | yeah, I figured you'd be statusing as things proceed | 16:35 |
| *** rh-jelabarre has joined #openstack-infra | 16:35 | |
| anteaya | anyone else want to take a peek at the draft email? https://etherpad.openstack.org/p/RfRCLEMT3J | 16:35 |
| clarkb | fungi: for lists I guess my plan then will be to stop services at 1700UTC then start running through the upgrade steps in a root screen | 16:35 |
| fungi | clarkb: sounds great. i'll attach once you indicate the session is initiated | 16:36 |
| fungi | or maybe i'll start the screen session now and you can attach | 16:36 |
| clarkb | wfm | 16:36 |
| anteaya | I'll post to the email list | 16:36 |
| clarkb | fungi: that way your terminal is correctly sized | 16:37 |
| fungi | and then you get to be subjected to my teensy terminal geometry ;) | 16:37 |
| clarkb | ++ | 16:37 |
| clarkb | that ++ was for both things | 16:37 |
| clarkb | the email and the terminal size | 16:37 |
| anteaya | thanks | 16:37 |
| anteaya | a + for each of us | 16:37 |
| fungi | clarkb: screen session running under root on lists.o.o now, it's the only session according to screen -list | 16:37 |
| clarkb | fungi: ok | 16:38 |
| *** tosky has quit IRC | 16:38 | |
| anteaya | oh subject line is good? | 16:38 |
| anteaya | I purposely did not use any tags on it | 16:39 |
| anteaya | is that the best way to reach all the people? | 16:39 |
| clarkb | anteaya: [all] is the suggested global tag now I think | 16:39 |
| anteaya | I can toss one of those in front | 16:39 |
| anteaya | I'll wait for a few minutes and then press send | 16:40 |
| fungi | lgtm | 16:41 |
| clarkb | before I forget | 16:41 |
| *** bgmccollum has quit IRC | 16:41 | |
| clarkb | #success Infra team has deployed its first letsencrypt cert to production | 16:42 |
| openstackstatus | clarkb: Added success to Success page (https://wiki.openstack.org/wiki/Successes) | 16:42 |
| mordred | clarkb: \o/ | 16:42 |
| fungi | i'll try to get the caa and hsts changes for grafana up today, lest i forget | 16:42 |
| *** jamesmcarthur has quit IRC | 16:44 | |
| clarkb | I'm pretty happy with the setup we ended up with. We largely avoid spofs and can create new instances without worrying about dns A/AAAA records beforehand and its all automated. Thank you ianw for putting that together | 16:44 |
| clarkb | still some minor details to sort out like caa records and restarting apache when certs update but I expect we'll get all of that sorted shortly | 16:45 |
| mordred | ++ | 16:47 |
| anteaya | thank you, sending | 16:47 |
| * mordred hands ianw this pie he found | 16:47 | |
| *** Lucas_Gray has joined #openstack-infra | 16:48 | |
| openstackgerrit | Merged openstack-infra/system-config master: Disable gerrit replication to GitHub for ara/ara-infra/ara-web https://review.openstack.org/650914 | 16:49 |
| anteaya | sent: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005067.html | 16:51 |
| *** bgmccollum has joined #openstack-infra | 16:51 | |
| *** jamesmcarthur has joined #openstack-infra | 16:51 | |
| *** cjloader has joined #openstack-infra | 16:53 | |
| openstackgerrit | Jeremy Freudberg proposed openstack-infra/project-config master: Minor/useful tweaks to Sahara repo descriptions https://review.openstack.org/652111 | 16:54 |
| *** tbachman has joined #openstack-infra | 16:57 | |
| clarkb | #status log Pre lists.openstack.org snapshot completed and is named lists.openstack.org-20190412-before-xenial-upgrade | 16:57 |
| *** ykarel has joined #openstack-infra | 16:57 | |
| openstackstatus | clarkb: finished logging | 16:57 |
| clarkb | fungi: ready? I'll stop mailman and exim services if so | 17:00 |
| fungi | yep, let's get this party started | 17:00 |
| *** ramishra has quit IRC | 17:00 | |
| clarkb | ok proceeding | 17:00 |
| fungi | the way i survive on a tiny terminal is by filtering out puppet deprecation notices ;) | 17:01 |
| *** jcoufal has quit IRC | 17:02 | |
| *** e0ne has joined #openstack-infra | 17:03 | |
| *** e0ne has quit IRC | 17:03 | |
| fungi | the service command needs a --wait option | 17:04 |
| fungi | sad that do-release-upgrade clears the terminal | 17:04 |
| *** jamesmcarthur has quit IRC | 17:05 | |
| fungi | then again, the existence of that command is very ubuntu to begin with | 17:05 |
| *** bgmccollum has quit IRC | 17:05 | |
| clarkb | fungi: I've added the iptables rule and reconnected via 1022 in another shell if you want to do that too | 17:05 |
| fungi | done | 17:05 |
| fungi | we're already at line 66 of https://etherpad.openstack.org/p/lists.o.o-trusty-to-xenial by the way | 17:07 |
| * mordred cheers fungi and clarkb on | 17:08 | |
| fungi | wow. we could stand to clean up some kernels on there at some point | 17:08 |
| clarkb | fungi: ya running apt-get autoremove in a cron might be a good idea | 17:09 |
| fungi | i think it's an option for unattended-upgrades | 17:09 |
| clarkb | oh even better | 17:09 |
| fungi | i don't think we need the old sysstat/sar files, no | 17:09 |
| clarkb | nope | 17:09 |
| *** bobh has joined #openstack-infra | 17:10 | |
| clarkb | I've done this a few times at this point and those go away | 17:10 |
| fungi | i take it including puppet would cause it to get started (or fail to start and then abort the upgrade) | 17:13 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add support for smart reconfigurations https://review.openstack.org/652114 | 17:13 |
| clarkb | fungi: that was my concern | 17:13 |
| clarkb | fungi: I actually never tested it when it was listed | 17:14 |
| fungi | since it's not running it doesn't need a restart, so either way that's fine | 17:14 |
| clarkb | exactly | 17:14 |
| fungi | we're waiting between lines 73 and 74 now i thin | 17:16 |
| fungi | k | 17:16 |
| clarkb | I removed it out of caution and doing so made the upgrade work | 17:16 |
| clarkb | oh I notice I had a "we should check exim doesn't need an update" comment in there and I did check :) | 17:16 |
| clarkb | the file updates were to macros not our actual configs | 17:16 |
| clarkb | and on my test server thinsg worked without a puppet run | 17:16 |
| fungi | okay, we're being prompted to replace configs now | 17:21 |
| fungi | and more configs | 17:23 |
| *** e0ne has joined #openstack-infra | 17:23 | |
| fungi | i expect the upcoming question about removing obsolete packages will clear out all those ancient kernels | 17:23 |
| clarkb | fungi: fwiw the process didn't differ much between lstis and afs so will likely apply to wiki as well | 17:24 |
| clarkb | if you end up planning to inplace wiki | 17:24 |
| fungi | at line 79 already | 17:24 |
| fungi | and yeah, i expect so but need to get puppet-mediawiki working with it first | 17:24 |
| fungi | at least for wiki-dev | 17:24 |
| *** adriancz has quit IRC | 17:28 | |
| clarkb | removing obsolete packages now | 17:28 |
| fungi | yep, almost to line 91 | 17:28 |
| fungi | er, 81 | 17:28 |
| *** e0ne has quit IRC | 17:29 | |
| *** jamesmcarthur has joined #openstack-infra | 17:29 | |
| fungi | time to reboot! | 17:30 |
| clarkb | ready? I'm going for it | 17:30 |
| fungi | yep | 17:31 |
| clarkb | fungi: when it comes back do you want to start another screen for me to attach to to do the packaging cleanups? | 17:31 |
| fungi | will do | 17:31 |
| clarkb | I'm in | 17:32 |
| fungi | screen ready, attach at will | 17:32 |
| fungi | looks like exim and mailman started | 17:33 |
| *** _erlon_ has joined #openstack-infra | 17:34 | |
| fungi | removal set lgtm | 17:36 |
| clarkb | k was just going to ask you to confirm | 17:37 |
| clarkb | ok that concludes the documented upgrade process | 17:39 |
| clarkb | fungi: we should probably reboot again to make sure the new kernels all come up as expected (we had trouble with that on some afs servers) | 17:40 |
| clarkb | but after that I think we reenable ansible/puppet and monitor the service | 17:40 |
| fungi | i would definitely reboot again | 17:40 |
| clarkb | fungi: ^ that sound good? if so I will issue a reboot command | 17:40 |
| clarkb | ok rebooting now | 17:40 |
| fungi | i'll start a new screen session as soon as it's back up | 17:40 |
| clarkb | its up | 17:41 |
| fungi | and screen is ready | 17:42 |
| clarkb | exim and mailman are running I checked that before attaching to screen. kernel looks correct | 17:42 |
| clarkb | ready to enable puppet? | 17:42 |
| fungi | yeah, go for it | 17:42 |
| clarkb | ok removed from the emergency file | 17:43 |
| clarkb | someone should send an email to a list :) | 17:43 |
| clarkb | I think puppet is about 30 minutes away | 17:43 |
| fungi | odds are a number of people have sent e-mail to some lists and they're waiting for the server to be up | 17:44 |
| clarkb | do we want to wait or run kick.sh? | 17:44 |
| clarkb | I suppose we should just kick.sh and keep moving | 17:44 |
| fungi | i'm fine waiting/testing in the interim too. either way | 17:44 |
| clarkb | /opt/system-config/tools/kick.sh lists.openstack.org look right to you? | 17:45 |
| clarkb | I'm going to run that now | 17:45 |
| fungi | yeah, that looks right | 17:45 |
| fungi | i'll see if there are any messages to approve in the moderation queues for a couple of lists | 17:46 |
| clarkb | I think it updated snmpd.conf so that should be a keep on future in place upgrades it also updated the sources.list | 17:47 |
| clarkb | otherwise looking good so far | 17:47 |
| fungi | i have one legit message waiting in the moderator queue for openstack-discuss | 17:49 |
| clarkb | Apr 12 17:48:56 lists puppet-user[14704]: (/Stage[main]/Mailman/File[/etc/mailman/mm_cfg.py]/mode) mode changed '0644' to '0444' | 17:49 |
| fungi | going to catch up my e-mail for that list and then approve it | 17:49 |
| clarkb | that is the only change from running puppet | 17:49 |
| clarkb | fungi: I think your grep wasn't working for some reason. I grepped just puppet-user from syslog and saw puppet ran | 17:49 |
| clarkb | I think we are good on the puppet side | 17:49 |
| fungi | yeah, that's strange | 17:49 |
| clarkb | fungi: well too late now with the -f | 17:50 |
| clarkb | in any case I think its happy | 17:50 |
| clarkb | now to double check functionality | 17:50 |
| fungi | strange that `|grep -v deprec` filtered out other lines too | 17:51 |
| clarkb | I can browse the archives of openstack-discuss | 17:51 |
| clarkb | I'll send an email to the infra list | 17:51 |
| *** jamesmcarthur has quit IRC | 17:51 | |
| fungi | waiting for the message i just approved out of moderation to get delivered | 17:52 |
| *** Lucas_Gray has quit IRC | 17:52 | |
| clarkb | my test email hasn't shown up yet either :/ | 17:54 |
| clarkb | I sent it via my mua and not nc though | 17:54 |
| clarkb | so maybe my smtp servers noticed it was offlien earlier and are queuing? | 17:54 |
| fungi | the one i approved just got delivered to me | 17:54 |
| clarkb | fungi: ok I see that one | 17:55 |
| clarkb | so maybe fastmail did notice we were offline and is queuing | 17:55 |
| clarkb | I shall strive to practice more patience | 17:55 |
| fungi | yeah, if someone else from there tried to send a message to one of our lists while it was offline, their mta may be holding all deliveries for the same subdomain to retry together | 17:55 |
| *** diablo_rojo has joined #openstack-infra | 17:56 | |
| clarkb | I think your approval email getting through confirms mailman is generally working as is exim outbound | 17:56 |
| clarkb | we just need to confirm exim inbound ? | 17:56 |
| fungi | yeah | 17:56 |
| anteaya | fungi: the email you approved, what list was it for? | 17:58 |
| fungi | openstack-discuss | 17:58 |
| anteaya | was it a post to a placement ptg thread? | 17:58 |
| clarkb | I think it was the telemetry core one | 17:59 |
| anteaya | a nominating thread? | 17:59 |
| clarkb | http://lists.openstack.org/pipermail/openstack-infra/2019-April/006309.html is my test email | 17:59 |
| clarkb | whihc I've not yet received back to myself yet | 18:00 |
| clarkb | anteaya: ya | 18:00 |
| clarkb | "Asking core privileges on Telemetry projects" | 18:00 |
| anteaya | I have this subject line [telemetry] Nominating Rong Zhu for Telemetry core | 18:01 |
| anteaya | I don't have 'Asking core privvileges on Telemetry projects' | 18:01 |
| anteaya | nor to I have clarkb's test post | 18:01 |
| clarkb | I see files in the mailman queues for openstack vhost | 18:04 |
| clarkb | so it is doing some processing of things at least | 18:04 |
| anteaya | this is the most recent email I have: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005068.html | 18:04 |
| *** dave-mccowan has joined #openstack-infra | 18:05 | |
| clarkb | the errors file for that vhost doesn't show any errors we didn't already have from prior to the upgrade | 18:06 |
| anteaya | the server recieved http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005070.html but thus far I have not | 18:07 |
| fungi | the one i approved was Subject: Re: [tc][telemetry] Asking core privileges on Telemetry projects | 18:07 |
| fungi | From: Trinh Nguyen | 18:07 |
| fungi | Date: Fri, 12 Apr 2019 22:28:50 +0900 | 18:07 |
| clarkb | fungi: any ideas on why the email I sent hasn't gone out the out queue? | 18:07 |
| *** imacdonn_ is now known as imacdonn | 18:08 | |
| fungi | most common cause is if the mailman qrunner for it hasn't started | 18:08 |
| fungi | but it looks like it's been running since reboot | 18:08 |
| anteaya | I have this one in my email client: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005069.html | 18:08 |
| *** irclogbot_0 has quit IRC | 18:08 | |
| clarkb | we have 5 outgoing qrunners which is the number I expect | 18:09 |
| clarkb | /srv/mailman/openstack/qfiles/out/1555091557.751896+3f57e87a69100bb36b456e8836d21b1f7e17e526.pck is the email I sent to test | 18:10 |
| clarkb | so it is sitting in the out queue | 18:10 |
| *** raissa has joined #openstack-infra | 18:10 | |
| *** irclogbot_0 has joined #openstack-infra | 18:10 | |
| fungi | i sent one to openstack-discuss a few minutes ago too | 18:10 |
| clarkb | smtp-failures doesn't show any failures we didn't have already (someone is subscribed with a no longer valid email addr looks like( | 18:11 |
| clarkb | strace implies it is trying to get the mailman lock file | 18:12 |
| clarkb | fungi: `strace -fp 1345` if you want to see that | 18:12 |
| fungi | i wonder if we have a stale lockfile | 18:12 |
| fungi | we could stop all the runners and then see if any lockfiles are left behind | 18:13 |
| anteaya | the server recieved fungi's email but not clarkb's and I have not recieved fungi's yet: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005071.html | 18:13 |
| clarkb | anteaya: the server got mine too | 18:13 |
| clarkb | fungi: ya I'll stop mailman-openstack now | 18:14 |
| anteaya | I don't see it in http://lists.openstack.org/pipermail/openstack-discuss/2019-April/date.html | 18:14 |
| fungi | he sent his to the infra ml | 18:14 |
| anteaya | ah sorry | 18:14 |
| anteaya | yes, there it is: http://lists.openstack.org/pipermail/openstack-infra/2019-April/006309.html | 18:15 |
| clarkb | now patiently waiting for the qrunner to actually stop | 18:15 |
| anteaya | you had shared the link with me earlier and I congitaively disassociated | 18:15 |
| clarkb | it is waiting on the outgoing qrunner to stop which is the one that wasn't able to get the locks (I think that is what it was doing) | 18:16 |
| clarkb | with the other processes stopped maybe that does point to a stale lock | 18:16 |
| clarkb | fungi: is the lockfile suffix the pid? | 18:18 |
| clarkb | beacuse 1537 doesn't exist as a pid | 18:18 |
| clarkb | but appears to be the current lock | 18:18 |
| fungi | yes, we might find it in old logs | 18:18 |
| *** tosky has joined #openstack-infra | 18:18 | |
| *** bgmccollum has joined #openstack-infra | 18:18 | |
| clarkb | Apr 12 17:32:33 2019 (1537) IncomingRunner qrunner started. | 18:19 |
| fungi | there you go | 18:19 |
| fungi | so that was from an earlier reboot | 18:19 |
| clarkb | ya I think so | 18:19 |
| fungi | the one right at the end of do-release-upgrade | 18:19 |
| clarkb | so now how do we safely remove that lock? do we clear it from the mailman.lock pointer file or rm it on disk? | 18:19 |
| fungi | the time matches | 18:19 |
| fungi | remove the file from disk should be fine | 18:20 |
| clarkb | ok I'm doing that now | 18:20 |
| clarkb | I don't see a change | 18:21 |
| clarkb | I think we might have to edit mailman.lock? | 18:21 |
| fungi | worth a shot | 18:22 |
| clarkb | or do I remove mailman.lock | 18:22 |
| clarkb | zuul doesn't have a mailman.lock | 18:22 |
| anteaya | what about rename mailman.lock? | 18:22 |
| clarkb | ya can move it aside | 18:22 |
| clarkb | fungi: ^ you good with that? | 18:24 |
| fungi | yep, it should be fine to remove the mailman.lock file when the qrunner isn't running | 18:24 |
| clarkb | well it is running | 18:24 |
| clarkb | oh but not the one that had the lock | 18:24 |
| clarkb | got it | 18:24 |
| clarkb | ok it stopped the service after that. I am starting it again | 18:25 |
| fungi | it was likely in a busywait for the file to disappear | 18:25 |
| clarkb | out/ contents disappearing | 18:25 |
| fungi | yay! | 18:25 |
| clarkb | which implies we should be getting emails | 18:25 |
| clarkb | yup I got my email | 18:25 |
| clarkb | ok I'll sanity check the other 4 vhosts don't have a similar situation | 18:26 |
| anteaya | clarkb: got yours | 18:26 |
| *** roman_g has quit IRC | 18:26 | |
| clarkb | the other vhosts do not appear to have the same problem | 18:27 |
| clarkb | so I think that may have been it and email seems to be flowing from input through mailman out exim again | 18:27 |
| clarkb | should I respond to my maintenance notice emails now with a note that we are done and expect things to be working? | 18:27 |
| clarkb | that will test all of the vhosts | 18:27 |
| anteaya | sounds reasonable | 18:28 |
| anteaya | I still don't have cdent's email nor fungi's from discuss | 18:28 |
| clarkb | I got them | 18:28 |
| clarkb | anteaya: so may just be a matter of exim getting around to sending them to you | 18:28 |
| anteaya | however my client seems to be occupied with qq spam | 18:28 |
| anteaya | so likely just me | 18:28 |
| fungi | they can take time to show up. openstack-discuss is around 1k subscribers now | 18:28 |
| clarkb | fungi: ^ anythign else you think we should check before I send my "we're done and think it is working" emails? | 18:29 |
| anteaya | yeah, so I'm likely to get them soon | 18:29 |
| fungi | nope, i think we're good. everything's exercised now | 18:29 |
| clarkb | alright working on that now | 18:29 |
| fungi | thanks for driving! | 18:29 |
| anteaya | nice work | 18:29 |
| anteaya | and there is fungi's and cdent's email | 18:31 |
| anteaya | thank you | 18:31 |
| clarkb | it'll have a bit more backlog than usual due to things queuing up while the runner was sad | 18:32 |
| clarkb | but should chew through it and be quicker soon enough | 18:32 |
| *** armax_ has joined #openstack-infra | 18:33 | |
| *** armax has quit IRC | 18:33 | |
| *** armax_ is now known as armax | 18:33 | |
| anteaya | clarkb: sent a reply to your -infra list email | 18:34 |
| anteaya | which hasn't been picked up yet: http://lists.openstack.org/pipermail/openstack-infra/2019-April/date.html | 18:35 |
| clarkb | anteaya: ya the incoming runner is running very quickly right now | 18:36 |
| clarkb | but it has the lock and strace shows it reading and writing so I think it is working just trying to get through its backlog | 18:36 |
| clarkb | we should keep monitoring it to see that it does reach a normal steady state | 18:36 |
| anteaya | very good | 18:36 |
| clarkb | fungi: ^ fyi | 18:37 |
| clarkb | ok in/ count is falling now | 18:38 |
| anteaya | good | 18:38 |
| clarkb | went from 360 to 300 so it seems to be processing | 18:38 |
| anteaya | going in the correct direction, anyhow | 18:38 |
| clarkb | I'm sure a good chunk of that is spam | 18:39 |
| clarkb | which is annoying | 18:39 |
| anteaya | good old qq | 18:39 |
| clarkb | now down to 268 | 18:39 |
| clarkb | and 224 | 18:39 |
| anteaya | I which I could just block everything coming from qq | 18:39 |
| anteaya | wish* | 18:40 |
| clarkb | so ya lets watch it but cautiously saying it still looks fine | 18:40 |
| anteaya | I wonder what email options folks have in china | 18:41 |
| anteaya | I'm curious how it would go over if we considered blocking everything coming from qq to our lists | 18:41 |
| clarkb | I think we haven't done it yet because it is a popular service with peopel and not just spammers | 18:42 |
| anteaya | I wonder if the people have other options? | 18:42 |
| clarkb | fungi: ^ or do we auto reject those now? | 18:42 |
| anteaya | lists with better filtering | 18:42 |
| fungi | i looked into it a while back and counted some number of legitimate posts to our lists from users with @qq.com addresses | 18:42 |
| anteaya | sorry email services with better filtering | 18:42 |
| anteaya | yeah there are some | 18:42 |
| clarkb | considering that things look ok other than the backlog I think I'm going to step out for a bit and then check on the backlog after a break | 18:43 |
| anteaya | I'm just wondering what the email landscape is for those humans | 18:43 |
| fungi | also, yes, the spam isn't actually coming from qq.com, it's spoofed to use @qq.com addresses so that people are *less likely* to block it | 18:43 |
| anteaya | clarkb: enjoy some fresh air | 18:43 |
| clarkb | fungi: I'm just looking at ls -l | wc -l output in the in/ and out/ qfile dirs | 18:43 |
| *** kjackal has quit IRC | 18:43 | |
| anteaya | fungi: oh, I didn't know that | 18:43 |
| *** kjackal has joined #openstack-infra | 18:43 | |
| clarkb | if people are wondering how I was trending that. And now taking a short break | 18:44 |
| anteaya | fooey | 18:44 |
| fungi | qq is a very popular messaging service in mainland china | 18:44 |
| clarkb | thanks for the help everyone | 18:44 |
| anteaya | oh it is messaging not just email | 18:44 |
| clarkb | I learned a lot about mailman doing this too | 18:44 |
| anteaya | clarkb: thanks for shouldering the load | 18:44 |
| anteaya | now the expertise is growing :) | 18:44 |
| fungi | yeah, qq is an integrated messaging platform which spans a number of protocols | 18:45 |
| anteaya | ah, I didn't know that | 18:45 |
| fungi | and acts as a freemail service provider | 18:45 |
| fungi | kinda like yahoo! or gmail | 18:45 |
| anteaya | all I know of it is that the addresses filling my spam folder are majority qq | 18:45 |
| anteaya | I wish they did a better job on spam filtering on their end | 18:46 |
| fungi | well, as i said, it's not actually coming from there | 18:46 |
| fungi | it's spoofed to look like it's coming from there | 18:46 |
| *** psachin has quit IRC | 18:46 | |
| anteaya | do we have any way of viewing the spoof and blocking it on our lists? | 18:47 |
| anteaya | and my reply to clarkb | 18:48 |
| anteaya | 's email has shown up: http://lists.openstack.org/pipermail/openstack-infra/2019-April/date.html | 18:48 |
| fungi | checking against spf records would be one way, though we'd need to implement a more full-featured spam filtering mechansim on the server (i have one in mind, but haven't had time to write a spec) | 18:49 |
| clarkb | this is why dkim/dmarc is so popular | 18:49 |
| clarkb | well one reason | 18:49 |
| *** jamesmcarthur has joined #openstack-infra | 18:50 | |
| anteaya | oh okay so possible and includs substantial work | 18:50 |
| anteaya | includes* | 18:50 |
| clarkb | dmarc/dkim create other problems for mailing lists | 18:50 |
| clarkb | unfortunately | 18:50 |
| fungi | substantial enough i may or may not get around to it, right | 18:50 |
| anteaya | so I have seen, yes | 18:50 |
| anteaya | okay fair enough, I certainly can't take anything off your plate, so won't feel right trying to add to it | 18:51 |
| anteaya | the qq spam does get redirected to my spam folder, there is just so much of it | 18:52 |
| fungi | i mean, i have a similar setup on my mailservers, using exim filters to score and reject messages over a certain threshold rather than queuing them, but extensive discussion would be required to determine if such a solution would be considered suitable for our listserv | 18:52 |
| anteaya | ah, we could get lost in the extensive discussion | 18:53 |
| anteaya | and never crawl out of the hole | 18:53 |
| fungi | i already am | 18:53 |
| anteaya | ha ha ha | 18:53 |
| *** apetrich has quit IRC | 19:00 | |
| *** bobh has quit IRC | 19:02 | |
| dmsimard | may I add myself to the openstack github org to initate the ara repository transfers ? | 19:08 |
| *** whoami-rajat has quit IRC | 19:09 | |
| dmsimard | thought I was in it, I guess not | 19:09 |
| fungi | dmsimard: it may be better to test adding our shared account temporarily to the ara org, since that's how we're going to ask other folks who aren't infra sysadmins to do it? | 19:10 |
| dmsimard | sure, what organization is it ? | 19:10 |
| dmsimard | rt | 19:10 |
| dmsimard | er, account* | 19:10 |
| fungi | it's documented in our credentials list | 19:11 |
| fungi | along with how to operate the one-time password generator to log into it | 19:11 |
| dmsimard | ack, will look | 19:11 |
| *** raissa has quit IRC | 19:14 | |
| clarkb | openstack qfiles dirs are all empty | 19:15 |
| clarkb | I think that means we are caught up | 19:15 |
| clarkb | and I see at least one email since I left so still looking good | 19:15 |
| *** raissa has joined #openstack-infra | 19:15 | |
| *** raissa has quit IRC | 19:15 | |
| *** raissa has joined #openstack-infra | 19:16 | |
| clarkb | and now I can go have celebratory lunch for stein release :) looks like it will be margaritas instead of helles. | 19:17 |
| mordred | clarkb: margaritas are always the right choice | 19:17 |
| clarkb | #status log Upgraded lists.openstack.org from trusty to xenial | 19:17 |
| openstackstatus | clarkb: finished logging | 19:17 |
| *** raissa has joined #openstack-infra | 19:18 | |
| *** raissa has quit IRC | 19:19 | |
| clarkb | fungi: I've hopped off that screen session if you want to close it feel free | 19:19 |
| *** ijw has joined #openstack-infra | 19:19 | |
| clarkb | mordred: "Margarita Factory" opened up down the street from us last week. Its a vancouver restaurant expanding from washington into oregon I guess. Gonna go full burbs mode margaritas | 19:19 |
| *** raissa has joined #openstack-infra | 19:19 | |
| *** raissa has quit IRC | 19:19 | |
| fungi | thanks, closed down the screen session now | 19:20 |
| fungi | clarkb: a stein full of margaritas would work | 19:20 |
| *** eernst has joined #openstack-infra | 19:21 | |
| fungi | though i had helles on wednesday. the local brewery down the road made a nice (if strong) attempt at a helles (well, maibock) | 19:22 |
| * cmurphy misses good helles | 19:23 | |
| *** eernst has quit IRC | 19:23 | |
| fungi | there's a bit of a revival of american microbreweries doing their takes on traditional german recipes | 19:24 |
| clarkb | ninkasi has one that is ok | 19:24 |
| *** eernst has joined #openstack-infra | 19:27 | |
| *** bgmccollum has quit IRC | 19:30 | |
| *** bgmccollum has joined #openstack-infra | 19:31 | |
| *** eernst has quit IRC | 19:32 | |
| *** kjackal has quit IRC | 19:32 | |
| *** kjackal has joined #openstack-infra | 19:33 | |
| *** eernst has joined #openstack-infra | 19:34 | |
| *** eernst has quit IRC | 19:38 | |
| *** eernst has joined #openstack-infra | 19:40 | |
| *** jamesmcarthur has quit IRC | 19:42 | |
| *** eernst has quit IRC | 19:45 | |
| *** eernst has joined #openstack-infra | 19:46 | |
| *** eernst has quit IRC | 19:50 | |
| *** eernst has joined #openstack-infra | 19:53 | |
| *** ykarel is now known as ykarel|away | 19:54 | |
| *** eernst has quit IRC | 19:57 | |
| *** eernst has joined #openstack-infra | 19:59 | |
| dmsimard | fungi, clarkb: tried the shared account for the transfer but the authentication requires a token when using 2fa: https://developer.github.com/v3/auth/#working-with-two-factor-authentication | 20:02 |
| dmsimard | should I create a token for the transfers and delete the token ? should we have a single token ? | 20:02 |
| *** eernst has quit IRC | 20:02 | |
| clarkb | so to auth for the transfer itself you need a new api token? | 20:03 |
| dmsimard | the token becomes the password | 20:03 |
| fungi | dmsimard: did you read the instructions in the credentials list? it tells you how to use the otp generator on bridge.o.o | 20:04 |
| fungi | i've used those instructions (thanks ianw!) a few times already and they've been working for me | 20:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove apport from servers https://review.openstack.org/652143 | 20:04 |
| dmsimard | fungi: yes, I am able to authenticate on the web UI but when running a dry-run with the script, it bails out: http://paste.openstack.org/show/749264/ | 20:05 |
| fungi | aha | 20:05 |
| *** eernst has joined #openstack-infra | 20:05 | |
| fungi | so to use a 2fa-enabled account you need to generate a code and add that to the api call, i guess? | 20:06 |
| dmsimard | oh, actually, in the docs I linked, there is a way to use OTP during the authentication | 20:06 |
| dmsimard | e.g, curl --request POST --url https://api.github.com/authorizations --header 'authorization: Basic PASSWORD' --header 'content-type: application/json' --header 'x-github-otp: OTP' --data '{"scopes": ["public_repo"], "note": "test"}' | 20:07 |
| dmsimard | I'm not sure how that works in practice since the otp changes over time ? | 20:07 |
| *** diablo_rojo has quit IRC | 20:08 | |
| mordred | fungi, clarkb, dmsimard, Shrews: ^^ patch a few lines ago - Shrews and I have been trying to figure out why zuul-preview is hanging ... | 20:08 |
| dmsimard | fungi: when you generate an API token, you authenticate against the API with the username and the token in-lieu of the password | 20:09 |
| mordred | in so doing have discovered that core files are configured to be piped in to apport | 20:09 |
| openstackgerrit | Merged openstack-infra/project-config master: Retire openstack-ansible unused roles https://review.openstack.org/650477 | 20:09 |
| mordred | which seems like a thing that provides negative value | 20:09 |
| mordred | I'm not 100% sure what is setting the /proc setting - but I'm sort of hoping uninstalling apport will set the pattern back to "core" | 20:09 |
| *** eernst has quit IRC | 20:09 | |
| mordred | does anyone have an issue with me uninstalling apport by hand on zp01 to verify if removing the package will revert the proc setting? | 20:10 |
| Shrews | mordred: it's not doing what it's supposed to anyway, i say go for it | 20:10 |
| mordred | of course, doing that makes it also want to remove ubuntu-server package | 20:11 |
| Shrews | of course | 20:11 |
| dmsimard | and removing ubuntu-server does what ? removes everything ? :p | 20:12 |
| mordred | k. uninstalling isn't needed - we can just stop the service and it causes the proc entry to properly revert | 20:12 |
| mordred | patch change coming | 20:12 |
| *** tjgresha_nope has quit IRC | 20:12 | |
| *** tjgresha has joined #openstack-infra | 20:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Disable apport on servers https://review.openstack.org/652143 | 20:14 |
| mordred | dmsimard, Shrews, fungi: ^^ | 20:14 |
| dmsimard | fungi, clarkb: the shared account is member of openstack but not owner and so doesn't have the necessary privileges https://github.com/orgs/openstack/people | 20:15 |
| dmsimard | I'll fix it | 20:15 |
| fungi | yeah, ever since gh upended their organization rbac model i haven't been able to make heads or tails of it | 20:16 |
| *** eernst has joined #openstack-infra | 20:16 | |
| *** ykarel|away has quit IRC | 20:17 | |
| *** ijw has quit IRC | 20:18 | |
| *** ijw has joined #openstack-infra | 20:19 | |
| *** eernst has quit IRC | 20:22 | |
| *** eernst has joined #openstack-infra | 20:23 | |
| *** ijw has quit IRC | 20:23 | |
| *** eernst has quit IRC | 20:25 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add script to automate GitHub organization transfers https://review.openstack.org/644937 | 20:26 |
| *** pcaruana has quit IRC | 20:26 | |
| dmsimard | transfers are done, there was a small bug that I fixed in ^ | 20:26 |
| dmsimard | I did it using a personal access token (which I've deleted already) | 20:26 |
| dmsimard | I'll add a note in the passwords file | 20:28 |
| * dmsimard googles how to save and exit emacs | 20:31 | |
| clarkb | ctrl x ctrl c iirc | 20:36 |
| *** zbr has quit IRC | 20:37 | |
| *** ijw has joined #openstack-infra | 20:39 | |
| *** ijw has quit IRC | 20:44 | |
| dmsimard | yeah, I'm just a vim person haha | 20:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Use splice instead of erase/push_front https://review.openstack.org/652148 | 20:45 |
| *** fried_rice is now known as efried_schoolrun | 20:58 | |
| *** kgiusti has left #openstack-infra | 20:59 | |
| fungi | gonna go find some dinner, but back in a little while | 21:04 |
| clarkb | I'm back from celebrating the stein release :) | 21:12 |
| clarkb | feeling extremely useless. I suppose I should review mordred and dmsimard's changes | 21:12 |
| clarkb | I still see email going through mailman so that looks good | 21:13 |
| *** jtomasek has quit IRC | 21:14 | |
| *** ijw has joined #openstack-infra | 21:16 | |
| *** jamesmcarthur has joined #openstack-infra | 21:19 | |
| clarkb | dmsimard: on https://review.openstack.org/#/c/644937 what did we end up goign with for the auth? a new token? or somethign else/ I don't see us documenting whatever the choice was and am thinking that would be good | 21:23 |
| *** jamesmcarthur has quit IRC | 21:23 | |
| mordred | clarkb: oh wow - the testing really doesn't like that apport change | 21:27 |
| clarkb | mordred: it failed centos too though | 21:28 |
| mordred | well, it CERTAINLY shouldn't have failed centos | 21:28 |
| clarkb | http://logs.openstack.org/43/652143/2/check/puppet-beaker-rspec-puppet-4-infra-system-config/473d0f9/job-output.txt.gz#_2019-04-12_20_24_09_919149 | 21:28 |
| clarkb | that is why it failed oncentos and on our images I think | 21:28 |
| clarkb | if the service isn't there (like on our minimal dib images) it fails | 21:28 |
| *** efried_schoolrun is now known as efried | 21:29 | |
| mordred | hrm | 21:29 |
| *** diablo_rojo has joined #openstack-infra | 21:29 | |
| *** betherly has quit IRC | 21:45 | |
| *** rosmaita has left #openstack-infra | 21:46 | |
| *** rlandy|ruck has quit IRC | 21:49 | |
| dmsimard | clarkb: I added a note in the passwords file but I can add it in the script too | 21:50 |
| dmsimard | I ended up creating abd t | 21:50 |
| dmsimard | and deleting a token | 21:50 |
| clarkb | got it | 21:51 |
| *** calebb has joined #openstack-infra | 22:00 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add script to automate GitHub organization transfers https://review.openstack.org/644937 | 22:01 |
| *** EvilienM is now known as EmilienM | 22:13 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Double checking pre merge behavior on trusted repo https://review.openstack.org/652159 | 22:14 |
| clarkb | mordred: pabelanger ^ sanity check on our end | 22:14 |
| *** Lucas_Gray has joined #openstack-infra | 22:15 | |
| pabelanger | clarkb: if you remove all, and leave noop, is that different (i wouldn't expect it) | 22:15 |
| clarkb | seems like it isn't running tox-pep8 in that change | 22:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Double checking pre merge behavior on trusted repo https://review.openstack.org/652159 | 22:16 |
| clarkb | pabelanger: ^ checking just noop in that ps | 22:16 |
| clarkb | still running openstack-zuul-jobs-linters | 22:17 |
| clarkb | so I don't think this is a general problem. Perhaps config or local code related | 22:18 |
| pabelanger | clarkb: I think I have a theory, can I PM? | 22:18 |
| *** calebb has quit IRC | 22:20 | |
| clarkb | sure | 22:20 |
| *** ijw has quit IRC | 22:28 | |
| *** ijw has joined #openstack-infra | 22:29 | |
| *** ijw has quit IRC | 22:34 | |
| *** ijw has joined #openstack-infra | 22:49 | |
| *** dave-mccowan has quit IRC | 23:00 | |
| *** tosky has quit IRC | 23:11 | |
| *** _erlon_ has quit IRC | 23:11 | |
| *** diablo_rojo has quit IRC | 23:24 | |
| *** nicolasbock has quit IRC | 23:30 | |
| fungi | the suspense is killing me | 23:35 |
| *** igordc has quit IRC | 23:38 | |
| *** aaronsheffield has quit IRC | 23:40 | |
| *** sthussey has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!