opendevreview | Ke Niu proposed openstack/constellations master: setup.cfg: Replace dashes with underscores https://review.opendev.org/c/openstack/constellations/+/885214 | 01:55 |
---|---|---|
frickler | knikolla: o.k., what do we need to do in order to proceed? some formal resolution? or just you telling me "go ahead"? | 04:56 |
fungi | in the past i've just rough taken consensus from tc members or approval of the tc chair as sufficient | 11:54 |
knikolla | frickler: you have my go ahead | 13:48 |
opendevreview | Merged openstack/governance master: Add cinder-nfs charm to OpenStack charms https://review.opendev.org/c/openstack/governance/+/879502 | 15:10 |
opendevreview | Merged openstack/governance master: Clarify expectations on keeping Python versions https://review.opendev.org/c/openstack/governance/+/882154 | 15:13 |
opendevreview | Merged openstack/governance master: Add openstack-helm PTLs missing irc nick name https://review.opendev.org/c/openstack/governance/+/884757 | 15:18 |
frickler | knikolla: thx, I've started with the first batch, will continue later or tomorrow | 16:35 |
frickler | sadly I cannot update the topic on this stack after merging, will have to take more care for the next ones https://review.opendev.org/q/I6cfa0f392b10edf5e086b130606ba079a651c2a1 | 16:35 |
frickler | 10 config errors down, 205 to go | 16:36 |
knikolla | tc-members: reminder that meeting is in ~1 hour and this is the one on video call. | 16:55 |
knikolla | frickler: awesome, thanks! | 16:55 |
spotz_ | knikolla: Thanks for the video reminder! | 17:20 |
opendevreview | Brian Rosmaita proposed openstack/governance master: Upstream investment: translations infrastructure engineer https://review.opendev.org/c/openstack/governance/+/885379 | 17:52 |
jamespage | knikolla: yep thanks for the reminder about the video call | 17:54 |
jamespage | awesome - zoom still seems to be working today | 17:56 |
knikolla | #startmeeting tc | 17:59 |
opendevmeet | Meeting started Tue Jun 6 17:59:51 2023 UTC and is due to finish in 60 minutes. The chair is knikolla. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:59 |
opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:59 |
opendevmeet | The meeting name has been set to 'tc' | 17:59 |
knikolla | #link https://us06web.zoom.us/j/87108541765?pwd=emlXVXg4QUxrUTlLNDZ2TTllWUM3Zz09 | 18:00 |
knikolla | #topic Roll call | 18:00 |
rosmaita | o/ | 18:00 |
gmann | o/ | 18:00 |
spotz_ | o/ logging in | 18:00 |
knikolla | o/ | 18:00 |
jamespage | o/ | 18:10 |
JayF | o/ omw | 18:10 |
knikolla | #topic Follow up on past action items | 18:12 |
knikolla | #link https://etherpad.opendev.org/p/tc-2023.2-tracker | 18:12 |
knikolla | #action knikolla To fix link redirect to release from docs | 18:13 |
knikolla | #topic Open Infra Summit and PTG in Vancouver, next week | 18:13 |
knikolla | #topic Gate health check | 18:21 |
gmann | this is something we need to spread or attract more members to do this #link https://docs.openstack.org/project-team-guide/testing.html#project-gating | 18:33 |
fungi | being "the openstack job fixer" has been the #1 source of burnout for long-term members of our community | 18:35 |
gmann | agree | 18:35 |
clarkb | I'm not able to dial in but this is sort of why I wondered if a project management type role might be somethng to investigate | 18:35 |
clarkb | they could help coordinate efforts across teams/groups ratherthan doing it all themselves and maybe that would help spread out the burnout factor | 18:36 |
fungi | if we take away the ability for random contributors to make recheck comments, they'll just rebase or push trivial new revisions instead | 18:37 |
fungi | laziness finds a way | 18:38 |
JayF | clarkb: nobody has a stick. TC doesn't. PTL doesn't. We can have someone who says "you, go fix the gate", but we have no method whatsoever for making those people -- or in many cases the bosses who set the unreasonable deadlines they are trying to meet -- dedicate their time where we want. | 18:38 |
JayF | This is where I struggle, it seems like in many cases saying "work on the commons instead of $interestingThing" may lead to loss of that contributors' effort rather than redirection of it | 18:39 |
clarkb | JayF: sure, but I don't think a stick is necessary. If such a role produces benefits people will go along with it. | 18:39 |
JayF | clarkb: IMHO, the root cause of the problem is that the benefits of contributing to the commons is mostly invisible, or difficult to track | 18:40 |
clarkb | its not about forcing people to do anything butkeeping track of where the issues are and pointing people in a common direction towards betterness | 18:40 |
JayF | My hypothesis is that such an effort would lead to the same cabal of people going in the pointed-direction while the same disengaged group continues working in their own direction | 18:40 |
clarkb | The problem today is that no one or no group exists for that so when eg ironic tests start timing out before anyone even does the most basic debugging they show up asking infra to fix it | 18:42 |
clarkb | essentially the infra team is acting as a poor proxy for this role/task today and I think there is a more efficient and productive way to do it | 18:42 |
knikolla | #link https://etherpad.opendev.org/p/tc-leaders-interaction-2023-vancouver | 18:43 |
fungi | clarkb: it's because they think we have a magic "make it work" switch | 18:43 |
gmann | thanks | 18:43 |
knikolla | #topic Broken docs due to inconsistent release naming | 18:45 |
knikolla | #topic Open Discussion and Reviews | 18:45 |
rosmaita | #link https://review.opendev.org/c/openstack/governance/+/885379 | 18:47 |
fungi | #openstack-i18n | 18:48 |
fungi | come join us! | 18:48 |
fungi | tonyb also replied to the cinder eol thread a few minutes ago | 18:49 |
fungi | again i'll float the proposal that we rename "extended maintenance" to "community maintenance" or maybe just "unmaintained" (and get rid of the separate distinction for the unmaintained phase we have now) | 18:51 |
fungi | but like i said in my reply, if nobody from the distros is actually proposing patches and volunteering to be the reviewers on those branches, i don't think we've succeeded in what we set out to do with them anyway | 18:52 |
fungi | they're the first to object when we say we want to eol a branch, but i almost never see them propose or review changes for them | 18:56 |
rosmaita | fungi: i think you're right about that | 19:03 |
rosmaita | does anyone know if there's a list of forum etherpads, or even just a URL-schema most people are using? | 19:04 |
tonyb | @diablo_rojo is collecting one ATM see recent list post | 19:05 |
gmann | rosmaita: this is page we might have but not currectly | 19:05 |
gmann | https://wiki.openstack.org/wiki/Forum/Vanvouver2023 | 19:05 |
rosmaita | fungi: also, i imagine everyone you know has pointed this out to you: https://thehill.com/policy/equilibrium-sustainability/4034986-fungi-may-offer-jaw-dropping-solution-to-climate-change/ | 19:05 |
knikolla | #endmeeting tc | 19:05 |
opendevmeet | Meeting ended Tue Jun 6 19:05:44 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:05 |
opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2023/tc.2023-06-06-17.59.html | 19:05 |
opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2023/tc.2023-06-06-17.59.txt | 19:05 |
opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2023/tc.2023-06-06-17.59.log.html | 19:05 |
knikolla | #endmeeting | 19:05 |
tonyb | One thing that confuses me is that the state of the branches down to train wouldn't have changed the amount of work for the 2023-2088 CVE | 19:06 |
rosmaita | tonyb: how do you mean? | 19:06 |
fungi | rosmaita: yes, we're doing great, slimy things for the environment | 19:06 |
tonyb | rosmaita: train is the base for RHOS-16, so even if the upstream branch was EOL, RH (you and co) would staill have backported it to train | 19:07 |
fungi | apparently the backport rh used wasn't something that would have been considered an acceptable stable branch change, as i heard it explained | 19:08 |
tonyb | Hmm okay. That's a little surprising | 19:09 |
rosmaita | basically, what fungi said | 19:09 |
fungi | having seen some of the things backported in rhel over the years, i don't find that in the least surprising ;) | 19:10 |
tonyb | for the recorded I'm totally fine with updating the current stable policy as we've largely shown that the existing one doesn't work. I just want to make sure we do it right for all projects | 19:10 |
fungi | where we likely need to strike a balance is between letting projects who *want* to support stable branches for longer do that, while not having branches open indefinitely because projects are so under-staffed there's nobody to ask to have them made eol | 19:12 |
knikolla | perhaps an opt-in vs opt-out for em? | 19:12 |
fungi | that's a huge problem these days, and the main reason most projects have their branches in em until someone mass-eol's ancient branches across the entirety of openstack | 19:13 |
fungi | they're not being used in most cases, but the only driver to close them out is bulk cleanup of last resort | 19:13 |
JayF | I'm going to be honest, regardless of branches supported and/or for how long, I don't think the EM distinction holds any value now whatsoever | 19:13 |
fungi | JayF: would it make any difference if we called those "unmaintained branches?" | 19:14 |
JayF | fungi: IMO, it should be more binary: maintained or EOL'd | 19:14 |
knikolla | fungi: as a minimum I think that would be a net improvement. | 19:14 |
fungi | or is it that having the branches open at all is a problem regardless of how we try to say we're not officially taking responsibility for them? | 19:14 |
JayF | fungi: because I don't think it's ever likely, in reality, that the promise of EM will be found in reality; primarily because as long as the people who developed $oldVersion are still around, it's going to be difficult to get them to ignore it as it rots | 19:14 |
JayF | fungi: so I'm saying, branches stay open/maintained as long as the projects want them to /are possible (with maybe even a default-EOL date to avoid the scenario you lay out) | 19:15 |
JayF | but that actual stage, the "supported but not by everyone" is just ... not that way in reality, and we're better off being able to keep performing releases, and just call it 'maintained' until it's not | 19:15 |
tonyb | JayF: it used to be binary and then operators and vendors wanted .... what we have now (more or less) so we built it with the expectation that " ... and they will come" only they didnt really | 19:16 |
fungi | so the suggestion is switch the coordinated em transition to be the new coordinated eol point, but allow projects to request some repositories have those branches held open for another cycle (and then renew that request each cycle if they're still committed to maintaining them)? | 19:17 |
JayF | fungi: something that general shape, yeah. I'm not sure that's exactly it, but that is the direction I'm thinking | 19:17 |
fungi | cool | 19:18 |
JayF | fungi: I think the complexities come with cross-project interactions ... if Ironic EOL's version X, and nova doesn't, how does nova continue to test that ironic integration | 19:18 |
tonyb | Yeah. | 19:18 |
JayF | those are kinda the problems that worry me about that shape, but AFAICT they exist now | 19:18 |
fungi | jobs need to be able to switch to installing releases or checking out tags instead of branches | 19:18 |
tonyb | should we see if there is room for a lareger discussion at the forum/ptg? rather than hijack the cinder room/table? | 19:18 |
dansmith | well, it's worse if a library has eoled and nova depends on a change in that library to fix itself | 19:19 |
fungi | devstack already does this for things that eol "early" while others are still in em | 19:19 |
tonyb | dansmith: Yeah os-brick goes EOL and then nova cant't backport a fix to it :/ | 19:19 |
fungi | tonyb: i do think this is something we've traditionally discussed at the forum and its predecessors, yes | 19:19 |
dansmith | right | 19:19 |
JayF | dansmith: yeah, that's a good ID of another edge that would be concerning for sure... but all these problems already exist, by evidenced by our numerous examples :D | 19:19 |
gmann | IMO, keeping EM just for backport and remove the integration tests at the start itself is way forward otherwise we keep fixing gate even we do not need to | 19:19 |
fungi | cinder's decision is their own of course, and allowed within the current policy, but it hints at the probability that the whole model needs a rethink | 19:20 |
JayF | I just don't like it when docs lie, and right now our docs about what EM is, and how we manage it, is a complete whopper of a story LOL | 19:20 |
dansmith | I'm just saying, it's not only about cases we can't test because some other service doesn't have a branch.. it would require a backport to a library that is eol *and* a release *and* a requirements change | 19:20 |
gmann | but how EM of lib work? as they cannot be releases, no update in constraints | 19:21 |
fungi | yes, i do think this is overall smoother when everything goes eol at the same time (like it did before em came about) | 19:21 |
dansmith | exactly | 19:21 |
JayF | gmann: this is what I'm saying, EM should no longer exist | 19:21 |
JayF | gmann: we would continue on a model where as long as a project is supported, it can release | 19:21 |
JayF | if something you depend on needs an upgrade to fix your bug, well, you should've been there to keep your dependency alive if your project cares | 19:22 |
gmann | JayF: IMO we can keep iit and remove the gate from start itself. a branch can have backport but test it before use. | 19:22 |
fungi | though what we generally said was that if a project (library is a vague distinction given the interconnectedness of our projects anyway) goes eol and then something needs a fix in it, just consider that impossible and eol the thing that needed it or stop running that test if possible or whatever the easiest and lowest cost solution is | 19:22 |
gmann | until we keep test (as long as they work) it create difficulties | 19:22 |
fungi | the main problem with the eol-once-things-break model is that there's no easy way to provide advance warning of something we can't predict | 19:23 |
JayF | fungi: as evidenced by email complaints to mailing list about EM branches going away; I think that distinction is overvalued | 19:24 |
knikolla | My personal opinion is that I don't think the benefits of EM make it worth all the downsides and difficulties that it brings. | 19:24 |
gmann | yeah and always do EOL in coordinated way is easy for everyone but difficult to do | 19:24 |
JayF | knikolla: lets be specific: do you mean "the benefits of maintaining branches older than 1.5y" or do you mean "EM, the state, as currently designed" | 19:24 |
JayF | knikolla: I'm trying to draw that line; because I think EM, the state as defined, is silly and bad; but that we can consider ridding ourselves of that state /without/ just killing all branches at 1.5yr mark | 19:25 |
knikolla | The benefits of maintaining branches that we don't support. Whatever our definition of support is. | 19:25 |
gmann | we have not made a clear distinguish of support here even doc tell it but in practical we are keeping it almost same. | 19:26 |
gmann | in ML, cinder is mostly raising point of testing cost and the bandwidth it need to get it merge | 19:27 |
JayF | this is why there's value in, if we are maintaining the "EM" branches longer, just calling them "maintained" and continue to do release activities | 19:27 |
JayF | when it's dead, it's dead, and goes straight to eOL | 19:27 |
JayF | that middle ground of "the branch is there but only for drive-by contributions or stable-dedicated contributors" does not exist | 19:27 |
fungi | i still think scheduling eol is likely to be of benefit to downstream consumers, as long as we publish that schedule | 19:28 |
gmann | yes | 19:28 |
JayF | The OpenStack Integrated Release goes EOL at the 18 month mark. Ironic/Nova/Glance/Others might continue to be supported after that | 19:28 |
knikolla | From a core reviewers perspective, I assume the following are true 1) I feel personally responsible for everything that merges (including EM branches) 2) I do not feel comfortable merging anything without tests (including in EM branches) | 19:28 |
JayF | just maybe not as the bundled package with a bow on top, with promises it works together and are tested together? | 19:28 |
JayF | I don't know if I love that | 19:29 |
* JayF == knikolla | 19:29 | |
gmann | I am ok to EM in between but make it more bakports-but-untested so that very minimum work for upstream members | 19:29 |
fungi | just marking things eol once we suddenly decide they're unmaintainable is likely to present a bigger problem to downstream consumers (we "warn" them that once it enters em it can eol at any time, but that's not the same as setting an eol date and sticking to it) | 19:29 |
gmann | not just downstream but for upstream coordinated and testing other dependent service also | 19:30 |
knikolla | We can make a resolution that 1) creates a maximum age for EM branches 2) reduces that maximum age every year until it reaches zero and we have no EM. | 19:31 |
knikolla | To make it less sudden. | 19:31 |
opendevreview | Brian Rosmaita proposed openstack/governance master: Upstream investment: translations infrastructure engineer https://review.opendev.org/c/openstack/governance/+/885379 | 19:32 |
gmann | this will help but not completely. frustration and time goes on testing failure and complexity to get backport merged due to gate health/complexity of integration testing | 19:33 |
fungi | also the vast majority, as we've observed, are simply ignoring those branches and waiting for a forcing factor (bulk cleanup) anyway | 19:34 |
JayF | knikolla: I would be -1 to a proposal that forces a project to EOL a branch before they are ready to. | 19:34 |
fungi | i think most of them have no expectation there will every be a change on those branches, but have no interest or incentive to close them early | 19:35 |
JayF | knikolla: if you haven't, I suggest reading fungi's comment at 19:17:12 UTC (at least in my client), this represents something I think is possible | 19:35 |
fungi | so the default is they stay open and unused until we say they have to get deleted | 19:35 |
fungi | which is fairly inefficient | 19:35 |
gmann | and backport as much you can even it miss some dependent lib backport | 19:36 |
knikolla | JayF: I appreciate the feedback. I'm open to the idea of keeping it opt-in for project teams that want that and requiring renewed commitment (by perhaps keeping the gate running) | 19:37 |
knikolla | This only applies for projects with few dependencies or no on other projects though. | 19:37 |
fungi | so yes, i do think that if projects are allowed to not participate in whatever the new coordinated eol schedule is, they need to not only opt out of the eol batch but also keep opting out explicitly. make the default eol instead of extension | 19:37 |
JayF | knikolla: yeah, keeping branches open for >18m being opt-in only I think is the change we need to bring that about | 19:37 |
dansmith | is it not okay to say those opt-in branches would be kept in a separate repo? | 19:37 |
fungi | and make it so that just because they opted out once, that doesn't mean we forget to ever eol the branch after the mass eol date has passed | 19:38 |
clarkb | I'd presonally prefer we not fork a bunch of repos. We already did that for packaing and I hate that its something I now have to support until the end of time | 19:38 |
JayF | Repository is our base level of organization, where ACLs are setup, how we reference things in governance, etc -- I would not be onboard for putting any openstack code belonging to $project into a different repo than $project | 19:38 |
fungi | extra copies of our repositories is not terribly efficient | 19:39 |
dansmith | a repo of nova-legacy where those branches are would really help convey the legacy-ness, but okay | 19:39 |
clarkb | ya it makes every gerrit upgrade take longer and replication for new git mirrors take logner and doubles our disk space requirements etc | 19:40 |
fungi | some of that is down to gerrit's design, it doesn't deduplicate commits between forks | 19:40 |
dansmith | right, I know gerrit and our job stuff makes that less easy than just github repos | 19:41 |
fungi | apparently github has its own secret sauce storage backend to make forking a low-resource action | 19:41 |
dansmith | might also be an opportunity to say "and these things no longer run tests because they've been moved out" :) | 19:41 |
knikolla | Could we have a "openstack-archive" that isn't in gerrit but just git? For those archived repos in a read-only manner? | 19:42 |
dansmith | yeah that ^ | 19:42 |
JayF | You'd create so much churn for downstreams; anyone consuming the old release from git would have to make positive changes | 19:42 |
gmann | but how read-only help ? no backport ? | 19:42 |
dansmith | it would be like the archive.ubuntu.org mirrors, where they're still there, people can install from them as needed, but it's clear they're not getting updates anymore | 19:42 |
JayF | we're literally adding a hurdle of the type a user specifically complained about on the list when a train EM branch we eol'd | 19:42 |
dansmith | yep, but a very clear one, | 19:43 |
dansmith | and once you do, it's in archive forever | 19:43 |
knikolla | gmann: this is just for the complaint about branches disappearing. For actual EM with testing and backports and merging it would have to be in Gerrit. | 19:43 |
dansmith | (but it's still a branch for branch-liking people) | 19:43 |
gmann | I think if we have branch somewhere in upstream people expect the backport | 19:44 |
knikolla | gmann: Not if it's clearly labeled as "openstack-archive", I hope. | 19:44 |
gmann | but keeping all branches as archive is not bad idea but that does not solve EM issue we have currently | 19:45 |
dansmith | well, you could even push backports to -archive if you want, but the thing that matters to me is that it becomes a very clear distinction from our supported branches in the main repo for the project | 19:45 |
knikolla | And for projects that are doing EM (not in openstack-archive), that expectation would actually be real, since they're going through the hassle of keeping the gates running and proving that they want to keep maintaining that branch. | 19:45 |
clarkb | dansmith: how is that any different than having a branch called archive-train? | 19:45 |
dansmith | I understand it's not super easy, so that's cool, but doing that would address multiple problems I have with it (I both think keeping branches that look maintained around is bad, but I also hate deleting branches and replacing with tagas) | 19:46 |
clarkb | I think to me it conveys the same message but one happens to be far more efficient and easier to deal with | 19:46 |
dansmith | clarkb: yeah, archive/train would be a lot better than what we have now which is stable/train honestly | 19:46 |
dansmith | in terms of messaging I think | 19:47 |
gmann | yeah *stable/* is confusing for EM for sure | 19:47 |
dansmith | I understand it's inefficient for non-human reasons, so that's fine, I just think a separate archive namespace makes a much clearer signal about what these things are | 19:47 |
dansmith | it sounds like separate repo/namespaces is a non-starter anyway, so we can just drop it | 19:48 |
gmann | there is no way to know what is EM until you looks into the release page | 19:48 |
knikolla | ++, stable/<release> works for me. | 19:48 |
knikolla | err, archive/ | 19:48 |
dansmith | however, to the CVE point, I think we really do need to stop maintaining things in archive/ or archive- or delete the branches if that's what we have to do.. because backporting minor fixes makes it look supported, regardless of what we say in the policy | 19:49 |
rosmaita | yes, that's exactly what cinder team is worried about | 19:50 |
rosmaita | it looks maintained because there's some activity, but it's dangerous to actually use it | 19:50 |
dansmith | I *do* think separating that activity into a separate repo changes the implication of that activity a little, but i definitely needs to stop at some point | 19:51 |
gmann | who issue is from what implementation/expectation of EM process when it was created in vancouver/sydney summit | 19:51 |
gmann | we always expected there will be a super hero team from somewhere for these EM and not the upsteam core team of that project :) | 19:52 |
knikolla | To summarize, my point is that JayF shows that there is a desire from some teams to actually "extend maintenance" of a branch in a process that involves clearly opt-in (that needs to be renewed) and clearing a bar of keeping gates and tests running. And for all other teams that can't maintain that level of activity and commitment, push them to archive. But also don't nuke the repo for archival purposes. | 19:52 |
gmann | if it is same core team has to do all work then it is becoming same as supported and no clear difference | 19:53 |
tonyb | but then we also have consumers complaing that we EOL too soon | 19:53 |
knikolla | gmann: exactly. In my view this is more about giving teams the chance to support a release beyond the EOL date of OpenStack releases. | 19:53 |
knikolla | tonyb: perhaps we do, but this is about aligning the expectations to what reality is. | 19:54 |
gmann | EOl complain was ok but idea was to allow them to help in the extended support of branches:) | 19:55 |
fungi | if consumers complain that we eol too soon, they should help fix that by assisting with maintenance of the branches we've still got open first. the em model clearly proved that they won't though | 19:55 |
knikolla | fungi: ++ | 19:55 |
gmann | and it end up on more supported branches with just two different name | 19:55 |
gmann | fungi: exactly | 19:55 |
fungi | my answer to "users will complain about this and yet do nothing to help out" is to ignore them, because they're dead weight | 19:55 |
fungi | if they want a free lunch, they're welcome to eat what we're serving | 19:56 |
knikolla | There is value in letting projects keep repos open if they have the proven ability to, but otherwise I do want to nuke EM branches by default. | 19:56 |
fungi | if they're willing to help out in the kitchen, then that's another matter | 19:57 |
tonyb | I guess it will be a super fun cinder session ;P | 19:58 |
knikolla | If people complain that we EOL things too soon, we can point them to the various way they can extend that. | 19:58 |
fungi | we can also point them to this rather lengthy experiment we ran for extending it, and welcome them to explain how it would be any different next time | 20:11 |
opendevreview | Brian Rosmaita proposed openstack/governance master: Upstream investment: translations infrastructure engineer https://review.opendev.org/c/openstack/governance/+/885379 | 21:06 |
fungi | tonyb: i agreed with your forum addition suggestion, you just won't see my reply because gmail says i look shifty | 23:53 |
tonyb | LOL | 23:54 |
tonyb | fungi: sounds good | 23:54 |
fungi | no idea why it randomly accepts and rejects messages i send to people. you'd think it would be consistent, but "Our system has detected that this message is likely unsolicited mail. To reduce the amount of spam sent to Gmail, this message has been blocked." | 23:54 |
tonyb | :( | 23:54 |
fungi | i sent three other messages to gmail addresses earlier today with no problem | 23:54 |
tonyb | I had that alot until I added DKIM and SPF to my domain | 23:55 |
tonyb | I do see your messages to lists, but google are "less picky" about them | 23:55 |
fungi | yeah, i'll just fall back on irc. i don't want to cater to the gmail mafia that badly | 23:55 |
tonyb | Okay, As long as Josh sees you're okay with it | 23:56 |
fungi | yeah, rackspace is far less picky | 23:56 |
fungi | but also i have out-of-band magic to pester him too, if it becomes necessary | 23:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!