openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: Clarify situation with ensure_pip_from_packages_with_python2 https://review.opendev.org/742571 | 01:01 |
---|---|---|
ianw | so the merge API is "POST /repos/:owner/:repo/merges" | 01:21 |
ianw | this comes under "contents" https://developer.github.com/v3/apps/permissions/#permission-on-contents | 01:21 |
ianw | #status log removed the r/w requirements for "contents" on the github zuul app ( https://github.com/apps/opendev-zuul) | 01:24 |
openstackstatus | ianw: finished logging | 01:24 |
ianw | Read access to code | 01:24 |
ianw | Read access to administration and metadata | 01:24 |
ianw | Read and write access to commit statuses, issues, and pull requests | 01:24 |
ianw | that looks right | 01:25 |
*** sgw has quit IRC | 02:32 | |
*** sgw has joined #opendev | 02:48 | |
*** sgw has quit IRC | 02:52 | |
*** sgw has joined #opendev | 02:57 | |
*** sgw has quit IRC | 03:40 | |
*** sgw has joined #opendev | 03:40 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Pre-install python3 for CentOS https://review.opendev.org/741868 | 03:51 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Drop dib-python requirement from several elements https://review.opendev.org/741877 | 03:51 |
*** xiaolin has quit IRC | 03:58 | |
*** sgw has quit IRC | 04:15 | |
*** sgw has joined #opendev | 04:15 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Pre-install python3 for CentOS https://review.opendev.org/741868 | 04:37 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Drop dib-python requirement from several elements https://review.opendev.org/741877 | 04:37 |
*** marios has joined #opendev | 05:21 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Pre-install python3 for CentOS https://review.opendev.org/741868 | 06:25 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Drop dib-python requirement from several elements https://review.opendev.org/741877 | 06:25 |
*** qchris has quit IRC | 06:51 | |
*** qchris has joined #opendev | 07:05 | |
*** DSpider has joined #opendev | 07:11 | |
*** tosky has joined #opendev | 07:22 | |
*** dougsz has joined #opendev | 07:35 | |
*** fressi has joined #opendev | 07:49 | |
*** elod_off is now known as elod | 07:54 | |
*** moppy has quit IRC | 08:01 | |
*** moppy has joined #opendev | 08:03 | |
*** jaicaa has quit IRC | 08:27 | |
*** jaicaa has joined #opendev | 08:29 | |
*** tristan888 has joined #opendev | 10:06 | |
*** qchris has quit IRC | 10:15 | |
*** dtantsur|afk is now known as dtantsur | 10:29 | |
*** zbr is now known as zbr|ruck | 10:30 | |
*** sshnaidm|afk is now known as sshnaidm|off | 11:11 | |
*** bhagyashris is now known as bhagyashris|afk | 11:54 | |
*** tristan888 has quit IRC | 12:07 | |
*** bhagyashris|afk is now known as bhagyashris | 12:22 | |
*** qchris has joined #opendev | 12:57 | |
*** qchris has quit IRC | 13:10 | |
*** qchris has joined #opendev | 13:14 | |
fungi | infra-root: just a heads up, i'm investigating several openstack release job failures from ~4 hours ago which suggest afs write errors... i think they may point to problems with specific executors, though the sample size is too small to say: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016064.html | 13:15 |
fungi | the two tarball write failures both ran from ze11, the docs failure from ze10 | 13:16 |
fungi | ze02, ze10 and ze11 have all unexpectedly rebooted within the past 24 hours, it looks like | 13:16 |
fungi | ze11 also logged a couple of "afs_pag_wait() PAG throttling triggered" lines in dmesg toward the end of the bootspam | 13:17 |
fungi | though that might just be a benign warning. there were no errors in dmesg around the times of the write permission problems recorded in the failed builds | 13:18 |
fungi | oh, though it also logged "clock went backwards, not PAG throttling" and dmesg times are >10 minutes earlier from the times syslog records the same entries | 13:26 |
fungi | so i wonder if something got confused by a giant time jump during boot | 13:27 |
fungi | ze10 has an eve larger discrepancy | 14:14 |
fungi | s/eve/even/ | 14:14 |
fungi | ze02 is not as extreme, but still several minutes difference | 14:15 |
clarkb | they are currently out of sync? | 14:17 |
clarkb | they are xenial so not using systemd's ntp service iirc. They should be ntpd? if that is the case wemay need to force them to make a jump to current time? | 14:18 |
fungi | nope, ntpd is doing its job just fine | 14:19 |
fungi | accurate at least to the second | 14:19 |
clarkb | oh I see it moved time by big amount in boot | 14:19 |
fungi | just wondering if the large time jump at start could have caused issues for afs or something | 14:19 |
clarkb | if afs's cache entries use system sourced timestamps I could see that | 14:20 |
fungi | i still need to manually test afs writes from them. trying to juggle this and yardwork before the heat advisory takes effect again | 14:20 |
clarkb | fs checkvolumes may help if so? | 14:20 |
clarkb | that made mirrors happy about the wheel cache dirs recwntly | 14:21 |
fungi | yeah, i mean, all three build failures happened within an hour of one another now 5+ hours ago, so i don't even know for sure if the issue persists | 14:21 |
fungi | i also haven't dug in the ze10 executor debug log yet to see what the actual error from the docs publication post_failure was | 14:23 |
dtantsur | folks, does anyone know if gertty is still developed? it doesn't seem to be compatible with python 3 | 14:24 |
fungi | dtantsur: can you expand on "doesn't seem to be compatible with python 3"? | 14:26 |
dtantsur | fungi: tries to call decode() on strings. crashes when opening any project. I'm writing a story as we speak. | 14:26 |
fungi | strange, i'm using it with python3 as we speak | 14:27 |
fungi | what minor version of the interpreter? | 14:27 |
dtantsur | 3.8 | 14:28 |
dtantsur | fungi: https://storyboard.openstack.org/#!/story/2007951 | 14:28 |
fungi | i normally run it under 3.5 but i'm testing with 3.8 on a newer machine now | 14:29 |
dtantsur | I *think* I used to use it with Python 3 as well, and probably that was 3.6 on the previous fedora | 14:29 |
*** mlavalle has joined #opendev | 14:44 | |
clarkb | did ply update to change the types there? | 14:45 |
clarkb | last release was 2018 unlikely | 14:46 |
fungi | dtantsur: i'm currently exercising gertty 1.6.0 on python 3.8.4 and it's not crashing for me | 14:53 |
fungi | same versions of ply and urwid you're using too | 14:55 |
fungi | dtantsur: i'm running from within a venv rather than with pip install --user, could that make a difference? | 14:56 |
fungi | i can open a project, open a review, check a diff, start a review comment, search with message:string and find changes, et cetera | 14:58 |
dtantsur | okay, lemme try with a venv | 14:58 |
fungi | i manually compile my python interpreters instead of using distro packages for them, but i doubt that makes a difference here | 14:58 |
dtantsur | nope, crashes the same for me Oo | 14:58 |
clarkb | same versions of gertty? | 14:59 |
dtantsur | pip freeze: http://paste.openstack.org/show/796250/ | 14:59 |
dtantsur | lemme try with 3.6 now | 14:59 |
fungi | this is my pip freeze: http://paste.openstack.org/show/796251 | 15:00 |
dtantsur | only urllib3 is different | 15:00 |
dtantsur | crashes the same in a 3.6 venv O__o | 15:00 |
fungi | your urllib is barely newer than mine, but other than that identical pip freeze too | 15:01 |
dtantsur | I wonder if my gertty.yaml has something that triggers the crash | 15:01 |
dtantsur | change-list-query: > | 15:02 |
dtantsur | status:open | 15:02 |
dtantsur | NOT message:DNM | 15:02 |
dtantsur | NOT message:"DO NOT MERGE" | 15:02 |
dtantsur | fungi: try the last part of the query ^^^ it triggers the issue for me | 15:04 |
fungi | Error: 'str' object has no attribute 'decode' | 15:05 |
dtantsur | so it's the same as https://storyboard.openstack.org/#!/story/2006484 | 15:05 |
fungi | bingo! | 15:05 |
dtantsur | apparently the quotation marks trigger it | 15:06 |
dtantsur | `NOT message:DNM` works normally | 15:06 |
fungi | yep, i concur | 15:07 |
fungi | i also get the same behavior on python 3.5 where i normally run gertty | 15:08 |
openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add ensure-pip to python release pre-run playbook https://review.opendev.org/742694 | 15:44 |
corvus | clarkb, fungi: ^ blocking zuul release (see #zuul) | 15:45 |
clarkb | should it gi before bindep? I'm checking in that now | 15:47 |
clarkb | I think it should since the bindep role also uses pip, though in this case it seems to find our preinstalled venv bindep? so is actually fine in either order for us | 15:49 |
clarkb | I've approvedut based on that | 15:50 |
fungi | it's easy to reorder that if we decide we should | 15:53 |
*** marios is now known as marios|out | 15:55 | |
*** zbr|ruck is now known as zbr | 16:02 | |
fungi | clarkb: corvus: ansible-lint on that change says "roles/ensure-pip/tasks/main.yaml:1: [E304] Environment variables don't work as part of command" | 16:05 |
fungi | um, what? | 16:05 |
fungi | also that only showed up in the gate pipeline | 16:05 |
corvus | i don't think we ran check to completion on that | 16:06 |
fungi | ahh | 16:06 |
fungi | right, we enqueue direct to gate and preempt check | 16:06 |
clarkb | it means that the command module doesnt apply env vars to the command execution env. need shell for that. I'm guessing this is anew check because ensure-pip haslanded elsewhere just fine? | 16:07 |
fungi | so that much is explained | 16:07 |
corvus | 304 is not ignored in zuul-jobs, so yeah, i'm thinking this may be new | 16:07 |
fungi | https://docs.ansible.com/ansible-lint/rules/default_rules.html | 16:08 |
fungi | version added: historic | 16:08 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: DNM: test lint on ensure-pip https://review.opendev.org/742699 | 16:08 |
corvus | fungi: that seems to unsubstatiate that theory | 16:08 |
fungi | i live to unsubstantiate theories | 16:09 |
fungi | but yeah, i'm baffled | 16:09 |
corvus | i mean, that's not a command task, that's shell, and env vars sure do work on shell | 16:10 |
corvus | i reprod that locally | 16:11 |
fungi | seems like a really thing to false positive match on | 16:12 |
fungi | er, a really strange thing | 16:12 |
clarkb | also there are no env vars? | 16:12 |
zbr | that is a very weird error to see, trying to reproduce locally | 16:12 |
clarkb | or is it finding them in the script itself? | 16:12 |
zbr | afaik 304 will only appear if it finds "command" and never on shell stuff. | 16:13 |
corvus | clarkb: yeah i think it's in the script | 16:13 |
corvus | like it's supposed to look for "command: ENV=foo bar" and tell you that doesn't work | 16:13 |
zbr | is using latest linter? | 16:13 |
corvus | are we doing anything like copying the zuul ansible modules in to the path? | 16:14 |
corvus | because we just fixed zuul's overridden command module | 16:14 |
corvus | so maybe it's tripping on that | 16:14 |
zbr | you are using ansible-lint<4 --- basically enjoy it! | 16:15 |
clarkb | ANSIBLE_ACTION_PLUGINS = {envsitepackagesdir}/zuul/ansible/base/actiongeneral | 16:15 |
clarkb | corvus: in tox.ini ^ I think that could be related | 16:15 |
zbr | bugs fixed years ago | 16:15 |
corvus | zbr: have you identified a bug? | 16:15 |
*** fressi has quit IRC | 16:15 | |
zbr | do a cat test-requirements.txt | 16:15 |
zbr | line 16 | 16:15 |
clarkb | we moved command out of action general into action and action trusted | 16:15 |
clarkb | so maybe if we change that to action or action trusted it will be happy? | 16:15 |
corvus | zbr: i believe you that we're running ansible-lint < 4. | 16:16 |
corvus | zbr: are you saying you know this is a bug that is fixed in 4? | 16:16 |
zbr | corvus: this is what i am trying to tell you | 16:16 |
clarkb | oh except it is installing zuul from pacakge nto source I think | 16:17 |
clarkb | and we don't have a packaged version with the fix yet? | 16:17 |
zbr | please look at https://pypi.org/project/ansible-lint/#history | 16:17 |
clarkb | zbr: just because there are releases doesn't mean this is a bug that has been fixed. We should investigate that | 16:17 |
*** fressi has joined #opendev | 16:18 | |
zbr | it did | 16:18 |
clarkb | can you point to that? | 16:18 |
clarkb | I think that is what corvus is asking about | 16:18 |
zbr | i run it locally, on same change, and no 304 was seen | 16:18 |
corvus | zbr: do you know what version we're using in zuul-jobs? | 16:18 |
zbr | yes, latest pre-release | 16:19 |
corvus | linters-requirements.txt:ansible-lint>=4.3.0a1,<5 | 16:19 |
corvus | ah we have a separate file for it | 16:19 |
zbr | which means latest (a5) | 16:19 |
zbr | copy same condition, it will work fine | 16:19 |
corvus | i'll look into updating it then | 16:20 |
zbr | you may want to add skips for 701 | 16:20 |
corvus | and 204 | 16:20 |
corvus | and 106 | 16:20 |
zbr | well, can copy the config itself too if you want | 16:20 |
zbr | 206 should be fixed instead | 16:21 |
corvus | i'm not gonna waste time on it :) | 16:21 |
zbr | i will add meta skip soon named "galaxy", which will skip those related to galaxy | 16:21 |
zbr | to make your life easier ;) | 16:22 |
corvus | oh sorry, yeah i'll fix 206, i misread :) | 16:22 |
zbr | in fact 4.3.0a5 is likely to become GA next week, we are only polishing it these days, preparing config changes, improvind docs. | 16:24 |
zbr | Also the current docs are located at https://ansible-lint.readthedocs.io/en/latest/ -- the ones on ansible website were not outdated in an year or so | 16:25 |
openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add ensure-pip to python release pre-run playbook https://review.opendev.org/742694 | 16:26 |
*** fressi has quit IRC | 16:26 | |
*** diablo_rojo_phon has joined #opendev | 16:30 | |
*** dougsz has quit IRC | 16:30 | |
clarkb | once things settle down on the zuul side I'm going to start prepping an etherpad for tomorrows gerrit project renaming (and followup on any needed changes for that etc) | 16:35 |
clarkb | one thing that comes to mind is last time we did this we had a problem because we ran manage-projects with a sequence of projects.yaml states. Do we have opinions on whether or not the changes to projects.yaml and friends should be squashed? I believe that if we properly stop ansible the expectation is that our automation will do the right thing? | 16:36 |
clarkb | things to think about, but I'll not dig in until after zuul is happy | 16:36 |
*** dtantsur is now known as dtantsur|afk | 16:37 | |
fungi | yeah, once the openstack release team meeting concludes i'll get back to troubleshooting those afs-related build failures | 16:37 |
fungi | and it just did | 16:37 |
clarkb | also I feel pulled in a hundred different directions so going to try and stay focused on the zuul side of things as that seems the most important at the moment | 16:38 |
openstackgerrit | Merged opendev/base-jobs master: Add ensure-pip to python release pre-run playbook https://review.opendev.org/742694 | 16:47 |
clarkb | corvus: ^ | 16:48 |
corvus | woot | 16:48 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update upload-logs-s3 with access/secret key variable documentation https://review.opendev.org/742710 | 16:52 |
*** marios|out has quit IRC | 17:01 | |
*** markmcclain has quit IRC | 17:14 | |
*** markmcclain has joined #opendev | 17:16 | |
AJaeger | config-core, infra-root, FYI, I'm on vacation the next 10 days and will go offline tomorrow night. | 18:31 |
corvus | AJaeger: enjoy! | 18:32 |
fungi | AJaeger: sounds great--hope you have a good time! | 18:32 |
AJaeger | I'll try my best - thanks! | 18:33 |
fungi | just to wrap up the earlier questions around afs write failures from ze11, i could not reproduce them. i was able to write to the same tree from ze11 (though i did it using my kerberos credentials, not the base job's) | 18:41 |
*** shtepanie has joined #opendev | 18:53 | |
openstackgerrit | Merged zuul/zuul-jobs master: Reinstate docker tag cleanup https://review.opendev.org/742461 | 18:58 |
mnaser | AJaeger: have fun!! | 19:24 |
clarkb | infra-root notes going up at: https://etherpad.opendev.org/p/gerrit-2020-07-24 | 19:34 |
clarkb | I'll write up the rename recording change shortly too | 19:34 |
openstackgerrit | Clark Boylan proposed opendev/project-config master: Add record for project renames on July 24, 2020 https://review.opendev.org/742731 | 19:39 |
clarkb | infra-root I think https://etherpad.opendev.org/p/gerrit-2020-07-24 and its associated changes are ready for reviewand the content is now complete | 19:45 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role https://review.opendev.org/742732 | 19:46 |
AJaeger | clarkb: want to add a WIP to https://review.opendev.org/#/c/738979/ ? I'll remove mine now so I don't forget. | 19:48 |
clarkb | AJaeger: you can leave yours as us gerrit admins can remove it too :) | 19:49 |
AJaeger | clarkb: Ok, then I leave it | 19:49 |
clarkb | and you should enjoy your vacation. I'm not sure what the istuation is like in your part of the wrold, but I was able to relax at home and get some things done around here when I took a week off. Not quite the same btu still worthwhile | 19:50 |
fungi | well, we *can* delete it but i think we have to delete the +2 at the same time | 19:50 |
clarkb | fungi: there is no +2 | 19:50 |
clarkb | so I think its fine | 19:50 |
fungi | on 738979? | 19:50 |
clarkb | oh the +2 was just added | 19:50 |
AJaeger | clarkb, gmann : -1 on https://review.opendev.org/#/c/739286 - could you check my comment, please? | 19:52 |
clarkb | looking | 19:52 |
AJaeger | clarkb: travelling inside Germany, it seems safe to travel again - with lots of masks ;) | 19:53 |
clarkb | AJaeger: oh good point on your comment, though I think we can also merge as is and figure it out after the rename if necessary. Let's see what gmann says | 19:53 |
clarkb | I saw the official jobs removal and that made sense and I attributed the other thing to that as well | 19:54 |
clarkb | btu the other thing is separate | 19:54 |
clarkb | gmann: ^ please take a look so that we are ready for tomorrow | 19:54 |
AJaeger | clarkb: yep, we could. I could also patch quickly either way... | 19:54 |
AJaeger | clarkb: removed the -1 and left a comment that a followup is fine as well. | 19:56 |
clarkb | thanks | 19:56 |
clarkb | I've double checked the fixes I found from the last rename and it seems we've merged them so I think we are good from that side of things | 20:01 |
openstackgerrit | Merged zuul/zuul-jobs master: Disable base role testing that runs code on localhost https://review.opendev.org/742537 | 20:04 |
openstackgerrit | Merged zuul/zuul-jobs master: Collect partial subunit files https://review.opendev.org/742527 | 20:04 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 20:06 |
*** DSpider has quit IRC | 20:15 | |
corvus | clarkb, fungi, ianw: https://github.com/pyca/cryptography/issues/5339#issuecomment-663193431 i'll etherpad a reply to this | 20:19 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role https://review.opendev.org/742732 | 20:25 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 20:25 |
fungi | corvus: ack, i also worry a little about our current arm64 test capacity depending on what their test volume realistically looks like | 20:39 |
fungi | but i expect we'll have to see it for ourselves | 20:42 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role https://review.opendev.org/742732 | 20:44 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 20:44 |
corvus | clarkb, fungi, ianw: https://etherpad.opendev.org/p/aQ_-tmssILd6UXInsOTT | 20:50 |
corvus | i apparently am having less trouble writing today than yesterday :) | 20:50 |
fungi | that makes one of us! ;) | 20:50 |
clarkb | that looks good. Maybe mention that our arm resources currently come from a single donor which makes their long term viability perhaps slightly less solid than our x86 options? That said linaro has done an excellent job and is involved and I'm not too worried | 20:52 |
corvus | how many clouds we got now? | 20:53 |
clarkb | for arm its one | 20:54 |
corvus | total | 20:54 |
clarkb | linaro, rax, inap, vexxhost, limestone, airship-citycloud (sort of when it owrks), ovh I think that is it? | 20:55 |
fungi | depending on whether you want to count regions | 20:55 |
corvus | i'll say "6+" :) | 20:55 |
fungi | also openedge presently offline but back soon | 20:55 |
clarkb | opendedge is offline ya | 20:56 |
clarkb | but intended to return | 20:56 |
corvus | The ARM resources also come from a single donor (whereas we have 6+ clouds providing x86 nodes); we consider that a risk, but the provider, Linaro, has been fairly reliable. | 20:56 |
corvus | clarkb: added that sentence | 20:56 |
fungi | lgtm | 20:56 |
clarkb | yup lgtm as well | 20:56 |
corvus | okay, i'll send | 20:56 |
fungi | thanks! | 20:57 |
corvus | done | 20:59 |
fungi | also just glad there's someone out there who uses parenthetical expressions as often as i do | 20:59 |
fungi | i treat english as a lisp variant | 20:59 |
corvus | there should be a name for the (closing parethetical smiley :) | 21:07 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role https://review.opendev.org/742732 | 21:07 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 21:07 |
corvus | fungi: also -- it must be said -- i read a lot of victorian novels --- which is probably where i developed my fondness for en-dashes (and em-dashes)! | 21:08 |
fungi | it happens | 21:19 |
fungi | and yeah, the place where emoticons and punctuation meet... therein lies chaos | 21:20 |
clarkb | I think that is why everyone uses animated gifs now | 21:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role https://review.opendev.org/742732 | 21:22 |
openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 21:22 |
*** JayF has quit IRC | 21:44 | |
*** JayF has joined #opendev | 21:45 | |
gmann | clarkb: AJaeger i can keep doc job in that patch itself, I am not sure if we can do it in in-repo (at least fast) as this repo acl group members are empty -https://review.opendev.org/#/admin/groups/994,members | 21:47 |
openstackgerrit | Ghanshyam Mann proposed openstack/project-config master: Rename transparency-policy from openstack/ to osf/ namespace https://review.opendev.org/739286 | 21:49 |
gmann | clarkb: AJaeger ^^ updated. | 21:49 |
clarkb | gmann: thanks | 21:49 |
JayF | Is there a way to subscribe to a project in gerrit? If I want to get an email about every new change posted to a project? I feel like it should be simple, but I can't find docs or figure it out in the web ui. | 22:06 |
JayF | In typical fashion; I've been looking for 15 minutes and find it right after asking. In case someone finds this later in logs: top right corner, click your name -> Settings -> "Watched Projects" | 22:07 |
clarkb | JayF: yes, gerrit calls it watching and you do it via https://review.opendev.org/#/settings/projects | 22:07 |
JayF | Thank you :) | 22:08 |
clarkb | ah you found it too :) | 22:08 |
clarkb | alternatively you can use gertty and set up project subscriptions in it as well | 22:08 |
JayF | That always happens. I give up, ask, then decide to look again and it's like I'm magnetized to the answer. | 22:08 |
*** tosky has quit IRC | 22:59 | |
*** mlavalle has quit IRC | 23:31 | |
*** shtepanie has quit IRC | 23:32 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!