| fungi | i'll help with that if i can find time | 00:00 |
|---|---|---|
| fungi | though i may not come up for air until mid-march at this rate | 00:00 |
| ianw | fungi: cool, i'm doing a change now :) we can review and approve at will and then just switch dns when we want production | 00:01 |
| * fungi dives deep to crush the spirits of more phd candidates | 00:01 | |
| ianw | good luck :) | 00:02 |
| *** ijw has quit IRC | 00:02 | |
| fungi | they've already been beaten black and blue, i'm sure. my feeble attempts to prod them to improve their talks are likely falling on deaf ears anyway | 00:03 |
| fungi | as someone who didn't even finish a bachelor's degree, i feel mildly dirty criticizing graduate students. or maybe not ;) | 00:03 |
| ianw | the conference treadmill is why i have a masters and not a phd :) | 00:04 |
| fungi | that's fair | 00:04 |
| ianw | i initially typod that as "dreadmill" and i'm not sure that's inaccurate | 00:05 |
| fungi | sounds about right to me | 00:05 |
| fungi | there was even a point in my life where i was convinced i was destined for a career in academia | 00:05 |
| *** artom has joined #openstack-infra | 00:05 | |
| fungi | and then i "graduated" to a more-than-full-time sysadmin job to pay the bills, and forgot to keep going to school | 00:06 |
| fungi | in retrospect, being a basement troll in some math theory department probably wouldn't have been as glamorous as i originally envisioned it | 00:07 |
| mordred | bah | 00:07 |
| mordred | it's the hight of glamour | 00:08 |
| mordred | height | 00:08 |
| mordred | can't spell | 00:08 |
| fungi | maybe you should have stayed in school ;) | 00:08 |
| *** mattw4 has quit IRC | 00:09 | |
| mordred | fungi: because that MFA in lighting design with a focus in Experimental Theatre would *definitely* have improved my spelling I'm sure ;) | 00:09 |
| fungi | you'd know how to spell every pantone color in the catalogue | 00:09 |
| *** tosky has quit IRC | 00:09 | |
| clarkb | I looked at graduate school but realized if I got a job I wouldnt have to move | 00:09 |
| mordred | indeed. and how to insert ironic punctuation into random words | 00:10 |
| clarkb | but had professor that wanted me to move east for grad school | 00:10 |
| clarkb | I hate moving | 00:10 |
| mordred | ++ | 00:10 |
| fungi | clarkb: now you and countless other techie drones have enabled the idea of graduate school that doesn't require leaving the house | 00:10 |
| fungi | problem solved | 00:10 |
| * mordred glares at remaining boxes | 00:10 | |
| mordred | fungi: can we start doing conferences that don't require leaving the house? but that don't involve staring at a video conference screen with an audio feed either? | 00:11 |
| fungi | i was sort of just in one for two days. it was unpleasant, but less unpleasant than travelling | 00:11 |
| fungi | i dialled into it with a telephone, and used text-based communication platforms as a secondary communication channel. could have been worse | 00:12 |
| fungi | only christine and the cats had to see my unkempt university basement troll wannabe appearance | 00:13 |
| fungi | and rob helped bring me pizza | 00:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk https://review.opendev.org/708018 | 00:16 |
| *** eernst has joined #openstack-infra | 00:18 | |
| ianw | fungi: is git-test something you were working on for the transition? | 00:18 |
| fungi | ianw: i have no idea what git-test is, nor what is the transition of which you speak | 00:19 |
| ianw | oh sorry i meant openstack->opendev | 00:19 |
| ianw | i have no idea what is either | 00:19 |
| fungi | where did you find a git-test? | 00:20 |
| ianw | it's a site that points to http://files.openstack.org/project/git-test/www/test.py | 00:20 |
| fungi | aha, https://opendev.org/opendev/system-config/src/branch/master/manifests/site.pp#L415-L421 | 00:21 |
| fungi | according to codesearch.o.o | 00:21 |
| fungi | git blame says you should ask corvus (it came in via change 634806) | 00:22 |
| *** eernst has quit IRC | 00:22 | |
| fungi | mordred: do you happen to know, is there something we have to enable to start showing gerrit's git notes in gitea? i recall a change merged to expose git notes months ago so is almost certainly included in the recent versions we've been running | 00:23 |
| fungi | i think the change we were waiting on for that, a while back, allowed specifying the base of the git note refs we wanted displayed, since gerrit sets a custom one and gitea had originally hard-coded a different expectation | 00:24 |
| fungi | ianw: so, the commit message in https://review.opendev.org/634806 has refreshed my memory ;) | 00:25 |
| fungi | we used that vhost to test git.o.o redirects for the cgit to gitea migration, yes | 00:26 |
| fungi | it can just disappear now | 00:26 |
| ianw | fungi: ok, well if we take this path of migrating to static.opendev.org we can just purge it when puppet files.openstack.org disappears | 00:27 |
| fungi | yup | 00:27 |
| ianw | i can add a task to the story to remove the afs dir | 00:27 |
| fungi | sounds great, we missed a reminder to clean it up when we were done. thanks! | 00:27 |
| ianw | ok, another one for the peanut gallery that i can figure out with due effort, do we control starlingx.io dns? | 00:29 |
| fungi | depends on the "we" | 00:32 |
| fungi | it's in rackspace dns | 00:32 |
| fungi | and the openstackinfra account has access to modify it | 00:33 |
| fungi | so if "we" includes folks who don't mind touching rackspace dns, then the answer is yes | 00:33 |
| ianw | cool, yep so as long as i can do _acme-challenge.docs.starlingx.io -> acme.opendev.org | 00:34 |
| fungi | yeah, it's "no worse" than openstack.org in that regard | 00:34 |
| fungi | we can treat it similarly | 00:34 |
| fungi | i think i brought it up as an example when we discussed the cross-domain cname plan for letsencrypt | 00:35 |
| fungi | (just so we weren't focusing solely on openstack.org) | 00:35 |
| *** ijw has joined #openstack-infra | 00:40 | |
| *** ijw has quit IRC | 00:44 | |
| *** ijw has joined #openstack-infra | 00:44 | |
| fungi | okay, here's some strange termcap-related voodoo that i've pickled away all the braincells for... in a recent upgrade my terminal and/or x11 switched my backspace key from sending ^? to ^H and i'm not sure how or where to undo that (or even who the culprit might be). i suspect an update of the terminal itself | 00:49 |
| fungi | does anyone recall the difference between ^? and ^H | 00:50 |
| fungi | modern web searches can't really accommodate that line of inquiry, so i'll probably resort to digging in ancient tomes of unix mysticism | 00:51 |
| fungi | yeah, it's gotta be my terminal, because ctrl-v,backspace in an xterm still shows ^? even on the updated systems | 00:52 |
| adriant | Hey, I'm currently adding a new tox check to my project called "black_check", how would I go about including that as a required test in zuul? Can I add it somehow to my .zuul.yaml? | 00:52 |
| *** yamamoto has joined #openstack-infra | 00:54 | |
| fungi | adriant: yeah, you could define a job parented on the tox or openstack-tox job which sets the tox testenv to black_check | 00:54 |
| adriant | fungi: where might I find an example of that? | 00:55 |
| fungi | adriant: i'll see if i can find you a good one | 00:56 |
| adriant | fungi: awesome! ty | 00:56 |
| adriant | i'm trying to use black to reformat my code, and include a check that ensures it always matches the format, and to include an entry for that massive change in '.git-blame-ignore-revs ' | 00:57 |
| adriant | so hopefully it should be a nice set of changes, and good going forward | 00:57 |
| *** igordc has quit IRC | 00:58 | |
| fungi | adriant: here's an example passing "linters" as the tox env parented on the "tox" job from zuul-jobs, though you could use the openstack-tox job as a parent if you want to take advantage of openstack requirements constraints list and such: https://opendev.org/zuul/zuul-jobs/src/branch/master/zuul.d/python-jobs.yaml#L135-L144 | 00:59 |
| fungi | tox_envlist is the magic variable name | 01:00 |
| adriant | fungi: ty! Will see what I can do :) | 01:00 |
| fungi | there's also an openstack-tox-linters job: https://docs.openstack.org/infra/openstack-zuul-jobs/jobs.html#job-openstack-tox-linters | 01:01 |
| fungi | you could just put your black invocation as a cmd in a [testenv:linters] in your tox.ini and then use that existing job | 01:01 |
| adriant | will try this first, and tend maybe fall back to linters if that seems easier | 01:04 |
| fungi | we're here if you need more help. feel free to point us at a review | 01:05 |
| adriant | will do :) | 01:05 |
| *** tetsuro has joined #openstack-infra | 01:07 | |
| *** yamamoto has quit IRC | 01:12 | |
| *** portdirect has quit IRC | 01:12 | |
| *** yamamoto has joined #openstack-infra | 01:13 | |
| *** gyee has quit IRC | 01:13 | |
| *** dancek has quit IRC | 01:14 | |
| *** mordred has quit IRC | 01:14 | |
| *** sdoran has quit IRC | 01:14 | |
| *** mordred has joined #openstack-infra | 01:15 | |
| *** portdirect has joined #openstack-infra | 01:16 | |
| *** sdoran has joined #openstack-infra | 01:16 | |
| *** dancek has joined #openstack-infra | 01:17 | |
| *** rfolco|doctor is now known as rfolco | 01:18 | |
| *** admcleod has quit IRC | 01:27 | |
| *** admcleod has joined #openstack-infra | 01:32 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] move afs sites from files.openstack.org to static.opendev.org https://review.opendev.org/709024 | 01:34 |
| *** rfolco has quit IRC | 01:34 | |
| *** yamamoto has quit IRC | 01:36 | |
| *** yamamoto has joined #openstack-infra | 01:40 | |
| *** yamamoto has quit IRC | 01:45 | |
| *** yamamoto has joined #openstack-infra | 01:49 | |
| adriant | fungi: seems to work: https://review.opendev.org/#/c/709023 | 01:51 |
| *** tetsuro has quit IRC | 01:56 | |
| *** tetsuro has joined #openstack-infra | 01:56 | |
| *** rlandy has quit IRC | 02:04 | |
| *** lbragstad has quit IRC | 02:11 | |
| *** lbragstad has joined #openstack-infra | 02:11 | |
| *** factor has joined #openstack-infra | 02:25 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] move afs sites from files.openstack.org to static.opendev.org https://review.opendev.org/709024 | 02:27 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: afs-release: also release project.releases https://review.opendev.org/709032 | 02:31 |
| openstackgerrit | Merged opendev/system-config master: static: add default site https://review.opendev.org/708972 | 02:32 |
| *** lbragstad has quit IRC | 02:38 | |
| adriant | ianw: regarding: https://review.opendev.org/#/c/708983/1 | 03:01 |
| adriant | adjutant-ui doesn't exist on pypi yet at all | 03:01 |
| adriant | I can't see a way to create it without publishing a release myself manually | 03:02 |
| adriant | I've only ever created projects on pypi using the publish tools in setup tools or now, twine. | 03:02 |
| prometheanfire | where generate-constraints is run for the daily bot update, is it run as a non-root user? | 03:02 |
| ianw | adriant: that's right, see the notes on a 0-version in https://docs.openstack.org/infra/manual/creators.html#give-openstack-permission-to-publish-releases | 03:05 |
| adriant | yeah, I see that now | 03:05 |
| adriant | not something I've done before... so just gotta work out exactly what that actually means :P | 03:05 |
| prometheanfire | https://github.com/storpool/python-storpool/commit/9f072befed9850643f5c31ee789f55b1d6cfa7f4 makes me sad | 03:05 |
| ianw | prometheanfire: iirc genreate-constraints is run from the same thing as the other proposal updates which would be non-root | 03:07 |
| prometheanfire | ok, well, I had to set an env-var to get it to run :( | 03:07 |
| prometheanfire | error: could not create '/usr/share/doc/python-storpool': Permission denied | 03:07 |
| *** tetsuro has quit IRC | 03:13 | |
| prometheanfire | ianw: I've bugged cinder about it, we'll see what happens, but if it continues I may be pushing for it's removal | 03:13 |
| ianw | prometheanfire: so that is breaking out of it's virtualenv to try and write stuff to /usr/share? | 03:15 |
| prometheanfire | yep | 03:15 |
| prometheanfire | at least it seemed to do so for me | 03:15 |
| prometheanfire | setting the var fixed it, but that should not be required imo | 03:16 |
| adriant | ianw: adjutant-ui now exists, and openstackci has owner, but I've deleted the release I uploaded | 03:16 |
| ianw | adriant: thanks, it should work now | 03:16 |
| *** ociuhandu has joined #openstack-infra | 03:19 | |
| *** tetsuro has joined #openstack-infra | 03:22 | |
| *** ociuhandu has quit IRC | 03:23 | |
| *** tetsuro_ has joined #openstack-infra | 03:28 | |
| *** tetsuro has quit IRC | 03:30 | |
| *** tetsuro_ has quit IRC | 03:33 | |
| *** tetsuro has joined #openstack-infra | 03:34 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: add releases.openstack.org site https://review.opendev.org/709009 | 03:35 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: move site setup to a loop https://review.opendev.org/709016 | 03:35 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] move afs sites from files.openstack.org to static.opendev.org https://review.opendev.org/709024 | 03:35 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Revert "Generate list of 404s for docs.o.o" https://review.opendev.org/709038 | 03:35 |
| *** ijw has quit IRC | 03:39 | |
| *** dchen has quit IRC | 03:42 | |
| openstackgerrit | Merged opendev/system-config master: afs-release: also release project.releases https://review.opendev.org/709032 | 03:47 |
| *** tetsuro_ has joined #openstack-infra | 03:47 | |
| *** tetsuro has quit IRC | 03:48 | |
| *** ykarel|away is now known as ykarel | 03:51 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] move afs sites from files.openstack.org to static.opendev.org https://review.opendev.org/709024 | 03:57 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Revert "Generate list of 404s for docs.o.o" https://review.opendev.org/709038 | 03:57 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: add static.openstack.org/files.openstack.org https://review.opendev.org/709042 | 03:57 |
| prometheanfire | 2020-02-21 03:20:50.555037 | ubuntu-bionic | error: could not create '/usr/share/doc/python-storpool': Permission denied | 04:07 |
| prometheanfire | yay | 04:07 |
| *** dchen has joined #openstack-infra | 04:30 | |
| *** imacdonn has quit IRC | 04:45 | |
| *** imacdonn has joined #openstack-infra | 04:45 | |
| *** dchen has quit IRC | 04:49 | |
| AJaeger | ianw: https://service-types.openstack.org/service-types.json does not exist, do you still have the data and can mirror it over? I think I know what happened: Our AFS publishing removed all files that don't exist, so we need to change the publish job to not delete. | 05:10 |
| ianw | hrm, upload-afs-synchronize should not delete? or is it using something else? | 05:11 |
| ianw | AJaeger: i can re-copy from static, just a sec | 05:11 |
| ianw | copied and released | 05:12 |
| AJaeger | ianw: thanks. We need to change - like we did with tarballs - to a role that does not delete... | 05:12 |
| AJaeger | ianw: I can look later today at it unless you're faster ;), that repo has very low volume of merges... | 05:13 |
| * AJaeger needs to take care of family breakfast now | 05:13 | |
| ianw | AJaeger: heh enjoy ... i'm not sure i'll get to it as i have to run to the airport sorry | 05:14 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: move afs sites from files.openstack.org to static.opendev.org https://review.opendev.org/709024 | 05:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Revert "Generate list of 404s for docs.o.o" https://review.opendev.org/709038 | 05:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: add static.openstack.org/files.openstack.org https://review.opendev.org/709042 | 05:15 |
| openstackgerrit | Merged openstack/project-config master: Add pypi publish to Adjutant UI https://review.opendev.org/708983 | 05:20 |
| *** dchen has joined #openstack-infra | 05:27 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: add static.openstack.org/files.openstack.org https://review.opendev.org/709042 | 05:28 |
| *** rh-jelabarre has quit IRC | 05:29 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: grafana afs: add project.releases volume https://review.opendev.org/709051 | 05:32 |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| AJaeger | ianw: no hurry... I think merging the two jobs is easier, will try that today... | 05:51 |
| *** ociuhandu has joined #openstack-infra | 06:01 | |
| *** ociuhandu has quit IRC | 06:06 | |
| *** ricolin has quit IRC | 06:15 | |
| *** Lucas_Gray has joined #openstack-infra | 06:22 | |
| *** lmiccini has joined #openstack-infra | 06:27 | |
| *** lajoskatona has joined #openstack-infra | 06:58 | |
| *** lajoskatona has left #openstack-infra | 06:59 | |
| *** ricolin has joined #openstack-infra | 07:21 | |
| *** faizy98 has joined #openstack-infra | 07:30 | |
| *** beagles_pto has quit IRC | 07:31 | |
| *** lpetrut has joined #openstack-infra | 07:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix promote-service-types-authority https://review.opendev.org/709060 | 07:45 |
| *** Wryhder has joined #openstack-infra | 07:45 | |
| AJaeger | config-core, this should fix the service-types-authority publishing, please review ^ | 07:45 |
| *** Lucas_Gray has quit IRC | 07:45 | |
| *** Wryhder is now known as Lucas_Gray | 07:46 | |
| *** ramishra has quit IRC | 07:49 | |
| *** rpittau|afk is now known as rpittau | 07:50 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 07:52 |
| *** ccamacho has joined #openstack-infra | 07:53 | |
| *** iurygregory has joined #openstack-infra | 07:55 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add --validate-tenants option to zuul scheduler https://review.opendev.org/542160 | 08:00 |
| *** slaweq has joined #openstack-infra | 08:00 | |
| *** ramishra has joined #openstack-infra | 08:03 | |
| *** Lucas_Gray has quit IRC | 08:12 | |
| *** tesseract has joined #openstack-infra | 08:16 | |
| *** amoralej|off is now known as amoralej | 08:20 | |
| *** tetsuro_ has quit IRC | 08:26 | |
| *** tosky has joined #openstack-infra | 08:27 | |
| *** dchen has quit IRC | 08:31 | |
| *** factor has quit IRC | 08:36 | |
| *** yamamoto has quit IRC | 08:36 | |
| *** factor has joined #openstack-infra | 08:36 | |
| *** ralonsoh has joined #openstack-infra | 08:37 | |
| *** jpena|off is now known as jpena | 08:43 | |
| *** Lucas_Gray has joined #openstack-infra | 08:47 | |
| *** ykarel is now known as ykarel|lunch | 08:47 | |
| *** tkajinam has quit IRC | 08:50 | |
| *** tetsuro has joined #openstack-infra | 08:51 | |
| *** ykarel|lunch is now known as ykarel | 08:55 | |
| *** tetsuro has quit IRC | 09:03 | |
| *** Wryhder has joined #openstack-infra | 09:04 | |
| *** Lucas_Gray has quit IRC | 09:04 | |
| *** Wryhder is now known as Lucas_Gray | 09:05 | |
| *** pkopec has joined #openstack-infra | 09:10 | |
| *** rcernin has quit IRC | 09:11 | |
| *** lucasagomes has joined #openstack-infra | 09:12 | |
| *** yamamoto has joined #openstack-infra | 09:15 | |
| *** ociuhandu has joined #openstack-infra | 09:16 | |
| *** dpawlik has joined #openstack-infra | 09:17 | |
| *** yamamoto has quit IRC | 09:20 | |
| *** derekh has joined #openstack-infra | 09:23 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Authorization rules: add templating https://review.opendev.org/705193 | 09:37 |
| *** yamamoto has joined #openstack-infra | 09:46 | |
| *** ociuhandu has quit IRC | 09:46 | |
| *** rcernin has joined #openstack-infra | 09:47 | |
| *** yamamoto has quit IRC | 09:51 | |
| openstackgerrit | Daniel Bengtsson proposed openstack/cookiecutter master: Update the minversion parameter. https://review.opendev.org/707397 | 09:51 |
| *** ociuhandu has joined #openstack-infra | 09:56 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Make test setup_config more pure https://review.opendev.org/709084 | 10:05 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Make test setup_config more pure https://review.opendev.org/709084 | 10:06 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Make test setup_config more pure https://review.opendev.org/709084 | 10:15 |
| AJaeger | infra-root, could you review https://review.opendev.org/#/c/709009/2 so that we can move releases website over, please? | 10:22 |
| *** xek_ has joined #openstack-infra | 10:23 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add --validate-tenants option to zuul scheduler https://review.opendev.org/542160 | 10:28 |
| *** faizy98 has quit IRC | 10:36 | |
| *** happyhemant has joined #openstack-infra | 11:08 | |
| *** haleyb|away has quit IRC | 11:15 | |
| *** ykarel is now known as ykarel|afk | 11:16 | |
| *** Lucas_Gray has quit IRC | 11:26 | |
| AJaeger | infra-root, so releases publish now to http://files.openstack.org/project/releases.openstack.org/ - please review 709009 so that releases.openstack.org points to the new AFS location. | 11:27 |
| *** Lucas_Gray has joined #openstack-infra | 11:33 | |
| *** ykarel|afk is now known as ykarel | 11:42 | |
| *** rpittau is now known as rpittau|bbl | 11:43 | |
| *** Lucas_Gray has quit IRC | 12:03 | |
| *** dtantsur|afk is now known as dtantsur | 12:06 | |
| *** ricolin has quit IRC | 12:14 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk https://review.opendev.org/708018 | 12:19 |
| *** rcernin has quit IRC | 12:28 | |
| *** Lucas_Gray has joined #openstack-infra | 12:30 | |
| *** jpena is now known as jpena|lunch | 12:42 | |
| *** iokiwi has quit IRC | 12:43 | |
| *** iokiwi has joined #openstack-infra | 12:43 | |
| *** iokiwi has quit IRC | 12:48 | |
| *** iokiwi has joined #openstack-infra | 12:48 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add ensure-bazelisk role https://review.opendev.org/708899 | 12:51 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add ensure-java role https://review.opendev.org/708901 | 12:52 |
| *** rfolco has joined #openstack-infra | 12:53 | |
| *** adriant has quit IRC | 12:53 | |
| *** adriant has joined #openstack-infra | 12:54 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add ensure-bazelisk role https://review.opendev.org/708899 | 12:56 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add ensure-java role https://review.opendev.org/708901 | 12:56 |
| *** rlandy has joined #openstack-infra | 12:59 | |
| *** rpittau|bbl is now known as rpittau | 13:00 | |
| *** rh-jelabarre has joined #openstack-infra | 13:02 | |
| *** Goneri has joined #openstack-infra | 13:09 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Dequeue items via buildset uuid https://review.opendev.org/709135 | 13:17 |
| *** ykarel is now known as ykarel|afk | 13:22 | |
| rosmaita | is it just me, or is openstack-discuss on pipermail stuck at Thu Feb 20 15:12:15 UTC 2020 ? http://lists.openstack.org/pipermail/openstack-discuss/2020-February/date.html | 13:22 |
| AJaeger | rosmaita: I did not receive anything newer in my inbox, so we agree on delivery side. | 13:23 |
| AJaeger | infra-root, do we have a delivery / receive problem that no mails have been sent out in the last 23 hours? | 13:24 |
| mordred | AJaeger, rosmaita: last delivery into mailmain on lists.openstack.org is 2020-02-21 13:29:36 UTC - so we're at least accepting messages | 13:32 |
| mordred | I'm not sure about next steps for debugging what might be happening with mailman - fungi should be up soon though | 13:33 |
| rosmaita | mordred: thanks for looking | 13:33 |
| mordred | I don't see any errors in mailman logs - but there's not much of anything in mailman logs | 13:33 |
| AJaeger | no mails coming in here - great, I can work further on my slides ;) | 13:33 |
| rosmaita | that is one bonus! | 13:34 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: test install-docker all-platforms https://review.opendev.org/709139 | 13:37 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: test install-docker all-platforms https://review.opendev.org/709139 | 13:39 |
| *** dpawlik has quit IRC | 13:40 | |
| *** jpena|lunch is now known as jpena | 13:42 | |
| *** dpawlik has joined #openstack-infra | 13:48 | |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| *** njohnston has quit IRC | 13:54 | |
| *** Goneri has quit IRC | 13:57 | |
| *** rpittau is now known as rpittau|afk | 13:57 | |
| *** njohnston has joined #openstack-infra | 13:58 | |
| *** dpawlik has quit IRC | 14:00 | |
| *** roman_g has joined #openstack-infra | 14:05 | |
| *** lbragstad has joined #openstack-infra | 14:08 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul master: ui: convert all spaces to commas for filter https://review.opendev.org/709147 | 14:10 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul master: ui: convert all spaces to commas for filter https://review.opendev.org/709147 | 14:12 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk https://review.opendev.org/708018 | 14:13 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Don't fetch pull request twice for status event https://review.opendev.org/709149 | 14:14 |
| *** amoralej is now known as amoralej|lunch | 14:23 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: WIP: Dequeue items via buildset uuid https://review.opendev.org/709135 | 14:24 |
| fungi | ianw: adriant: regarding 708983, maybe we should do a better job of documenting that the release jobs will automatically create the project on pypi if one doesn't exist yet. if you're worried that someone will come along and steal the project name out from under you before you get around to requesting a release, that's the only time you'd really need to worry about precreating it in pypi to "reserve" | 14:24 |
| fungi | the name for your use | 14:24 |
| fungi | rosmaita: AJaeger: mordred: i'll check out mailman now | 14:26 |
| rosmaita | fungi: ty | 14:26 |
| rosmaita | and good morning! | 14:26 |
| fungi | the last line in /srv/mailman/openstack/logs/post is: Feb 20 15:12:16 2020 (29781) post to openstack-discuss from... | 14:33 |
| fungi | though /srv/mailman/openstack/logs/smtp has a number of "smtp to openstack-discuss" messages after that time, most recently 12:00:09 utc | 14:34 |
| fungi | aha! | 14:35 |
| fungi | [Thu Feb 20 15:17:50 2020] Out of memory: Kill process 32112 (python) score 147 or sacrifice child | 14:35 |
| fungi | and the symptoms are consistent with a queue runner not running | 14:36 |
| corvus | based on ps, it looks like maybe only the news and retry runners are running for one of the domains (probably openstack) | 14:37 |
| fungi | i'll restart them all just to be sure | 14:37 |
| fungi | oh, okay i can do just those | 14:37 |
| corvus | fungi: well, those are the ones running, the arch, bounce, command, incoming, outgoing, and virgin runners are the ones not running. at any rate, restarting the 'mailman-openstack' service should take care of it | 14:38 |
| corvus | (but we may need to check for lock files) | 14:38 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: prometheus: add options to start the server and process collector https://review.opendev.org/599209 | 14:39 |
| fungi | ahh, yeah, i misread you and my process count wasn't matching | 14:39 |
| fungi | anyway, i've stopped them for openstack and am checking for locks | 14:39 |
| fungi | looks like /srv/mailman/openstack/locks/ is empty so we should be fine there? | 14:40 |
| fungi | started back up now | 14:40 |
| fungi | hopefully we'll see deliveries of the pending posts shortly | 14:40 |
| fungi | #status log restarted the mailman-openstack runners on lists.openstack.org following a 15:17:50 oom killer event yesterday | 14:41 |
| openstackstatus | fungi: finished logging | 14:41 |
| fungi | yep, here they come | 14:43 |
| * fungi now has a catchy primus song stuck in his head | 14:43 | |
| fungi | i suspect some of the others may need restarting too. at the moment i count only 46 mailman processes | 14:44 |
| *** ociuhandu has quit IRC | 14:44 | |
| *** rfolco has quit IRC | 14:47 | |
| fungi | on, i wasn't limiting it to the runners | 14:48 |
| fungi | we have 5 mailmanctl processes and 40 qrunner processes so that looks right for 5 sites | 14:48 |
| fungi | (airship, opendev, openstack, starlingx, zuul) | 14:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk https://review.opendev.org/708018 | 14:49 |
| *** Lucas_Gray has quit IRC | 14:51 | |
| *** rfolco has joined #openstack-infra | 14:54 | |
| *** ociuhandu has joined #openstack-infra | 14:55 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: test install-docker all-platforms https://review.opendev.org/709139 | 14:56 |
| *** Lucas_Gray has joined #openstack-infra | 14:56 | |
| *** ociuhandu has quit IRC | 15:00 | |
| *** eharney has joined #openstack-infra | 15:03 | |
| *** yamamoto has quit IRC | 15:07 | |
| openstackgerrit | Merged zuul/zuul master: ui: convert all spaces to commas for filter https://review.opendev.org/709147 | 15:13 |
| AJaeger | mordred: could you review https://review.opendev.org/709060 , please? And once that's in, let's test it with https://review.opendev.org/709062 - this is service-types-authority | 15:15 |
| *** yamamoto has joined #openstack-infra | 15:19 | |
| *** amoralej|lunch is now known as amoralej | 15:19 | |
| donnyd | fungi: How would I go about disabling FN all together? I am just about done with the move, and my last steps require the public ip address space the old build is currently sitting in | 15:23 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add nodesets API route https://review.opendev.org/605877 | 15:23 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add ensure-bazelisk role https://review.opendev.org/708899 | 15:31 |
| *** ykarel|afk is now known as ykarel | 15:34 | |
| *** ociuhandu has joined #openstack-infra | 15:35 | |
| fungi | donnyd: i think we would need to set the remaining max-servers entries for the special node types to 0 and optionally pause image uploads | 15:36 |
| *** Goneri has joined #openstack-infra | 15:36 | |
| fungi | donnyd: oh, and turn off swift uploads for logs there if we ended up turning them back on at some point? (though i thought they were still off) | 15:37 |
| donnyd | The api endpoint urls will also be changing | 15:37 |
| donnyd | I think that is already disabled | 15:37 |
| donnyd | Where do I pause image uploads? | 15:37 |
| *** ociuhandu has quit IRC | 15:39 | |
| *** jpena is now known as jpena|off | 15:39 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/diskimage-builder master: Added ironic-agent pkg-map for CentOS-8 https://review.opendev.org/709159 | 15:40 |
| *** dpawlik has joined #openstack-infra | 15:41 | |
| *** jpena|off is now known as jpena | 15:42 | |
| *** ociuhandu has joined #openstack-infra | 15:44 | |
| fungi | donnyd: hrm... i thought we could pause uploads of images on a per-provider basis but so far i'm only finding examples of globally pausing builds of specific images. still digging | 15:45 |
| donnyd | fungi: can we just remove FN from the zuul image upload config? | 15:45 |
| *** yamamoto has quit IRC | 15:45 | |
| fungi | though also if nodepool ceases to be able to contact the api it's probably moot. it'll just keep trying and eventually after dns gets updated or whatever it will work again (may need a builder restart though) | 15:46 |
| *** yamamoto has joined #openstack-infra | 15:46 | |
| donnyd | if its not going to break anything or make something angry, then can I just shut it down when its time? | 15:47 |
| AJaeger | fungi: doesn't setting to -1 pause image uploads as well? | 15:47 |
| fungi | donnyd: yeah, we design it to be robust in the face of total cloud outages | 15:47 |
| fungi | AJaeger: maybe? for some reason i thought that didn't apply to nodepool v3 but i'll check the docs | 15:47 |
| Shrews | fungi: i think we can just remove the diskimage reference from the provider? that would delete any uploads of that image though, iirc | 15:48 |
| *** admcleod has quit IRC | 15:48 | |
| donnyd | Shrews: the old controller (and images) will be gone anyways | 15:48 |
| donnyd | I finally have something that will have some better availibility | 15:48 |
| fungi | AJaeger: ftr, the last occurrence i find of setting max-servers:-1 in nodepool.yaml was in 2017 | 15:49 |
| AJaeger | wow ;( | 15:49 |
| donnyd | not quite full HA, but much better than what I have previously had | 15:49 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add ensure-java role https://review.opendev.org/708901 | 15:49 |
| Shrews | fyi, https://zuul-ci.org/docs/nodepool/operation.html#removing-from-the-builder | 15:49 |
| Shrews | fungi: donnyd: ^^ | 15:49 |
| Shrews | setting to empty list lets us cleanup all-the-things properly | 15:50 |
| donnyd | oh nice | 15:51 |
| fungi | yeah, if the whole environment is being rebuilt and glance will be a blank slate then we'll need to do that as well | 15:51 |
| donnyd | is that is project-config | 15:51 |
| fungi | yep, openstack/project-config specifically | 15:51 |
| donnyd | ok cool | 15:51 |
| *** yamamoto has quit IRC | 15:51 | |
| donnyd | I will get my edits in soon | 15:51 |
| donnyd | thank you fungi Shrews AJaeger | 15:52 |
| fungi | thanks to you as well donnyd! looking forward to the new and improved fn | 15:52 |
| donnyd | I am sure I will need to tune a few things... but its seems to be much faster to me... and now I have some real object storage | 15:53 |
| donnyd | 10 nodes of ceph... much mo betta than two nodes of swift | 15:53 |
| fungi | nice! | 15:54 |
| donnyd | not that there is anything wrong with swift - just needed to dual hat that part of the array for one specific task | 15:54 |
| donnyd | I have one last thing to buy - and that is new UPS | 15:54 |
| donnyd | power around this time of year is usually not an issue | 15:55 |
| donnyd | They should be in by the end of apr | 15:55 |
| mordred | infra-root: I have to run to the DMV - I'm taking my laptop in case it's a complete disaster zone | 15:56 |
| donnyd | mordred: I am not jealous of you. You should also take a mobile speaker so you have music... it will lessen the misery of the DMV | 15:57 |
| Shrews | mordred: when is the dmv ever NOT a complete disaster zone? | 15:57 |
| fungi | if you do bring music, be sure to listen to primus's "dmv" | 15:58 |
| fungi | anyone who's been there knows exactly what i mean | 15:58 |
| zbr | fungi: what should I do about broken platforms and I add tests for our roles? like https://review.opendev.org/#/c/709139/ | 16:01 |
| zbr | can I make the broken ones -nv manually? so we can merge it and later fix them incrementally? | 16:01 |
| fungi | zbr: are those generally broken, or only broken with the addition of that change? | 16:03 |
| zbr | fungi: they were broken because they where never tested, this change only enable testing. | 16:03 |
| fungi | zbr: yeah, then i would either skip those platforms for now or set them non-voting. i tend to prefer the former unless someone is actively trying to get them to work | 16:04 |
| fungi | since the latter just means we needlessly run broken jobs and waste resources | 16:05 |
| zbr | fungi: can i use alternative tag for all-platforms, like listing only those that I want? or I only need to manually remove the magic tag and the extra jobs? | 16:06 |
| zbr | i ended up with this matrix, due to the use of the tool, but what if I want to enable multi-platform with a limited set? | 16:06 |
| fungi | i'll have to read up on how that's implemented | 16:07 |
| fungi | 709139 seems to actually add those jobs | 16:08 |
| fungi | in zuul-tests.d/container-roles-jobs.yaml | 16:08 |
| fungi | so could you just not add the broken jobs? | 16:08 |
| *** lpetrut has quit IRC | 16:09 | |
| AJaeger | any infra-root around to help push releases.o.o to AFS? It needs a review of https://review.opendev.org/#/c/709009 (and stack as well for some cleanups) and then changing DNS once that is in. The publishing job is already using AFS, so we currently publish old content... | 16:10 |
| openstackgerrit | Merged openstack/project-config master: Fix promote-service-types-authority https://review.opendev.org/709060 | 16:10 |
| *** rfolco is now known as rfolco|eats | 16:11 | |
| fungi | AJaeger: i'm about to disappear for an early lunch so am hesitant to approve stuff like that and run, but i can take a look when i get back if nobody beats me to it | 16:11 |
| AJaeger | thanks, fungi. Lunch is more important ;) | 16:12 |
| fungi | my stomach agrees with that sentiment ;) | 16:13 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Enable testing of install-docker on multiple platforms https://review.opendev.org/709139 | 16:16 |
| openstackgerrit | Merged zuul/zuul master: Disable misfire grace time of apscheduler job https://review.opendev.org/707842 | 16:18 |
| zbr | fungi: i updated the review and added only working ones and centos-8 as nv, which I will fix in a follow-up. | 16:18 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Enable testing of install-docker on multiple platforms https://review.opendev.org/709139 | 16:20 |
| openstackgerrit | Merged zuul/zuul master: Spec for allowing circular dependencies https://review.opendev.org/643309 | 16:24 |
| *** Wryhder has joined #openstack-infra | 16:31 | |
| fungi | headed out for food, back as soon as i can be | 16:32 |
| *** Lucas_Gray has quit IRC | 16:32 | |
| *** Wryhder is now known as Lucas_Gray | 16:32 | |
| clarkb | AJaeger: looking | 16:32 |
| *** rfolco|eats is now known as rfolco | 16:33 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Fix install-docker on centos-8 https://review.opendev.org/709184 | 16:36 |
| AJaeger | thanks, clarkb. | 16:37 |
| AJaeger | clarkb: want to review rest of stack as well? | 16:38 |
| clarkb | AJaeger: yup working through it too | 16:38 |
| AJaeger | thanks | 16:38 |
| *** lmiccini has quit IRC | 16:39 | |
| clarkb | looks like we decided to go from files.openstack to static.opendev. I guess because static.opendev is bionic and ansible configured so represents way of future | 16:42 |
| smcginnis | Was it ever determined what all these "Waiting on logger" messages are from in the logs? | 16:46 |
| *** iurygregory has quit IRC | 16:46 | |
| *** admcleod has joined #openstack-infra | 16:48 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove implemented specs https://review.opendev.org/701435 | 16:49 |
| clarkb | smcginnis: ansible doesn't do streaming logs out of the box. Zuul hacks around that by having the initial command task on the remote side start a daemon that tails a log file and serves it on port 19885. On the zuul executor side running the job it waits for port 19885 to start listening. That wait is what causes those messages | 16:50 |
| clarkb | smcginnis: unless the wait is signficant (like more than several seconds) its probably fine to ignore | 16:51 |
| clarkb | AJaeger: ianw I think https://review.opendev.org/#/c/709024/5 needs some cleanup. AJaeger its straightforward if you want to give making those changes a go. This should add the new sites without stopping to serve them on the old server meaning its pretty safe to make mistakes here as long as we check before updating DNS. But getting it right is good too :) | 16:52 |
| clarkb | corvus: ^ that change will move where zuul-ci.org is hosted if you want to give it a look over too | 16:52 |
| AJaeger | clarkb: let me try... | 16:52 |
| *** lucasagomes has quit IRC | 16:53 | |
| clarkb | fungi: when you return from lunch I think your input on https://review.opendev.org/#/c/709038/3 would be valuable as that our only case of simple server log parsing and I know you have been looking at that sort of thing for zuul recently. Maybe there is a better tool we can replace it with globally for all the vhosts instead | 16:54 |
| *** mattw4 has joined #openstack-infra | 16:56 | |
| smcginnis | clarkb: Thanks for the explanation. Just concerned about the spam in the logs, but it's not too much. | 16:58 |
| AJaeger | clarkb: should I remove the --insecure? | 16:58 |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: static: move afs sites from files.openstack.org to static.opendev.org https://review.opendev.org/709024 | 16:59 |
| clarkb | AJaeger: no I think that is necessary for the LE testing to work because our test jobs don't actually get real LE certs | 16:59 |
| clarkb | AJaeger: I just wanted to point out why the testing didn't catch the bug | 16:59 |
| *** gyee has joined #openstack-infra | 16:59 | |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Revert "Generate list of 404s for docs.o.o" https://review.opendev.org/709038 | 16:59 |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: static: add static.openstack.org/files.openstack.org https://review.opendev.org/709042 | 17:00 |
| clarkb | fungi: after a quick google goaccess has an anonymize-ip option. That might work? | 17:00 |
| *** tesseract has quit IRC | 17:00 | |
| AJaeger | clarkb: Ok, I got it right - thanks | 17:00 |
| clarkb | smcginnis: we may want zuul to log total time waiting once it stops waiting? | 17:01 |
| clarkb | problem is if it never stops :/ | 17:01 |
| AJaeger | config-core, two cleanups for review, please: https://review.opendev.org/#/c/708920/ and https://review.opendev.org/#/c/708925/ | 17:01 |
| smcginnis | clarkb: Looks like it times out of something and does log the times. Would probably be nice if it could just do something like "Previous message repeats 199 times", but it's not a big concern. | 17:02 |
| clarkb | smcginnis: the problem is it is a live stream | 17:03 |
| clarkb | smcginnis: you don't know how many times it will repeat until you get there and once you get there it has been written | 17:04 |
| clarkb | (and I don't think zuul should be in the job of rewriting logs) | 17:04 |
| smcginnis | eah | 17:04 |
| smcginnis | Yeah | 17:04 |
| clarkb | fungi: the version of goaccess on bionic is too old to have the anonymize-ip option :/ | 17:05 |
| clarkb | however they do publish repos on their own deb repo | 17:06 |
| *** Lucas_Gray has quit IRC | 17:06 | |
| openstackgerrit | Merged opendev/system-config master: static: add releases.openstack.org site https://review.opendev.org/709009 | 17:12 |
| *** dtantsur is now known as dtantsur|afk | 17:16 | |
| AJaeger | clarkb: do you want to make the DNS changes, please? ^ | 17:19 |
| clarkb | AJaeger: yes, we'll need to wait for that to apply and then test it using local /etc/hosts override but once that looks good I can update dns | 17:20 |
| AJaeger | ok, thanks | 17:21 |
| clarkb | fungi: ok goaccess anonymize-ip only truncates the last octet | 17:21 |
| clarkb | its not a very good anonymization. The data it produces is probably the sort of thing we would want though | 17:21 |
| clarkb | I wonder if we can convince them to do a salted one way hash version | 17:22 |
| *** tosky has quit IRC | 17:23 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Fix install-docker on centos-8 https://review.opendev.org/709184 | 17:30 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-website master: Add redirect for /docs/zuul/user/ https://review.opendev.org/709195 | 17:32 |
| *** evrardjp has quit IRC | 17:34 | |
| *** ociuhandu_ has joined #openstack-infra | 17:35 | |
| *** evrardjp has joined #openstack-infra | 17:35 | |
| *** ociuhandu has quit IRC | 17:38 | |
| *** ociuhandu_ has quit IRC | 17:39 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Enable testing of install-docker on multiple platforms https://review.opendev.org/709139 | 17:39 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Fix install-docker on centos-8 https://review.opendev.org/709184 | 17:40 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Fix install-docker on centos-8 https://review.opendev.org/709184 | 17:43 |
| *** amoralej is now known as amoralej|off | 17:44 | |
| fungi | back from lunch and catching up now | 17:44 |
| *** pkopec is now known as pkopec|afk | 17:46 | |
| *** pkopec|afk has quit IRC | 17:46 | |
| *** jpena is now known as jpena|off | 17:47 | |
| fungi | clarkb: i think the sort of details the osf marketing folks are interested in is going to require something along the lines of web bugs, but i believe we could cover that at a local url at the same domain as each site by doing apache mod_proxy to a local socket. my current likely candidate is https://www.goatcounter.com/ which doesn't seem to leak any ip network details, just compares to a geoip list | 17:47 |
| fungi | and then discards addresses | 17:47 |
| fungi | for 404 tracking, i suspect we'd need a custom 404 page which added the reference to cause the browser to hit the proxied url | 17:48 |
| fungi | (also goatcounter is distributed under an osi-recognized license, the "pricing" details confusingly included in that page are if you want to use their hosted service instead of running it yourself) | 17:49 |
| clarkb | fungi: does goatcounter not list the 404 hits? goaccess does | 17:49 |
| fungi | goatcounter doesn't perform log analysis, so i don't think it would be aware of 404 hits unless it was called from a custom 404 page. one of the obvious downsides to web bugs vs log analysis | 17:50 |
| clarkb | oh I see | 17:50 |
| fungi | i believe popular solutions like urchin/ga work the same way in that regard | 17:51 |
| clarkb | fungi: fwiw all of the data on goatcounters page except for screensize is available via log analysis | 17:51 |
| clarkb | that makes me wonder if we wouldn't be better served by log analysis instead | 17:51 |
| clarkb | (since it can give us request result info too) | 17:51 |
| fungi | yeah, if we can get goaccess to do a more thorough job of hiding ip address info it might be an interesting option too | 17:52 |
| fungi | fwiw, this is the first i've heard of goaccess so still reading up on it | 17:52 |
| clarkb | fungi: it occured to me that we could preanonymize the ips we feed into goaccess | 17:52 |
| clarkb | so I'm fiddling with that idea now | 17:52 |
| clarkb | fungi: process would be make log copy, edit ips somehow, run goaccess | 17:53 |
| clarkb | using salted one way hash with a different salt for each run may be sufficient? | 17:53 |
| fungi | and i only discovered goatcounter a month or so ago... seems there are new solutions to these problems cropping up as gdpr and other forces steer sites to less risky traffic analysis | 17:53 |
| clarkb | Then we run that daily and you shouldn't be able to brute force the one way hash over the 32bit ipv4 addr space | 17:53 |
| fungi | well, the risk with pseudonymous unique hashes is those still represent individual users, so you could in theory work some of them out through behavior analysis | 17:54 |
| AJaeger | fungi: we have for docs.openstack.org a custom 404 page, can add content to it if you want | 17:54 |
| clarkb | ah I see | 17:54 |
| clarkb | AJaeger: ya one of the changes in the files.o.o stack removes that | 17:55 |
| clarkb | AJaeger: whcih is why I started looking at alternatives that are richer that we could possibly apply across the board | 17:55 |
| AJaeger | clarkb: I know - and I'm not really happy about it. I use it twice a year and always find something. | 17:55 |
| fungi | i did this for a living at one point... unique hashes can be reversed to people through non-cryptographic means. anonymization goes hand in hand with aggregation | 17:55 |
| clarkb | gotcha | 17:56 |
| fungi | "safety in numbers" | 17:56 |
| clarkb | that is probably why goaccess truncates the last octect of ipv4 addrs | 17:56 |
| fungi | so as long as what we serve is aggregate daya i think that's fine | 17:56 |
| clarkb | except that still produces largely unique users unless you have significant traffic | 17:56 |
| AJaeger | clarkb: just pushed four or five changes to get rid of the top 4 lines from http://files.openstack.org/docs-404s/2020-02-21.txt | 17:56 |
| clarkb | fungi: aggegate in what way? | 17:56 |
| fungi | disassociating individual statistics from their constituent identities, and totalling them separately | 17:57 |
| fungi | you don't need to know how many people with this browser string visited that page. it's sufficient to know how many people had this browser string and how many people visited that page, separately | 17:58 |
| *** derekh has quit IRC | 17:58 | |
| clarkb | fungi: yes goaccess does that | 17:59 |
| *** diablo_rojo has joined #openstack-infra | 17:59 | |
| clarkb | they are separate tables and you can't cross reference from what is presented to you (at least I haven't figured out how yet) | 17:59 |
| *** jamesmcarthur has joined #openstack-infra | 17:59 | |
| fungi | ultimately it's a balancing act. the more precise information you have on visitors the more insights you can infer, however the more precise information you publish on visitors the easier you make it for others to figure out who's doing what | 17:59 |
| clarkb | fungi: you can take a look at files.openstack.org:/home/clarkb/goaccess-report.html | 18:00 |
| fungi | sure! | 18:00 |
| zbr | i am really concerned that one of you may be able to figure out, which docs pages I am visiting, based on my browser signature. | 18:02 |
| openstackgerrit | Merged zuul/zuul master: Remove implemented specs https://review.opendev.org/701435 | 18:02 |
| fungi | clarkb: wow that html file is really not human readable ;) | 18:02 |
| clarkb | zbr: that is true of any website on the internet? | 18:02 |
| clarkb | zbr: an unfortunate reality is that in order to make tcp connections and functional http requests we give up a lot of information about ourselves to servers | 18:03 |
| zbr | clarkb: yep, that prevents me from sleeping at night | 18:03 |
| clarkb | typically mitigation for such things involves TOR and clean browser installs | 18:03 |
| fungi | zbr: it's not so much "one of us" as we want to be able to publish this data across all the websites we're running | 18:03 |
| clarkb | (extensions are a major fingerprinting breadcrumb aiui) | 18:04 |
| fungi | because other organizations want to know details about the popularity of what we're hosting, and we don't feel comfortable having "privileged" relationships where we give organizations exclusive private access to possibly personally identifying information | 18:05 |
| clarkb | fungi: ya you'll want to view it in a browser | 18:05 |
| zbr | security and privacy, two mirages of this millennia | 18:05 |
| fungi | while i agree that too many folks are working to make security and privacy hard (or impossible) i don't feel like that means we should just give up on the idea ourselves | 18:06 |
| clarkb | fwiw that report doesnt' cross reference any of the tables | 18:06 |
| fungi | better that we set an example | 18:06 |
| clarkb | so you can't know which chrome users have which IPs or which windows uers are using chrome etc | 18:06 |
| clarkb | you only get each table independently (which depending on the data set could still be cross referenceable I think) | 18:06 |
| fungi | zbr: the "easy" way out would be to just publish all our web server logs. clearly that leaks a lot of pii, and may even be outright illegal under some circumstances (as well as ethically poor judgement on out part) | 18:07 |
| fungi | zbr: also, we have gleaned insights in the past looking at webserver logs that suggest some organizations which may be using our software who may not want that information to be public, and wouldn't feel right outing them, especially since it could be counter-productive | 18:10 |
| zbr | it was a long week for me, and legal stuff is not the best way to end it with, but I do see the points. | 18:11 |
| fungi | clarkb: yeah, the only one which worries me is the "Visitor Hostnames and IPs" table, would be especially useful if they could replace that with a geoip lookup so you get some attempt at global visitor distribution (inaccurate as that tends to be) | 18:11 |
| clarkb | fungi: there is geoip support in goaccess, but I expect that is in addition to, not exclusion of the visitor/ip table | 18:11 |
| fungi | i wonder if we there's an option to just leave that data out of the report then? | 18:12 |
| clarkb | oh we can actually provide a list of panels we want | 18:12 |
| clarkb | ya | 18:12 |
| clarkb | let me test this more | 18:12 |
| zbr | when you have few minutes maybe you can help me merge these two: https://review.opendev.org/#/c/709184/4 (and parent) | 18:12 |
| fungi | the other thing which is possibly helpful is if it can break out "local" (same site) referrers from the rest. a lot of web log analysis tools do that so maybe it can too | 18:13 |
| *** jamesmcarthur has quit IRC | 18:14 | |
| fungi | the split between referrers urls and referring sites gets close to that though, so not strictly necessary | 18:14 |
| fungi | but nice to know if a specific blog post on some site is driving a lot of traffic to specific pages | 18:14 |
| fungi | zbr: yep, looking at those now. thanks for the heads uip | 18:15 |
| fungi | er, heads up | 18:15 |
| zbr | i am slowly learning about update-test-platform, even make small bugfix. | 18:15 |
| zbr | abstract jobs are not supposed to endup in check/gate. | 18:16 |
| clarkb | ya --ignore-panel HOSTS seems to work | 18:16 |
| fungi | zbr: correct, an abstract job should never be run directly | 18:16 |
| clarkb | fungi: considering ^ we wouldn't even need --anonymize-ip support in the ubuntu version because we'd just delete the panel entirely | 18:16 |
| fungi | there's also the opposite, a "final" job, which should never be inherited from | 18:16 |
| zbr | i kinda find confusing the anchor compression logic. | 18:16 |
| fungi | clarkb: yeah, sounds fine to me | 18:17 |
| fungi | thanks for working on this. i ran into a bunch of dead ends with all the tools i tried up to now | 18:17 |
| fungi | but also never found goaccess | 18:17 |
| fungi | very promising | 18:17 |
| clarkb | fungi: `./goaccess /path/to/apache/access.log -o report.html --log-format=COMBINED --ignore-panel HOSTS` | 18:18 |
| fungi | neat | 18:19 |
| clarkb | I'm checking now if my local compile compiled geolocation support | 18:19 |
| clarkb | doesn't look like it so I can't easily test that functionality but in theory it exists | 18:20 |
| clarkb | fungi: I think the last thing we need to potentially worry about is referers. We can disable that panel too. I'm not quite sure myself yet if posting that is good or bad | 18:20 |
| fungi | generally it's safe as long as get parameters are stripped | 18:21 |
| clarkb | I can see how marketers would really like that data, but it could also point out people using software that may not wish to be called out | 18:21 |
| clarkb | fungi: ya the default is only domain name (the urls are stripped too) | 18:21 |
| clarkb | fungi: you can see that in my example | 18:21 |
| *** jamesmcarthur has joined #openstack-infra | 18:22 | |
| fungi | but yeah, say some large company has links in an internal wiki or bug tracker and then people are arriving at the site by clicking those links... | 18:22 |
| AJaeger | clarkb: thanks for looking into these 404s | 18:22 |
| AJaeger | (and fungi as well) | 18:22 |
| fungi | clarkb: yeah, i agree that's potentially risky | 18:22 |
| *** jamesmcarthur_ has joined #openstack-infra | 18:23 | |
| fungi | whatever data we're opting to exclude, we should document the reasons for those choices to so we don't have to remember when folks ask us | 18:23 |
| clarkb | ++ | 18:23 |
| clarkb | also we can start conservative and have a minimal set of data there | 18:23 |
| clarkb | then people can argue to turn on extra bits | 18:23 |
| fungi | but yeah, what i don't want is to start sharing potentially sensitive visitor data with people under a non-disclosure agreement | 18:24 |
| *** ykarel is now known as ykarel|away | 18:24 | |
| clarkb | maybe start with just: visitor count, requested files, static requests, 404s, http status codes | 18:25 |
| fungi | i think we should draw the line that we provide whatever visitor information we deem safe in public, for anyone to consume if they find it useful | 18:25 |
| fungi | and we don't have special relationships to provde sensitive data to anybody | 18:25 |
| clarkb | ++ | 18:25 |
| clarkb | let me generate my proposed list report as I think that fits your proposal | 18:25 |
| *** jamesmcarthur has quit IRC | 18:26 | |
| fungi | this could also be a good spec, and maybe the first test of our new governance | 18:27 |
| fungi | because i don't want it to just be the two of us having this discussion | 18:27 |
| AJaeger | I would love to have: 404 page, referer, count | 18:30 |
| fungi | AJaeger: see above discussion about referrers... it can leak internal organizational information for visitors | 18:30 |
| clarkb | cool I think that works. We can --ignore-panel every panel. Then --enable-panel the ones we want | 18:31 |
| AJaeger | fungi: argh ;( Could we limit it to our own pages? My goal is to fix broken links if possible | 18:31 |
| clarkb | fungi: check files.openstack.org:/home/clarkb/goaccess-clean-report.html | 18:31 |
| fungi | AJaeger: we might be able to preprocess the logs to filter outside domains | 18:31 |
| AJaeger | ++ | 18:31 |
| clarkb | AJaeger: fungi goaccess 404s reporting doesn't report the referrer | 18:32 |
| AJaeger | or just summarize "unknown" or "outside" | 18:32 |
| AJaeger | so, no improvement to what we have ;( | 18:32 |
| clarkb | you get hits, request type, and url | 18:32 |
| fungi | so if the domain in the referrer url is not on the list of local domains, then we stream edit to replace it with a common placeholder? | 18:32 |
| *** openstackstatus has quit IRC | 18:33 | |
| fungi | (before feeding to the analyzer) | 18:33 |
| clarkb | fungi: well ist a noop right now beacuse analyzer doesn't look at it | 18:33 |
| clarkb | you don't get the data either way | 18:33 |
| fungi | clarkb: yeah, i mean as far as AJaeger's request to include referrers so we know where 404 hits come from, though i think this tool isn't going to tell us that either way | 18:34 |
| clarkb | right. That said preprocessing the input data is a reasonable appraoch I think | 18:34 |
| clarkb | assuming tools mostly support what we want | 18:34 |
| fungi | we may still need a separate report to list (our own) references to 404 pages | 18:35 |
| fungi | knowing what nonexistent pages people are requesting is useful, but not as useful as knowing where we're linking to pages which don't exist | 18:35 |
| fungi | the sanitized view lgtm, btw | 18:35 |
| *** diablo_rojo has quit IRC | 18:36 | |
| *** diablo_rojo has joined #openstack-infra | 18:37 | |
| clarkb | fungi: I've also clicked the export json button to make sure we aren't just avoiding render of the other panels. But the json lacks that data too | 18:37 |
| * fungi did the same, yep | 18:37 | |
| clarkb | I think this could be run against all of our apache access logs pretty safely | 18:37 |
| fungi | that's definitely what i've been looking for | 18:38 |
| fungi | might be interesting to get the same for gitea, where there's no apache | 18:38 |
| fungi | i wonder if we can get apache-format access logs from it | 18:38 |
| clarkb | fungi: you can configure log formats on the goaccess side too | 18:38 |
| fungi | oh, nice | 18:39 |
| clarkb | that was actually where I leanred about goaccess. I used it to debug haproxy when we had gitea problems in the past | 18:39 |
| clarkb | I should've written down my log format for haproxy logs somewhere :/ | 18:39 |
| fungi | haproxy has no insight into our https traffic though because we terminate crypto on the backends | 18:39 |
| clarkb | fungi: correct I was trying to identify traffic rates by source and backend | 18:40 |
| clarkb | which was all layer 3 data | 18:40 |
| clarkb | it mostly worked but was clunky. | 18:40 |
| clarkb | I think feeding gitea logs in directly would largely just work | 18:42 |
| clarkb | just need to write the log parser format tring | 18:42 |
| clarkb | also it can run as a real time server | 18:44 |
| fungi | neat | 18:45 |
| AJaeger | clarkb: did ansible run for releases.o.o change? | 18:46 |
| clarkb | AJaeger: looking | 18:46 |
| clarkb | AJaeger: it did but it had a problem verifying the LE cert. Looks like we didn't update dns for the acme challenge | 18:49 |
| clarkb | I'll do that now which should fix the cert and when that is fixed we can swap main dns | 18:49 |
| AJaeger | ah | 18:50 |
| *** jamesmcarthur_ has quit IRC | 18:52 | |
| AJaeger | clarkb: so, wait another hour? | 18:52 |
| clarkb | AJaeger: yes, sorry about that. I didn't think to check the acme dns record | 18:54 |
| clarkb | I'm tailing the acme.sh log on the server now so should see when that goes through | 18:54 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Enable testing of install-docker on multiple platforms https://review.opendev.org/709139 | 18:55 |
| clarkb | I'm going to step out for a few while we want for that. Need breakfast/lunch | 18:55 |
| clarkb | fungi: re the server version of goaccess that might be tricky because it listens on a port for websockets. We'd have to set up a separate websocket for each report I think which on a host like static.opendev.org might get painful | 18:57 |
| AJaeger | clarkb: no worries - just wanted to get an estimate. Get your food! | 18:57 |
| clarkb | its possible we could feed it all the access logs and it would break it down by vhost not sure | 18:58 |
| clarkb | and ya I should eat | 18:58 |
| fungi | clarkb: i don't feel like rt reporting is critical. running it each time we do log rotation is probably fine | 18:59 |
| *** jamesmcarthur has joined #openstack-infra | 18:59 | |
| fungi | config-core: anyone free for a second review on https://review.opendev.org/708961 (an extremely barebones project creation change)? | 19:01 |
| clarkb | fungi: ya I agree | 19:01 |
| *** jamesmcarthur_ has joined #openstack-infra | 19:01 | |
| *** jamesmcarthur has quit IRC | 19:04 | |
| AJaeger | fungi: I just asked back whether it needs a git import, otherwise fine. Let me +2 and we can +A once the submitter answers | 19:04 |
| fungi | AJaeger: yeah, i'm pretty sure msdisme (he's not in here, but lurking in #opendev and #openinfralabs) plans to start with an empty repo, but it's good to ask to be sure. thanks | 19:05 |
| openstackgerrit | Michal Arbet proposed openstack/project-config master: Add xstatic-** projects for vitrage-dashboard https://review.opendev.org/704133 | 19:09 |
| *** igordc has joined #openstack-infra | 19:10 | |
| *** ianychoi_ has joined #openstack-infra | 19:15 | |
| *** ianychoi has quit IRC | 19:18 | |
| *** ijw has joined #openstack-infra | 19:21 | |
| *** ociuhandu has joined #openstack-infra | 19:22 | |
| AJaeger | config-core, two cleanups for review, please: https://review.opendev.org/#/c/708920/ and https://review.opendev.org/#/c/708925/ | 19:23 |
| openstackgerrit | Merged zuul/zuul-jobs master: Enable testing of install-docker on multiple platforms https://review.opendev.org/709139 | 19:25 |
| *** ociuhandu has quit IRC | 19:26 | |
| openstackgerrit | Merged opendev/system-config master: static: move site setup to a loop https://review.opendev.org/709016 | 19:27 |
| *** ociuhandu has joined #openstack-infra | 19:34 | |
| *** diablo_rojo has quit IRC | 19:37 | |
| *** diablo_rojo has joined #openstack-infra | 19:37 | |
| *** jamesmcarthur_ has quit IRC | 19:38 | |
| mordred | infra-root: back from DMV ... I miss anything good? | 19:38 |
| fungi | just several hours of your life | 19:40 |
| fungi | oh, and discussions about publishing web access stats | 19:41 |
| mordred | fungi: those sound similar? | 19:42 |
| fungi | shouldn't you be carnivaling anyway? | 19:43 |
| mordred | fungi: well - parades don't start until 16:45 today - which is actually a bit early but last night the weather was bad enough that they cancelled Muses, so it's running this afternoon instead | 19:43 |
| mordred | fungi: BUT | 19:43 |
| mordred | fungi: we've already mardied much gras and may hide from a good bit of the gras from this point on | 19:44 |
| mordred | in other news .... | 19:44 |
| mordred | https://review.opendev.org/#/c/708018/ is GREEN! | 19:45 |
| fungi | ahh, you accidentally staggered onto bourbon street? | 19:45 |
| fungi | that would certainly end it for me | 19:45 |
| mordred | fungi: oh good golly no | 19:46 |
| *** jamesmcarthur has joined #openstack-infra | 19:46 | |
| AJaeger | mordred: could you review https://review.opendev.org/#/c/709062/ , please? Hope it's publishing correctly now... | 19:46 |
| mordred | fungi: but we did parades last fri-sun then a parade and ball in mobile on tuesday | 19:46 |
| fungi | yeah, that sounds like plenty | 19:47 |
| mordred | AJaeger: done! | 19:47 |
| AJaeger | thanks, mordred | 19:48 |
| *** jamesmcarthur has quit IRC | 19:52 | |
| clarkb | AJaeger: can you test with 23.253.245.150releases.openstack.org in /etc/hosts ? | 19:53 |
| clarkb | AJaeger: it seems to be workign for me now and if you confirm that is the correct up to date content I will update dns for releases.openstack.org to be a CNAME to static.opendev.org | 19:53 |
| AJaeger | clarkb: looks correct for me | 19:55 |
| AJaeger | yep, contains just released packages... | 19:56 |
| AJaeger | mordred: service-type publish failed, I'm on it | 19:58 |
| clarkb | AJaeger: I will update dns shortly then | 19:58 |
| *** eharney has quit IRC | 19:59 | |
| mordred | AJaeger: :( | 20:00 |
| mordred | AJaeger: let me know if you want me to look in to anything | 20:00 |
| *** ralonsoh has quit IRC | 20:01 | |
| mordred | clarkb: https://review.opendev.org/#/c/708018/ and its depends-on changes are fun if you want to look at them | 20:01 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix service-type publishing https://review.opendev.org/709231 | 20:03 |
| AJaeger | mordred: this one, please ^ | 20:03 |
| AJaeger | config-core, service-types-authority patching is too tricky for me - could I get another review, please? ^ | 20:13 |
| clarkb | looking as soon as I update dns for releaes | 20:23 |
| clarkb | ok dns has been updated and it had a short 5 minute ttl before so should go into production quickly | 20:25 |
| clarkb | AJaeger: ^ maybe you can confirm at 20:30utc | 20:25 |
| *** eharney has joined #openstack-infra | 20:32 | |
| AJaeger | clarkb: LGTM, thanks | 20:36 |
| openstackgerrit | Merged openstack/project-config master: Fix service-type publishing https://review.opendev.org/709231 | 20:36 |
| AJaeger | mordred: could you do another review, please: https://review.opendev.org/#/c/709232/ - to test the job | 20:37 |
| openstackgerrit | Clark Boylan proposed opendev/infra-specs master: Bump blockdiag dep so that we build on newer python3 https://review.opendev.org/709235 | 20:44 |
| openstackgerrit | Clark Boylan proposed opendev/infra-specs master: Add spec for website stat tracking effort https://review.opendev.org/709236 | 20:44 |
| clarkb | infra-root ^ that first change I'll just push through if CI passes on it. fungi mordred that second one tries to being to capture what we learned this mroning so that it isn't all gone tomorrow | 20:45 |
| clarkb | its still very WIPy but I think has the important breadcrumbs for turning this into something real | 20:45 |
| clarkb | AJaeger: hrm I didn't catch it earlier but the http://releases.openstack.org redirect to https is redirecting to https://static.opendev.org not https://releases.openstack.org | 20:46 |
| clarkb | AJaeger: I'm working to sort that out now | 20:47 |
| clarkb | oh hrm it works in a second browser so that may just be bad caching when I was debugging the LE cert issue | 20:48 |
| *** ociuhandu has quit IRC | 20:48 | |
| clarkb | can someone else check http://releases.openstack.org when they have a moment? | 20:48 |
| *** ociuhandu has joined #openstack-infra | 20:49 | |
| *** ociuhandu has quit IRC | 20:51 | |
| *** ociuhandu has joined #openstack-infra | 20:51 | |
| clarkb | mordred: comment on https://review.opendev.org/#/c/708018/15 I think that is more like -0 than -1 but I was pretty confused by what was going on there for a bit | 20:54 |
| clarkb | I think separating as suggested would help avoid that confusion and also make future changes to the gerrit image builds cleaner | 20:55 |
| mordred | clarkb: yeah - that's a good point. It also might be easier to just make a tiny role in zuul-jobs that can be reused so that we don't have to do the add-zuul-to-the-deps thing just to lint that one fiole | 20:58 |
| mordred | file | 20:58 |
| clarkb | mordred: the other downside to that change is now we can't build images locally as easily | 21:00 |
| clarkb | mordred: beacuse docker build isn't sufficient | 21:00 |
| clarkb | but I'm not super concerned about that, the steps to get a release are still there in the ansible and can be used via ansible or reproduced locally | 21:01 |
| clarkb | ok I've fast approved that infra-specs dep change as it fixes things for me locally | 21:07 |
| clarkb | and CI was happy | 21:07 |
| *** ociuhandu has quit IRC | 21:07 | |
| *** ociuhandu has joined #openstack-infra | 21:08 | |
| AJaeger | mordred: thanks! publishing worked, check https://service-types.openstack.org/ | 21:08 |
| *** ociuhandu has quit IRC | 21:09 | |
| AJaeger | clarkb: redirect works correctly with chromium for me | 21:09 |
| mordred | AJaeger: yay! | 21:09 |
| *** ociuhandu has joined #openstack-infra | 21:09 | |
| clarkb | AJaeger: ya I think because we have it as a permanent redirect that when things weren't working with LE before it still cached the wrong thing | 21:09 |
| mordred | clarkb: yeah - I had the same concern - but I think ultimately that should be pretty unfrequent thing we need to do | 21:10 |
| clarkb | AJaeger: it worked fine for me in a different browser | 21:10 |
| AJaeger | great! | 21:10 |
| openstackgerrit | Merged opendev/infra-specs master: Bump blockdiag dep so that we build on newer python3 https://review.opendev.org/709235 | 21:11 |
| AJaeger | Now we have two cleanups for review for the static AFS changes for review, please: https://review.opendev.org/#/c/708920/ and https://review.opendev.org/#/c/708925/ | 21:11 |
| fungi | clarkb: http://releases.openstack.org correctly redirects to https for me, yep | 21:14 |
| fungi | in an incognito ff, so shouldn't have that redirect cached either | 21:15 |
| *** happyhemant has quit IRC | 21:17 | |
| clarkb | fungi: if you get a chance your thoughts on https://review.opendev.org/709236 would be great. Also feel free to push a new ps with more info since i Know you had investigated this a bit before | 21:20 |
| AJaeger | Have a great weekend everybody! | 21:25 |
| AJaeger | thanks, clarkb for driving releases.o.o through! | 21:25 |
| clarkb | you too! | 21:34 |
| *** ociuhandu has quit IRC | 21:51 | |
| clarkb | I'm goign to pop out for a bike ride while the weather is nice. Back in a bit | 21:51 |
| *** rfolco has quit IRC | 21:52 | |
| *** ociuhandu has joined #openstack-infra | 21:53 | |
| *** ociuhandu has quit IRC | 21:57 | |
| *** ociuhandu has joined #openstack-infra | 21:57 | |
| fungi | and clarkb yep, already had it up in gertty | 22:00 |
| fungi | have a good weekend AJaeger! | 22:00 |
| *** rlandy has quit IRC | 22:10 | |
| *** xek_ has quit IRC | 22:22 | |
| *** dpawlik has quit IRC | 22:22 | |
| *** exsdev has quit IRC | 22:24 | |
| *** icey has quit IRC | 22:24 | |
| *** dosaboy has quit IRC | 22:25 | |
| *** ociuhandu has quit IRC | 22:27 | |
| *** ociuhandu has joined #openstack-infra | 22:27 | |
| *** ijw has quit IRC | 22:28 | |
| openstackgerrit | Merged openstack/project-config master: Create openinfralabs/contrib repo https://review.opendev.org/708961 | 22:29 |
| *** ijw has joined #openstack-infra | 22:30 | |
| *** ijw has quit IRC | 22:32 | |
| *** ociuhandu has quit IRC | 22:32 | |
| *** ijw has joined #openstack-infra | 22:32 | |
| *** ijw has quit IRC | 22:34 | |
| *** mattw4 has quit IRC | 22:35 | |
| *** mattw4 has joined #openstack-infra | 22:36 | |
| *** ijw has joined #openstack-infra | 22:36 | |
| *** rfolco has joined #openstack-infra | 23:03 | |
| *** bdodd has quit IRC | 23:05 | |
| *** matt_kosut has quit IRC | 23:06 | |
| *** matt_kosut has joined #openstack-infra | 23:06 | |
| *** bdodd has joined #openstack-infra | 23:07 | |
| *** rfarr has joined #openstack-infra | 23:11 | |
| *** matt_kosut has quit IRC | 23:11 | |
| *** dave-mccowan has quit IRC | 23:11 | |
| *** rfarr has quit IRC | 23:13 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: DNM Replace kubectl snap with apt repo https://review.opendev.org/709253 | 23:18 |
| mordred | infra-root: ^^ that's a placeholder patch to remind me to write that patch | 23:18 |
| *** ociuhandu has joined #openstack-infra | 23:22 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add pause-buildset-registry role https://review.opendev.org/709256 | 23:26 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add pause-buildset-registry role https://review.opendev.org/709256 | 23:28 |
| *** dave-mccowan has joined #openstack-infra | 23:28 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Full Disable of FortNebula https://review.opendev.org/709257 | 23:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk https://review.opendev.org/708018 | 23:32 |
| mordred | clarkb: ^^ I think that should be cleaner and should address your concern from earlier | 23:33 |
| donnyd | fungi: if you are still kicking around, please lmk if I did that correctly | 23:34 |
| clarkb | mordred: thanks will rereview | 23:35 |
| * donnyd waves at clarkb | 23:35 | |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Use pause-buildset-registry role https://review.opendev.org/709258 | 23:35 |
| donnyd | things are moving along quite well over here, and I am hoping to have something ready in 7(ish) days. | 23:36 |
| *** ijw has quit IRC | 23:38 | |
| fungi | donnyd: yep, looking now, thanks! | 23:38 |
| *** ijw has joined #openstack-infra | 23:39 | |
| *** ociuhandu has quit IRC | 23:39 | |
| fungi | lgtm, just want to get a second pair of config-core eyes on that | 23:40 |
| *** ociuhandu has joined #openstack-infra | 23:40 | |
| *** ociuhandu has quit IRC | 23:45 | |
| *** mattw4 has quit IRC | 23:49 | |
| *** ijw has quit IRC | 23:49 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: k8s: allow port forwarding https://review.opendev.org/709259 | 23:49 |
| *** ijw has joined #openstack-infra | 23:49 | |
| *** exsdev has joined #openstack-infra | 23:50 | |
| corvus | donnyd: is it shutting down permanently? | 23:51 |
| corvus | oh | 23:51 |
| fungi | it's being resurfaced, apparently | 23:51 |
| corvus | i just saw the msg from 23:36, sorry | 23:51 |
| *** icey has joined #openstack-infra | 23:51 | |
| fungi | so the old images are going away | 23:51 |
| corvus | that makes sense | 23:51 |
| *** dosaboy has joined #openstack-infra | 23:51 | |
| corvus | (added a note in my review :) | 23:52 |
| fungi | you also removed your vote, presumably caught gertty in the midst of syncing it | 23:52 |
| donnyd | corvus: no, it's being rebuilt to have better availability and now has some real object storage | 23:53 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Stream output from kubectl pods https://review.opendev.org/709261 | 23:54 |
| corvus | fixed | 23:56 |
| fungi | thanks! | 23:56 |
| clarkb | ok that took longer than I expected because laptop crashed | 23:57 |
| *** ociuhandu has joined #openstack-infra | 23:57 | |
| clarkb | donnyd: ohai | 23:58 |
| donnyd | I am going to shut down the mirror node when this change merges. I mainly just need a couple of the v4 IPs to bring the new stuff online | 23:58 |
| clarkb | my old laptop has been very flaky with modern linux | 23:59 |
| clarkb | might need to consider a reaplacement | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!