| auristor | also update afs01.dfw to 1.8.5 first because that is the server that is performing the sending | 00:00 |
|---|---|---|
| fungi | thanks for the reminder, we definitely do need to upgrade it | 00:00 |
| auristor | there are many rx improvements in 1.8 over 1.6. Its effectively the auristor rx code base as of 2012 | 00:00 |
| fungi | oh, right, i remember you mentioning that now | 00:01 |
| ianw | yeah, i at least was working under the lore that we couldn't mix the versions, i don't know where i picked that up from | 00:01 |
| fungi | so definitely worth trying again with 1.8 | 00:01 |
| auristor | it still isn't anywhere close to the current auristor rx performance but its much better than 1.6 | 00:01 |
| auristor | on the fileserver, you have to install a version that is compatible with your vice partitions | 00:02 |
| auristor | what kernel is running on afs01.dfw, afs02.dfw and afs01.ord? | 00:02 |
| clarkb | they are currently ubuntu xenial, but we'd likely upgrade them to at least bionic if doing openafs upgrades | 00:03 |
| ianw | yeah, i'd say skip to focal for a new server and give us more lts | 00:03 |
| clarkb | the volume data is on removable iscsi cinder volumes we can shift around | 00:03 |
| ianw | that said, focal ships a ~pre release again, so we'd be using our PPA 1.8.5 packages | 00:04 |
| auristor | do you remember what kernels were on the systems when you gave up on afs01.ord? | 00:04 |
| clarkb | no that was a while back | 00:04 |
| clarkb | and they were trusty era kernels then | 00:04 |
| fungi | ubuntu continues to pull packages from debian/unstable (which has been doing prereleases of openafs) instead of debian/testing (which has actual release versions of openafs) | 00:05 |
| ianw | auristor: you mean the kafs work ... let me dig up the page | 00:05 |
| auristor | the linux kernel udp performance was turbocharged sometime just before the 3.10 kernel | 00:05 |
| fungi | one of the places where debian and ubuntu support concepts don't quite line up | 00:05 |
| ianw | oh, no you mean the server | 00:05 |
| auristor | iamw: I literally mean the kernel version on the afs01.dfw server and afs01.ord server | 00:05 |
| fungi | it's a patched linux 4.4 | 00:06 |
| auristor | the udp performance is excellent now; it used to be a bit like a second class citizen to tcp | 00:06 |
| fungi | lists itself as "4.4.0-184" but it's basically 184 patches from (mostly) the 4.4 stable branch backported to 4.4.0 | 00:06 |
| openstackgerrit | Merged opendev/system-config master: rax-dns-backup: fix copy file typo https://review.opendev.org/736063 | 00:08 |
| *** mlavalle has quit IRC | 00:15 | |
| mordred | ianw: I'd also say skip to focal | 00:16 |
| mordred | because then we'd be potentially good for a _while_ | 00:16 |
| mordred | I'm excited to hear that we might get better performance from a re-spin of ord if we wanted to try it again | 00:17 |
| mordred | or - even like an afs01.sjc1.vexxhost instead | 00:17 |
| mordred | (get _actual_ off-site) | 00:17 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-virtualenv: call ensure-pip for Xenial https://review.opendev.org/736067 | 00:23 |
| ianw | mordred: any ideas how we could gate test the server bits? | 00:23 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-virtualenv: call ensure-pip for Xenial https://review.opendev.org/736067 | 00:38 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: remove Xenial venv workaround https://review.opendev.org/736068 | 00:38 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-virtualenv: call ensure-pip for Xenial https://review.opendev.org/736067 | 00:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: remove Xenial venv workaround https://review.opendev.org/736068 | 00:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: install from EPEL for Python 2 https://review.opendev.org/736070 | 00:53 |
| *** Meiyan has joined #opendev | 00:57 | |
| ianw | 2020-06-17 01:02:11 | Running vos release. | 01:03 |
| ianw | 2020-06-17 01:02:26 | Released volume mirror.fedora successfully | 01:03 |
| * fungi cheers | 01:05 | |
| ianw | fungi: if you have a sec; https://review.opendev.org/#/c/735773/ will add f32, which should be a good larger incremental test | 01:07 |
| fungi | +2, feel free to self-approve if you're going to keep an eye on it (i didn't check whether there's sufficient quota, presumably you have) | 01:10 |
| ianw | yeah i can, and there is from the graphs | 01:14 |
| *** olaph has quit IRC | 01:14 | |
| *** xiaolin has joined #opendev | 01:40 | |
| openstackgerrit | Merged opendev/system-config master: mirror-update: mirror Fedora 32 https://review.opendev.org/735773 | 01:42 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Remove the -plain job variants https://review.opendev.org/735774 | 01:48 |
| auristor | rsync to mirror.fedora is underway | 02:51 |
| auristor | approximately 7500 store operations so far | 02:52 |
| ianw | it's up to "b" | 03:00 |
| ianw | i probably should have held the lock manually but i'm hoping it copies before the timeout | 03:01 |
| *** ysandeep|away is now known as ysandeep | 03:55 | |
| *** ykarel|away is now known as ykarel | 04:08 | |
| AJaeger | ianw: do we need ensure-pip before revoke-sudo or is afterwards fine as well? See https://review.opendev.org/#/c/736043/1/playbooks/release/pre.yaml | 04:45 |
| AJaeger | ianw: do you have time to fix https://review.opendev.org/736032 up, please? I need to run some errands first but think this is important to have | 04:46 |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove the -plain job variants https://review.opendev.org/735774 | 04:48 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-pip: install from EPEL for Python 2 https://review.opendev.org/736070 | 04:53 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-virtualenv: call ensure-pip for Xenial https://review.opendev.org/736067 | 04:53 |
| *** diablo_rojo has quit IRC | 04:53 | |
| auristor | about 2.3GB of data written in 300,000 rpcs so far. I'm done for the night. | 05:06 |
| *** factor__ has joined #opendev | 05:19 | |
| *** icarusfactor has quit IRC | 05:22 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Ensure pip is installed for propose-update-constraints https://review.opendev.org/736032 | 05:56 |
| AJaeger | ianw: can we include a role in pre-task? https://review.opendev.org/#/c/736032/2/playbooks/proposal/pre.yaml uses pip in pre_task... | 05:58 |
| ianw | AJaeger: sorry, just looking at it now :) missed that bit | 05:58 |
| AJaeger | ianw: one step at a time - thanks! | 05:59 |
| ianw | do we want to install it in a venv and link it to /usr/local/bin maybe? | 05:59 |
| AJaeger | we have only a few users for that one, so can change those as well | 05:59 |
| AJaeger | playbooks/proposal/propose_update.sh is the user - hope the only one | 06:00 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Ensure pip is installed for propose-update-constraints https://review.opendev.org/736032 | 06:03 |
| ianw | AJaeger: ^ how about that? | 06:03 |
| AJaeger | thanks, looks good | 06:04 |
| AJaeger | ianw: now it needs ensure-virtualenv as well, doesn't it? | 06:06 |
| ianw | AJaeger: to use '{{ ensure_pip_virtualenv_command }}' we don't need virtualenv | 06:07 |
| AJaeger | ah | 06:07 |
| ianw | it will use venv everywhere now | 06:07 |
| AJaeger | great | 06:09 |
| AJaeger | so, should we merge that now to have it fixed - or wait? | 06:10 |
| ianw | if it passes lint i think i'm happy with it | 06:11 |
| ianw | there are certainly more instances requiring ensure-pip than i thought. it's probably good to have them explicitly calling it out as a dependency though | 06:16 |
| AJaeger | yes, that's more than expected. | 06:17 |
| AJaeger | and mordred and corvus had a discussion in #zuul last night that now ensure-pip needs sudo - while before those were baked in. So, that might need some further changes... | 06:18 |
| AJaeger | ianw: do you have time for some reviews later, please? https://review.opendev.org/736016 https://review.opendev.org/735932 https://review.opendev.org/735819 https://review.opendev.org/735937 | 06:21 |
| AJaeger | linters are happy on 736032 | 06:21 |
| ianw | AJaeger: oh i haven't been through that, but we had a big conversation about that at some point; basically ensure-pip doesn't/shouldn't need sudo if pip is already installed | 06:22 |
| AJaeger | ianw: and that one would be nice as well https://review.opendev.org/#/c/735929/ | 06:22 |
| AJaeger | ianw: ah, good point | 06:22 |
| ianw | that was the general gist of the ensure-* roles (ensure-tox similar) | 06:23 |
| ianw | so basically if people have static nodes/other images, if they bake those things in first, the ensure-* should be a noop | 06:23 |
| ianw | AJaeger: what do you think about returning the gpg bits as artifacts themselves? makes it easier than digging through logs | 06:31 |
| openstackgerrit | Merged openstack/project-config master: Turn -plain nodes down to min-ready 0 https://review.opendev.org/735777 | 06:32 |
| AJaeger | ianw: nice idea, let's see what fungi thinks | 06:34 |
| AJaeger | fungi, this is about 735929 | 06:36 |
| * AJaeger needs to step out for a bit, bbl | 06:36 | |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-pip: remove Xenial venv workaround https://review.opendev.org/736068 | 06:37 |
| openstackgerrit | Merged openstack/project-config master: Replace build-sphinx-docs jobs https://review.opendev.org/736016 | 06:39 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Add vexxhost/ansible-role-frrouting https://review.opendev.org/736115 | 06:55 |
| *** rpittau|afk is now known as rpittau | 06:56 | |
| *** DSpider has joined #opendev | 07:16 | |
| *** roman_g has joined #opendev | 07:19 | |
| AJaeger | frickler, ianw , could you +2A https://review.opendev.org/#/c/736032/ as well, please? | 07:29 |
| *** roman_g has quit IRC | 07:35 | |
| *** roman_g has joined #opendev | 07:35 | |
| *** roman_g has quit IRC | 07:35 | |
| *** tosky has joined #opendev | 07:36 | |
| openstackgerrit | Merged openstack/project-config master: Ensure pip is installed for propose-update-constraints https://review.opendev.org/736032 | 07:53 |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** ykarel is now known as ykarel|lunch | 08:13 | |
| AJaeger | argh, publish-openstack-python-branch-tarball is still failing | 08:49 |
| *** priteau has joined #opendev | 08:49 | |
| AJaeger | https://zuul.opendev.org/t/openstack/build/af6e45cad84245e7bdad1e4034ce505d - "install wheel" complains about missing sudo password ;( | 08:52 |
| *** roman_g has joined #opendev | 08:55 | |
| AJaeger | mordred: left a comment on https://review.opendev.org/#/c/736001 | 08:56 |
| *** tkajinam has quit IRC | 08:57 | |
| *** ykarel|lunch is now known as ykarel | 09:03 | |
| smcginnis | The weird thing about 736001 is that doesn't appear related to the pip and virtualenv changes. | 09:06 |
| *** aannuusshhkkaa has quit IRC | 09:20 | |
| AJaeger | smcginnis: it is - we have less installed in our images and thus need to install things now | 09:24 |
| smcginnis | Ah, so it ended up being a noop before, therefore it never hit the issue. | 09:25 |
| smcginnis | So sort of unrelated, but a direct result of that change. | 09:26 |
| *** ysandeep is now known as ysandeep|afk | 09:26 | |
| openstackgerrit | Merged opendev/irc-meetings master: Remove the Tricircle team meeting https://review.opendev.org/728922 | 09:27 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Add openstack-ansible-os_adjutant repo https://review.opendev.org/736142 | 09:31 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Add openstack-ansible-os_adjutant repo https://review.opendev.org/736142 | 09:35 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Add os_adjutant zuul project https://review.opendev.org/736145 | 09:40 |
| *** ysandeep|afk is now known as ysandeep | 09:54 | |
| *** Meiyan has quit IRC | 09:55 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Add openstack-ansible-os_adjutant repo https://review.opendev.org/736142 | 10:05 |
| *** rpittau is now known as rpittau|bbl | 10:18 | |
| frickler | infra-root: devstack jobs seem to be lacking lots of data collection. can this be related to the ansible upgrade? https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0e5/735640/2/check/devstack-platform-opensuse-15/0e5543e/controller/logs/ | 10:35 |
| *** sshnaidm_ has joined #opendev | 11:32 | |
| *** sshnaidm has quit IRC | 11:33 | |
| *** sshnaidm_ is now known as sshnaidm | 11:35 | |
| fungi | AJaeger: ianw: i'd be fine returning the checksums and signatures as artifacts, but should we also be returning the wheels and sdist tarballs as artifacts too in that case? | 11:36 |
| fungi | frickler: that job failed fairly early (couldn't find pip), so perhaps there were simply fewer files to collect? | 11:39 |
| *** tosky has quit IRC | 11:56 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/project-config master: [Grafana] Update label of neutron-functional-with-uwsgi job https://review.opendev.org/736168 | 11:58 |
| *** tosky has joined #opendev | 11:59 | |
| *** rpittau|bbl is now known as rpittau | 12:06 | |
| AJaeger | fungi, ianw, let's merge fungi's change and we can return those as followup... | 12:06 |
| openstackgerrit | Merged openstack/diskimage-builder master: update grub cmdline to current kernel parameters https://review.opendev.org/735445 | 12:15 |
| openstackgerrit | Merged zuul/zuul-jobs master: Record artifact checksums and signatures to stdout https://review.opendev.org/735929 | 12:15 |
| *** hashar has joined #opendev | 12:18 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Revert "Make sure wheel is installed for python releases" https://review.opendev.org/736183 | 12:25 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Avoid become on build-python-release https://review.opendev.org/736185 | 12:32 |
| frickler | fungi: iurygregory mentioned this in -qa , missing tempest.conf/.log, I assumed that that was similar, but maybe two different issue really https://zuul.opendev.org/t/openstack/build/da4a72ab684f41a499c6677ff87b7681 | 12:37 |
| iurygregory | missing the useful logs for ironic (ironic-bm-logs folder also) XD | 12:37 |
| frickler | and what I'm missing in the opensuse log is e.g. the local.conf generated for devstack, that should be in place before stack.sh is started | 12:40 |
| *** mrunge has quit IRC | 12:41 | |
| auristor | ianw: by my estimate it took a bit over 7h to rsync f32 into mirror.fedora which required approximate 550,000 storedata rpcs. The vos release at 07:17:27 UTC completed. | 12:43 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: Avoid become on build-python-release https://review.opendev.org/736185 | 12:52 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: Avoid become on build-python-release https://review.opendev.org/736185 | 13:06 |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Make sure wheel is installed for python releases" https://review.opendev.org/736183 | 13:10 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: Avoid become on build-python-release https://review.opendev.org/736185 | 13:12 |
| *** mrunge has joined #opendev | 13:27 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: Avoid become on build-python-release https://review.opendev.org/736185 | 13:35 |
| *** rpittau is now known as rpittau|brb | 13:39 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: Avoid become on build-python-release https://review.opendev.org/736185 | 13:40 |
| *** priteau has quit IRC | 13:40 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: Avoid become on build-python-release https://review.opendev.org/736185 | 13:44 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: Avoid become on build-python-release https://review.opendev.org/736185 | 13:45 |
| *** rpittau|brb is now known as rpittau | 13:53 | |
| *** priteau has joined #opendev | 13:56 | |
| *** lpetrut has joined #opendev | 13:56 | |
| *** ysandeep is now known as ysandeep|brb | 14:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix missing repos for zuul tenant https://review.opendev.org/736226 | 14:08 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: Avoid become on build-python-release https://review.opendev.org/736185 | 14:09 |
| *** rchurch has quit IRC | 14:14 | |
| *** rchurch has joined #opendev | 14:15 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Add openstack-ansible-os_adjutant repo https://review.opendev.org/736142 | 14:18 |
| openstackgerrit | Merged openstack/project-config master: Add vexxhost/ansible-role-frrouting https://review.opendev.org/736115 | 14:23 |
| openstackgerrit | Merged openstack/project-config master: Add noop-jobs for neutron-fwaas and neutron-fwaas-dashboard projects https://review.opendev.org/735807 | 14:25 |
| mnaser | AJaeger: ^ | 14:28 |
| mnaser | i guess opendev/infra-manual doesn't push notifications here.. | 14:28 |
| mnaser | wait, it is supposed to.. | 14:29 |
| mnaser | https://review.opendev.org/#/c/736238/ should have pushed a notification here | 14:29 |
| mnaser | infra-root: i think maybe something is borked in openstackgerrit ? https://opendev.org/openstack/project-config/src/branch/master/gerritbot/channels.yaml#L99 should have shown the opendev/infra-manual change above | 14:31 |
| AJaeger | mnaser: we haven't updated gerritbot for some time, waiting for ansiblification of it | 14:32 |
| fungi | mnaser: its config is way out of date because it hasn't been switched form puppet to ansible+containers yet | 14:32 |
| mnaser | ah, my bad, sorry, i haven't kept up to date with all of this | 14:32 |
| mordred | blast. sorry - that's on my plate - I realy need to finish that | 14:32 |
| fungi | it runs on review.opendev.org directly at the moment, and we're wanting to move it to eavesdrop where most of the other irc bots reside | 14:33 |
| fungi | i think the only other one not running there is elastic-recheck's irc bot component | 14:33 |
| mnaser | i would try my hand at it but i feel like it wouldn't progress as much | 14:33 |
| mnaser | most i can help with is containerizing accessbot if the source code is somewhere | 14:33 |
| mnaser | but the rest is a bit harder as i cant see whats happening inside as much | 14:34 |
| *** mlavalle has joined #opendev | 14:34 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Add openstack-ansible-os_adjutant repo https://review.opendev.org/736142 | 14:36 |
| mnaser | oh looks like it already has containers | 14:37 |
| mnaser | so i guess the rest is re-wiring it all up | 14:37 |
| mordred | mnaser: yeah. it really sholdn't be hard- I just keep getting distracted | 14:38 |
| *** mrunge has quit IRC | 14:45 | |
| *** mrunge has joined #opendev | 14:46 | |
| *** ysandeep|brb is now known as ysandeep | 15:04 | |
| openstackgerrit | Merged openstack/project-config master: Add openstack-ansible-os_adjutant repo https://review.opendev.org/736142 | 15:12 |
| *** sshnaidm is now known as sshnaidm|bbl | 15:17 | |
| AJaeger | ianw: could you check the promote failure on https://review.opendev.org/736230 - is it correct that the afs jobs runs there? | 15:17 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: Avoid become on build-python-release https://review.opendev.org/736185 | 15:19 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add os_adjutant zuul project https://review.opendev.org/736145 | 15:23 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update guideline docs for os specific tasks https://review.opendev.org/731606 | 15:24 |
| *** hashar has quit IRC | 15:38 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Update guideline docs for os specific tasks https://review.opendev.org/731606 | 15:40 |
| mordred | clarkb: I'm working on the puppet manifest split because of the broken puppet applu job | 15:41 |
| mordred | clarkb: I haz a question - in the ansible world we have things set up to not run things on backup-server normally | 15:41 |
| mordred | but we seem to have openstack_project::backup_server and openstack_project::server running in puppet unconditionally | 15:42 |
| mordred | I feel like the puppet is vestigal and should just go away from looking at it | 15:43 |
| mordred | oh - I get it | 15:43 |
| mordred | it isn't run - because backup servers aren't int eh puppet group | 15:44 |
| mordred | so it's just cruft | 15:44 |
| fungi | in the past we manually ran those when we could observe and confirm we didn't break the server | 15:44 |
| fungi | we just didn't want an automatic configuration management run to break the backup host, have nobody notice, and then later suffer an outage where we needed to restore from backup and find out we've not been backing anything up for months | 15:45 |
| mordred | fungi: yeah - that part makes sense - I was just confused because it looked like we were _always_ running the puppet | 15:53 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split site.pp into smaller files https://review.opendev.org/736282 | 15:56 |
| *** aannuusshhkkaa has joined #opendev | 15:57 | |
| mordred | infra-root: that's a giant patch, but I think it's 80% of the way there - I'm guessing some of the hostnames are wrong, and there's probably other issues | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split site.pp into smaller files https://review.opendev.org/736282 | 16:01 |
| mordred | corvus, clarkb, frickler, AJaeger: I have a job in openstackclient that uses unittests as a base job (and thus revoke-sudo) - and then the job content is a shell script which calls virtualenv do to some things. what do we think is the right way to go about doing ensure-virtualenv? | 16:08 |
| mordred | if I do ensure-virtualenv in a pre playbook, it'll do the install as not-root into a venv right? and then I need to use the variable to get the virtualenv command? | 16:09 |
| mordred | wait - no, it's goingto install virtualenv via packages | 16:10 |
| mordred | which means it needs root | 16:10 |
| mordred | which means I can't use the unittests base job | 16:10 |
| mordred | nevermind - we don't do revoke-sudo in unittests. cool | 16:12 |
| *** rpittau is now known as rpittau|afk | 16:17 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Make build-python-release use python3 https://review.opendev.org/736185 | 16:19 |
| auristor | mirror.ubuntu, mirror.centos mirror.epel, mirror.ubuntu-cloud, mirror.deb-jewel, mirror.debian, mirror.opensuse, mirror.deb-luminous, project.zuul, mirror.yum-puppetlabs, and more updated this morning without copying the entire volume | 16:20 |
| fungi | yep, the mirror.centos, mirror.epel, mirror.opensuse and mirror.yum-puppetlabs ones there are relevant to the rsync change | 16:20 |
| fungi | the debian/ubuntu package mirrors are built with a different tool (reprepro) so didn't have the problem to begin with | 16:21 |
| clarkb | mordred: aya backup server puppet is just old | 16:21 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: POC: test build-python-release using python https://review.opendev.org/736293 | 16:29 |
| *** ykarel is now known as ykarel|away | 16:33 | |
| *** ysandeep is now known as ysandeep|away | 16:36 | |
| clarkb | infra-root looks like the dns backups are running now | 16:51 |
| clarkb | which I think means we are good to start cleaning up records /me finds that etherpad | 16:51 |
| clarkb | https://etherpad.opendev.org/p/rax-dns-openstack-org | 16:52 |
| clarkb | one thing I noticed on there is smokestack.openstack.org, that was the old dprince thing right and thus should be totally safe to remove? | 16:52 |
| fungi | yes | 16:54 |
| fungi | wow i can't believe that's still in there | 16:54 |
| fungi | well, okay, i can | 16:54 |
| fungi | but still, that takes me waaaay back | 16:55 |
| clarkb | ya | 16:55 |
| clarkb | if others want to look that over in the next few hours I'm happy to click the buttons in the web ui to clean stuff up after my bike ride and lunch today | 16:55 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Ensure wheel exists for build-release-python https://review.opendev.org/736185 | 16:56 |
| corvus | all those "_" afs/krb things should stay, right? | 16:56 |
| corvus | i just did a strikethrough on those so you can see what i propose to delete from the etherpad (in order to indicate that the records should remain) | 16:57 |
| clarkb | corvus: yes afs/krb things should stay. I think ianw was suggested we simply remove the records from the etherpad if we want to keep them (out of sight out of mind) | 16:59 |
| clarkb | but I'll look for explicit DELETE to delete things in any case | 17:00 |
| *** diablo_rojo has joined #opendev | 17:00 | |
| *** lpetrut has quit IRC | 17:02 | |
| corvus | done | 17:02 |
| clarkb | thanks! | 17:03 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Zookeeper: listen on plain and TLS ports https://review.opendev.org/736309 | 17:39 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Switch to ZK TLS in Nodepool and Zuul https://review.opendev.org/736310 | 17:39 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove Zookeeper plain-text listener https://review.opendev.org/736311 | 17:39 |
| corvus | okay, that's the new plan for zk tls | 17:39 |
| fungi | and we should be able to merge 736309 immediately, yeah? | 17:40 |
| corvus | last 2 changes are WIP waiting for us to be ready to use it. | 17:40 |
| corvus | fungi: yep | 17:40 |
| corvus | that will switch the cluster to tls internally, and stage certs on all the clients but not use them | 17:40 |
| clarkb | corvus: that will require a complete restart of the cluster right? | 17:41 |
| clarkb | rather than rolling? since we switch to ssl quorum | 17:41 |
| corvus | clarkb: i believe so | 17:49 |
| corvus | (strictly speaking, i think we could do the client ssl without quorum ssl, but i think it makes sense to get this out of the way; it seemed to work fine) | 17:50 |
| mordred | corvus: +2 from me | 17:51 |
| mordred | clarkb: look - _some_ of the jobs passed! https://review.opendev.org/#/c/736282/ :) | 17:51 |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/735782 | 18:08 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 18:28 |
| *** hashar has joined #opendev | 18:29 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 18:33 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 18:37 |
| *** roman_g has quit IRC | 18:49 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 18:52 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 18:55 |
| *** sshnaidm|bbl is now known as sshnaidm | 19:09 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 19:40 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Ensure wheel exists for build-release-python https://review.opendev.org/736185 | 19:46 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 19:47 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add ensure pre-run policy to ansible-lint https://review.opendev.org/736367 | 20:01 |
| clarkb | corvus: when you get a sec can you review https://review.opendev.org/#/c/735969/2 which adds testing for our zuul and nodepool restart playbooks. Mordred and I kinda collabed on that so would be good to get a second reviewer that isn't one of us | 20:05 |
| corvus | clarkb: +w great -- i have a minor concern that maybe that could be an issue doing a restart right before testinfra or something if our testinfra tests get better; but i bet we can deal with that. | 20:08 |
| clarkb | ya and I think the regular bring up in theory has the same race so we're relying on timing either way and would have ot address it if it started to fail | 20:11 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: phoronix-test-suite: extract ensure- role from run playbook https://review.opendev.org/736368 | 20:11 |
| *** tosky_ has joined #opendev | 20:12 | |
| *** tosky has quit IRC | 20:13 | |
| *** tosky_ is now known as tosky | 20:13 | |
| *** JasonF has joined #opendev | 20:14 | |
| *** JayF has quit IRC | 20:18 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add linting rule to enforce no-same-owner policy https://review.opendev.org/727642 | 20:27 |
| *** aannuusshhkkaa has quit IRC | 20:37 | |
| *** shtepanie has quit IRC | 20:38 | |
| *** knikolla has quit IRC | 20:40 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 20:42 |
| *** knikolla has joined #opendev | 20:42 | |
| *** priteau has quit IRC | 20:49 | |
| *** knikolla has quit IRC | 20:52 | |
| *** shtepanie has joined #opendev | 20:52 | |
| openstackgerrit | Merged opendev/system-config master: Run restart playbooks to test they work https://review.opendev.org/735969 | 20:55 |
| openstackgerrit | Merged opendev/system-config master: Add stop and start playbooks for nodepool https://review.opendev.org/736031 | 20:55 |
| *** shtepanie has quit IRC | 20:57 | |
| *** diablo_rojo_phon has joined #opendev | 21:00 | |
| *** JasonF is now known as JayF | 21:06 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add linting rule to enforce no-same-owner policy https://review.opendev.org/727642 | 21:07 |
| *** noonedeadpunk has quit IRC | 21:12 | |
| openstackgerrit | Ahmad Mahmoudi proposed zuul/zuul-jobs master: (fix) Changed pip to pip3 https://review.opendev.org/736379 | 21:16 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add ensure pre-run policy to ansible-lint https://review.opendev.org/736367 | 21:19 |
| *** xiaolin has quit IRC | 21:27 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 21:31 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Fix ansible-lint rules tests https://review.opendev.org/736387 | 21:34 |
| *** aannuusshhkkaa has joined #opendev | 21:34 | |
| clarkb | I'm doing the dns cleanups now | 21:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add linting rule to enforce no-same-owner policy https://review.opendev.org/727642 | 21:35 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Fix ansible-lint rules tests https://review.opendev.org/736387 | 21:37 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add linting rule to enforce no-same-owner policy https://review.opendev.org/727642 | 21:37 |
| *** knikolla has joined #opendev | 21:40 | |
| *** shtepanie has joined #opendev | 21:42 | |
| clarkb | #status log Cleaned up old unnecessary records in openstack.org DNS zone. | 21:50 |
| mordred | clarkb: \o/ | 21:50 |
| openstackstatus | clarkb: finished logging | 21:50 |
| fungi | thanks clarkb! | 21:51 |
| *** hashar has quit IRC | 21:52 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove elasticsearch01 https://review.opendev.org/736389 | 21:52 |
| clarkb | minor related cleanup that I noticed as I went through that list | 21:52 |
| *** tosky has quit IRC | 21:56 | |
| *** tosky has joined #opendev | 21:56 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Fix ansible-lint rules tests https://review.opendev.org/736387 | 21:58 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Ensure wheel exists for build-release-python https://review.opendev.org/736185 | 21:59 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add option to install tox into a path location https://review.opendev.org/736330 | 22:05 |
| fungi | clarkb: i also marked a few more in the pad | 22:09 |
| clarkb | fungi: thanks looks like some of those may need a bit more investigating | 22:11 |
| fungi | yeah, some are candidates for asking arund | 22:12 |
| fungi | around | 22:12 |
| clarkb | fungi: for trystack I think we want to keep the name for the redirect? | 22:12 |
| fungi | clarkb: maybe, though i don't know that trystack.openstack.org was ever circulated | 22:12 |
| fungi | it was originally going to be where we hosted the trystack site, but we never hosted the trystack site | 22:13 |
| clarkb | ah | 22:13 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Fix ansible-lint rules tests https://review.opendev.org/736387 | 22:13 |
| fungi | trystack.org was the main domain name used in urls while trystack was in operation | 22:13 |
| fungi | but the effort dried up around the time we were trying to move hosting into infra | 22:13 |
| fungi | and as part of that move we said let's turn the vanity domain into a redirect to something in the openstack.org domain | 22:14 |
| fungi | but instead that never finisied so we redirected the openstack.org subdomain to the trystack.org domain and watched it finish dying | 22:15 |
| clarkb | gotcha we likely can clean it up in that case. Should we cross check with the foundation? | 22:15 |
| clarkb | (thinking out loud here, maybe we do a bit more investigating on those you call out that may need ti then in near future do another pass of deletes) | 22:16 |
| clarkb | fungi: maybe you can add a DELETE next to the ones you're sure can go and then we'll ask about the iffy ones? | 22:19 |
| ianw | here's something ... interesting ... the fedora 32 bits seem to want to rsync themselves again | 22:19 |
| ianw | >f..T...... Cloud/x86_64/images/Fedora-Cloud-32-1.6-x86_64-CHECKSUM | 22:20 |
| clarkb | ianw: I think that is a side effect of no -t? | 22:20 |
| ianw | An alternate value of T means that the modification time will be set to the transfer time, which happens when a file/symlink/device is updated without ‑‑times and when a symlink is changed and the receiver can't set its time. | 22:21 |
| ianw | i have no idea what that sentence is supposed to mean | 22:21 |
| clarkb | the manpage warns about it but says it has other optimizations to reduce amount of data transfered anyway | 22:21 |
| clarkb | iirc | 22:21 |
| clarkb | "in other words, a missing -t or -a will cause the next transfer to behave as if it used -I, causing all files to be updated (though rsync’s delta-transfer algorithm will make the update fairly efficient if the files haven’t actuallychanged, you’re much better off using -t)." | 22:22 |
| ianw | sigh | 22:22 |
| ianw | i guess for the extant mirrors, that were run with "-t", we haven't notice this, it's only the fresh mirroring bits? | 22:23 |
| ianw | perhaps the best solution is to build a later rsync with the "don't update nanosecond if not required" fixes mentioned in our ppa | 22:23 |
| clarkb | hrm you're thinking that having a pass with -t addresses this problem then subsequent ones are better? and since this was never run with -t its in a sadder state? thta could be I guess | 22:24 |
| clarkb | I was thinking about newer rsync and wondered if maybe a container would be better? | 22:24 |
| clarkb | then we don't have to rely on ppas and use our container infrastructure (and possibly just a newish debuntu) | 22:25 |
| clarkb | mordred: ^ you've been fiddling with afs in containers any concerns with that as an option? | 22:25 |
| fungi | or upgrade mirror-update.opendev.org to focal? | 22:25 |
| mordred | so... | 22:25 |
| mordred | the quick story about afs client in containers | 22:25 |
| clarkb | fungi: ya I guess that is an option too if rsync is new enough there | 22:26 |
| mordred | is that if you just do afs client in docker, all docker containers will share the PAG | 22:26 |
| mordred | HOWEVER | 22:26 |
| mordred | if you wrap the call in zuul-bwrap inside of docker, zuul-bwrap gets an new PAG | 22:26 |
| fungi | clarkb: the suspected fix is in rsync 3.1.3 which is what focal ships, bionic has 3.1.2 | 22:26 |
| mordred | all that said, I think it's not an issue given our usage on mirror-update | 22:26 |
| clarkb | (fwiw I'm not opposed to using a ppa, but wanted to call out other options we might consider) | 22:27 |
| mordred | it should just be noted that with only docker we get not auth separation between containers | 22:27 |
| fungi | at the moment, the only things mirror-update.opendev.org does are run rsync and act as an openafs client, so in theory upgrading to (rebuilding on) focal shouldn't be terrible | 22:28 |
| mordred | of course, it should be fairly easy to test on mirror-update if newer rsync is happier using a zuul/zuul-executor image | 22:28 |
| mordred | before upgrading to focal | 22:28 |
| ianw | yeah, a focal rebuild sounds probably the way forward | 22:28 |
| mordred | but I also agree with fungi, a focal rebuild also sounds straightforward | 22:29 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add linting rule to enforce no-same-owner policy https://review.opendev.org/727642 | 22:29 |
| ianw | i have a git rsync built anyway (which is how i realised that recent rsync doesn't set the modification time, because the breakpoint i set on it never hit) | 22:30 |
| ianw | so maybe i'll do two runs with that -- one to bring it back in with timestamps, then a second to test zero delta updates | 22:30 |
| ianw | if we're not seeing some OTHER F**##$ING PROBLEM ... which seems very possible ... we can probably go with an upgrade | 22:31 |
| mordred | ++ | 22:31 |
| fungi | sounds right | 22:33 |
| fungi | clarkb: i've updated most of those i feel comfortable can go with DELETE by them, the others i suggest who to check with | 22:42 |
| clarkb | thanks | 22:43 |
| clarkb | ttx: can we delete forumtopics.openstack.org DNS records? clenaup details are at https://etherpad.opendev.org/p/rax-dns-openstack-org | 22:44 |
| openstackgerrit | Merged zuul/zuul-jobs master: Ensure wheel exists for build-release-python https://review.opendev.org/736185 | 22:48 |
| *** DSpider has quit IRC | 22:51 | |
| *** seongsoocho has joined #opendev | 22:52 | |
| fungi | clarkb: ttx: i found notes that i deleted the forumtopics server 2019-03-20 so i almost certainly just failed to clean it up in dns | 22:53 |
| clarkb | fungi: and forum topic stuff got folded into the regular summit cfp too so should all be fine | 22:54 |
| fungi | right | 22:54 |
| fungi | i'll update the pad | 22:54 |
| *** tkajinam has joined #opendev | 22:54 | |
| *** shtepanie has quit IRC | 23:05 | |
| *** tosky has quit IRC | 23:09 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!