Tuesday, 2021-02-09

diablo_rojofungi,  lol got it. Fixing now.00:02
openstackgerritKendall Nelson proposed openstack/project-config master: Setup OpenInfra Channels  https://review.opendev.org/c/openstack/project-config/+/77455000:03
openstackgerritJeremy Stanley proposed opendev/git-review master: Add missing -p/-P/-w/-W/--license to manpage  https://review.opendev.org/c/opendev/git-review/+/77456700:17
*** tosky has quit IRC00:20
openstackgerritMerged openstack/project-config master: Setup OpenInfra Channels  https://review.opendev.org/c/openstack/project-config/+/77455000:32
openstackgerritIan Wienand proposed opendev/system-config master: borg-backup-server: add script for pruning borg backups  https://review.opendev.org/c/opendev/system-config/+/77456100:33
openstackgerritIan Wienand proposed opendev/system-config master: borg-backup-server: volume space monitor  https://review.opendev.org/c/opendev/system-config/+/77456400:33
openstackgerritIan Wienand proposed opendev/system-config master: gerrit: only backup accountPatchReviewDb  https://review.opendev.org/c/opendev/system-config/+/77456900:33
clarkbone thing I am realizing is that we'll want to do a prepass on the refs/users/ side of things to avoid the preferred email is missing when we then clean up external ids00:45
clarkband if we do the refs/users/ side of things first we should end up without new errors00:46
fungimakes sense, yeah00:46
clarkbbut looking at the external ids so far the vast majority really do seem to be there are 2-3 accounts with conflicting emails. Only one them was ever used. So we retire the other(s) to avoid preferred email missing errors then remove the conflicting external ids from the retired accounts00:47
clarkbthe tricky bit is in deciding which account to retire as sometimes it is the newer one or the older one and in some cases both were used and so you have to apply some judgement to decide which is active now00:47
clarkbone thing I've used for that is looking up whether or not the internet claims people still work at the company that one or both of the accounts has an email address with00:48
clarkbin some cases I think the company doesn't even exist anymore :/00:48
*** openstackgerrit has quit IRC00:49
*** mlavalle has quit IRC00:53
*** rchurch has quit IRC00:57
*** openstackgerrit has joined #opendev00:58
openstackgerritMerged opendev/system-config master: Setup OpenInfra Channels  https://review.opendev.org/c/opendev/system-config/+/77456300:58
*** rchurch has joined #opendev01:00
clarkbthis is a fun one. I've just found a pair of accounts where one pushed code the other did reviews. The preferred email in one matches the openid emails on the other and the preferred email on the other matches the openid emails from the first one01:00
clarkbits like the perfect set of criss crossed wires :/01:00
clarkbAnd I've just written down "I don't know what to do with this situation" in my notes01:01
clarkbwe could maybe flip the preferred emails so they match their openids then remove the conflicting mailto? that seems pretty heavy handed.01:01
clarkbBoth accounts haven't been used in half a decade though so maybe we can just retire them both and if the user returns we sort it out with them?01:02
clarkband now time to figure out dinner01:04
openstackgerritIan Wienand proposed opendev/system-config master: doc: update backup instructions  https://review.opendev.org/c/opendev/system-config/+/77457001:18
*** openstackstatus has quit IRC01:20
*** openstack has joined #opendev01:22
*** ChanServ sets mode: +o openstack01:22
fungiclarkb: yeah, i've run into that when trying to help folks resolve duplicate accounts in the past, but like you say, it may be easiest to just disable both and remove the conflicts from them01:27
fungii would actually lean toward disabling the one which is being used for change uploading, because the user can always set a username and ssh key via the webui with the one they logged into for reviewing01:29
fungithough one missing piece of info we may want to incorporate is to cross-reference all of them against group memberships, since if one account in a conflict set is a member of a group that's probably the one which should be kept/fixed01:31
*** DSpider has quit IRC01:43
*** diablo_rojo has quit IRC02:12
*** dviroel has quit IRC02:43
ianwle sigh ... anyone know why bridge can't talk to rax any more?02:55
ianwMax retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd79f6e8e90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))02:55
ianwit works with 5.0.1 in my home directory, nto with 5.2.1 from the container02:59
*** TheJulia has quit IRC03:26
*** walshh_ has quit IRC03:26
*** mwhahaha has quit IRC03:26
*** snbuback has quit IRC03:27
*** parallax has quit IRC03:27
*** CeeMac has quit IRC03:27
*** ildikov has quit IRC03:28
*** TheJulia has joined #opendev03:28
openstackgerritMerged opendev/system-config master: gerrit: only backup accountPatchReviewDb  https://review.opendev.org/c/opendev/system-config/+/77456903:28
*** mwhahaha has joined #opendev03:29
*** ildikov has joined #opendev03:29
*** walshh_ has joined #opendev03:29
*** snbuback has joined #opendev03:31
*** CeeMac has joined #opendev03:31
*** parallax has joined #opendev03:31
ianwi think we need to look at unattended-upgrades during node creation.  i think it's installing every package completely separately and running more inbetween; it's been going for about 40 minutes03:33
ianwit recreated the initfs 10 times according to the logs03:35
openstackgerritIan Wienand proposed opendev/system-config master: Add refstack01.openstack.org  https://review.opendev.org/c/opendev/system-config/+/77457903:39
fungiianw: urllib3 5.2.1?03:50
ianwsorry openstackclient03:51
ianwalthough i'm sure the dependencies installed are also many and varied03:51
fungiyeah, first suspicion is rax uses some older ssl parameters which newer <something> has decided is better to throw an opaque error about rather than use03:52
fungiwhat's the full url? could try passing it to requests.get() in the repl and seeing if it gives any more useful error03:57
openstackgerritMerged opendev/system-config master: Deploy refstack with ansible docker  https://review.opendev.org/c/opendev/system-config/+/70525803:58
fungimmm, trying it myself, and it's taking a while to return03:59
fungiFailed to discover available identity versions when contacting https://identity.api.rackspacecloud.com/v2.0/. Attempting to parse version from URL.03:59
ianwyeah, that look like it04:00
fungiahh, there we go04:00
fungiit's obscured by a retry04:00
fungihowever /v2.0/tokens presumably needs a post or something, so requests.get() just receives a 40504:02
openstackgerritMerged opendev/system-config master: Add refstack01.openstack.org  https://review.opendev.org/c/opendev/system-config/+/77457904:24
*** ykarel has joined #opendev04:33
*** ysandeep|away is now known as ysandeep|rover04:48
*** whoami-rajat__ has joined #opendev05:13
openstackgerritIan Wienand proposed opendev/system-config master: refstack: add production image and deployment jobs  https://review.opendev.org/c/opendev/system-config/+/77458605:19
openstackgerritIan Wienand proposed opendev/system-config master: refstack: move non-private variables to public  https://review.opendev.org/c/opendev/system-config/+/77458705:19
openstackgerritIan Wienand proposed opendev/system-config master: borg-backup: save PIPESTATUS before referencing  https://review.opendev.org/c/opendev/system-config/+/77458805:23
*** ykarel is now known as ykarel|mtg05:38
*** marios has joined #opendev06:10
*** ralonsoh has joined #opendev07:36
*** eolivare has joined #opendev07:40
*** fressi has joined #opendev07:55
*** sboyron has joined #opendev07:58
*** rpittau|afk is now known as rpittau07:59
*** hashar has joined #opendev08:00
*** slaweq has joined #opendev08:03
*** jpena|off is now known as jpena08:15
*** andrewbonney has joined #opendev08:19
*** fressi has quit IRC08:22
*** fressi has joined #opendev08:23
*** ykarel|mtg is now known as ykarel08:26
*** DSpider has joined #opendev08:43
*** ralonsoh has quit IRC09:02
*** tosky has joined #opendev09:10
*** marios has quit IRC09:21
*** marios has joined #opendev09:24
*** dtantsur|afk is now known as dtantsur09:45
dtantsurHi folks! storyboard gives HTTP 500 on login, is it known already09:51
*** dviroel has joined #opendev10:58
*** guilhermesp has quit IRC11:10
*** icey has quit IRC11:10
*** guilhermesp has joined #opendev11:11
*** icey has joined #opendev11:13
*** ysandeep|rover is now known as ysandeep|brb11:38
*** hashar is now known as hasharAway11:44
*** calcmandan has quit IRC12:00
*** calcmandan has joined #opendev12:01
*** whoami-rajat__ has quit IRC12:08
*** andrewbonney has quit IRC12:08
*** guilhermesp has quit IRC12:08
*** ysandeep|brb is now known as ysandeep|rover12:09
*** dviroel has quit IRC12:09
*** whoami-rajat__ has joined #opendev12:10
*** andrewbonney has joined #opendev12:10
*** dviroel has joined #opendev12:10
*** guilhermesp has joined #opendev12:10
*** calcmandan has quit IRC12:10
*** calcmandan_ has joined #opendev12:14
*** jpena is now known as jpena|lunch12:18
*** calcmandan_ is now known as calcmandan12:18
dtantsuranother bad news this morning: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982182 seems to affect debian-buster images12:21
openstackDebian bug 982182 in grub-pc "grub-pc: Regression on buster 10.8 release with non-interactive update" [Important,Open]12:22
dtantsurat least bifrost fails on installing grub-efi-amd64-signed12:22
dtantsurwe may work around by downloading and unpacking it instead (we don't really need it install), but it seems the bug will be triggered by anything that will cause upgrade of grub-pc12:22
*** hasharAway is now known as hashar12:35
zigodtantsur: Again, I don't think you need to go into unpacking and such, just preseeding grub should be enough.12:47
zigodtantsur: Do you know how preseeding works?12:47
dtantsurno idea, I'm not the one creating these images12:48
zigodtantsur: It's as if you pre-answer the prompts from debconf.12:48
dtantsurI don't think *I* can do that12:49
dtantsurI neither know nor care how this image is booted12:50
zigoIt would be something like that (example with grub-pc/chainload_from_menu.lst):12:53
zigoecho "grub-pc grub-pc/chainload_from_menu.lst boolean false12:53
zigogrub-pc grub-pc/chainload_from_menu.lst seen true" | debconf-set-selections12:53
*** whoami-rajat__ has quit IRC12:57
*** CeeMac has quit IRC12:58
*** walshh_ has quit IRC12:58
*** walshh_ has joined #opendev13:00
*** CeeMac has joined #opendev13:01
*** iurygregory has quit IRC13:04
*** iurygregory has joined #opendev13:06
*** whoami-rajat__ has joined #opendev13:07
dtantsurif this is required by debian, it should be built into the images13:19
*** jpena|lunch is now known as jpena13:21
*** ykarel_ has joined #opendev13:23
*** ykarel has quit IRC13:25
*** ykarel_ is now known as ykarel13:26
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: Update upload-logs-swift  https://review.opendev.org/c/zuul/zuul-jobs/+/77465013:55
*** ykarel is now known as ykarel|away13:56
*** mlavalle has joined #opendev13:58
*** sshnaidm is now known as sshnaidm|afk14:00
*** ykarel|away has quit IRC14:32
dtantsurinfra folks, would it be possible to build the steps zigo mentions in the images?14:46
*** stand has joined #opendev14:51
*** lourot has quit IRC15:00
*** lourot has joined #opendev15:00
*** ysandeep|rover is now known as ysandeep|away15:21
*** hashar is now known as hasharAway15:27
fungidtantsur: storyboard's been fixed for a few minutes, i'm still catching up on scrollback15:29
fungiunattended-upgrades apparently decided python-pymysql was a cool thing to automatically remove for no reason, i haven't gotten to the root cause there but i manually reinstalled the package and that seemed to solve things15:29
fungidtantsur: which files from the grub-efi-amd64-signed package do you need? maybe there's a better source for them?15:33
dtantsurfungi: this one: https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-ironic-install/defaults/required_defaults_Debian_family.yml#L1015:34
fungithe *.efi.signed files i guess15:34
fungiand where are you getting that /usr/lib/shim/shimx64.efi.signed file from?15:36
dtantsurfrom shim-signed15:36
fungiaha, okay, so also installing a package of that15:37
fungii wonder if you could temporarily stub out /usr/sbin/policy-rc.d to make dpkg not invoke the maintscripts for that15:39
fungibecause you don't actually want to muck with the bootloader on the system where you're installing it15:40
fungii need to prep for a meeting, but can revisit this later15:41
*** sshnaidm|afk is now known as sshnaidm15:44
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: Update upload-logs-swift  https://review.opendev.org/c/zuul/zuul-jobs/+/77465015:55
clarkbfungi: ya in the vast majority of cases it seems pretty clear whihc one is the one to preserve. That one with the criss crossed conflicts was the first really ambiguous one I have run across in the externalid conflict set15:57
clarkbI have not been cross checking against groups, but have looked at reviewedby:id and owner:id and that usually makes it clear15:57
*** hasharAway is now known as hashar15:58
*** diablo_rojo has joined #opendev16:06
diablo_rojofungi, looks like we only have smcginnis left to drop from the board channel.16:07
fungii'm not sure public shaming works on him ;)16:14
fungialso i merged your earlier changes for the other channels16:14
fungidiablo_rojo: were you going to push a change to add yourself to the operators list in the accessbot config?16:16
diablo_rojoI thought that I did that?16:20
fungioh, yep, you used a separate review topic, that's why i missed it16:23
diablo_rojoOh sorry!16:23
diablo_rojoI suppose I should have made them all the same16:23
fungino need to apologize, i was spacey last night16:23
fungianyway, there's already a comment asking you to insert yourself in alpha order16:24
diablo_rojofungi, whoops. Will do that after the staff call.16:27
*** chandankumar is now known as raukadah16:33
dtantsurFYI test-release-openstack has been broken with cryptography for hours16:34
clarkbdtantsur: you should let the openstack release team know16:34
dtantsurwill do16:34
dtantsurfungi: I'm pondering downloading and unpacking the package instead of installing it16:35
dtantsurit's definitely much more annoying but also more robust16:36
diablo_rojofungi, we are good to go with the board channel!16:54
diablo_rojoGot OP and registered it and guarded it and added the openstackinfra access16:56
*** slaweq is now known as slaweq|away16:56
fungidtantsur: yeah, it would work, i just wonder if there's a better source for that file, since the package is focused more on deploying it for booting the system on which it's installed17:03
fungithanks diablo_rojo! so two more patches for that channel i guess and we'll be all set for the next phase17:03
clarkbfungi: dtantsur: is this happening in a dib context? I want to say that dib does the grub install very late and only after the image has been constructed17:04
dtantsurclarkb: no, not this time. this is bifrost, it requires this file to build a deployment environment.17:04
*** marios is now known as marios|out17:14
*** jpena is now known as jpena|off17:17
fungiokay, so update on the storyboard outage. it looks like there was a flurry of python module packages removed in the wake of us uninstalling bup17:25
fungii still need to spider the dependency tree there, but my guess is that when we installed bup it pulled in python-pymysql as an indirect dependency and the package was marked as automatically installed17:25
fungionce bup was removed, there were no remaining explicitly installed debian packages depending on python-pymysql, so it got cleaned up17:26
fungii think the puppet apt module just checks to see whether a specified package is installed, and doesn't know/care if it's marked as automatically installed17:27
clarkbfungi: so since we pulled it in automatically first puppet never updated that flag and it got removed when nothing directly depended on it even though we tell puppet to install it explicitly?17:28
fungithat certainly what it seems like, yeah17:29
fungii expect that bup was installed before puppet checked to see whether it needed to install python-pymysql17:29
fungiin fact, we probably were using python-mysqlclient at the time bup was installed on that server, and then transitioned to python-pymysql for storyboard later17:30
dtantsurit may be an argument for turning storyboard into a package (or a container)17:38
clarkbit is already a container aiui, we just have to update the deployment to use it17:39
*** dtantsur is now known as dtantsur|afk17:50
*** rpittau is now known as rpittau|afk17:56
*** eolivare has quit IRC18:00
openstackgerritKendall Nelson proposed openstack/project-config master: Add diablo_rojo to AccessBot Operators  https://review.opendev.org/c/openstack/project-config/+/77455518:03
openstackgerritKendall Nelson proposed openstack/project-config master: Setup OpenInfra-Board Channel  https://review.opendev.org/c/openstack/project-config/+/77470518:06
openstackgerritKendall Nelson proposed opendev/system-config master: Setup OpenInfra-Board Channel  https://review.opendev.org/c/opendev/system-config/+/77470618:08
diablo_rojofungi, ^^ all done.18:08
*** hashar is now known as hasharAway18:23
*** marios|out has quit IRC18:30
mtreinishfungi: yeah, I saw that cryptography added a rust extension, but I didn't think they made it mandatory in the recent release18:37
mtreinishsetuptools_rust is probably required to be installed before running setup.py because it most likely gets imported at the top of the setup.py, but I haven't looked at how they did the packaging18:38
clarkbmtreinish: the github issue made it sound like they talked about having an alternative C set of bindings but that they didn't have the necessary tools or time to support that too18:39
clarkbalso it seems the invovled individuals aren't interested in writing more C18:39
clarkbanyway its only an issue if you don't get a wheel (and that seems to happen more often than it should, possibly due to stale pypi indexes?)18:39
clarkbor if your platform doesn't have a wheel I ugess18:40
clarkbalpine in particular seems affected there because it uses musl18:40
mtreinishah, yeah looking at their setup.py it looks like it's what I guessed. setuptools-rust is a prereq to run setup.py and they put it in pyproject.toml, but the pyproject.toml is hit or miss in my experience in actually installing required things18:42
clarkbmtreinish: https://github.com/pyca/cryptography/blob/master/pyproject.toml they use a pyproject.toml18:42
clarkbmtreinish: I think it depends on having a new enough pip to support it?18:43
mtreinishI worked around it in my rust/python lib by doing this super hacky thing: https://github.com/Qiskit/retworkx/blob/master/setup.py#L2-L10 which solves it for most cases18:43
mtreinishclarkb: I dunno, the pyproject.toml never seems to work for me locally18:43
mtreinisheven with the latest pip18:43
mtreinishlike the last time I tried to remove that try except block CI failed because setuptools rust couldn't be found18:44
mtreinishalthough I just tested it locally now and it seems to work fine, so I dunno. The last time I tried it was like a year ago18:48
*** andrewbonney has quit IRC18:52
fungidtantsur|afk: clarkb: yep, we build container images and upload them to dockerhub for every commit which lands to storyboard and storyboard-webclient, we're just overdue for deploying those with ansible instead of installing from git into the system context with puppet19:10
fungiclarkb: mtreinish: i believe the issue ultimately is they published an abi3 wheel, which the older pip shipped with centos 8 doesn't know to look for, so it tries to install the sdist instead, but the version of pip there is also old enough to not know what to do with the pyproject.toml so it doesn't install setuptools_rust either19:12
fungii think ubuntu-bionic is in the same boat19:12
fungiunfortunately, that's still our default nodeset (we need to talk about switching our default to ubuntu-focal, that's also overdue)19:13
clarkbaha its the abi that causes the sdist to be pulled19:16
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: Update upload-logs-swift  https://review.opendev.org/c/zuul/zuul-jobs/+/77465019:17
mordredfungi, clarkb: latest openstacksdk works just fine talking to rackspace using the clouds.yaml entry on bridge19:27
clarkbcould be an osc specific issue then?19:27
fungiyeah, even something as simple as server list is breaking for me19:28
fungisudo openstack --os-cloud openstackci-rax --os-region-name DFW server show afs01.dfw.openstack.org19:28
fungion bridge19:28
mordredI'm working on trying osc now19:28
mordredkk - thanks19:28
fungithat spins for a while for me and then complains about the tokens method19:29
mordredurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='identity.api.rackspacecloud.com', port=443): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bcf0bcfd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))19:36
mordredI think there is some sort of networking issue19:36
clarkbtemporary failure in name resolution sounds like dns problems19:37
clarkbweird that it would work with sdk if dns was sad though?19:37
mordredwell - I tried sdk from my laptop19:38
mordredI also tried curl-ing the main keystone url from bridge and that is working19:38
mordredso I'm checking what's going on inside of the container right now19:38
fungimight it be something listed in the catalog isn't actually in dns?19:39
mordredooh - yeah19:39
mordredI booted a container19:39
mordredand I can't do things in it19:39
mordredroot@5fb36de8a300:/# apt-get update && apt-get install curl19:39
mordredErr:1 http://deb.debian.org/debian buster InRelease19:39
mordred  Temporary failure resolving 'deb.debian.org'19:39
mordredthat's in an osc container running bash on bridge19:39
fungiooh, we're running osc from a container! so it's container networking which is at fault?19:40
mordredso I think the issue is "dns is broken inside of containers on bridge"19:40
mordredand we just happened to notice via the osc container19:40
fungithanks, that seems much more tractable. i forgot we had containerized osc19:40
mordredfungi: yah - openstack on bridge is a wrapper script installed by ansible19:40
mordredcat /usr/local/bin/openstack19:40
mordred"Apparently the docker0 network bridge was hung up."19:43
mordredmight be worth starting with just restarting docker daemon19:43
mordredshall I try that?19:44
clarkbseems reasonable19:45
fungidouble-plus good19:45
mordredI restarted both docker and containerd for good measure19:46
mordredall works now19:47
fungithanks mordred! i should have remembered there was a container layer there now19:48
fungimy mind immediately jumped to the problem preventing us from using osc to talk to rackspace's cinder19:48
mordred++ - that will eventually get fixed with the ongoing osc->sdk work19:51
ianwoh cool.  yeah the container wrapper has hit me trying to upload image files ... the container can't see them.  but i didn't think of container networking20:04
clarkbianw: -1 for a job name thing on the refstack jobs change20:06
clarkblunch and exercise next for me, but then I've got the borg changes top of my todo list when I return20:06
ianwclarkb: thanks will look20:07
openstackgerritIan Wienand proposed opendev/system-config master: refstack: move non-private variables to public  https://review.opendev.org/c/opendev/system-config/+/77458720:11
openstackgerritIan Wienand proposed opendev/system-config master: refstack: add production image and deployment jobs  https://review.opendev.org/c/opendev/system-config/+/77458620:11
mordredianw: we should do a $something to mount in a working dir20:13
mordredianw: maybe we should add something like -v$(pwd):/work or something like that so you could still use it for image uploads20:13
diablo_rojofungi, so what's next?20:30
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add zuul-storage-proxy in zuul namespace  https://review.opendev.org/c/openstack/project-config/+/77236420:35
*** Dmitrii-Sh has quit IRC20:39
fungidiablo_rojo: what's next is i need to review your changes ;)20:40
*** Dmitrii-Sh has joined #opendev20:40
diablo_rojofungi, lol and then I do the rest of the steps in the renaming an irc channel section?20:40
fungidiablo_rojo: pretty much, yeah20:42
fungiand thinking about the channels involved, we probably don't need to do a ton of advance warning (it was already discussed on the ml anyway)20:43
fungiconfig-core: diablo_rojo is volunteering to help with irc channel management, and is working on some foundation channel moves to the #openinfra channel namespace: https://review.opendev.org/77455520:44
fungi"/bin/bash: ansible: command not found" huh, that's... unexpected https://zuul.opendev.org/t/openstack/build/e07f15fd4c7f43f09565eeda6521752520:47
fungithat job was just passing yesterday20:47
ianw" It will also require most users to update the way they install the linter as they now need to mention which version of Ansible they want to use it with."20:49
fungidid that just happen?20:49
ianwa few hours ago it seems, 5.0.0 release20:49
fungi"Released: about 6 hours ago"20:50
* fungi takes another look at the change for dropping ansible-lint jobs from system-confg20:51
openstackgerritMerged openstack/project-config master: Setup OpenInfra-Board Channel  https://review.opendev.org/c/openstack/project-config/+/77470520:56
*** whoami-rajat__ has quit IRC21:03
*** d34dh0r53 has quit IRC21:07
openstackgerritIan Wienand proposed opendev/system-config master: Stop running ansible-lint on this repo  https://review.opendev.org/c/opendev/system-config/+/73340621:08
*** d34dh0r53 has joined #opendev21:08
ianwcorvus / fungi: ^ that's an update if we don't want to get to the bottom of this21:08
*** hasharAway has quit IRC21:14
ianw  bash -c "ANSIBLE_INVENTORY_PLUGINS=./playbooks/roles/install-ansible/files/inventory_plugins ansible -i ./inventory/base/hosts.yaml not_a_host -a 'true'"21:14
ianwwe are probably relying on ansible-lint to pull in ansible for that21:15
fungiright, i'm guessing it no longer expresses a dep on it21:15
fungii'm working on reproducing locally now21:15
fungiyeah, reproduces here for me, trying just adding ansible to the deps list for the linters testenv now21:17
fungithat seems to get past the error21:19
ianwyeah, and i mean that probably fixes ansible-lint too21:20
openstackgerritJeremy Stanley proposed opendev/system-config master: Install ansible for ansible-lint testing  https://review.opendev.org/c/opendev/system-config/+/77473521:21
fungialternative if we want to keep it21:21
fungiat least the solution seems trivial21:21
ianwwell yeah, we're directly calling ansible so need it in the linters tox environment21:21
ianwthe usefulness of linting with uncapped ansible when bridge is running, whatever it's running is perhaps questionable21:22
fungiright, previously we were relying on ansible-lint to drag it in for us21:22
fungiand i agree, i waffled on adding a cap there, but figured i'd start with that and see what folks want21:22
ianwi'd like the gate to work :)21:23
fungiup to now we were relying on whatever version of ansible an uncapped ansible-lint depended on21:23
fungiso i figured uncapped ansible was effectively the same21:23
ianwyeah, it's no worse.  i feel like i had a change to update bridge ansible21:24
fungiokay, so we may need to squash them21:59
fungilooks like 774735 does also get ansible-lint to run21:59
fungiunfortunately 5.0.0 has lots of new things to say about the content in system-config21:59
corvusmaybe just add the requirement to 406?22:01
funginearly >4.5k lines of ansible-lint complaining about that repo22:04
fungiyeah, i'll revise 733406 with that22:05
openstackgerritJeremy Stanley proposed opendev/system-config master: Stop running ansible-lint on this repo  https://review.opendev.org/c/opendev/system-config/+/73340622:09
*** gmann is now known as gmann_afk22:13
*** Dmitrii-Sh has quit IRC22:27
*** Dmitrii-Sh has joined #opendev22:34
clarkbis there a tldr for which change(s) I shoudl review to fix ansibel lint stuff (if any)22:40
ianwclarkb: just have to wait for https://review.opendev.org/c/opendev/system-config/+/733406 to go through22:41
clarkbcool I'll catch up on refstack and backup reviews then22:41
clarkbsince the -1s appear related to the ansible lint thing22:42
ianwthanks; now we have a plan for pruning i'll add wiki to borg manually; that's the last thing outstanding22:43
ianwfungi: i know we had this discussion, but we need basically all /opt backed up?22:43
fungiianw: yes, the way i was trying to reorganize it on wiki-dev would put the configuration-managed stuff in a separate path from the precious data, but on the existing server even the deployment of the software is precious because it was done by hand22:46
fungiand everything runs out of git checkouts/submodules at specific refs22:47
ianwnp; the user/keys are all setup, i just need to copy the backup script and cron jobs into it22:48
clarkbianw: does `sudo FOO=bar -s <<'EOF' echo $FOO EOF` do proper var substituation in that heredoc with the env vars set prior to the -s?22:50
clarkbfor context I'm looking at https://review.opendev.org/c/opendev/system-config/+/774561/3/playbooks/roles/borg-backup-server/files/prune-borg-backups.sh and trying to replicate that behavior and haven't successfully gotten it to work with my local sudo and shell22:50
ianwclarkb: for this script we don't want to do that because we're passing the variables in via the sudo22:50
clarkbianw: if I do `sudo FOO=foo -s echo $FOO` it echos a blank newline not foo22:51
fungiin that example your shell is interpreting $FOO (and replacing it with a null string)22:52
clarkbok ya echo \$FOO works22:52
ianwyeah, the <<'EOF' stops that and passes it raw into the shell sudo has started22:52
clarkbTIL you can set env vars in the sudo command like that22:52
fungi$FOO needs to be part of the string which the subshell executes under sudo if you're going to rely on sudo's environment options22:52
ianwi admit, it's a little crazy -- but i wanted to have a single prune script and wanted to make sure to run as the user so we don't leave behind root files that might block further backups22:52
clarkbianw: ya I think the sudo -u is a really good idea22:53
clarkbI just had to reconcile the env var stuff in my own head. Thank you for walking me through that22:53
fungithe resulting syntax is a good bit cleaner than trying to accomplish the same with su -c22:53
clarkbianw: and checking the test logs we have https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_acb/774561/3/check/system-config-run-borg-backup/acb9e1d/borg-backup01.region.provider.opendev.org/prune-borg-backups.log22:58
clarkbianw: do we expect it to emit a bit more output tehre (saying which things it is pruning? I can't remember what it says for my local backups)22:58
ianwhrm, it may not if there is nothing to prune ... or the rather hacky way i ran that in testinfra might not be capturing it's output22:59
clarkbianw: I wonder if those tests all run in order? coudl be the prune runs before the archives happen?22:59
ianwi did think it would run in file sequential order ... but indeed that is another possibility23:00
ianwthe html log should tell us23:00
ianwdoes look like it ran last https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_acb/774561/3/check/system-config-run-borg-backup/acb9e1d/bridge.openstack.org/test-results.html23:01
ianwlet me run in noop mode and see what happens when i capture on servers23:01
ianwyeah, i can see output coming out.  it must use another fd23:03
clarkbcool, fwiw I really like the use of testing here to help ensure things are happenign the way we expect :)23:04
clarkbthis is a testing only issue for now so I'll leave it up to you if you want to fix it now or in a followup23:04
openstackgerritIan Wienand proposed opendev/system-config master: borg testing: catch stdout and stderr from test prune correctly  https://review.opendev.org/c/opendev/system-config/+/77474523:05
ianwor indeed we could do ^ ... i always get that mixed up23:05
ianwthanks, good catch23:05
clarkbianw: left a question on the docs change too23:06
clarkbianw: for the redirect fix shouldn't 2>&1 and &> be equivalent in bash ? 2>&1 is the sh specific version?23:07
* clarkb is learning lots about bash and sudo today23:08
ianwi had the order wrong though, the file after the 2>&123:08
ianw&> is just a shorthand so you don't make the mistake i did :)23:09
clarkbhttps://www.gnu.org/software/bash/manual/html_node/Redirections.html confirms the order is important, I feel like I remember that whenever I do it wrong too23:09
clarkbianw: I've +2'd the stack but didn't approve as I awsn't sure the lint fix is in the gate yet23:12
clarkblooks like 733406 is just about to merge so approving things is probably ok23:12
*** Dmitrii-Sh has quit IRC23:15
fungiyeah, `foo >bar 2>&1` does basically the same as `foo &> bar`23:15
fungiexcept there's an implied 3>&1 and 4>&1 and so on, i believe23:16
fungiahh, nope, bash manpage says it's just fd 1 and 223:17
fungialso acknowledges you can write it >& but that notation is ambiguous and limits what characters the target can start with23:17
*** Dmitrii-Sh has joined #opendev23:18
*** sboyron has quit IRC23:18
* mordred always find &> weird and too new23:23
mordredor >& or whatever it is23:23
ianwapropos nothing, we mentioned node launch in the meeting ... there was one slight problem with unattended-upgrades23:24
ianwi don't know exactly what it's doing, but it seems to install every single package it wants to update separately.  this takes like close to an hour to get through, it rebuilt the initramfs liek 10 times23:25
clarkbianw: I wonder if it is really built to be run daily and if you boot something that is quite behind it doesn't work well23:26
clarkbits possible we may just want to replace it with apt-get update && apt-get dist-upgrade -y ?23:26
ianwyeah, i think it's supposed to be run on shutdown or something and does it like that to avoid blocking things23:26
clarkbI think I added that bit to the script and it was just an easy way to use existing tooling to get the image up to date before we proceeed23:26
ianwyeah, i think we need that earlier in the launch script.  i'll take a look, anyway just remembered that23:26
mordredwe should be careful with dist-upgrade -y - probably want to make sure we've applied our apt config first (because we turn off recommends iirc) - and dist-upgrade -y could wind up installing some things that we wouldn't otherwise install23:37
mordredI mean - I thnk it's probably a good idea - just saying we should make sure our apt config stuff has applied23:38
fungian excellent reminder23:39
clarkbmordred: ya I think the current code runs after applying base configs23:39
clarkbthat pulls in our unattended upgrades at least and then we just run that23:39
clarkbwhich was why I seem to recall using it, since it was going to use our existing policy for updates and that was easy mode23:39
openstackgerritMerged opendev/system-config master: Stop running ansible-lint on this repo  https://review.opendev.org/c/opendev/system-config/+/73340623:40
*** tosky has quit IRC23:48
*** DSpider has quit IRC23:54

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!