Wednesday, 2022-08-10

clarkbianw: that whole stack lgtm now. But we probabl want ot double hceck the correct versions are installed before approving00:08
ianwyeah i'll pull out the log references00:09
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run-borg-backup: rename hosts to distro
opendevreviewIan Wienand proposed opendev/system-config master: create-venv: add role; use in install-borg
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run: bump base timeout to 3600
opendevreviewIan Wienand proposed opendev/system-config master: install-borg: update to borg 1.1.18
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run-borg-backup: add jammy test host
opendevreviewIan Wienand proposed opendev/system-config master: gate-groups: remove old backup group
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run-borg-backup: rename hosts to distro
ianwsigh, typo00:15
*** clarkb is now known as Guest16600:18
Guest166hrm I don't want to load the passwd manager to reauth this evening. I'll be a guest for now00:19
*** dasm|off is now known as Guest16700:20
fungiwe're very welcoming to guests00:21
Guest166seems like my connectivity to oftc has been much flakier lately00:21
fungime too apparently00:31
fungilooks like it's probably rolling server restarts00:31
*** rlandy is now known as rlandy|out00:45
*** ysandeep|out is now known as ysandeep01:23
opendevreviewIan Wienand proposed openstack/project-config master: nodepool: update package maps for Rocky 9
opendevreviewMerged openstack/project-config master: project-config-grafyaml: drop job
opendevreviewMerged opendev/system-config master: create-venv: add role; use in install-borg
*** ysandeep is now known as ysandeep|afk02:44
fungiianw: looks like infra-prod-service-borg-backup failed in deploy for 85248703:32
fungii haven't looked at the logs yet03:32
ianwsigh, looking04:44
ianwThe task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'borg_user'04:46
ianwwhich is weird ...04:46
ianw"item": ""04:47
ianwoooohhhh i think i see04:53
*** ysandeep|afk is now known as ysandeep04:57
ianwoh for heavens sake .. it's translate0105:01
ianwthe venvs look broken on some of these old hosts.  i'm probably going to have to punt on this one until tomorrow05:22
*** marios is now known as marios|ruck05:37
*** pojadhav is now known as pojadhav|rover05:43
*** prometheanfire is now known as Guest19407:00
*** ysandeep is now known as ysandeep|afk07:09
*** Guest194 is now known as Guest20007:28
*** jpena|off is now known as jpena07:36
*** ysandeep|afk is now known as ysandeep07:37
*** Adri2000_ is now known as Adri200008:29
*** ysandeep is now known as ysandeep|lunch09:34
*** rlandy|out is now known as rlandy10:37
*** ysandeep|lunch is now known as ysandeep11:06
*** dviroel|out is now known as dviroel11:24
*** bhagyashris_ is now known as bhagyashris11:32
fungibroken at some point in the past and we've been needing to blow away and recreate them?11:42
*** gthiemon1e is now known as gthiemonge12:09
fungiwe should probably brace for an impending setuptools 64.0.0 release with the new pep 660 editable installs implementation, since i wouldn't be surprised if some of our communities turn up corner cases the maintainers aren't expecting12:35
opendevreviewSimon Westphahl proposed zuul/zuul-jobs master: Supply missing error message on S3 upload failure
*** ysandeep is now known as ysandeep|break13:30
*** ysandeep|break is now known as ysandeep14:34
*** ysandeep is now known as ysandeep|out14:47
*** dviroel is now known as dviroel|lunch15:38
*** marios|ruck is now known as marios|out15:49
*** Guest200 is now known as prometheanfire15:59
*** pojadhav|rover is now known as pojadhav|out16:22
*** dviroel|lunch is now known as dviroel16:47
*** Guest166 is now known as clarkb16:50
*** jpena is now known as jpena|off16:52
fungifinally have a moment to dig into the borg venvs on xenial servers... they do look thoroughly borg^Hked18:14
fungicalling `pip list` in then results in a cascade of tracebacks18:14
fungiSyntaxError: invalid syntax18:14
clarkbfungi: they probably had the same issue we fixed for bionic but since the condition matched only bionic they broke18:14
clarkbI think we need to rebuild them with pip and setuptools caps for python3.518:14
clarkbbasically do the same thing we did for bionic18:15
fungii think we ended up installing the bionic (py36) version on xenial (py35) yes18:15
fungiagreed, should be a pretty straightforward patch18:15
clarkbI think its worse than that. We install python3.7 versions on xenial18:15
clarkbeg latest18:15
fungier, yeah but not py35 versions at any rate18:15
fungii'll see what we need to pin back to for that18:15
fungiof course, we don't have python3-venv installed, so i'm not sure how these venvs/virtualenvs could even have been created18:16
fungiis there some magic back door to make a venv?18:17
clarkbis virtualenv installed? I think once the venv exists ansible doesn't care about the install comamnd so our attempts to use -m venv would just be ignored18:18
fungino, at least not in the module search path for python318:18
clarkbmaybe it used python2 virtualenv?18:19
fungithe installed libpython3.5-stdlib does seem to provide /usr/lib/python3.5/venv though, it's the lack of ensurepip which seems to be at issue18:19
fungifungi@storyboard01:~$ /opt/borg/bin/python --version18:19
fungiPython 3.5.218:19
fungibetter still, the python3-venv package actually is installed18:22
fungiaha, we had pending upgrades for python 3.518:25
fungiunfortunately, we're getting 401 unauthorized from esm.ubuntu.com18:26
clarkbfungi: I guess we should email them about extending the existing setup?18:27
clarkbwe're down to just a handful of nodes now18:27
fungioh, i guess maybe it expired?18:27
clarkbyes I think they gave us a year and that year ran out.18:27
clarkbIn theory we can ask them to extend it and then it will just work again? I dunno the documentation and info on that stuff is very light (probably intentionally)18:28
fungiyeah, so in short i don't have access to any python3.5 systems i can easily create a venv on, so i'll inspect package metadata instead18:37
fungii even deleted the old 3.5 build on my workstation to free up space a while back18:37
clarkbfungi: docker run -it --rm ubuntu:xenial should still work18:38
fungijust append whatever commands i want to run in a xenial container after that?18:38
clarkbfungi: based on pip changelog pip<2118:38
clarkbfungi: ya I would usually run bash18:39
clarkband that will give you a shell prompt that you can interact with18:39
clarkb`docker run -it --rm ubuntu:xenial bash` is the full command18:39
clarkbthen when you exit bash docker will clean up the container18:39
fungioh, yep it started a root shell for me18:39
fungiso looks like pip<21 setuptools<5118:40
clarkbwe may need to manually clear out the old venv and let ansible rebuild it after we update our ansible. Or manually fix it up and then let ansible enforce that state going forward18:44
clarkbbasically I don't expect the current state to be recoverable using the ansible tooling unless we add some rm's to the ansible too18:44
fungiyeah, i expect that as well18:48
fungiwhat's the ansible syntax if i want to compare against a list of strings?18:49
fungiright now our fallthrough is "when: ansible_distribution_release != 'bionic'"18:50
fungiand i'd like it to be neither xenial nor bionic18:50
fungiahh, i can do this instead i think:18:51
fungiwhen: ansible_distribution_versi18:51
fungion is version('18.04', '>=')18:51
fungier, i guess i want >=20.04 for the fallthrough18:52
opendevreviewJeremy Stanley proposed opendev/system-config master: Also pin pip/setuptools when creating Xenial venvs
fungiclarkb: ianw: ^ that maybe?18:55
Clark[m]That looks right. CI should provide useful feedback 18:57
clarkbfungi: that didn't actually run the backup test job. I think we should update the job to trigger on updates to that create-venv role (at least temproarily)19:33
clarkbdo you want to make that update or shoudl I push that?19:33
fungii can give it a shot19:33
opendevreviewJeremy Stanley proposed opendev/system-config master: Also pin pip/setuptools when creating Xenial venvs
fungiclarkb: ^19:35
clarkbcool zuul has enqueued the borg job this time19:36
clarkbwe should be able to confirm it installs the right packages on bionic (we don't have a xenial test node in that job) which is probably good enough19:37
*** rlandy is now known as rlandy|mtg20:16
fungisystem-config-run-borg-backup TIMED_OUT20:27
clarkbfungi: I think that shows the bionic host installing older pip and setuptools nad that focal and jammy install the latest and greatest20:57
clarkb+2 from me20:58
*** rlandy|mtg is now known as rlandy21:03
clarkbit seems that error has gone away now that I'm doing the alembic check21:41
clarkbbut hard to say if I'm just getting lucky or that is helping21:41
opendevreviewAlfredo Moralejo proposed opendev/system-config master: Use rackspace mirror to sync centos stream repos
*** dviroel is now known as dviroel|out21:48
clarkbrandomly my brightness change buttons work again22:02
ianwthanks for looking at the venv stuff!22:05
ianwthat was my assumption too that we'd pulled in later versions22:05
ianwon the timeout, the change to bump the timeouts hasn't merged yet because of ... timeouts :)22:07
fungii would have recheck-spammed it in earlier today if i'd remembered22:08
fungiwe could also bypass zuul to merge that22:08
fungifalls into the catch-22 category where that is marginally acceptable22:09
*** rlandy is now known as rlandy|bbl22:17
ianwi'll give it one more chance otherwise take corrective action22:21
clarkbI suspect that I may have misinterpreted the SMTP_HOST variable's purpose and that is why ianw had issues signing up22:31
clarkbI think its meant to designate the source ip for sending email and it sends directly?22:32
clarkbso setting it to localhost is just all wrong22:32
fungichecking whether that has a direct analog in mm222:34
clarkbnevermind here it says it wants to send through the host itself22:37
fungithere's a SMTPHOST='localhost' in which we don't seem to override22:37
clarkbthe web readme says it is the ip to send from22:37
clarkbok so I've probably got that bit correct. Will have to dig in properly tomorrow22:37
fungialso SMTP_HELO_HOST=DEFAULT_URL_HOST which may be more of what you were thinking of22:38
clarkbianw: oh interesting note on the issue being upstream and possibly deserving further investigation22:39
clarkber previously I mean and maybe this time. We don't know about this time22:39
* fungi grumbles at another timeout and rechecks again23:15
ianwso basically i'll manually remove /opt/borg venv on xenial hosts and let them get recreated23:27
clarkbonce that change lands23:28
ianwthat's going to be the other thing about upgrading borg to 1.2 series,; it's 3.8+23:29
clarkbso we'd need everything on focal or newre. Which is in progress but progress is slow :)23:30

Generated by 2.17.3 by Marius Gedminas - find it at!