Monday, 2023-01-09

*** rlandy is now known as rlandy|out00:30
*** JasonF is now known as JayF00:54
*** yadnesh|away|pto is now known as yadnesh03:59
ianwi've sent a mail about the columns for labels in gerrit 3.6 @
*** ysandeep is now known as ysandeep|ruck05:42
fricklerlooks like a number of mirror volumes are getting close to being full, also afs01.dfw is > 90% now 05:46
ianwa lot of upward slanting lines there05:56
ianwmaybe reducing usage here would be a good carrot for pushing on purging xenial work05:58
*** marios is now known as marios|rover05:58
*** ysandeep|ruck is now known as ysandeep|lunch07:34
*** jpena|off is now known as jpena08:49
*** priteau_ is now known as priteau08:58
*** ysandeep|lunch is now known as ysandeep09:38
noonedeadpunkhi everyone!10:37
noonedeadpunkI'm quite stuck with investigating on why job does fail in CI, as I can't reproduce it. And nasty thing that I don't really see the way of how to debug it, as ansible module for lxc_container does have `--quiet` flag hardcoded
noonedeadpunkSo any chance to get a hold on the task?10:38
noonedeadpunkPatch in topic is and job `openstack-ansible-deploy-hosts_lxc_zfs-ubuntu-focal` (will need to recheck it though)10:39
fricklernoonedeadpunk: I'll set up a hold in a minute and recheck the job, please let us know your ssh key, too11:44
fricklerslaweq: there's still a held node for 865822, I assume that it can be removed now? (re-asking from last week)11:48
*** rlandy|out is now known as rlandy12:09
*** ysandeep is now known as ysandeep|afk12:54
fricklernoonedeadpunk: seems your job has passed now, let me know if you want to keep retrying or maybe modify your patch, or if I should cancel the hole12:56
noonedeadpunkugh, that's annoying... Maybe depends on provider...12:58
noonedeadpunkI will try to recheck more12:58
jrosseri am thinking extra-disk / not extra-disk.....12:58
noonedeadpunkYeah, seems it's super close to truth12:59
fungiwhat was the symptom? just the task exiting with a nonzero return code?13:03
noonedeadpunkfor 1 specific scenario (zfs)13:03
noonedeadpunkOr well, the task is run of lxc_container module, that has hardcoded `--quiet` option in their code13:04
*** ysandeep|afk is now known as ysandeep13:58
*** ysandeep is now known as ysandeep|ruck13:59
*** dasm|off is now known as dasm14:01
noonedeadpunkfrickler: it failed now :)14:38
slaweqfrickler yeah, sure, sorry that I forgot to tell You that14:44
fricklernoonedeadpunk: good, I guess? anyway root@ is now yours14:48
*** ysandeep|ruck is now known as ysandeep14:48
fricklerslaweq: np, cleaned up now14:48
noonedeadpunkyep, good enough14:48
slaweqfrickler: thx a lot14:48
*** ysandeep is now known as ysandeep|ruck14:51
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: use-buildset-registry: support microk8s
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Switch ensure-kubernetes role default to cri-o from docker
noonedeadpunkfrickler: thanks, you can release hold now! That was very helpful15:11
funginoonedeadpunk: what did the problem wind up being?15:28
noonedeadpunkwe have different path on how to create zfs pools when provider has extra drive that should be used for data15:29
noonedeadpunkand we appeared to have different pool names in tasks depending on the path15:29
*** yadnesh is now known as yadnesh|away15:39
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Clean up tox.ini for tox v4
*** ysandeep|ruck is now known as ysandeep|dinner16:48
*** marios|rover is now known as marios|out16:52
fricklernoonedeadpunk: done, thx for the feedback16:54
opendevreviewClark Boylan proposed opendev/base-jobs master: Add opendev nox docs promotion
clarkbcorvus: ^ something like that?16:58
opendevreviewClark Boylan proposed opendev/base-jobs master: Add opendev nox docs promotion
opendevreviewClark Boylan proposed opendev/base-jobs master: Fix tox.ini for tox v4
jrosserclarkb: would you be able to help me looking at some connectivity trouble?17:23
clarkbjrosser: sure17:24
jrossera traceroute/mtr to from would be great if you could do one17:24
opendevreviewMerged zuul/zuul-jobs master: Remove kubernetes docker jobs
clarkbinfra-root reviews on 869580 and 869582 appreciated (fixes tox v4 problems in base-jobs then adds nox jobs to help us move away from tox)17:44
*** jpena is now known as jpena|off17:45
*** rlandy is now known as rlandy|brb17:54
clarkbinfra-root meeting agenda is coming together. Please take a look and add anything I've missed over the holidays. Or let me know what I've missed and I'll add it18:08
*** ysandeep|dinner is now known as ysandeep|out18:17
*** rlandy|brb is now known as rlandy18:29
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: use-buildset-registry: support microk8s
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Switch ensure-kubernetes role default to cri-o from docker
fungijrosser: clarkb: did you ever manage to track down the bbc/gitea connectivity problem?20:15
jrosserfungi the trace routes look pretty dimilar from each side but the throughout is/was just 50-100kbit/s20:18
jrosseryes, it seems sensitive to routing/peering at this end. picking a geographically similar server that went via cogent there was no issue20:20
jrosserwe had someone in osa irc the other day saying their deployment was taking many hours for no apparent reason, and it’s possibly the same thing, the symptom is just huge slowness20:21
fungijrosser: yes, i used to see that in my old job wrangling backbone providers and massaging bgp prefs, it wasn't at all uncommon for two backbones to get into an argument over who was going to pay to repair or upgrade a peering interconnect, and then you'd get fun packet loss or latency depending on what intermediate routes your traffic followed20:24
fungicould persist for days or months20:24
fungiwe'd add all manner of weird conditional pads/prepends to try to route around those situations20:26
opendevreviewMerged zuul/zuul-jobs master: use-buildset-registry: add URL to buildset_registry_namespaces
opendevreviewMerged zuul/zuul-jobs master: ensure-kubernetes: add microk8s support
opendevreviewMerged zuul/zuul-jobs master: use-buildset-registry: support microk8s
ianw    ssh_client.ssh('(ifdown eth0 && ifup eth0) || true')22:11
ianwAttributeError: 'NoneType' object has no attribute 'ssh'22:11
ianwthis is a new one trying to launch a new nb04 in osuosl22:11
*** dasm is now known as dasm|off22:20
Clark[m]I feel like this is the issue I hit with paramiko not supporting rsa sha222:20
Clark[m]It was hard to debug because ssh failed in this weird way but once I updated paramiko to latest it worked22:21
ianwhrm, interesting.  this is *after* it has connected as the cloud user on the image, copied the auth bits and now trying to connect back as root22:22
Clark[m]I don't think I ever got that far previously with the rsa sha2 problem. Hrm 22:23
fungicould the failure mode be specific to the user keys? i.e. affecting root's but not the cloud user's?22:27
ianwi'm wondering same ... i'm trying again, maybe it was a fluke22:28
ianwok, not a fluke ... 22:36
ianwbut i also launched the mirror node in linaro just before xmas, and that didn't do this22:37
ianw"PermitRootLogin no"22:47
ianwi think the osuosl 22.04 base image might have added that22:47
fungiseems likely22:48
corvusClark: left a comment on
clarkbcorvus: can I set the var as a variant in the new project-template? I guess that would be a better way to express this if so22:51
corvusclarkb: yeah i like that idea22:52
corvus(also, i didn't -1 because i'm not opposed to making a new job with that default, but i do think i have a small preference for trying to avoid that -- basically trying to stick to the original design)22:52
clarkbok, I'll spin up a new patchset that does this shortly22:52
opendevreviewClark Boylan proposed opendev/base-jobs master: Add opendev nox docs promotion
clarkbcorvus: ^ like that?23:03
ianw... sigh, i've uploaded a generic image to osuosl, but that doesn't seem to connect23:19
clarkbianw: could also edit the ssh config as the ubuntu user before flipping over23:20
ianwyeah, that's my next option :)23:20
ianwi may have just missed a image property or wrong format, etc.  it seems easier to not debug it though23:21
corvusclarkb: ++23:36
clarkbok last call for meeting agenda edits. I'll send that out in about 20 minutes23:42
opendevreviewIan Wienand proposed opendev/system-config master: launch: permit ssh root login on base image
ianwyeah, the other server is stuck in uefi somehow23:50
ianw(the one with the image i uploaded)23:50

Generated by 2.17.3 by Marius Gedminas - find it at!