Tuesday, 2020-08-11

fungiianw: so the problem boils down to wireless bridging, and isn't really ipv6-specific00:10
fungi802.11 isn't ethernet, is the crux of the challenge00:11
fungiwith ethernet, you can have a virtual ethernet bridge on your laptop, and connect the virtual machine's interface to it with a distinct mac00:12
ianwright, that's what i have done when i'm at my office desk in the dock, where i'm wired into the network00:13
fungiyeah, the ethernet part is the straightforward part00:13
fungii've not tried bridging a virtual interface to a wlan, i'll need to look around and find out what's out there for that00:14
fungii know it exists, because folks set up linux machines as wireless access points all the time00:14
fungiso they need to be able to bridge ethernet frames/addresses to wifi and back00:15
ianwwell yeah, i can't add my wifi card to a bridge on my laptop00:16
fungiianw: arch wiki says to use hostapd to map it: https://wiki.archlinux.org/index.php/Network_bridge#Wireless_interface_on_a_bridge00:16
ianwhrm, this must be something like virtualbox does under the hood00:17
fungialso links to the debian wiki which suggests alternative of translating via ebtables: https://wiki.debian.org/BridgeNetworkConnections#Bridging_with_a_wireless_NIC00:17
ianwthat allows me to to select a network of "bridged adapter" and select the wifi interface00:18
ianwwhich is great, but it doesn't work00:18
fungiianw: an alternative would be to set up a router on your laptop with a separate routed subnet... how much v6 space do you get? can you request larger or multiple allocations via dhcp6-pd or similar?00:19
fungithe idea would be to configure downstream prefix delegation to your laptop, and then have it run a local dhcpd on the virtual bridge and something like radvd to announce that network, and then route between the vm and your laptop's gateway00:22
ianwi have a /56 ... but a) my preference would be not hard-code it; e.g. to handle the case of when i occasionally tether to 4g and am not on the home network, and b) how does that subnet know to go via the ethernet or wifi00:22
fungiyeah, i'm suggesting you don't hard-code it, but... it's multiple daemons to have to manage00:22
fungiif you also run dhcp6 on your router it can serve an additional /64 to your laptop which your laptop then in turn announces on the virtual bridge00:23
fungibut i do think the layer 2 solution is likely to be less work than the layer 3 solution00:24
ianwyeah, i'm running a unifi security gateway.  it really only provides for slaac on a network; but you can define multiple networks, each with a prefix from the /5600:24
fungiit may not be flexible enough in that case. it's one of the reasons my home router is a sbc with straight up openbsd installed00:25
fungimakes it possible to play around with much more complex network configurations00:25
ianw(which is exactly what I *don't* want in the house :)00:26
fungiso anyway, one of the layer 2 solutions involving either hostapd or ebtables on your laptop is probably what you need00:27
ianwi think that must be similar to what virtualbox is doing, which unfortunately seems to have issues https://www.virtualbox.org/ticket/550300:28
ianw"The benefit of using an IP / Layer 3 solution is that it will work when the outward-facing interface is a wireless ethernet client, without using WDS and without resorting to NAT. Simple Layer 2 bridging does not work in this case due to the vagaries of wireless AP client behaviour; WDS may help but AP support can be patchy. "00:31
donnydianw: I  would think there is a path where your laptop becomes a router instead of a NAT device. If you advertise a subnet from your laptop - you should be able to route v6 networks..00:42
donnydunless what you are looking for is something like what libvirt does by default - which is just "nat all outbound traffic from this bridge to whatever"00:43
ianwdonnyd: bridge, NAT, carrier pigeon would be fine, i'd take anything at this point :)00:45
fungidonnyd: yep, we covered routing above... it should work but if he wants to avoid hard-coding prefixes it likely involves running a dhcp6d somewhere to subdelegate the additional routed prefix00:54
fungiand then route announcements from the laptop to the local lan for routing and onto the bridge for slaac and default route00:55
ianwi'm not sure how that knows to send packets out the wired or wireless interface though00:56
fungihere i'm assuming you're only ever connecting one at a time00:56
fungihaving more than one interface routing to the internet at the same time gets... weird00:57
ianwwell, yes, only one at a time i guess.  networkmanager is generally handling this00:57
fungi(for v6 or v4)00:57
fungiyeah, so your laptop would route those packets "normally" based on whichever interface knows how to get packets where they're destined00:58
ianwdo people just not use vm's on wifi?01:02
fungithat's my suspicion01:03
fungii don't anyway01:03
ianwi'm willing to hardcode prefixes because i guess i'm not going anywhere for 6 more weeks :/  i would like the ability to switch to 4g and have it "just" work ...01:03
fungii also mostly launch virtual machines out on the internet rather than locally01:04
ianwi've always had a "work" vm, that keeps work keys etc. separate and never logs into any personal accounts etc.01:04
ianwit's also nice when the laptop breaks that it's just a cp of the image to get going again01:05
fungiahh, yeah, i just use separate physical machines, but i also don't ever take machines with work keys on them out of the house, so permanently wiring them to the lan is the lazy solution in my case01:07
ianwit looks like in NM i can create an ethernet interface with an option "shared to other computers" for ipv601:08
donnydYou can also use BGP on your laptop (a little heavy), but it will surely work01:19
fungiyeah, but radvd or similar should do the trick in this case01:20
fungiibgp just for v6 would definitely be overkill01:20
donnydyes, but fancy and fun01:20
fungiheck, even simple rip becomes a useful igp for ipv601:21
donnydI think you *can* also NAT ipv6 just like you can ipv4 as well.. I just don't know how to do it.01:22
donnydso you could mock the rules up in ipv6 that are there by default for v401:22
fungiyeah, i expect iptables could do it01:23
funginat just makes me queasy01:23
donnydyea, its not very ipv6y01:23
donnydrouting is the preferred method I would imagine - but for this use case, I do believe you can just masquerade a v6 subnet just like a v4 one and get the same thing01:24
donnydianw: are you trying to connect inbound to these VM's or are you just trying to get them a v6 address01:25
donnydI would imagine the end state would require know what exact you want on the instances01:25
ianwdonnyd: i don't care about inbound.  i just want to ssh to nodes via ipv6 basically :)01:25
fungiyeah, you could just use linklocal on the bridge and nat it then01:26
donnydfrom the laptop while you are on it, or from other non local machines01:26
donnydnon-localhost machine I should say01:27
fungioh, wait, not linklocal. i don't think you can convince the laptop to route that, even via nat01:27
fungii think there is block set aside as non-global v6 for nat though01:27
donnydyea, you would have to setup dhcpv6 and give them a real address so it can be NAT properly I would think01:28
* fungi checks iana01:28
donnydI have never actually tried any of it though - I use the bgp functions in openstack for my v6 needs01:28
ianwfungi: yeah, i looked into that.  i don't think there is an equivalent01:29
donnydAs a result, the IETF reserved the address block fc00::/7 in October 2005 for use in private IPv6 networks and defined the associated term unique local addresses.01:31
fungiyep, that one01:31
donnydthat one I think should work01:31
donnydI am real curious to see how this works out - I may even have to give this a whirl myself01:32
fungiis what i was looking for01:32
fungiinterestingly, rfc 4193 doesn't mention nat01:33
fungiit seems to be concerned more with vpn use cases01:33
fungibut i don't immediately see why it wouldn't work to do layer 4 pat/overload nat for it to a single global address01:34
ianwSince 6.5.0 it is possible to enable NAT with IPv6 networking. As noted above, IPv6 has historically done plain forwarding and thus to avoid breaking historical compatibility, IPv6 NAT must be explicitly requested.01:34
ianw... maybe i've missed something ...01:34
ianwv6.5.0 (2020-07-03)01:38
ianwok, maybe i can be forgiven for missing it was added in the last month ... still .. this is promising maybe01:39
ianwok dokie, i've got the libvirt preview copr installed with 6.6.0 so ... quitting irc now to see if i can get this to work ...01:55
donnydfungi: does this work for you `curl -k https://[2001:470:8:2e9::1]:5000`02:01
donnydianw: reminded me to tighten up my firewall rules for v6 - I could have pretty much been an interwebs router the way I had it - LOL02:02
*** icey has quit IRC02:05
*** icey has joined #opendev02:08
donnydI can also confirm that ipv6 MASQUERADE does work - at least with my setup02:14
donnydip6tables -t nat -I POSTROUTING -s fc00::/64 -o $YOUR_INTERFACE -j MASQUERADE02:14
ianwdonnyd: it works for me!!!!!!!!!!!!!!!!!!02:38
ianwi have ipv6 over my wifi connection!!!!!!02:38
ianw    inet6 fc00:dead:beef:55:2450:cdd1:93cc:da6e/64 scope global dynamic noprefixroute02:39
ianw       valid_lft 3476sec preferred_lft 3476sec02:39
donnydwoot woot!!!02:40
donnydnow when you switch between does it still work?02:40
donnydand did you set it up, or did libvirt?02:40
ianwyes, it seems to work!  i am not on wired connection02:43
ianwi mean i am *now*02:43
ianwi can still hit 2e9::102:43
ianwi do not really understand how it is working, and there's something about having to set accept_ra on the wifi card that i might have to loop via libvirt mailing list for02:44
ianwit also seems that things aren't connecting via ipv6 by default02:46
ianwi wonder if that's because it's a fc00 address02:46
*** zbr9 has joined #opendev03:03
*** auristor has quit IRC03:04
*** zbr has quit IRC03:04
*** dpawlik2 has quit IRC03:04
*** zbr9 is now known as zbr03:04
donnydthat is possible03:11
donnydwhat is in your dnsmasq config03:11
donnydhere is what I ended up using03:11
*** auristor has joined #opendev03:14
donnydand here are the results https://usercontent.irccloud-cdn.com/file/46tvwUCG/image.png03:15
donnydianw: I have been wanting to get my v6 stuff cleaned up, so thank you for the reminder :)03:17
ianwhrm, i guess libvirt must be running it in the background, let me poke03:18
ianwthat's basically it03:19
ianwi guess something lin the 2001:'s is better03:43
ianw2001:10::/28 deprecated (formerly ORCHID)03:44
ianwmaybe that's a good place that will never conflict03:44
donnydwell that was to show my public v6 address is from my edge router03:45
donnydand what is on my machine is being NAT outbound03:46
donnydI am happy you got it working03:46
donnydand I am also happy you reminded me to clean up my ipv6 mess in my edge machine03:46
ianwdonnyd: me too :)  dropped a message with some of the bumps and see what comes of it : https://www.redhat.com/archives/libvirt-users/2020-August/msg00042.html03:47
*** raukadah is now known as chkumar|rover04:43
*** ysandeep|away is now known as ysandeep05:04
*** danpawlik has joined #opendev06:37
*** auristor has quit IRC06:47
*** hashar has joined #opendev06:49
*** danpawlik has quit IRC06:54
*** danpawlik has joined #opendev06:55
*** hashar has quit IRC07:05
*** tosky has joined #opendev07:41
*** hashar has joined #opendev07:51
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** DSpider has joined #opendev08:09
*** ysandeep is now known as ysandeep|lunch08:25
*** priteau has joined #opendev08:26
*** priteau has quit IRC08:42
*** priteau has joined #opendev08:58
yoctozeptomorning channel09:04
yoctozeptodid something zuul/nodepool-related changed today (2020-08-11) between 0 and 6 (UTC)? asking because kolla started getting permission errors in jobs; could be new images but it must be something common to all distros09:06
ianwyoctozepto: i'm not aware of anything deliberately changing ... not that is much help :)09:10
ttxlourot: all set at https://github.com/openstack/charm-keystone-kerberos -- now it will sync at the next push.09:10
ttxI'll have a look and see if it would be complicated to trigger that sync after the repo is first created09:11
yoctozeptoianw: thanks, no problem, glad to know it was not on purpose :-)09:12
yoctozeptoFWIW, ansible seems to have released during that time frame https://github.com/ansible/ansible/releases09:19
lourotttx, thanks a lot!09:22
*** ysandeep|lunch is now known as ysandeep09:32
*** tkajinam has quit IRC09:36
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: add-build-sshkey: call cmd with command  https://review.opendev.org/74564609:41
openstackgerritVanou Ishii proposed opendev/puppet-openstackci master: Fix Misuse of Markdown Syntax in reStructuredText  https://review.opendev.org/74564709:50
openstackgerritVanou Ishii proposed opendev/puppet-openstackci master: Fix Misuse of Markdown Syntax in reStructuredText  https://review.opendev.org/74564709:57
yoctozeptoianw: it's to be fixed on our side but notice there is a breaking change in stable ansible backports: https://github.com/ansible/ansible/pull/7022110:04
yoctozeptodefault mode has changed10:04
yoctozepto"for security reasons"10:04
*** priteau has quit IRC10:11
*** dirk has joined #opendev10:43
*** hashar has quit IRC10:53
frickleryoctozepto: poor kitty ;)11:01
yoctozeptofrickler: :-)11:06
*** hashar has joined #opendev11:12
*** hashar has quit IRC11:12
*** auristor has joined #opendev11:58
*** iurygregory has quit IRC12:25
*** iurygregory has joined #opendev12:26
*** ryohayakawa has quit IRC12:27
*** hashar has joined #opendev12:36
fungii mentioned it over in #zuul too12:45
fungiif this turns out not to be a temporary regression, the service-discuss and zuul-discuss mailing lists might also warrant a heads up12:46
openstackgerritRiccardo Pittau proposed openstack/diskimage-builder master: Do not install python2 packages in ubuntu focal  https://review.opendev.org/74566512:52
openstackgerritClark Boylan proposed opendev/system-config master: Two small gerrit base image cleanups  https://review.opendev.org/74559513:00
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: source-repositories: git is a build-only dependency  https://review.opendev.org/74567813:53
*** mlavalle has joined #opendev13:57
*** smcginnis has quit IRC14:01
openstackgerritClark Boylan proposed opendev/system-config master: Two small gerrit base image cleanups  https://review.opendev.org/74559514:02
openstackgerritThierry Carrez proposed openstack/project-config master: Allow TC to review governance-sigs changes  https://review.opendev.org/74567914:03
*** smcginnis has joined #opendev14:05
*** chkumar|rover is now known as raukadah14:42
openstackgerritMerged openstack/project-config master: Allow TC to review governance-sigs changes  https://review.opendev.org/74567914:55
openstackgerritJulia Kreger proposed openstack/diskimage-builder master: Handle NetworkManager for dhcp-all-interfaces  https://review.opendev.org/74569815:12
*** mlavalle has quit IRC15:15
*** mlavalle has joined #opendev15:17
openstackgerritClark Boylan proposed opendev/system-config master: Two small gerrit base image cleanups  https://review.opendev.org/74559515:23
clarkbI think that change has fixed the codemirror-editor issue (likely simply due to setting the tag checkout)15:26
clarkbstill working on the javamelody plugin issue btu I think that latest patchset is close15:27
clarkbopendev has finished. I'm going to take a potentially long break now and see if I can get a bike ride in too. I'll be back for the meeting15:40
*** ysandeep is now known as ysandeep|away15:56
*** tosky has quit IRC16:59
*** hashar has quit IRC17:10
clarkbhttps://review.opendev.org/745595 seems to fix the java melody plugin issues on 2.16. I need to run through all images again and test them but that change is looking good17:59
clarkbinfra-root ^ if you'd prefer I can split it up into a few changes too17:59
clarkbbut I think I need to prep for the meeting, then after the meeting I want ot try and deploy gerritbot container on eavesdrop18:00
fungiclarkb: seems to break the 3.0 image build, just pulling up logs now18:13
fungiERROR: Skipping 'plugins/javamelody:javamelody-deps_deploy.jar': no such target '//plugins/javamelody:javamelody-deps_deploy.jar': target 'javamelody-deps_deploy.jar' not declared in package 'plugins/javamelody' (did you mean 'javamelody_tests_deploy.jar'?) defined by /home/zuul/src/gerrit.googlesource.com/gerrit/plugins/javamelody/BUILD18:14
clarkbhttps://gerrit.googlesource.com/plugins/javamelody/+/refs/heads/stable-3.0/src/main/resources/Documentation/build.md seems to show that the extra copy isn't needed in 3.018:15
clarkbbut also the target is gone too18:16
fungiahh, so we can version-bound that step?18:16
clarkbya I'll get a new ps up shortly18:16
fungino rush, just making sure it wasn't something more complicated18:16
openstackgerritMerged zuul/zuul-jobs master: Allow ara-report to run on any node  https://review.opendev.org/74297118:18
openstackgerritClark Boylan proposed opendev/system-config master: Two small gerrit base image cleanups  https://review.opendev.org/74559518:24
clarkbI think ^ should do it.18:24
*** mlavalle has quit IRC18:54
*** mlavalle has joined #opendev18:54
openstackgerritMerged opendev/gerritbot master: Add option to disable daemonization  https://review.opendev.org/74524019:27
*** hashar has joined #opendev19:31
openstackgerritMerged zuul/zuul-jobs master: add-build-sshkey: call cmd with command  https://review.opendev.org/74564619:34
clarkbfungi: apparnetly my glob hack with Dockerfile does not work19:56
clarkbthe internets lied to me ! :)19:56
clarkbI guess we may need to split the dockerfile into 219:56
clarkbcorvus: ^ do you have any better ideas for that?19:56
fungiit's less makefile-like than i would have hoped20:02
corvusclarkb: i'm guessing it's failing on no files; so it would work for >=1 file20:03
corvusclarkb: could put a readme in there :)20:03
corvusclarkb: if you can put that last, you could make a multi-stage build and set the target to that on the 2.x builds20:05
clarkbcorvus: oh thats a neat hack20:05
clarkbya I think it can go last20:05
clarkbfwiw I've written the split too and its not too bad, though having one would be nice I think20:06
corvusclarkb: also, fyi, i think we could use the bazel-build role from zuul jobs, but not if we modify it for javamelody20:06
corvusclarkb: but maybe we just keep this one around until 3.x, then drop it and use zuul-jobs20:06
clarkbya 3.x seems to be a lot cleaner build wise20:07
*** hashar has quit IRC20:07
clarkbhrm not sure how the target stuff would work, it will bother me that 3.x is uploaded from a different baser target than 2.x20:09
*** hashar has joined #opendev20:09
clarkbmeh maybe thats ok20:09
openstackgerritLogan V proposed openstack/project-config master: Disable limestone provider  https://review.opendev.org/74573320:09
openstackgerritClark Boylan proposed opendev/system-config master: Gerrit image cleanups/fixes  https://review.opendev.org/74559520:17
clarkbok I think ^ should do it20:17
fungilogan-: i've gone ahead and approved that ^ but be aware that nodepool is going to keep uploading images to glance. if that's not desirable we can also adjust that20:17
fungialso our ansible is going to continue trying to manage configuration for the mirror instance hosted there20:18
fungiwhich we can also disable if needed20:18
clarkbI'm approving the gerritbot on eavesdrop change now20:18
clarkbI'll disable gerritbot on review.o.o once it looks like deploy will get close after merging20:18
fungilogan-: similarly we have automation checking/updating ssh keys and security groups20:18
logan-fungi: Thanks, it _should_ be OK, we'll have to reboot the mirror and probably cold-migrate it to a new host once we start upgrading, but it would only be a few minutes of outage for that. API access might be spotty for image uploads and other operations. No issue from my end leaving that turned on.. my only concern is if there's potential for API inaccessibility to cause grief in opendev's ansible / nodepool instances.20:20
clarkblogan-: no it should be fine, we'll error and then try again until we stop erroring20:22
fungilogan-: yeah, that's all fine. i was more thinking like if the entire api is going to be offline for weeks20:22
logan-Ok sounds good. I'm not expecting anything like that but if something comes up where we're looking at extended downtime I'll give y'all a heads up!20:24
fungithanks again, we really appreciate all the help!20:24
clarkbhow does this look #status notice Gerritbot will be offline for a short period while we redeploy it on a new server20:24
clarkbI'll send that when I stop the review.o.o service?20:24
fungisure, can't hurt20:26
fungior maybe "the openstackgerrit irc bot (gerritbot) will be..."20:27
fungijust in hopes folks don't misread and think it's gerrit going down20:27
clarkbnew rev: #status notice The openstackgerrit IRC bot (gerritbot) will be offline for a short period wile we redeploy it on a new server20:28
clarkbzuul says we are about half an hour from the change merging then we need infra-prod-service-eavesdrop to run20:29
openstackgerritMerged openstack/project-config master: Disable limestone provider  https://review.opendev.org/74573320:32
openstackgerritMerged opendev/system-config master: Add ansible role to manage gerritbot  https://review.opendev.org/74479520:50
clarkb#status notice The openstackgerrit IRC bot (gerritbot) will be offline for a short period while we redeploy it on a new server20:50
openstackstatusclarkb: sending notice20:50
-openstackstatus- NOTICE: The openstackgerrit IRC bot (gerritbot) will be offline for a short period while we redeploy it on a new server20:51
*** openstackgerrit has quit IRC20:52
* fungi watches as his weechat channel activity bar lights up from all the statusbot notices20:52
clarkbinfra-prod-service-eavesdrop is queued20:52
clarkbnow we wait20:52
openstackstatusclarkb: finished sending notice20:54
clarkbthe process seems to be running. We got a warning about review's host key but that doesn't seem to be fatal as we are getting gerrit events20:58
clarkbbut so far nothing that would produce an irc message?20:58
clarkbanyone have a change they want to update?20:58
* fungi checks21:00
*** hashar has quit IRC21:00
clarkbhrm according to the logs it should've sent stuff to #openstack-neutron but not seeing that21:01
clarkbwe are currently logging at INFO level and not DEBUG because DEBUG was super verbose but we can switch to DEBUG and restart again and see if that gives any more info (not see any reason for lack of logging to #openstack-neutron)21:02
clarkbalso we're double logging that should be an easy fix21:03
fungiclarkb: i see one in #openstack-tripleo!21:04
fungier, #tripleo21:05
clarkbok I see that one logged too21:05
clarkbam I perhaps netsplit from the bot?21:05
fungimeetbot logged it at least21:05
clarkbya so its somewhat working21:06
fungialso the join message said "openstackgerrit (~openstack@eavesdrop01.openstack.org)" so coming from the correct server21:08
funginot accidentally restarted on review0121:08
smcginnisIs this work relevant to the #openstack-stable changes I'm trying to do in https://review.opendev.org/#/c/744947/21:09
clarkbfungi: http://eavesdrop.openstack.org/irclogs/%23airshipit/%23airshipit.2020-08-11.log.html#t2020-08-11T20:59:18-2 that one worked too. That was the first one then tripleo was second. I'm wondering if it hit a problem after those and then broke21:10
clarkbbut skimming the logs I don't see that happening21:10
clarkbsmcginnis: yes, this should get us back to auto loading new gerritbot configs21:10
clarkbfungi: ya the first 3 messages sent all show up on eavesdrop meetbot channel logs21:11
clarkband I see the airshipit events in my client so wasn't split at athat point (so probably not split at all)21:11
clarkbthere was an newer airshipit event a few minutes ago that I don't see so not channel specific21:12
clarkbah here we go server not connected errors now21:13
clarkbfungi: I'm going to switch us to debug logging and restart21:14
clarkbtripleo, ansible and stable should've all just gotten events21:20
fungii just pushed a pbr change too, which would have notified #openstack-infra and #openstack-oslo (at least)21:21
clarkbI see the three I called out21:22
clarkbnow looking at logs for why your pbr chang emay have failed21:22
fungiremote:   https://review.opendev.org/744720 Add Release Notes to documentation21:24
clarkbya it saw the event from gerrit21:25
clarkband says it sent things to oslo and friends21:25
clarkbnetstat shows we're connected to card.freenode.net21:25
clarkbits like the tcp connection is dying after it works for a bit and it doesn't seem to know why21:26
clarkbI'm hoping that with debug logging we'll get more info once it notices its current connection is dead21:27
clarkbwould a failed ping pong cause the server to kill our tcp connection? I wonder if it is protocol level like that21:29
clarkbreading the code we seem fairly defensive to this sort of thing, its just not throwing exceptions ?21:34
*** DSpider has quit IRC21:34
clarkbthis feels like a: we updated python irc lib and python2 to python3 sort of problem21:38
clarkblike its going to be a byte vs string or synchronization type problem21:38
clarkbok it finally errored and there is no additional debugging putput21:40
clarkbhttps://github.com/jaraco/irc/issues/171 ya that looks suspicious21:43
clarkbwe only handle output ourselves though but maybe that breaks the sending of PONG?21:44
clarkbit looks like it should handle that via reactor.process_forever within the irc client for the bot21:56
clarkbhowever I've set logging to debug level and I don't see the debug message for process_forever so I'm not convinced yet it is working properly21:56
clarkbI also don't see the I've connected debug message so that may just be a logging config problem21:56
fungisorry, got told to put the computer away during dinner, back now ;)21:57
fungiso the problem could be that gerritbot is effectively blocking the (only) thread?21:59
fungiand so the pong handler never gets a chance to wake up?21:59
clarkbthat is what it looks like21:59
clarkbsince we work until we'd hit a ping timeout21:59
clarkbthough the logging doesn't yet point to that directly21:59
fungidebug level logging doesn't log the ctcp ping/pong exchange though, i guess22:00
fungiwhich either means it's not happening or it's just not logged22:00
clarkbya and there are log.debug calls in the irc bot code22:02
fungibut yeah, after around 5 minutes we're seeing "-- openstackgerrit (~openstack@eavesdrop01.openstack.org) has quit (Ping22:02
fungitimeout: 256 seconds)22:02
clarkbsetting root logger level to NOTSET means all messages will be logged /me tries this22:05
clarkbok got it to log irc client things by dropping our logging config22:11
clarkbnow we should see what is going on22:11
clarkbAttributeError: module 'irc.client' has no attribute 'VERSION_STRING'22:12
clarkbhttps://github.com/jaraco/irc/blob/master/irc/bot.py#L310-L315 is hitting that22:13
clarkbhttps://github.com/jaraco/irc/blob/313620f625c1e8562447c2cbf0cc021c73470dc3/CHANGES.rst#v1900 there is testing https://github.com/jaraco/irc/blob/dd68925ba224d4b9c623ca6dc7b9033cd9d5cc2d/irc/tests/test_client.py#L10 and yet :/22:14
fungidoes it want to call this instead? https://github.com/jaraco/irc/blob/master/irc/bot.py#L310-L31522:15
fungier, sorry, https://github.com/jaraco/irc/blob/master/irc/client.py#L645-L64722:15
clarkbno I think we need to do the importlib lookup as described in that change22:16
clarkbbut ideally in the irc bot?22:16
fungiahh, you already found that22:18
fungii guess this calls for a pr22:18
clarkbremote:   https://review.opendev.org/745755 Pin to irc==18.0.022:19
clarkbfungi: I think we can start at ^22:19
clarkbthen ya do a PR then unpin22:19
clarkbI'm running out of energy after the early start today so https://review.opendev.org/745755 seems like a reaosnable place to start. I'll stop gerritbot on eavesdrop now and start it on review.o.o as we seem to grok the problem now22:20
fungiwhee! https://github.com/jaraco/irc/issues/17422:20
clarkbI did docker-compose down on eavesdrop which should prevent that container from starting after a reboot and systemctl started gerritbot on review.o.o22:22
clarkbwe should be functional again22:22
clarkband https://review.opendev.org/745755 should allow us to flip them back over a second time :)22:22
*** qchris has quit IRC22:22
* clarkb takes a break22:24
fungii single-core approved it a few minutes ago since we at least know that's a problem (though whether it's also hiding other problems, no clue of course)22:24
*** mlavalle has quit IRC22:30
*** qchris has joined #opendev22:35
*** openstackgerrit has joined #opendev22:48
openstackgerritMerged opendev/gerritbot master: Pin to irc==18.0.0  https://review.opendev.org/74575522:48
*** tkajinam has joined #opendev22:54
fungiclarkb: i've got a patch passing their unit tests and style checks, can push a pr unless you've already got one in progress22:56
clarkbfungi: I do not. Go for ut22:57
fungicool, will have it up in a minute or two and mark it as closing #17422:58
clarkbfungi: should I give swapping the gerritbot hosts another try now that the irc pin has merged?23:03
fungiif you're up for it, sure23:03
clarkbok doing that now23:03
*** openstackgerrit has quit IRC23:04
clarkbit is running on eavesdrop again23:04
*** logan- has quit IRC23:05
clarkband in ten minutes or so we push a change and check that it works?23:06
clarkbit just posted to #tripleo successfully23:07
clarkbfungi: smcginnis I think there is one more piece that we need for picking up config changes and that is running service-eavesdrop on changes to the gerritbot config23:10
clarkbI won't get to that today and am out tomorrow but can pick that up as well as deduping gerritbot logs thursday if no on beats me to it23:11
clarkb3 minutes ago we sent events to #tripleo23:19
clarkbwhich is well past the previous "timeout"23:19
clarkbI think we are likely good23:19
clarkb#status log Moved gerritbot from review.openstack.org to eavesdrop.openstack.org. Cleanup on old server needs to be done and we need to have project-config run infra-prod-service-eavesdrop when the gerritbot config updates.23:20
openstackstatusclarkb: finished logging23:20
fungiand after the usual 20 minutes reminding myself how to use teh githubz, https://github.com/jaraco/irc/pull/17523:20
clarkbfungi: it runs automated tests23:21
clarkbsomething tells me they will fail :)23:21
clarkbyup first one just failed23:22
fungiahh, yeah, tox was working for me on my default python 3.823:22
fungiolder pythons don't have importlib.metadata23:22
clarkbfungi: well I also expect it will fail otherwise why would they release with broken tests23:23
clarkb(its probably normal for them to be broken)23:23
fungialso looking more closely at 313620f i suspect the real reason for that commit was to remove calls to importlib23:24
fungiso all i'm doing is reintroducing them23:25
fungii think the one place we're hitting this is probably the only one that matters, that's where it responds to `ctcp version` from the server23:32
fungithe other uses are a welcome banner (cosmetic) and unit testing (unnecessary now)23:32
clarkbit continues to send events. Really think we are set now23:33
ianwSTARTTLS failed! SSL connect attempt failed error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed at /usr/libexec/git-core/git-send-email line 1548.23:56
ianwand people say dealing with gerrit is too hard ...23:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!