Thursday, 2020-08-27

ianwclarkb / corvus : fyi ... we're in a pretty good position generating the wheels I think01:14
*** sgw has quit IRC01:14
clarkbianw: that means the elfpatch thing was sorted out?01:14
ianwclarkb: not quite but i think the cryptography images are carrying a patch01:15
*** sgw has joined #opendev01:33
*** xiaolin has joined #opendev01:59
*** ysandeep|away is now known as ysandeep02:39
ianwError from server (Forbidden): error when creating "test-pod.yaml": pods "test" is forbidden: error looking up service account default/default: serviceaccount "default" not found03:37
ianwhrm, this is in the zuul-jobs kubernets something something test :/03:37
ianwkevinz: not sure if you've seen
ianwkevinz: in short, if you start a container on a guest and try to access pypi/pythonhosted via ssl it just hangs03:38
ianwkevinz: basically fastly CDN.  turning down the MTU makes it work03:40
ianwit's got to be a container, and i guess behind the default NAT networking or whatever docker sets up03:40
ianwwe did it! :)04:47
*** DSpider has joined #opendev04:51
*** bhagyashris|away is now known as bhagyashris05:02
openstackgerritOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml
*** ysandeep is now known as ysandeep|afk06:47
*** yoctozepto has quit IRC07:34
*** dtantsur|afk is now known as dtantsur07:40
*** tosky has joined #opendev07:47
*** ysandeep|afk is now known as ysandeep07:55
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
openstackgerritMerged zuul/zuul-jobs master: bindep: Fixed runtime warnings
openstackgerritMark Goddard proposed openstack/project-config master: kolla-cli: deprecation - retiring master branch
openstackgerritStephen Finucane proposed openstack/project-config master: Add Backport-Candidate label for nova deliverables
openstackgerritStephen Finucane proposed openstack/project-config master: Add Backport-Candidate label for placement deliverables
*** yoctozepto has joined #opendev08:55
*** andrewbonney has joined #opendev08:59
openstackgerritStephen Finucane proposed openstack/project-config master: Drop +2/-2 Backport-Candidate label values for oslo
openstackgerritStephen Finucane proposed openstack/project-config master: Consistently set the stable ref permissions
*** hashar has joined #opendev09:45
openstackgerritMichal Nasiadka proposed openstack/diskimage-builder master: Add cloud-init-disable-growpart
openstackgerritMichal Nasiadka proposed openstack/diskimage-builder master: Fix grubenv link in latest grub2 CentOS packages (EFI)
*** sshnaidm|afk is now known as sshnaidm10:30
*** jaicaa has quit IRC10:41
*** jaicaa has joined #opendev10:44
*** hashar is now known as hasharLunch10:48
*** ysandeep is now known as ysandeep|afk11:13
*** ysandeep|afk is now known as ysandeep11:34
*** xiaolin has quit IRC11:37
*** hasharLunch is now known as hashar12:01
*** ysandeep is now known as ysandeep|brb12:50
*** ysandeep|brb is now known as ysandeep13:02
openstackgerritThierry Carrez proposed opendev/system-config master: Explain "why opendev" on index page
*** ysandeep is now known as ysandeep|away14:34
AJaegermnaser: please review as this is vexxhost related14:52
clarkbianw: !! success14:52
*** qchris has quit IRC14:57
AJaegerconfig-core, please review
*** andrewbonney has quit IRC14:59
openstackgerritMerged openstack/project-config master: Normalize projects.yaml
*** qchris has joined #opendev15:10
openstackgerritClark Boylan proposed opendev/base-jobs master: Set file modes explicitly
clarkbnext up zuul-jobs though that is quite a bit more involved16:06
*** ysandeep|away is now known as ysandeep16:14
openstackgerritMerged openstack/project-config master: Remove legacy-tempest-dsvm-neutron-{ipv6only,serviceipv6}
*** dtantsur is now known as dtantsur|afk16:23
*** sshnaidm is now known as sshnaidm|afk16:24
openstackgerritClark Boylan proposed opendev/base-jobs master: Set file modes explicitly
openstackgerritClark Boylan proposed zuul/zuul-jobs master: WIP: Address ansible-lint E208
*** tosky has quit IRC16:45
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Partial address ansible-lint E208
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes
*** kevinz has quit IRC17:58
*** tosky has joined #opendev18:35
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes
clarkbcorvus: if you have a moment zbr asked for rereview on that change. I think the logging at least will help us identify when that happens and help with debugging. If we are happy with it I can include it in the gerritlib releaes needed for jeepyb branch handling20:52
clarkbif not I'll make the release without it20:52
zbrclarkb: in fact event == select.POLLIN is a clear but, because that is a bitmas20:59
zbrevent should never be checked with ==20:59
corvuszbr: i continue to disagree with you on that as i wrote in the comment on the change.  but i'm not going to argue any more.  i've left a +1 on the change to indicate i'm not opposed, but i'm not going to +w it since i don't have time to deal with potential fallout21:21
corvusclarkb: ^21:21
clarkbya I expect its going to be something with OSX's tcp stack and understanding how to make it happy woul dbe a good thing21:22
clarkbbut I don't have an OSX setup to test on so adding logging seems like a reasonable intermediate step21:23
*** hashar has quit IRC21:23
corvusagreed; and if we accept "osx sends a pollpri that's safe to ignore for some reason no one understands" as a true statement, that change lgtm.  my lack of +w is due to the small chance something in that stamement is wrong, or merging and deploying this reveals some new issue.  just being extra conservative due to limited time.21:25
clarkbwfm thanks for looking21:25
* JayF is on an OSX machine and is happy to do a thing if it's helpful21:28
clarkbJayF: basically gerritlib's ssh connections get POLLPRI events back at them when running on pyton3.8 on osX21:35
clarkbJayF: we don't know if there is an important signal in that or not21:35
clarkbJayF: adds logging  you could test and see if the log show up if connecting to's event stream21:35
JayFwell, if you have a thing and need it run with debug logging on OSX I'm happy to. I have nearly-zero low level OSX knowledge21:35
JayFI'm assuming the test case would be like a 10 line python script using  that code, for someone who knew gerritlib?21:36
clarkbJayF: ya it would basically be pip install that checkout, then import gerritlib ; g = gerritlib.Gerrit('', 'yourusername') ; g.startWatching()21:38
JayFk, give me one sec21:38
JayFinstalling 3.8.1 via pyenv to test21:39
JayFhmm no Gerrit in gerritlib21:43
* JayF looks21:43
JayFaha seems to be gerritlib.gerrit.Gerrit, perhaps21:44
JayF...or not21:44
JayFimport gerritlib.gerrit; gerrit.Gerrit21:45
clarkbaha sorry21:46
JayFthat was... underwhelming21:46
JayFgist incoming21:46
JayFperhaps I'm not popular enough on gerrit? lol21:46
clarkbJayF: its because the startWatching() call starts a daemon thread that fills a queue objet21:47
clarkbyou can add a while True: g.getEvent(); to block and pull those off21:47
JayFso I probably need to consume that queue and print things, or put in a busy loop21:47
JayFI'm getting borderline-spammed with  `POLLPRI event 3 received, see`21:48
clarkbJayF: cool are you getting events too?21:49
clarkb(I'm curious if the underlying event stream manages to function)21:49
JayFI'm looking now, restarted it with the getEvent wrapped in a print21:49
JayFlooks like it tbh21:49
JayFbut only  one so far21:49
clarkbwill depend on people doing gerrit things21:49
clarkbI guess its good the behavior is reproduceable21:50
JayFget to work, $people! :D21:50
clarkbfor debugging it I guess we need to write some code to read the side channel data21:50
JayFmy hunch is it's going to print those for every event it gets21:50
JayFyou got a simple PR for me to +1 to create an event? lol21:50
JayFoh, there goes another21:50
JayFlooks like I'm getting that debug log followed by the event pretty reliably21:51
JayFwhere datapoints=221:51
JayFalthough TBF I wouldn't know if I was missing events21:51
JayFI have certainly had zero cases of *getting* an event without also getting that debug log21:51
JayFoh wow, there it goes, yep, can confirm, lots of data points21:52
clarkbsomething probably merged21:52
clarkbcreates a lot of events21:52
clarkbon linux that should mean there is out of band data21:53
clarkbon osx I wonder if it can mean something else21:53
clarkbare osx man pages a thing that exist and if so are they on the internets?21:53
JayFpasted those  comments into the pr for you21:53
JayFwhat do you want, a man 2 poll or something?21:53
clarkbcurious what it says about POLLPRI21:54
JayF     POLLPRI        High priority data may be read without blocking.21:54
clarkbwow so does that mean we'll basically always get it on osx?21:54
JayFgoing to gist the whole manpage for you21:54
clarkbJayF: I found an iphone manpage thatsays that too21:55
JayFadded as a comment here
clarkbthe internet says if you get a pollpri then pollin will also be set21:58
JayFI'll keep that venv alive in case you all have something else you want me to run on it later21:58
clarkbrandom internet person says pollpri does not mean out of band data on osx
fungiit's not just osx/darwin, seems to maybe be a general bsdism?22:01
clarkbI wonder if osx is identifying the traffic as ssh somehow (ports or whatever) and deciding it should all be priority because it may be an interactive session?22:02
fungithough that manpage does go on to also say that "...on OpenBSD, the POLLPRI and POLLRDBAND flags may be used to detect when out-of-band socket data may be read without blocking."22:03
clarkbcorvus: going back to your original comment concerned about getting pollpri and pollin together apparently that is normal22:04
clarkbmaybe not necessarily always the case though22:05
clarkb(which would be important to understand if were gonna change that loop I think)22:05
clarkbya so POLLIN means there is any data to read22:07
JayFPOLLPRI means there's high-priority data to read?22:07
clarkbya so POLLIN should aways be set if POLLPRI is set I think22:07
clarkbso maybe it is fine to do & POLLIN|POLLPRI22:08
clarkbwe wouldn't read the priority data first but we'd get to it whatever it may be on osx22:08
clarkbhowever I think POLLPRI on linux isn't necessarily something that means there is regular stream data to read22:09
clarkbwhich makes me think we need to differentiate there to properly handle things22:09
clarkbactually what if we checked if bitmask == POLLIN or bitmask == POLLPRI & POLLIN22:10
*** mtreinish has quit IRC22:10
clarkbthen we'd be checking that there is data to read and could read (and ignore the out of band on linux and whatever it means on osx)22:10
clarkbthe problem with == POLLIN was that we were skipping events on osx because of the extra mask but the problem with an | is that it could be either or (also on that last bitmask == check it should be a | not a &)22:11
* clarkb is trying to compose thoughts and will leave a response on the change22:13
*** mtreinish has joined #opendev22:16
clarkbJayF: if you're still able to help can you do a packet capture of that traffic and see if the URG flag is set22:25
clarkbI wonder if on OSX that is what it means but on linux its like meh you lose22:25
clarkbJayF: that should all be in the tcp framing outside of the encrypted bits so you'll see it just doing a naive capture22:26
clarkbJayF: and we would expect that on the packets from gerrit to you22:26
JayFhmm, I don't have all the tools for that installed22:27
JayFlet me see how tough it is22:27
JayFand disconnect from vpn lol22:27
clarkbwireshark will do it and I think is cross platform22:28
clarkbosx likely has a tcpdump too22:28
JayFyeah, just gotta brew install wireshark and disconnnect from VPN so I don't set off klaxons at Verizon SOC :D22:28
clarkbnow I'm going to sanity check that my local python3.8 can stream events on linux without POLLPRI22:29
clarkb(it may also be a python3.8 thing?)22:29
JayFI'm surprised it's not a "what headers python compiles  against" thing22:30
JayFhomebrew appears to be downloading approximately half of hte internet22:30
clarkbI'm sorry :(22:31
JayFeh, I don't mind22:31
clarkbits not really that important right this instant if you hav ebetter things to do22:31
JayFjust annoyed  when things don't install instantly22:32
JayFI have unreasonable expectations22:32
clarkbconfirmed the old code seems to work on linux with python3.8 so it likely an osx thing not a python version thing22:33
JayFthis all you needed?22:34
JayF15:33:43.490580 IP > Flags [P.], seq 3648:3920, ack 1, win 33, options [nop,nop,TS val 1138593608 ecr 651404337], length 27222:34
JayF15:33:43.490728 IP > Flags [.], ack 3920, win 2043, options [nop,nop,TS val 651404840 ecr 1138593608], length 022:34
JayFI don't know if URG is a tcp flag or not, doubtful because it didn't sound familiar22:34
JayFbut my network-foo is extremely dusty22:34
corvusi hope there's a tcp flag called 'urg'22:34
clarkbok commnt left on the change with my new thoughts22:35
JayFclarkb: was that what you neede?22:36
JayFI have one loaded up in wireshark now too, but don't see much URG-y there22:36
clarkbya URG is a flag along with SYN ACK FIN etc22:36
JayF> The URG flag is used to inform a receiving station that certain data within a segment is urgent and should be prioritized.22:36
clarkbis the P flag there for PSH?22:36
clarkbI wonder if that is it22:36
JayFUrgent - not set22:36
clarkbFlags [P.] <- it may be that osx bubbles that up as POLLPRI?22:37
JayFthat makes a lot of sense22:37
JayFpush is like, tcp_nodelay22:37
JayFand reading that as POLLPRI would not seem crazy to me22:37
JayFThe sending application informs TCP that data should be sent immediately.22:37
JayFThe PSH flag in the TCP header informs the receiving host that the data should be pushed up to the receiving application immediately.22:37
clarkbJayF: do you think youcan add that info to the change (your capture bits and the theory that osx exposes psh that way)22:38
clarkbnot yet sure what that means from a client perspective but I bet that is it22:39
clarkband I'm being pushed out the door to do a family walk. Back in a bit. Thanks again for helping debug that on osx22:39
JayFI'll most likely be a pumpkin when you return :) glad to help22:40
JayFI know these issues will bounce around with people guessing about behavior22:40
JayFwhen I can provide real data, I'm happy to22:40
clarkbI think the next step is to do a similar packet capture on linux (whichI can do) and see if the PSH flag is set there too22:40
clarkband if so we'll need to handle the poll events differently for different platforms potentially22:41
JayFoh that'd be nuts22:41
JayFif it's an ssh client difference22:41
JayFwell, no, it'd have to be a server difference, right? if it's a tcp flag?22:41
clarkbya its coming from the server but it may be acting differently based on the cleint (I really expect its sending PSH in all cases and linux just ignores it for being silly and osx does not)22:42
clarkbif it sends PSH in all cases I think linux should keep the existing behavior to fail if there is out of band data22:42
clarkband osx can mask POLLPRI in too22:42
JayFwhat is "out of band" data?22:42
JayFno, nevermind22:42
JayFgo walk22:42
clarkbJayF: its what you get when URG arrives22:42
clarkbURG implies PSH but PSH does not imply URG22:43
clarkbI think that is why linux is acting different here22:43
clarkband ya popping out now. Back in a bit22:43
*** DSpider has quit IRC23:15
clarkbok just tested and the PSH happen on linux too23:18
clarkbbut they start with the local side23:18
clarkbI wonder if that is a paramiko thing at the end of the day but then linux and osx expose it differently23:18
*** tosky has quit IRC23:59

Generated by 2.17.2 by Marius Gedminas - find it at!