Wednesday, 2020-06-03

*** gtema has joined #opendev00:10
*** gtema has quit IRC00:14
*** DSpider has quit IRC00:16
openstackgerritmelissaml proposed openstack/diskimage-builder master: Switch to newer openstackdocstheme and reno versions
*** Meiyan has joined #opendev00:50
*** _mlavalle_1 has joined #opendev01:07
*** mlavalle has quit IRC01:10
*** ysandeep|away is now known as ysandeep01:38
openstackgerritIan Wienand proposed opendev/system-config master: [wip] arm64 role tests
*** xiaolin has joined #opendev02:11
*** gtema has joined #opendev02:11
*** gtema has quit IRC02:16
ianwso something interesting, with_first_found switching on architecture doesn't work, because the lookup is on the local host, not the remote host02:29
ianwi.e. our openafs roles work more by luck than design02:30
*** sshnaidm has quit IRC02:34
openstackgerritIan Wienand proposed opendev/system-config master: [wip] arm64 role tests
*** diablo_rojo has quit IRC02:37
openstackgerritIan Wienand proposed opendev/system-config master: openafs-client: arch fixes
*** xiaolin has quit IRC02:41
*** xiaolin has joined #opendev03:17
*** sshnaidm has joined #opendev03:33
*** diablo_rojo has joined #opendev03:51
mordredianw: yay for luck?04:02
mordredianw: oh - jeez re: architecture04:03
*** gtema has joined #opendev04:12
*** gtema has quit IRC04:17
*** ykarel|away is now known as ykarel04:31
*** sshnaidm_ has joined #opendev04:43
*** sshnaidm has quit IRC04:45
diablo_rojo#cinder clean04:49
diablo_rojo#glance clean04:49
diablo_rojo#monasca clean04:49
mordreddiablo_rojo: I feel like you're cleaning04:51
diablo_rojomordred, I am.04:51
diablo_rojoMy OCD does not like all the 'end of day see you tomorrow' stuff04:51
mordreddiablo_rojo: days end?04:52
diablo_rojo..except its not cleaning..04:52
diablo_rojofor certain teams yes..04:53
diablo_rojoWrong channel would do it04:53
* diablo_rojo facepalms04:53
diablo_rojomordred, my bad ..04:55
diablo_rojoIn my defense I only napped for an hour today and I was awake before 7 AM.04:56
fungiyep, i was about to say...04:56
fungii napped far longer, but not well04:56
fungiat least i have 7 hours between this session and the next04:56
ianwrax cleanup script
diablo_rojofungi,  yeah my hour wasnt awesome04:57
diablo_rojoAfter I am done on the help desk I will get to sleep..04:58
* diablo_rojo maths04:58
diablo_rojoAbout 6 hours?04:58
*** sgw has quit IRC05:01
*** _mlavalle_1 has quit IRC05:09
*** _mlavalle_1 has joined #opendev05:09
openstackgerritIan Wienand proposed openstack/project-config master: Remove periodic openafs jobs
clarkbfwiw monasca and neutron look like they may have been big meetpad users based on logs05:47
clarkbthey may be good sources of feedback on how to make clients happy and what could be improved05:47
clarkbdiablo_rojo: I told larissa that yesterday it felt like I had jetlag even though I only had to fly to my office across the hallway05:48
clarkbtoday was much easier for me thankfully05:48
diablo_rojoclarkb, lol did you have early meetings? or stay up late?05:49
clarkbya I was up super early yesterday then between meetings and kids being crazy beacuse they knew it was a day before their birthday it was a tiring day05:50
diablo_rojoOh yeah, how was their birthday?05:51
clarkbit was quiet. I think welearned some things about socially dostant kid birthdays05:53
diablo_rojoWhat did you do?05:55
clarkbmade cupcakes and put together lego friends turtle rescue05:56
*** Meiyan has quit IRC06:08
*** _mlavalle_1 has quit IRC06:14
*** _mlavalle_1 has joined #opendev06:15
*** gtema has joined #opendev06:15
*** dtantsur|afk is now known as dtantsur06:43
*** tkajinam has quit IRC06:56
*** tkajinam has joined #opendev06:56
*** bolg has joined #opendev06:57
*** gtema has quit IRC07:18
*** tosky has joined #opendev07:24
*** ravsingh has joined #opendev07:34
AJaegerianw: do we build dib images with python3? See - this would break if we use python207:38
*** ravsingh has quit IRC07:44
*** gtema has joined #opendev07:48
*** olaph has quit IRC07:53
*** DSpider has joined #opendev07:55
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** diablo_rojo has quit IRC08:07
*** ravsingh has joined #opendev08:17
fricklerAJaeger: seems python is python 3.8.2 in the nodepool-builder container, just checked that on nb01, so this should be fine08:24
*** redrobot has quit IRC08:35
*** _mlavalle_1 has quit IRC08:38
*** _mlavalle_1 has joined #opendev08:39
*** ysandeep is now known as ysandeep|lunch08:39
*** roman_g has joined #opendev08:41
*** tkajinam has quit IRC08:53
*** tkajinam has joined #opendev08:54
*** tkajinam has quit IRC09:09
AJaegerthanks, frickler09:17
ianwAJaeger: yeah, we should, and we just dropped py2 support in dib09:26
*** ysandeep|lunch is now known as ysandeep09:33
*** gtema has quit IRC10:02
*** _mlavalle_1 has quit IRC10:27
*** gtema has joined #opendev10:37
*** gtema has quit IRC10:43
*** gtema has joined #opendev10:56
*** priteau has joined #opendev10:59
*** hashar has joined #opendev11:26
*** rpittau|afk is now known as rpittau11:57
*** sgw has joined #opendev12:01
*** ysandeep is now known as ysandeep|afk12:06
*** hashar has quit IRC12:24
*** hashar has joined #opendev12:24
*** ysandeep|afk is now known as ysandeep12:55
*** tkajinam has joined #opendev12:57
*** mlavalle has joined #opendev13:02
openstackgerritMerged zuul/zuul-jobs master: Add the namespace in wait-for-pods role
*** gtema has quit IRC13:14
*** gtema has joined #opendev13:17
*** Guest27280 has joined #opendev13:22
*** Guest27280 is now known as redrobot13:25
*** gtema has quit IRC13:26
*** gtema has joined #opendev13:26
*** lpetrut has joined #opendev14:10
*** diablo_rojo has joined #opendev14:11
corvusclarkb: did you use the native jitsi-meet android app with meetpad?  did it open the etherpad on start?14:23
*** lpetrut has quit IRC14:30
*** hashar has quit IRC14:35
*** hashar has joined #opendev14:35
openstackgerritThierry Carrez proposed opendev/system-config master: Start meetings with video muted
clarkbcorvus: I havent but I think ttx does/did14:55
clarkbhe suggested the mobile vlients after successfully using them with meetpaf14:55
corvusi just tried it, and it's a bit different -- the etherpad takes up the whole screen15:01
corvusclarkb, ttx: was trying to think about the appropriate response to the concern here:  -- what do you think?15:01
clarkbcorvus: I think status quo on mobile may be a good idea given that description15:04
clarkbthe doc is still there but you can manage the call without being stuck (or feeling that way) in an etherpad15:04
corvusyeah; maybe i'll just update it with a doc comment indicating the behavior difference15:05
*** mlavalle has quit IRC15:07
*** gtema has quit IRC15:15
*** mlavalle has joined #opendev15:20
*** gtema has joined #opendev15:21
*** gtema has quit IRC15:33
*** gtema has joined #opendev15:36
openstackgerritgugug proposed openstack/project-config master: Retire puppet-congress project - step 1 end project gating
*** gtema has quit IRC15:46
*** jaicaa has quit IRC15:50
fungiafter the fix merged to get our periodic jobs running again, got a new ssl cert, though seems not to have yet15:50
*** jaicaa has joined #opendev15:50
fungii'll take a closer look in a bit15:51
corvusmaybe in a few hours if the ptg board is clear of meetpad entries we could land ?  or do we just want to defer that until next week?15:53
fungii'm fine merging when there are folks around to test/troubleshoot15:55
clarkbya I think w ecan land that during the break between blocks15:55
fungii just didn't want to do it and then leave someone else to clean up the mess if something went unexpectedly wrong15:55
fungicorvus: also your opinion on 733159 would be good, maybe we want to put that in at the same time?15:56
corvusfungi: ++ on both15:57
*** gtema has joined #opendev15:59
fungifwiw, meetpad audio has been rock-solid for me once i realized that low bandwidth mode would turn off receiving video streams16:00
fungii just lack the cpu horsepower and/or hardware acceleration features necessary to handle video from more than a handful of participants. even on audio-only mode it runs around a load average of 316:00
fungias soon as i turn on video, even low definition, the system becomes starved enough for cpu that the audio starts cutting out16:01
fungiand it's not just a jitsi-meet problem, the html5 client for zoom is similarly resource-hungry for me16:02
toskyan old intel graphic card?16:06
*** ykarel is now known as ykarel|away16:07
*** dtantsur is now known as dtantsur|afk16:13
clarkbmlavalle: it looks like neutron may have been using meetpad yesterday. Is that the case? How many people did you have in those sessions? any other meetpad feedback we should try to incorporate?16:16
*** rpittau is now known as rpittau|afk16:24
*** gtema has quit IRC16:30
fungitosky: i don't know what counts as "old" but haswell (~2013) and cherrytrail (~2015) based systems with on-board intel video chipsets, yes16:31
fungii prefer to stick to 100% mainline kernel video drivers. i don't trust the tainted/proprietary out of tree binary blob drivers16:32
toskyI guess it also depends on the browsers blacklisting some cards16:32
*** ysandeep is now known as ysandeep|away16:33
*** hashar has quit IRC16:34
*** hashar has joined #opendev16:34
clarkbfungi: fwiw I don't think there are out of tree intel drivers16:39
fungiright, that's what i meant. i haven't been buying systems which needed out of tree video drivers16:39
fungiwhich is why i'm using the intel video controllers16:39
clarkbI've checked graphite's logs and it shows a cert renewal16:43
clarkbbut the cert served by its web server expires July 116:44
clarkbI think this is a case of not properly restarting servicesfor some reason16:44
openstackgerritPierre Riteau proposed openstack/project-config master: Add Backport-Candidate label for Kolla deliverables
fungiclarkb: yep, that's where i've gotten to as well16:47
fungichecking apache logs next16:47
openstackgerritPierre Riteau proposed openstack/project-config master: Add Backport-Candidate label for Kolla deliverables
clarkbfungi: looking at the service-letsencrypt logs I think we are running the touch file handler which is noopy for graphite16:48
clarkboh I remember16:48
clarkbwe touch the file which puppet is then supposed to key off of iirc16:49
fungiyeah, /var/log/apache2/error.log shows the last apache reconfig was at 06:25:02 during log rotation16:49
clarkbbut I think that we can probably just switch that to restarting apache16:49
fungibut the cert file wasn't updated until 06:3416:49
fungiso i agree, apache just hasn't done a reload yet16:50
openstackgerritClark Boylan proposed opendev/system-config master: Restart apache on graphite when LE updates certs
clarkbfungi: ^ I think maybe that is a fix? I can't see nayhting that is supposed to consume that stamp and restart apache (but I think that was one of our really early designs before we realized we could just have ansible do it even if puppet is managingthe apache configs)16:52
clarkbmaybe we can get ianw to double check 733247 later today and if there isn't something we are missing land that and manually restart apache16:54
*** gtema has joined #opendev17:04
*** gtema has quit IRC17:09
fungiclarkb: even if that change merges, we'll need an apache reload (graceful restart) to load the new cert before the next time the le role renews it17:12
fungiany objection to me doing a reload now?17:12
fungiit'll be after july 1 before another cert gets issued17:13
clarkbfungi: ya, my only concern us maybe something is supposed to use that stamp file that we touch and it is broken (I cant find where that is fwiw)17:14
clarkbdebugging that may be more difficult if we manually reload/restart17:14
fungifair, there's no hurry, we've got a month still17:14
openstackgerritPierre Riteau proposed openstack/project-config master: Propose Review-Difficulty label for Kayobe
clarkbdiablo_rojo: we are in the ptg break time now right?17:16
mlavalleclarkb: the Neutron's team experience with meetpad has been great. In fact, this morning we joined the edge team in their zoom room for a x-project session and I experienced problems with zoom that I haven't had with meetpad17:16
clarkbinfra-root I think we can go ahead and approve those two jitsi meet config changes now if we are in the break period17:16
mlavalleclarkb: the only problem mentioned in the session yesterday was that some team members were unable to see the integrated etherpad. I had no problems whatsoever, but other team members mentioned it17:17
clarkbmlavalle: that is great to hear. Do you have a rough idea for peak co current users?17:17
corvusclarkb: done17:17
mlavalleclarkb: the peak has been somewhere 15 to 2017:17
mlavallebut consistently we have at least 1017:17
diablo_rojoclarkb, yes we are17:17
fungimlavalle: awesome feedback, thanks!17:17
fungimlavalle: and yes, i also have trouble getting the shared document to remain visible. no idea why yet17:18
corvusmlavalle: in the menu on the bottom right, there's an option to "open shared document" which should toggle the etherpad17:18
corvusmlavalle: that might help get it to show up if it doesn't automatically17:18
mlavallecorvus: I am aware of that... but still some team members had trouble with the document17:18
corvusmlavalle: okay, good to know that sometimes there's still trouble even using that17:19
mlavallecorvus: I'll mention it tomorrow and see what feedback we get. we are done for today17:19
corvusmlavalle: thanks!17:19
fungibut like in my case, i have yet to find the necessary magic to get it to stay visible. it goes away again as soon as someone starts talking and the client wants to go back to displaying their "camera" but i'm out of troubleshooting ideas17:19
mlavalleI'll report back here tomorrow17:19
corvuswe're about to merge some "tweaks" that we hope might improve things for folks (but no direct fixes for that issue)17:19
mlavalleok, let's see if with those improvements the doc problem goes away17:20
*** ravsingh has quit IRC17:20
corvusmlavalle: sorry, i mean we don't expect the changes to affect the doc problem; they're all related to video stuff17:20
*** ysandeep|away is now known as ysandeep17:20
mlavalleah ok17:21
corvus(but hey, who knows? it's a complex system :)17:21
fungiso we're clear to merge those now? i can go ahead and approve them if so17:21
corvusfungi: i just did17:21
fungieven better, thanks!17:21
*** hashar has quit IRC17:22
*** hashar has joined #opendev17:23
openstackgerritPierre Riteau proposed openstack/project-config master: Propose Review-Difficulty label for Kayobe
*** hashar has quit IRC17:35
openstackgerritMerged opendev/system-config master: Start meetings with video muted
openstackgerritMerged opendev/system-config master: Indicate firefox is suboptimal in meetpad
clarkbonce those are applied I'll spin up a room on firefox and double check I get the warning17:53
clarkband I guess we want to confirm that we enter muted too17:54
corvusi'll get my laptop w/ camera ready17:54
clarkbbut then I'm long overdue for a bike ride so if jitsi is still happy I'll try to go out on one of those17:54
mordredcorvus, mlavalle: re: viewing etherpad, I've had both experiences - I've had moments where I can't get it to show me the etherpad, and moments where it all works as expected with no intervention. I'm *mostly* in the "it just works" camp, but a few times for reasons I don't yet know I joined the other camp17:57
corvusyeah, that's been my experience too17:57
corvusbut mostly the 'just works' camp recently17:57
mordredcorvus: I had a fun one this morning17:58
clarkbwhen it happens it seems to be triggered by jitsi trying to show the actively speaking person17:58
clarkbit seems that in some scenarios that overrides the "focus on shared doc" state17:58
mordreda couple fo times gtema disconnected, and every time he did I lost audio and had to reload the browser window17:58
mordreds/disconnected/was unexpectedly disconnected/17:58
corvusmordred: fascinating17:58
clarkbdeploy is done with both of those changes. I'll be hopping on with FF to check the mute state and the warning17:59
fungii'm outside trying to wrestle the lawn into submission, so no cam handy but can at least double-check dd17:59
fungier, ff17:59
clarkbFF warning worked I seem to be unmuted17:59
* clarkb tries chrome to see if mute state is set there17:59
corvusmuted video btw, not audio :)18:01
*** ysandeep is now known as ysandeep|away18:01
clarkbfor those watching here everything seems to work as expected18:02
fungiyeah, that's what i expected from the change (muted video, not audio)18:03
corvusin ff, i saw the etherpad-not-pinned behavior18:03
corvusbut in chrome, etherpad-pinned worked as expected18:03
fungiinteresting. i saw it not start pinned in chromium18:03
fungias well as in ff18:03
corvusi wonder if it's a race18:03
mordredI also did not start with a pinned etherpad18:04
mordredalthugh - the last meetpad I did before this one I had switched to viewing not-etherpad18:04
mordredI wonder if there is a cookie memory?18:04
corvusmaybe some saved state/cookie?18:04
mordredwell - I ujst re-joined the test meetpad and this time I had etherpad18:05
mordredso if it is saving state it's doing a really bad job of it18:05
fungicould be. i'll try blowing away my chromium profile in a bit and starting fresh. i've only been using it for meetpad so there's nothing to lose18:05
corvusi really want to find out how to watch the xmpp messages; i feel like that could be important to debugging this.18:05
mordredcorvus: tcpdump ;)18:06
* mordred does not want to watch xmpp messages via tcpdump18:06
corvusmore than that right now, i want to get exercise, so back after lunch18:06
clarkbme too re exercise.18:07
fungicorvus: mordred: also not sure if you've seen, but a couple of nova contributors in prc reported not being able to get audio working, so the nova team switched to zoom presuming meetpad is blocked by the gfwoc18:09
fungii have my doubts that's really the case, but would love to get to the bottom of it18:09
fungii suppose it's possible prc considers xmpp traffic "suspicious"18:11
openstackgerritGhanshyam Mann proposed opendev/irc-meetings master: Fix the policy popup team meeting id
mnasergetting 502s in gerrit18:32
fungimy default gerrit dashboard is loading fine for me... any specific urls a problem?18:35
fungii've pulled up some changes, no problem18:35
fungialso diff views are loading for me18:35
fungimnaser: did you get any error detail message with the 502?18:36
mordredmaybe those are the special 502s I configured just to mess with mnaser18:39
mnaseri just had it when i clicked on a change id18:39
mnaserit was the UI as an error message :X18:39
mnaseri mean, it went back to being ok after, but i figured it may have been the start of osmething18:39
fungiwill definitely keep an eye out, but sounds like it may have been an isolated incident18:42
fungikeep in mind the gerrit webui is a monolithic js app, which is making gerrit api calls, so odds are the api returned a 502 to the client on some operation(s)18:42
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: test-playbooks: improved syntax
*** hashar has joined #opendev19:05
*** hashar has quit IRC19:13
*** hashar has joined #opendev19:13
*** sshnaidm_ is now known as sshnaidm19:19
*** gtema has joined #opendev19:19
*** hashar_ has joined #opendev19:24
*** gtema has quit IRC19:24
*** hashar_ has quit IRC19:24
*** hashar has quit IRC19:25
*** hashar has joined #opendev19:25
corvusmordred: remind me -- we need this to be green before we can do the thing where we set up a gate-specific set of zuul connections, right?
fungiAJaeger: if you get a few minutes in the coming days, ildiko is trying to work out how to get content searches enabled for, starting with but there's presumably more to it than just that20:10
fungii'm guessing their docs repo likely needs to add the search page to the toc as well, but i'm really not sure yet20:11
AJaegerfungi: stephenfin did that work AFAIR, so better to ask him.20:11
fungiAJaeger: perfect, thanks!20:11
AJaegerfungi: but I'll check as well20:11
fungiit's not urgent, just figured you might have deeper understanding of sphinx's search bits20:11
fungideeper than mine that is20:12
AJaegerfungi: no, not really - there're some areas I avoid ;)20:12
*** priteau has quit IRC20:12
fungiyeah, i completely sympathize. dark magicks20:12
clarkbI'm trickling back in now. Anything ai should check once tacos are consumed?20:14
*** prometheanfire has quit IRC20:16
AJaegerfungi: left a review on the theme, just a shot in the dark ;)20:16
*** prometheanfire has joined #opendev20:16
fungiclarkb: nothing so important as tacos, no20:16
fungithanks AJaeger!20:17
AJaegerclarkb: want to review and (I think the latter is wrong but didn't find the time to dig deep enough) for wheel building, please?20:18
*** smcginnis has quit IRC20:18
*** smcginnis has joined #opendev20:19
mordredcorvus: yes20:23
mordredcorvus: I have *NO IDEA* what's wrong with the linter - I tried reproducing locally and can't20:25
mordredcorvus: and by linter here - I mean the yaml parser is doing something sort of bonghits20:25
corvusERROR: InvocationError for command /bin/bash -c 'find roles playbooks -type f -regex '"'"'.*.y[a]?ml'"'"' -print0 | xargs -t -n1 -0 ansible-lint' (exited with code 123)20:25
mordredcorvus: I'm lying to you20:25
mordredwe do not need that patch to work20:26
corvusis that really all the error message we get out of that?20:26
mordred(although I still don't understand why it's not woirking20:26
mordredcorvus: we got a different error in the previous run of the job20:26
mordredcorvus: we need for your thing20:26
mordredcorvus: 731258 is just simplifying something using the new ability20:26
corvusmordred: but isn't 258 the one that shows us that 991 really works? :)20:26
mordredyes - that's right - but I think 258 is showing it works, other than the linter derp20:27
corvusi mean, 991 dosen't break, but 258 is the first thing to use the facility, right?20:27
mordredthat's right20:27
corvusi agree we could proceed with the info we have.  though i would have to put this table i just flipped over back upright and then flip it again if i spend a bunch of time doing that only to see "exit code 123" from ansible-lint.20:27
mordredcorvus: that's the error I got on the _previous_ run20:28
mordredwhich makes less than zero sense20:28
corvusoh that's the same i think20:29
mordredcorvus: we're getting the same error in the 123 ...
corvusit's just in a slightly different order20:29
mordredoh - wait ...20:29
mordredI think the traceback is making slightly more sense ... it's complaining about an undefined varialbe in a really weird way I think?20:30
corvusweird; the only variables removed there are bridge_hostvar* and git grep shows no hits for that20:32
clarkbAJaeger: I'm not an rpm expert but I can definitely review the job stuff20:33
corvuscould it be that extra_users isn't available in some context?20:33
mordredcorvus: yeah - maybe the linter run isn't picking that up because it doesn't have enough inventory context20:33
corvusoh interesting20:34
mordredcorvus: but ... like ... it woudl get no values that way20:34
AJaegerfungi: I was wrong, stephenfin did not work on the search - but still the best contact to ask20:34
mordredso I still don't understand20:34
*** hashar has quit IRC20:35
AJaegerclarkb: the RPM looks ok20:35
clarkbAJaeger: ya looking at the publish side now20:36
*** xiaolin has quit IRC20:36
AJaegerI think the parent is wrong for a promote, this is mixing publish and promote.20:37
clarkbAJaeger: ya I think you may be right20:38
clarkbI want to say we are promoting system config docs via afs now right? so that may be a pattern to hceck /me looks20:38
corvusmordred: thisk seems very likely to impact similar changes; how do you think we should proceed?20:40
corvusmordred: should we try to figure out the ansible-lint bug and fix or work around it?  or should we consider removing ansible-lint from system-config (istr we were on the verge of doing that several times in the past already)20:41
AJaegerthanks, clarkb ! Calling it a day now...20:41
mordredcorvus: I'd vote for removal for now20:42
mordredcorvus: I have failed for the last 2 days to figure it out20:42
corvusmordred: ok.  i'll run 'tox -e linters' locally real quick just to get one more data point, then i'll push something up20:42
mordredcorvus: ok cool20:42
mordredclarkb: if you get a sec, could you review and its parents? corvus is going to need it for his zuul work20:44
mordredcorvus, clarkb: and needs 2 reviews20:44
* mordred afk for errand20:48
corvusmordred: hrm, i'm able to repro locally20:48
corvusmordred: i'm running ansible-lint with python3.620:49
clarkbmordred: can you check the comment on
*** roman_g has quit IRC20:50
clarkbfungi: does openstackid use any of hte php stuff removed in ?20:52
corvusmordred: it's the testinfra_extra_data thing.20:54
corvusmordred: i have no idea why your change would trigger that.20:54
clarkbfungi: grepping in puppet-openstackid says no, but I know its got php of some sort in there20:55
fungiclarkb: good question, looking20:55
openstackgerritJames E. Blair proposed opendev/system-config master: Override bridge hostvars directly
corvusmordred: i fixed it there, but i get really unhappy whenever we write incorrect code to satisfy a linter20:57
openstackgerritJames E. Blair proposed opendev/system-config master: Stop running ansible-lint on this repo
corvusso that's there for us to evaluate.21:00
clarkbI guess that our integration testing should be sufficient to catch the mismatch ' in yaml that ansible hates as well as other related problems21:01
corvusprobably in most cases?21:04
corvusclarkb: i'm curious why 999 did not fail testing.21:04
clarkbcorvus: me too21:04
clarkbcorvus: I triple checked my human python parser and ended up deciding that the code there wasn't quite right21:05
clarkbbut ya it should've failed I think?21:05
clarkb(I'm trying to get through the stack before sorting that out)21:07
corvustestinfra/[ansible://] PASSED [ 98%]21:07
corvussorry, this is the relevant one:21:08
corvustestinfra/[ansible://] PASSED [ 99%]21:08
corvuswhich means this must be false: if host.backend.get_hostname() == '':21:08
corvusi don't really understand why that isn't just in its own file21:09
corvuswith its own test21:09
corvusthat's used in a number of places and seems really dangerous21:11
corvuslike, there's no way to tell if that matches or not21:11
corvusi'm working on a patch21:17
clarkb is quite the change, but I think it looks good now21:22
clarkbcorvus: thanks, let me know if I can help, I'm through the bulk of that stack now21:23
openstackgerritJames E. Blair proposed opendev/system-config master: Stop using backend hostname in zuul testinfra tests
corvusclarkb: ^ take a look and see if that makes sense? :)21:24
corvusi'm going through and looking for similar pitfalls21:24
corvusi think the test_backups usage is okay, because it's used to skip entire tests (so we can verify it ran)21:26
corvussame for nodepool21:26
corvusletsencrypt is more complicated, but it's also a case where we know the test either ran (with *one* of the datasets, though we can't confirm which) or was skipped.  on balance, it's probably okay too.21:27
clarkbcorvus: I like it, but I think I found a bug (noted on the change)21:28
corvusso that was the only usage where we could fail open and not tell from the logs (the others can fail open by skipping, but that's also true by just setting testinfra_hosts wrong)21:28
corvusclarkb: heh i thought i typed that in :)21:28
openstackgerritJames E. Blair proposed opendev/system-config master: Stop using backend hostname in zuul testinfra tests
corvuswe could restructure that back into one file, and use the 'skip' approach used in the other files... maybe i'll do that as a followup and we can see which we like better21:30
corvusthat's cool, i just lost a file due to git review21:38
corvusi think i see what i did wrong, but man it was really easy to do :(21:39
clarkbhrm its always had a sad when it does the rebasing if there are outstanding edits21:40
clarkb(I use git stash a lot to deal with that)21:40
corvusyeah, i accidentally did a git add before the git rebase abort21:40
corvusi feel like git and i disagree about what should happen when i abort a rebase after doing a git add.21:41
*** roman_g has joined #opendev21:41
corvusit's okay.  no big deal.  i'll just write the entire change again.  i type fast.21:41
clarkbyou might be able to get it out of the reflog?21:42
clarkbbut no ref would've been created ?21:42
clarkbthat is a weird git behavior21:42
corvuscorrect, there is no commit with my file in it21:43
fungiremind me again why git-review performs a test rebase before pushing?21:46
corvusto see if it's going to merge-conflict after uploading21:46
clarkbfungi: to avoid pushing unmergable commits21:46
fungiyeah, so just to catch that case21:46
fungiwhich zuul would quickly report on the change anyway in our case21:46
corvusyeah, though i'd never see it :)21:46
openstackgerritJames E. Blair proposed opendev/system-config master: Recombine zuul testinfra tests into one file
fungiyou would presumably eventually notice that zuul left a verify -1 on the change, but yeah not necessarily any time soon21:47
fungiit seems like there ought to be a more atomic way of testing locally for potential merge conflicts without leaving the worktree in a dirty mid-rebase state21:48
fungii guess it could immediately --abort and make the user initiate a git rebase themselves?21:49
corvusi want to say that we thought auto-aborting could be risky?  but then we do just tell users to do that, so... :/21:50
fungiyeah, for me the biggest confusion with that scenario usually comes from mental context, i didn't say rebase, and if i don't read carefully i forget it's left me mid-rebase21:51
clarkbI run with git status stuff in my PS1 and that helps, but git-review doesn't set that up for people (and it probably shouldn't)21:51
fungii guess we could add a secondary worktree and perform the rebase in that21:53
fungijust as an ephemeral copy21:53
clarkbin a tmpdir maybe21:53
fungithough on a very large repo that might be slow21:53
fungithough maybe not if it's all on the same fs21:53
clarkbit wouldn't be for me if using /tmp21:54
clarkbsince i've got my homedir on a separate partition and fs from other things21:55
clarkband I don't know if we can nest git repos safely?21:55
mnaserare we okay to use the check-arm64 pipeline in our jobs (asking on behalf of osa)21:59
clarkbmnaser: I think so. With annotation that resources are limited and we still run into occasional multi arch issues that challenge our expectations22:00
clarkbmnaser: hrw and ianw are likely the best resources for confirming that though22:00
mnaserclarkb: yeah -- i think the other thing is wheel builds dont seem to happen for arm (yet)22:01
mnaserso that might take a hit on our jobs22:01
clarkbmnaser: its in progress right now iirc22:01
clarkbaddingthem I mean22:01
mnaseri almost got excited seeing :)22:01
fungii wonder if git review could try to instead do a `git merge --no-ff --no-commit`22:03
fungimnaser: yeah, it's coming ;)22:03
fungithe latest hangup is that there are lots more wheels we need to build for arm64 than for amd6422:04
fungibecause far fewer projects publish arm64 wheels to pypi22:04
fungiso we need to outfit the build job with a bunch of additional build requirements22:05
fungiand also it will take waaay longer to run, not only because the nodes may be slower, but also just the sheer number of additional builds from sdist it will require22:05
fungibasically our amd64 jobs get to take advantage of the fact that a lot of dependencies which have c extensions get "manylinux1" platform wheels on pypi, but those are still architecture-specific and most are only x86-64/amd6422:07
fungivery few have aarch64/arm64 manylinux1 wheels, but that's slowly improving and people are finding that, say, travis-ci has the ability to build them22:08
clarkbright there are ~3 types of wheels we deal with. The trivial type that have no c bindings and are all python. These run on any platform and many are already on pypi. The next is those with c bindings. Many are on pypi but only for x86-64. And finally those that aren't built for anything on pypi and we always build them ourselves22:09
fungiand our projects can reach out to upstreams for their dependencies with pull requests adding configuration to build those22:09
fungibut that's going to be a long road22:10
clarkbcorvus: fwiw I think I prefer the split out testinfra tests22:17
corvusclarkb: yeah, i think they're more future-proof22:17
clarkbit makes things nice and clear without reading test content super carefully to see what is being tested22:17
clarkbhowever its still passing without the % ip ?22:18
clarkbthe followup has the %ip but not the split out one22:18
clarkbboth are +1'd22:18
clarkbis it because we don't run the testinfra tests against that change?22:20
clarkbI think that may be it22:20
corvusyeah, that's why i intentionally did not fix thi bug :)22:21
ianwmnaser/clarkb/fungi : all of would help with arm64 wheels :)22:29
ianwif we get the openafs builds published, i think we have enough to publish the wheel builds with the current code22:30
funginoted, i'll try to take a look once i finish with my weapons of grass destruction22:30
ianwand i'll work on moving them to afs via the executor after22:31
clarkbianw: AJaeger and I had a comment on the afs rpm pblishing change but I think it can proably go ahead if that was the intended publishing location22:31
fungialso, ops meetup reports they used meetpad and had essentially zero problems. kept video off most of the time, didn't miss it because most of them had the etherpad up anyway. averaged around 17-20 participants. all very happy with it. success!22:32
openstackgerritJames E. Blair proposed opendev/system-config master: Stop using backend hostname in zuul testinfra tests
openstackgerritJames E. Blair proposed opendev/system-config master: Recombine zuul testinfra tests into one file
corvusfungi: \o/22:32
fungiianw: yeah, afs on the executor means adding additional architectures no longer depends on working openafs support for those architectures22:33
fungiwhich would be awesome22:33
ianwclarkb: yeah, i intended to switch the location in system-config afs-client role once they're in that location.  as you say, i think just the consistent location should be ok22:33
ianwfungi: yep, i want to move all the copy and index generation into o-z-j and then we can gate test that portion, and the executor just has to rsync22:34
ianw(notes in
fungiooh, i missed the multi-arch sig session i guess. meant to hit that one but it probably conflicted with something else for me22:36
*** roman_g has quit IRC22:38
ianwi think that one would have been the night shift for you :)22:39
clarkbianw: we noticed that graphite.o.o's LE handler was the touch file handler and not restart apache22:43
clarkbianw: was that intentional? if not can we land ?22:43
clarkb(looks like thatneeds a recheck /me does this)22:43
clarkbthe cert has been updated on the server but apache is still serving the old cert hwich expires july 122:44
clarkbthat change is expected to fix it in the future but we'll manually reload apache this time if this was an oversight22:44
*** roman_g has joined #opendev22:45
ianwoh, no i don't think it was intentional.  maybe it had something to do with initial ideas to containerise it22:45
openstackgerritMerged openstack/project-config master: Move wheel build role/job outside project-config
clarkbcool fungi ^ I think that means we can do the manual reload now. Did you still want to do it? I can do it if you are calling it a day or otherwise busy22:46
*** DSpider has quit IRC22:48
fungii can do it, still watching seagulls land on the handle of my mower and answering e-mails22:51
fungiNot After : Sep  1 05:34:35 2020 GMT22:52
fungi#status log manually reloaded apache on graphite.o.o to pick up renewed le cert, fixes longer term22:53
openstackstatusfungi: finished logging22:53
fungii almost sent out a search party for statusbot22:53
fungiokay, guess i should scare off the gulls and finish the lawn22:54
clarkbthanks for getting that sorted22:55
ianwother thing on openafs and arm64 etc, there was some interest in having particularly arm64 testing with openafs upstream's gerrit22:58
ianwthey currently seem to use buildbot to do a compile test on changes, we could do a end-to-end install and functionality test like we do in system-config gate22:59
ianwwe would have to figure out how to build on debuntu from source; has some interesting thigns for rpm generation from trunk23:00
clarkbianw: we could even add their gerrit to zuul potentially23:07
clarkbmaybe thats what you meant?23:07
*** factor has joined #opendev23:09
fungiianw: what specific openafs packages are we looking to build on arm64? debian already builds a bunch they distribute:
fungifor example, openafs-client 1.8.6~pre1-3 is built for arm64 in sid23:11
fungias is openafs-fileserver23:11
clarkbfungi: I thinkwe want to avoid the pre packages?23:11
clarkbthat was the source of our problems on bionic23:11
fungiright, but we could still likely use the packagING just not the packagES23:11
clarkbwe've got a ppa building packages for bionic on a proper release23:11
clarkbfungi: I think we are?23:12
clarkbbut via ppa23:12
clarkb(maybe my asumption we can extend the ppa to debian is flawed)23:12
fungithen i'm not clear on what "figure out how to build on debuntu from source" means23:12
clarkbfungi: I think thats for building tip of master23:12
fungiyeah, just saying the packaging for 1.8.6~pre1-3 in debian/sig can likely be used to package tip of master as well23:13
fungiat least it's where i'd start23:13
fungihappy to help with that if there are more specifics on what's needed23:14
fungii had a zoo of a dozen different major processor architecture representatives running debian i used to use to do ports of packages to less popular architectures, and it's usually pretty straightforward23:15
fungibut in this case most of the porting work is likely already figured out23:15
fungisince debian has packages ported to arm64 from source close to master23:16
ianwyeah, debian is better than bionic/focal that have shipped pre versions23:16
ianwi did mean connect our zuul to their gerrit23:16
ianwcurrently we install from our ppa -- which i agree it would be better if we had local jobs that built the .debs we install, as having it out-of-band is a bit of an overall anti-pattern23:17
ianwso if we came up with something similar to the rpm build jobs but for the .debs23:17
ianwand if that could build debs not just from releases but from zuul checkouts ...23:18
ianwwe'd be 90% of the way there23:18
ianw(to useful testing for upstream gerrit)23:18
ianwone sticking point is that the .spec is part of the openafs source, while the debian/ packaging is maintained separately afaik23:19
ianwthere's probably little point if we spend more time dealing with broken packaging than actually testing the packaged source code23:20
clarkbas a heads up I've realized that we've got kid doctor visits on the 12th which is the day we pencilled in for project renaming in gerrit23:20
clarkbI don't have to AFK until about 1800UTC so going earlier is probably best if I'll be helping23:21
clarkbbut I'm also not even sure if they'll allow multiple adults in at the same time so it may be a non issue23:21
fungiianw: to be fair, debian is *why* ubuntu shipped ~pre builds... because debian has been including them in its sid (unstable) suite, which is where ubuntu pulls its packages from23:23
fungidebian's just smart enough not to migrate those to its testing and stable suites23:23
fungiit's more about how ubuntu auto-syncs packages it doesn't care about from debian23:23
ianwyeah, i guess it's the don't care bit because the bionic bug has been open since bionic23:24
ianwas i'm using arm64 more -- the mirror is the most unreliable part :
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: Add python_version for tox27
ianwit drops afs connection like that constantly23:25
fungiianw: are these possibly connectivity/responsiveness problems with that environment?23:25
ianwyeah, i mean i guess it must be, the traceroute from the mirror to rackspace is only 11 steps23:27
fungido you hit similar errors with proxied files or just afs-served?23:28
ianwi don't really have exposure to jobs that are using a lot of the proxy side of things, only the packages really23:28
*** mlavalle has quit IRC23:41
*** mlavalle has joined #opendev23:44
ianw"I will bump the severity and work on this once I am caught up" : Date: Mon, 11 Apr 201623:49
ianwfamous last words :)23:49
*** tosky has quit IRC23:50
*** factor has quit IRC23:52
*** mlavalle has quit IRC23:52
*** factor has joined #opendev23:52
*** roman_g has quit IRC23:57
*** mlavalle has joined #opendev23:57
ianw23:54:56.639157 IP > ICMP host unreachable - admin prohibited, length 10223:57
ianwi wonder what that means23:57
ianwi can ping between mirror and afs01 just fine, doens't drop a single packet23:58
clarkbadmin prohibited is a firewall I think23:59
clarkblikely not ours, but check syslog for iptables logging to rule it out?23:59

Generated by 2.17.2 by Marius Gedminas - find it at!