Thursday, 2020-09-03

fungithe second pvmove on afs01.dfw is running in a root screen session in case anyone needs to check on it00:19
*** qchris has quit IRC00:43
*** qchris has joined #opendev00:56
openstackgerritIan Wienand proposed opendev/system-config master: Update README.rst  https://review.opendev.org/74961500:57
openstackgerritMerged opendev/system-config master: puppet: don't run module install steps multiple times  https://review.opendev.org/74943102:02
openstackgerritIan Wienand proposed opendev/system-config master: docs: Update some of sysadmin details  https://review.opendev.org/74963003:56
*** mwhahaha has quit IRC04:39
*** mwhahaha has joined #opendev04:40
*** ysandeep|away is now known as ysandeep04:40
*** osmanlicilegi has quit IRC04:40
*** osmanlicilegi has joined #opendev04:40
*** osmanlicilegi has quit IRC04:40
*** DSpider has joined #opendev04:56
*** osmanlicilegi has joined #opendev04:59
*** diablo_rojo has quit IRC05:01
*** raukadah is now known as chandankumar05:22
openstackgerritMerged openstack/project-config master: Add ansible-collections for system-config -devel job  https://review.opendev.org/74759605:48
*** jaicaa has quit IRC05:49
openstackgerritMerged openstack/project-config master: Move github reporting to checks API  https://review.opendev.org/74682705:50
*** jaicaa has joined #opendev05:52
*** icey has quit IRC05:59
openstackgerritIan Wienand proposed opendev/system-config master: docs: Update some of sysadmin details  https://review.opendev.org/74963006:01
*** fressi has joined #opendev06:16
*** qchris has quit IRC06:20
*** icey has joined #opendev06:28
*** icey has quit IRC06:29
*** icey has joined #opendev06:29
*** qchris has joined #opendev06:33
*** icey has quit IRC06:35
*** icey has joined #opendev06:35
*** fressi has quit IRC06:46
*** hashar has joined #opendev06:50
*** sshnaidm|afk is now known as sshnaidm06:50
openstackgerritIan Wienand proposed opendev/system-config master: Update README.rst  https://review.opendev.org/74961507:02
openstackgerritIan Wienand proposed opendev/system-config master: docs: Update some of sysadmin details  https://review.opendev.org/74963007:02
*** yoctozepto2 has joined #opendev07:20
*** yoctozepto has quit IRC07:22
*** yoctozepto2 is now known as yoctozepto07:22
*** andrewbonney has joined #opendev07:28
*** tosky has joined #opendev07:40
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** fressi has joined #opendev08:18
*** fressi has quit IRC08:22
openstackgerritRadosław Piliszek proposed openstack/project-config master: kolla-cli: deprecation - remove noop jobs  https://review.opendev.org/74967208:30
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes  https://review.opendev.org/74849808:33
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes  https://review.opendev.org/74849808:41
*** priteau has joined #opendev08:49
*** hashar is now known as hasharAway09:00
*** dtantsur|afk is now known as dtantsur09:06
*** pushparajkvp has joined #opendev09:10
*** pushparajkvp has quit IRC09:22
openstackgerritRadosław Piliszek proposed openstack/project-config master: kolla-cli: deprecation - Mark as retired in Gerrit  https://review.opendev.org/74968809:32
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes  https://review.opendev.org/74849809:58
*** ysandeep is now known as ysandeep|afk10:00
*** hasharAway is now known as hashar10:10
*** pushparajkvp has joined #opendev10:29
*** priteau has quit IRC10:33
*** ysandeep|afk is now known as ysandeep10:34
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Add ensure-ansible role  https://review.opendev.org/74970611:03
*** hashar is now known as hasharAway11:25
*** priteau has joined #opendev11:44
*** priteau has quit IRC11:44
*** lpetrut has joined #opendev11:47
*** hasharAway is now known as hashar11:54
*** priteau has joined #opendev12:01
*** mattd01 has joined #opendev12:01
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes  https://review.opendev.org/74849812:11
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Add ensure-ansible role  https://review.opendev.org/74970612:13
hasharhi,  if a patch has CR+2 and W+1 , do I have a way to retrigger the gate pipeline ?12:34
hasharthe change is https://review.opendev.org/#/c/688446/  and 'recheck' apparently only trigger the check pipeline12:34
hasharreverify  apparently did the trick :]12:39
yoctozeptoinfra-root: could someone let me op myself on #openstack-masakari so that I can set the channel topic? :-) (like I do on #openstack-kolla)12:52
openstackgerritThierry Carrez proposed opendev/system-config master: Explain "why opendev" on opendev.org index page  https://review.opendev.org/74826313:14
openstackgerritRadosław Piliszek proposed openstack/project-config master: kolla-cli: deprecation - Mark as deprecated in Gerrit  https://review.opendev.org/74968813:16
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add openstack/senlin to vexxhost tenant  https://review.opendev.org/74973013:22
fricklerinfra-root: I'm suprised to see that the reverify ^^ seems to actually have been executed. didn't we dump that?13:22
frickleror only for the openstack tenant?13:22
Open10K8Shi teamplease check this PS  https://review.opendev.org/749730 Add openstack/senlin to vexxhost tenant13:23
fungifrickler: we long ago got rid of the separate reverify because the clean-check requirement made it redundant/confusing, but yes that decision was really only carried over into the openstack tenant. in other tenants where we don't require a positive (or any) check result before enqueuing, adding a reverify command to rerun in the gate pipeline made sense13:26
fricklerfungi: thx, so I would almost have given some wrong advise if I'd seen this earlier, you never stop learning ...13:28
hashar:]]]13:29
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes  https://review.opendev.org/74849813:29
fungifrickler: according to this recheck should have worked too: https://opendev.org/opendev/project-config/src/branch/master/zuul.d/pipelines.yaml#L5513:30
hasharfrickler: fungi: what I dont get though is that others did "recheck" on the change but that only triggered the check pipeline13:30
hasharI went with "verify" cause it is not a keyword for the check pipeline, it is solely for the gate pipeline13:31
hasharmy understanding is a recheck should add the change in both check and gate and the change get removed from the check pipeline cause gate has "supercedes: check"13:32
* yoctozepto wishes he could reverify some patches to avoid long-running check jobs13:32
fungiyeah, definitely strange... for the opendev tenant the gate pipeline is set to supercede check13:32
fricklerhashar: "recheck" would have triggered the check pipeline and the new verified+1 vote would then have triggered the gate13:32
hasharseems like a recheck is sufficient for the gate, as long as the change has workflow+113:33
fungiyoctozepto: the openstack tenant is configured not to allow directly (re)enqueuing into the gate because it wrecks lengthy dependent queues13:33
hasharOH no13:34
hasharso it used to be that gate did not react to recheck / reverify and that got added ~ 2 months ago with https://opendev.org/opendev/project-config/commit/d530aaa8bf595047c2daad6b8cac8ed845395c5413:34
hasharwhich is why my "reverify" managed to trigger the gate13:34
fungiyoctozepto: basically people kept reverifying broken changes and making things take forever to merge as a result13:34
hasharwhen previous "recheck" comments did not cause they have been made before that change ;]  mystery solved13:35
fungihashar: aha! yep, that certainly explains it13:35
fungiso working as expected (probably)13:35
hashar:]]13:36
yoctozeptofungi: ah, that makes sense13:37
fungiyoctozepto: also if a change introduces a bug which only surfaces 50% of the time, the odds that a recheck will allow it to go through passing builds twice (in both check and gate) are 25%, so that requirement reduced the effect of folks reverify-spamming inconsistent failure bugs into projects13:40
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add openstack/senlin, openstack/senlin-tempest-plugin to vexxhost tenant  https://review.opendev.org/74973013:41
Open10K8SHi team. Please check this PS, i just updated once more. waiting for review :)13:55
fungimoving the third afs01.dfw is going slowly. i'm not sure if that's because either the source or destination device has performance issues, or has lots more churn than the first two, or the server just got busy14:08
fungiit's been underway for roughly 10 hours and is 81% complete so far14:09
fungier, i should say, moving the third pv for afs01.dfw14:09
openstackgerritSorin Sbarnea (zbr) proposed opendev/bindep master: Improve tox.ini setup  https://review.opendev.org/60561314:11
openstackgerritSorin Sbarnea (zbr) proposed opendev/bindep master: Improve tox.ini setup  https://review.opendev.org/60561314:11
openstackgerritSorin Sbarnea (zbr) proposed opendev/bindep master: Improve tox.ini setup  https://review.opendev.org/60561314:12
openstackgerritMattias Jernberg proposed opendev/git-review master: Allow the default of notopic to be configurable  https://review.opendev.org/69744814:19
*** ysandeep is now known as ysandeep|away14:32
*** lpetrut has quit IRC14:50
openstackgerritAndreas Jaeger proposed openstack/project-config master: Final step for networking-l2gw and networking-l2gw-tempest-plugin retirement  https://review.opendev.org/73804014:51
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes  https://review.opendev.org/74849815:00
openstackgerritMerged openstack/project-config master: Disable nested-virt-centos-8 from vexxhost-ca-ymq-1  https://review.opendev.org/74821015:00
*** lpetrut has joined #opendev15:03
openstackgerritMerged openstack/project-config master: Add openstack/senlin, openstack/senlin-tempest-plugin to vexxhost tenant  https://review.opendev.org/74973015:08
clarkblooks like zbr's updated ask.o.o message has been applied to the server and I don't think anyone manually applied it. That should indicate ianw's run_once cleanup is functioning as expected15:15
openstackgerritwes hayutin proposed zuul/zuul-jobs master: Make .sh browsable on swift logs  https://review.opendev.org/73179515:17
fungiclarkb: did we expect that to actually make ask read-only yet? because i got a notification this morning of a reply on a question matching one of my subscribed tags15:19
*** mlavalle has joined #opendev15:20
fungihrm, though if i pull it up, it says it was last updated august 3015:21
fungibut for some reason the notification i received was dated today15:21
fungiyeah, maybe it just batches those up or something15:21
clarkbyes it is expected to be read only (not sure why email notifications are late)15:22
clarkbsetuptools has gotten a 50.0.2, 50.0.3, and 50.1.0. I'm looking into what fixes hav ebeen made and whether or not we can start removing our pins15:23
fungiexcellent15:23
clarkbfrom 50.1.0 "Setuptools reverts using the included distutils by default."15:24
clarkband "Platform maintainers and system integrators and others are strongly encouraged to set SETUPTOOLS_USE_DISTUTILS=local to help identify and work through the reported issues with distutils adoption,"15:24
*** iurygregory has quit IRC15:24
clarkbso ya I expect this will work until 52.0.0 when they switch the dfeault again :)15:24
fungiyeesh15:25
openstackgerritClark Boylan proposed opendev/system-config master: DNM Test fixed setuptools 50.0.x  https://review.opendev.org/74976615:29
clarkbfungi: ^ thats a sanity check of the bug fixes that landed before the recent revert15:29
clarkbmostly looking for information on how bad older python3 situation is15:29
zbrclarkb: regarding ask going readonly, I know that I asked why not using stackoverflow instead about 2-3 years ago. I do remember being opposed. It took two more years to happen. :D15:29
fungizbr: yep, many of us have asked repeatedly for years if we can take that down. glad it's finally happened15:30
*** dtantsur is now known as dtantsur|afk15:31
clarkbif the 50.0.3 change is happy I'll push up an unpin of setuptools in our image venvs15:32
clarkbI guess I'll try 50.1.0 first15:32
clarkbbut thats the end goal here i sdetermining if the images can remove those pins15:32
*** iurygregory has joined #opendev15:42
openstackgerritAndreas Jaeger proposed openstack/project-config master: Add zuul/zuul-client to gerritbot  https://review.opendev.org/74977015:48
clarkbsetuptools 50.0.3 does seem to fix our venv issues in the puppet apply jobs15:59
openstackgerritClark Boylan proposed opendev/system-config master: Revert "Install older setuptools in puppet apply jobs"  https://review.opendev.org/74976616:00
clarkbthat update removes the pin entirely, if that passes I'll get a change up to remove the pin on the image side16:00
openstackgerritNoam Angel proposed openstack/diskimage-builder master: 00-1-centos-python3 runs before red hat subscription and failed any image build that using red hat subscriptions based repositories   https://review.opendev.org/74977316:01
openstackgerritNoam Angel proposed openstack/diskimage-builder master: 00-1-centos-python3 runs before red hat subscription and failed any image build that using red hat subscriptions based repositories   https://review.opendev.org/74977316:02
openstackgerritClark Boylan proposed openstack/project-config master: Revert "Pin setuptools<50 in our image venvs"  https://review.opendev.org/74977716:19
clarkband ^ is the image cleanup16:20
*** priteau has quit IRC16:25
*** lpetrut has quit IRC16:57
*** hashar has quit IRC16:58
*** mattd01 has quit IRC17:01
*** ysandeep|away is now known as ysandeep17:02
*** ysandeep is now known as ysandeep|away17:16
fungithe third pvmove for afs01.dfw has finally completed, and i've started the last one for it now17:28
fungihopefully this one will go more quickly (already seems to be) and then i can clean up the old volumes and repeat the process for afs01.dfw17:28
fungier, repeat the process for afs02.dfw17:29
clarkb"We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default." <- has anyone looked at that for our python things?17:40
clarkbI don't expect openstack with constraints will ahve many issues because constraints should win17:40
clarkbbut anything not using a constraints or lockfile may be sad17:40
fungiprometheanfire did some testing with installing the requirements list17:46
fungiand compared the result of constraints list building with that option on and off, seeing no change in behavior17:47
fungithe openstack global requirements list, i should clarify17:47
*** mattd01 has joined #opendev17:55
clarkbthat is what I would expect with constraints, do you mean the installations were done without constraints?17:56
fungino, the script we use to generate the constraints list18:06
clarkbgot it, so ya that would run without constraints to produce a new(er) constraints list18:07
fungiwhich installs the requirements list (without constraints) and then does a pip freeze to create the constraints list18:07
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: render-diff: fix typo in failure messsage  https://review.opendev.org/74980918:07
clarkband if there is no delta in resulting pacakge versions its likely that things are ok18:07
fungiyep, that's at least the scenario prometheanfire describe having tested18:07
fungier, described18:07
*** priteau has joined #opendev18:24
clarkbfungi: your migrations for afs server volumes had me looking at https://etherpad.opendev.org/p/2020-10-rax-dfw-volume-maint again and I think that etherpad's volume backs its database18:34
clarkbwe may want to move that one into the migrate list18:35
fungiahh, sure, please do18:36
*** mattd01 has left #opendev18:39
*** priteau has quit IRC18:47
*** sshnaidm is now known as sshnaidm|off18:48
clarkbI think I'm going to continue and try to dig out of my review backlog today. Feel free to call out important changes otherwise I'm just doing my best looking at things like system-config and zuul and nodepool18:53
clarkbbut first lunch18:53
prometheanfirefungi: yep, tested installing / generating constraints with the new pip stuff19:09
openstackgerritMerged zuul/zuul-jobs master: render-diff: fix typo in failure messsage  https://review.opendev.org/74980919:14
*** andrewbonney has quit IRC19:29
*** tosky has quit IRC19:45
*** xiaolin has joined #opendev20:20
*** johnavp1989 has left #opendev20:36
*** johnavp1989 has joined #opendev20:36
clarkbthat was weird I hit the post buttin in gerrit to leave a review ocmment which worked but it also seems to have hit the cherry pick button which was overlaid by post20:41
clarkbDon't think I've ever seen that before20:41
fungiin our current gerrit, or testing upgraded gerrit?20:46
clarkbcurrent gerrit20:47
clarkbI think it may be related to my network connection. I'm doing code reviews in the front yard20:47
openstackgerritClark Boylan proposed openstack/project-config master: Retire the devstack-plugin-zmq project  https://review.opendev.org/74872420:52
fungii should, but christine doesn't like me hanging out on the patio when there's a heat advisory (even though it's actually really nice out there, she refuses to believe me)20:52
clarkbits nice in the shade here if a bit warm20:52
corvusfungi: that's like a standard sci-fi plot right?  trapped in the underground bunker for generations for protection from the apocalypse unaware of the garden of eden above?20:53
fungipretty much20:53
corvusfungi: so you're saying the apocalypse is over and i can go outside again?20:53
fungi40c here, 50%rh and a 15kph wind coming off the water20:54
fungiideal really20:54
corvusthe air quality map says i should stay in the underground bunker :(20:54
fungifrom the fires?20:55
corvusyeah20:55
fungi:(20:55
corvushighly variable depending on the wind; wind is apparently bad for us now20:55
fungihere's hoping my cybernetic replacement body won't care about air quality20:56
openstackgerritClark Boylan proposed openstack/project-config master: Retire devstack-plugin-pika project  https://review.opendev.org/74871220:57
clarkbonly 31C here according to the internet21:08
fungi#status log all four cinder volumes for afs01.dfw have been replaced and cleaned up21:08
openstackstatusfungi: finished logging21:08
fungithat went remarkably smoothly, aside from osc being unable to handle volume names in rackspace now21:08
fungigetting started on afs02.dfw next21:08
clarkbfungi: thank you for taking care of that21:08
fungiit's pretty straightforward, and i can have it going in the background while i work on other stuff, so no sweat21:09
fungifour new volumes created, attached, partitioned, pvcreated, vgextended, and now the first of four pvmoves is in progress in a root screen session on afs02.dfw21:18
fungigoing along at a nice clip too, 2.5% complete already21:22
openstackgerritMerged openstack/project-config master: kolla-cli: deprecation - remove noop jobs  https://review.opendev.org/74967221:29
openstackgerritMerged openstack/project-config master: Final step for networking-l2gw and networking-l2gw-tempest-plugin retirement  https://review.opendev.org/73804021:35
clarkbfungi: ianw I left a comment on https://review.opendev.org/#/c/749604/1 and +2'd and didn't approve to ensure we could consider that first21:38
openstackgerritSean McGinnis proposed openstack/project-config master: Add openstack/osops  https://review.opendev.org/74983421:39
openstackgerritSean McGinnis proposed openstack/project-config master: Add openstack/osops  https://review.opendev.org/74983421:40
*** pushparajkvp has quit IRC21:44
*** xiaolin has quit IRC21:54
ianwclarkb: yeah, i think that all contexts that we're running that localhost == bridge?22:11
clarkbianw: yes, I think that is the case (for ci too even)22:12
clarkbI just wanted to call it out so others could reaffirm that22:12
ianwclarkb: on running things out-of-band ... yeah i'm not sure.  i'm happy to write a procedure if we have one :)22:14
ianwi  think the first thing is you need your own system-config22:14
clarkbianw: I think the new setup as root is to run the service-foo.yaml playbook on bridge out of zuuls homedir22:15
ianwand then you can run ansible-playbook --limit=hostofinterest ./system-config/playbooks/service-<foo>22:15
clarkbafter running disable-zuul or whaetever its called22:15
clarkbianw: even if we want to put a TODO there I think it would be good as a reminder that we want to have that22:15
*** hashar has joined #opendev22:16
openstackgerritIan Wienand proposed opendev/system-config master: docs: Update some of sysadmin details  https://review.opendev.org/74963022:24
ianwclarkb: ^ re-added, moved just under the disable section22:24
clarkbthanks, looking22:24
clarkbthat looks right to me.22:25
*** mlavalle has quit IRC23:01
clarkbianw: I'm looking at nb03 dockerization now and am wondering if we should deploy an nb03.opendev.org rather tha nconvert the existing server?23:14
clarkbit apperas thats how we handled the x86 builders23:15
clarkbianw: if we do add a new nb03.opendev.org then we'll need to add a project-config/nodepool/nb03.opendev.org file23:17
clarkbbut then I think we can just launch node and go from there. What I'm not sure about is if we've got room in the linaro cloud for the old and new server to run side by side23:18
clarkband then also coordinating if we need to disable the service on the old one to keep them from deleting the images? Though I think if they have the same config that bit is fine23:18
openstackgerritClark Boylan proposed openstack/project-config master: Add config for nb03.opendev.org  https://review.opendev.org/74985223:32
openstackgerritClark Boylan proposed opendev/system-config master: Remove nodepool builder puppetry and nb03.openstack.org  https://review.opendev.org/74985323:35
clarkbcorvus: ianw ^ I started writing some of the required hcanges to do the new nb03.opendev.org idea. There is a bit in the middle of those two changes which is add nb03.opendev.org to our inventory23:35
clarkbI'm not in a good spot to start that with the rest of my day, but can pick that up tomorrow if we like that plan23:36
clarkb(i've not booted nb03.opendev.org which is why I have nothing to add to inventory)23:36
clarkbalso if I've missed a better approach feel free to point at it :)23:36
*** cloudnull has quit IRC23:47
*** DSpider has quit IRC23:51
hasharhello :]23:59
corvushashar: o/23:59
hasharI tried to subscribe to the service-discuss mailing list on lists.opendev.org and somehow I am not receiving any email.  I guess it bounces somehow23:59
corvushashar: i'll check the logs23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!