| opendevreview | melanie witt proposed openstack/devstack-plugin-ceph master: DNM testing --pg-num-max https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/898787 | 01:25 |
|---|---|---|
| opendevreview | Pranali Deore proposed openstack/tempest master: Add tests for new add location api of glance https://review.opendev.org/c/openstack/tempest/+/891026 | 07:20 |
| ralonsoh | fungi, just a heads-up: I've received the confirmation from the sys admins in redhat. The mimecast issue is solved now. | 07:32 |
| frickler | ralonsoh: yes, a lot of mails that seem to have been queued showed up. I assume you didn't get any information on what the fix looked like? that might help us next time around | 08:04 |
| ralonsoh | no sorry, only the confirmation that is solved | 08:04 |
| frickler | ralonsoh: ack, thx for confirming anyway | 08:04 |
| kopecmartin | frickler: wdyt about this https://review.opendev.org/c/openstack/devstack/+/887184/ ? the regression will be fixed by the depending patch | 09:08 |
| frickler | kopecmartin: honestly I think we should revert the whole thing, it makes for a really bad user experience when using devstack locally. neutron should fix their OVN installation code instead | 09:12 |
| frickler | but I also think that prehistoric branches should not be changed in such a way anyway, who really cares about them? | 09:13 |
| kopecmartin | yeah, i wanted to either merge it or abandon, wallaby is quite old and nothing suggests that someone needs that merged there .. soo we can let it be then | 09:18 |
| fungi | ralonsoh: thanks for the update! glad they fixed whatever the issue was | 12:44 |
| opendevreview | Merged openstack/tempest master: Fix cleanup for object_storage tests https://review.opendev.org/c/openstack/tempest/+/896011 | 13:35 |
| opendevreview | Merged openstack/devstack master: Add support volume backup_driver config option https://review.opendev.org/c/openstack/devstack/+/897402 | 13:35 |
| *** haleyb_ is now known as haleyb | 13:41 | |
| clarkb | sean-k-mooney: do you know why the global venv breaks on centos/rocky? Naively I would've expected global venv to hide some of the distro specific behaviors since all the code gets installed to a common location | 16:32 |
| clarkb | unfortunately the bug doesn't really seem to identify the underlying cause just that it occurs on centos | 16:33 |
| sean-k-mooney | clarkb: i think perhaps some of the paths are diffent | 16:49 |
| sean-k-mooney | but to be clear i have only used ubuntu since that change landed because i rearly use fedora or centos | 16:49 |
| sean-k-mooney | i really dont think it would be hard to make it work | 16:49 |
| sean-k-mooney | i can maybe give it a try before that ptg session | 16:49 |
| clarkb | its not that important. Mostly just hoping if someone knows we could update the bug with more concrete info | 16:50 |
| sean-k-mooney | i feel like its somethign dumb that just needs someone to look at it | 16:50 |
| sean-k-mooney | ya | 16:50 |
| sean-k-mooney | so im doing something else right now but ill kick of a centos 9 devstack install and see what explode | 16:51 |
| clarkb | sean-k-mooney: we can just push a patch to devstack that enables global venv on those platforms and let zuul do it for us | 16:52 |
| sean-k-mooney | ya ill do that in paralle | 16:54 |
| sean-k-mooney | i need to power on my home cloud a break popped during the weak and i havent powered it on since | 16:55 |
| clarkb | need some lower power servers :) | 16:55 |
| sean-k-mooney | actully it was the trip switch (rcd) splashed some water on the side of my coffee machine and it also touched the sink for the rcd triped and killed all power to all sockets | 16:57 |
| sean-k-mooney | but i proably should invest in a ups | 16:58 |
| sean-k-mooney | lower power servers would also not hurt from a power persective i guess | 16:58 |
| clarkb | suddenly I'm imagining a rack of servers in the kitchen next to the fridge | 16:59 |
| clarkb | "we can cook an egg on top of the rack" | 16:59 |
| sean-k-mooney | hehe nope they are in the attic space between the roof and the insulation | 17:00 |
| dansmith | sean-k-mooney: you not having a dedicated circuit (and a UPS) is weaksauce. I'm disappointed. | 17:00 |
| sean-k-mooney | i know right. its on my todolist when i get solar/carchager and some other stuff done | 17:01 |
| sean-k-mooney | im thinking about geting the main pannel repalced and upgradign all the ciruts to RCBOs instead of just MCBs with a shared rcd for ground faults | 17:02 |
| sean-k-mooney | that would have ment that only hte kitchen sockte woudl have tripped but finding time to get someone to do that is... | 17:03 |
| dansmith | yeah we tend to put our RCDs at the point of use | 17:06 |
| dansmith | but generally nothing downstream of them that aren't in wet locations specifically for that "the kitchen shut off the tv" sort of situation | 17:06 |
| opendevreview | sean mooney proposed openstack/devstack master: ignore dbcounter sub dirs https://review.opendev.org/c/openstack/devstack/+/898844 | 17:14 |
| opendevreview | sean mooney proposed openstack/devstack master: [DNM] testing centos9/rocky9 jobs with global venv enabled https://review.opendev.org/c/openstack/devstack/+/898845 | 17:14 |
| sean-k-mooney | lets see what ci says i guess ill see if i can find time to repodcue locally but proably not today | 17:15 |
| frickler | so I think the last thing kopecmartin did in debugging was find out that pip places the wsgi scripts into a differen location than the cli scripts, causing the wsgi services to fail | 17:47 |
| sean-k-mooney | ya when i was fixing fedora reboot how wsgi was handeled was slightly diffent so that is not surpising | 17:49 |
| sean-k-mooney | altough pbr which generates teh wsgi script should use the same location | 17:49 |
| frickler | for debuntu both end up in venv/bin, for centos it was something in ~/.local iirc | 17:50 |
| sean-k-mooney | for all distos | 17:50 |
| clarkb | huh | 17:50 |
| sean-k-mooney | odd but that shoudl be a trival path fix for the systemd service files | 17:50 |
| sean-k-mooney | prehaps it has somehtign to do with distutils differnece betwen rpm vs deb distos or something | 17:51 |
| JayF | if it's in ~/.local that implies it wasn't installed in the venv but instead as pip --user; I wonder if there's a weird patch on pip or something... hmm. I don't use RPM-based distros very much though so it's just conjecture, I'll be interested to see what you land on | 17:53 |
| sean-k-mooney | same i more or less only use them when i have to debug an issue that only happens on them | 17:54 |
| * JayF blinks | 17:57 | |
| sean-k-mooney | my employer has nothign to do with my prefence in linux distro so i dont use rpm distos unless i have too :) | 17:59 |
| * sean-k-mooney is currently using debian sid to type this | 17:59 | |
| JayF | I've used a fair bit of debian :) (well, I've used a fair bit of everything). Always come back to Gentoo. Been running Gentoo for a long time now | 18:09 |
| sean-k-mooney | the only time i considerd gentoo was when i was considering doing LinuxFromScratc i decided LFS would be more educational so i did that instead | 18:13 |
| JayF | Well if you think I know a lot about linux; blame Gentoo. If not, blame me ;). Gentoo gives you a lot more tools to work with than LFS really. Gentoo is more like building your own personal customized distribution which will keep going without much effort after the up-front | 18:15 |
| sean-k-mooney | ya i used nixos for about 18 months until i got a new laptop in may for a similar reason | 18:16 |
| sean-k-mooney | i was tired fo fedora breaking things randomly so i wanted something talored and minimal | 18:17 |
| JayF | at least when something like gentoo or nixos breaks, you can kinda blame yourself and that makes it easier to kick myself in gear and fix it | 18:17 |
| sean-k-mooney | when i got a new laptop i tried fedroa 38 kinotie and it broke after 3 weeks use so i said screwit and went back to debian | 18:17 |
| JayF | and it's nice to be able to fix something since openstack never ever breaks :P | 18:17 |
| sean-k-mooney | well first rule of openstack is if you install it on your laptop do it in a vm :) | 18:18 |
| JayF | Well there's the gentoo laptop, and the gentoo hypervisor in the corner I use virt-manager to spin up devstacks on :D | 18:32 |
| sean-k-mooney | i normally do much of my devstack based dev on my home server which has singel node kolla-ansible on ubunutu but i ocatlly create devstack vms locally too | 18:33 |
| sean-k-mooney | speakign of devstack i wonder if the ci reported back | 18:34 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/c012dd2c8d0e406587208cdb5502452b | 18:35 |
| sean-k-mooney | so ya keystone failed to start which makes sense that is the first wsgi servce we woudl start | 18:35 |
| sean-k-mooney | ok so its doing pip install here https://zuul.opendev.org/t/openstack/build/c012dd2c8d0e406587208cdb5502452b/log/controller/logs/devstacklog.txt#4878 | 18:37 |
| sean-k-mooney | which expands to /opt/stack/data/venv/bin/pip install | 18:38 |
| sean-k-mooney | so we are defintly atempting to use the venv | 18:38 |
| dansmith | frickler: okay I got one grenade run that took like 62s to do the sync, so perhaps setting that long is reasonable.. 10m seems maybe too long, so how about 3m, we merge, and see if that fail signature goes away? | 18:39 |
| sean-k-mooney | ok and hten it creates an editable wheel adn installs that. unfortunetly it does not tell us what files that creates | 18:39 |
| dansmith | frickler: honestly I was skeptical that if it took >60s it was anything other than just stuck, but apparently not | 18:39 |
| sean-k-mooney | i can confirm we are expecign the wsgi script to be in the bin dir | 18:41 |
| sean-k-mooney | write_uwsgi_config /etc/keystone/keystone-uwsgi-public.ini /opt/stack/data/venv/bin/keystone-wsgi-public /identity | 18:41 |
| sean-k-mooney | so it might just be as simple as create a simlink or update that path to point to the correct localtion | 18:42 |
| sean-k-mooney | it sound like we coudl confirm where the fiel is place just by cloning keystone, creating a venv and pip installing it on rocky/centos | 18:42 |
| sean-k-mooney | lol of course that put it in ~/.venv/bin | 18:47 |
| sean-k-mooney | so on rocky pip installing just works | 18:47 |
| sean-k-mooney | from within a venv | 18:47 |
| sean-k-mooney | ill try that again without activating the venv to see if that chagnes thigns | 18:48 |
| sean-k-mooney | same result so this does not appre to work in a non devstack venv. | 18:52 |
| sean-k-mooney | * this does not appear to be broken in a non devstack env | 18:55 |
| sean-k-mooney | ill take a look at this again during the ptg | 18:56 |
| opendevreview | Dan Smith proposed openstack/grenade master: Increase cinder verify sync timeout https://review.opendev.org/c/openstack/grenade/+/898745 | 19:18 |
| dansmith | gmann: after a bunch of rechecks, this ^ seems to be a reasonable timeout for that cinder sync during upgrade which has been failing a bunch lately | 19:18 |
| gmann | dansmith: ack, will check today. | 19:19 |
| sean-k-mooney | so it is the ssh that is failing not the server ping. | 19:20 |
| sean-k-mooney | i was not sure which command in that function was failing | 19:20 |
| dansmith | no, its not the ssh | 19:30 |
| dansmith | it's ssh'ing, running 'sync' and that is taking a long time (legitimately) | 19:30 |
| dansmith | due to super slow IO bandwidth likely | 19:30 |
| dansmith | sometimes it takes zero time, other times over 60s | 19:31 |
| sean-k-mooney | oh ok. well that partly what i ment the ssh command | 19:31 |
| dansmith | well, distinct from a "can't ssh to instance" sort of problem | 19:32 |
| sean-k-mooney | ya by the way https://review.opendev.org/c/openstack/devstack/+/890693 is still up for review. that the zswap/ksmtuned change | 19:33 |
| sean-k-mooney | are we sill interested in that | 19:33 |
| sean-k-mooney | the ci results are rotated so i should proably rebase that qickly | 19:35 |
| opendevreview | sean mooney proposed openstack/devstack master: add support for zswap and ksmtuned https://review.opendev.org/c/openstack/devstack/+/890693 | 19:35 |
| sean-k-mooney | thats more memory io then disk io but i do actully tune some of the sysctl disk io tunables there too | 19:36 |
| dansmith | potentially. I've seen a number of job timeouts again this week and cinder folks were saying they've got lots of OOMs | 19:36 |
| dansmith | zswap should reduce how much we swap to disk so it should help with io as well | 19:36 |
| sean-k-mooney | well its runing in ci again now so we will have fresh results | 19:37 |
| sean-k-mooney | ya it shoudl help in 2 ways (really one) since the io will be comperess we write less byte to disk, that also menas we can swap faster within a givne io budget | 19:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!