| wxy-xiyuan | Once nodepool-builder use diskimage-buidler>=3.28.0, the openEuler CI job timeout problem will be fixed I think. Let's wait more. | 00:54 |
|---|---|---|
| wxy-xiyuan | ] | 06:07 |
| *** jpena|off is now known as jpena | 07:42 | |
| *** ykarel_ is now known as ykarel | 13:21 | |
| opendevreview | Merged openstack/devstack master: Add config options for cinder nfs backend https://review.opendev.org/c/openstack/devstack/+/868672 | 14:20 |
| *** jpena is now known as jpena|off | 16:33 | |
| dansmith | clarkb: could we have some way to not even start the expensive jobs until the linter job(s) pass? | 16:48 |
| dansmith | I am certainly not without guilt here, submitting patches that fail linters, but.. maybe we could improve behavior a bit by saying "we don't run your stuff on devstack if it failed pep8" | 16:49 |
| clarkb | dansmith: yes zuul allows you to define the jobs in a pipeline such that one or more must pass before others start. We've looked at it in the past and found that often that creates more round trips since things like pep8 are often not fatal | 16:51 |
| dansmith | clarkb: yeah, non-fatal technically, but we could call it fatal for our purposes | 16:52 |
| clarkb | right. The idea is/was that pep8 could fail and the unittests and devstack all on different things and you could correct the three in a single new patchset rather than two or three. I think this made sense because we expected people to at least run linters pre push (and I'll admit I skip it sometimes too when I'm like this is fine and find out nope) | 16:53 |
| dansmith | I have shamelessly submitted patches (usually for configs that I can't easily replicate locally) that fail pep8 for SyntaxError or other fatal things and don't realize until later | 16:53 |
| dansmith | probably not very common, but.. we *could* say you have to pass pep8 to run the expensive stuff | 16:53 |
| dansmith | yep, it's always making a tiny change I'm sure of that I fail :D | 16:54 |
| dansmith | anyway, just curious if it was an option | 16:54 |
| JayF | I've even pushed patches which I remembered to run (and make them pass!) unit tests, then realized post-push I didn't do lint ... I should get the pre-commit-hooks Sean was talking about | 18:36 |
| opendevreview | Artom Lifshitz proposed openstack/whitebox-tempest-plugin master: WIP/DNM: try using barbican client from their plugin https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/878711 | 18:46 |
| sean-k-mooney | JayF: for what its worth reboot on centos 9 stream is broken i belive becasue of the apache <=> uwsgi interaction | 19:50 |
| sean-k-mooney | curl -g -i -X GET http://localhost/identity/v3/ -H "Accept: application/json | 19:50 |
| sean-k-mooney | gives a 503 | 19:50 |
| sean-k-mooney | but keystone steams to be fine looking at journalctl output so if that was fixed it proably would work | 19:51 |
| clarkb | what is the uwsgi interaction? | 19:51 |
| JayF | I avoid CentOS Stream tbh, so I don't think I'd hit that one. | 19:51 |
| sean-k-mooney | i think its really the proxy behavior | 19:52 |
| sean-k-mooney | clarkb: basically sudo journalctl -u devstack@keystone -xe seams to show keystone working away fine | 19:53 |
| sean-k-mooney | but if i curl it i just get a 503 | 19:53 |
| clarkb | right so apache is saying the service isn't ready for some reason | 19:55 |
| clarkb | apache should log why somewhere | 19:55 |
| sean-k-mooney | ya i didnt see anything in the error log or access log but im sure its somewhere | 19:55 |
| sean-k-mooney | i dont think this happens on ubuntu for what its worht | 19:56 |
| clarkb | sean-k-mooney: is there a keystone specific log file? I want to say there is | 19:56 |
| clarkb | keystone specific apache log file I mean | 19:56 |
| sean-k-mooney | [stack@controller ~]$ sudo ls /var/log/httpd/ | 19:57 |
| sean-k-mooney | access_log error_log horizon_access.log horizon_error.log ssl_access_log ssl_error_log ssl_request_log | 19:57 |
| sean-k-mooney | not that i can see i would assume just in the error_log | 19:57 |
| sean-k-mooney | this is not super important i was just looking at this becasue fo the tc converstation | 19:57 |
| clarkb | grepping in devstack I see `ErrorLog /var/log/%APACHE_NAME%/keystone.log` | 19:58 |
| clarkb | but maybe that template isn't in use | 19:58 |
| clarkb | yup thats in the wsgi config for mod_wsgi and uwsgi uses something else | 19:59 |
| sean-k-mooney | thats the wsgi config for keystone https://termbin.com/2r5z | 20:02 |
| sean-k-mooney | from /etc/keystone/keystone-uwsgi-public.ini | 20:02 |
| sean-k-mooney | i dont see any extra log cofnig | 20:02 |
| sean-k-mooney | and the apache one is tiny | 20:02 |
| clarkb | is the socket "file" present? | 20:03 |
| sean-k-mooney | https://termbin.com/oyhzo | 20:04 |
| sean-k-mooney | ill check | 20:04 |
| sean-k-mooney | yep its in /var/run/uwsgi/ | 20:04 |
| clarkb | just talking out loud here: I wonder if apache starts before the uwsgi service creates the socket it caches the lack of the socket and returns errors? | 20:04 |
| clarkb | might be worth doing a `systemctl restart httpd` to see if that changes naything | 20:05 |
| sean-k-mooney | actully apache does not start automaticallly | 20:05 |
| sean-k-mooney | i started it myself | 20:05 |
| sean-k-mooney | but i can stop and start it again | 20:05 |
| clarkb | easy to try but seems unlikely to be a race in service startup given that | 20:05 |
| sean-k-mooney | i know tha the wsgi dir used to be a probelm but that was fixed with tmpfiles.d i think | 20:06 |
| clarkb | is /var/run a tmpfs too? that could be am ounting order of operations | 20:07 |
| sean-k-mooney | tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=1574272k,nr_inodes=819200,mode=755,inode64) | 20:07 |
| sean-k-mooney | i dont see var/run but ill check again | 20:07 |
| sean-k-mooney | ah its a simlink which is what i was expecting | 20:08 |
| sean-k-mooney | lrwxrwxrwx. 1 root root 6 Aug 25 2022 /var/run -> ../run | 20:08 |
| sean-k-mooney | i think this is a routing thing | 20:08 |
| sean-k-mooney | when devstack deploys it moves the ip to an ovs bridge | 20:09 |
| sean-k-mooney | after the reboot it goes back to eth0 or whatever | 20:09 |
| sean-k-mooney | hum but actully that does nto really make sense | 20:09 |
| sean-k-mooney | that should not casue a 503 | 20:09 |
| clarkb | ya 503 should mean no access to the backing service or running out of threads | 20:09 |
| clarkb | since the backend service is on a socket and not via tcp it shouldn't care about eth0 | 20:10 |
| sean-k-mooney | lol | 20:10 |
| sean-k-mooney | ok its working now want to guess | 20:10 |
| sean-k-mooney | [stack@controller devstack]$ openstack service list | nc termbin.com 9999 | 20:11 |
| sean-k-mooney | https://termbin.com/y92d6 | 20:11 |
| sean-k-mooney | clarkb: it was Selinux... | 20:11 |
| sean-k-mooney | set it to permissive and it worked | 20:12 |
| sean-k-mooney | im going to change that in the config and reboot again and see if it works | 20:12 |
| clarkb | huh did rebooting switch it from permissive to enforcing? | 20:12 |
| clarkb | ++ | 20:12 |
| sean-k-mooney | yep we only disable it at runtime in devstack | 20:12 |
| sean-k-mooney | we dont persit it to the file | 20:12 |
| sean-k-mooney | ok works so only two thing are misisng for the fedora famaily of distors | 20:16 |
| sean-k-mooney | disable selinux in the file. and enable httpd so it starts on reboot | 20:16 |
| sean-k-mooney | JayF: ^ | 20:16 |
| sean-k-mooney | i can proably hack a pacth up for that quickly | 20:16 |
| sean-k-mooney | that also makes sense why it works on ubuntu apache2 is enable by default when you install and apparmor does not block access to uwsgi sockets | 20:17 |
| sean-k-mooney | as a side note this si a 2 node devstack i deploy with a molecue evn using vagrat/libvirt during the tc call in 1 command using the zull devstack roles | 20:18 |
| JayF | sean-k-mooney: clarkb: very nice find!!! | 20:19 |
| sean-k-mooney | https://github.com/openstack/devstack/blob/540ec94a043056d14a5e3eaff9ea776b5a2f67bb/tools/fixup_stuff.sh#L36-L40 | 20:21 |
| sean-k-mooney | so that just need to be replace with something that survives reboot | 20:21 |
| sean-k-mooney | crudini is prbably the best way to do that | 20:22 |
| sean-k-mooney | actully i think devstack uses iniset instead fo crudini driectly | 20:24 |
| sean-k-mooney | actully i can jsut use a heredoc | 20:24 |
| sean-k-mooney | now the real question is do i test this locally or push cause obviously its correct :P | 20:32 |
| JayF | just make sure to run lint ;) | 20:32 |
| sean-k-mooney | does devstack have ath i assume bashate | 20:33 |
| sean-k-mooney | hehe good thing i checked my indetation was broken in the heredoc | 20:36 |
| sean-k-mooney | 766 second that much better | 20:50 |
| sean-k-mooney | ok that seams to work | 20:51 |
| sean-k-mooney | to test this properly i would need a clean vm but i can try that tomorrow | 20:52 |
| opendevreview | sean mooney proposed openstack/devstack master: Fix reboot on fedora like nodes https://review.opendev.org/c/openstack/devstack/+/878718 | 20:58 |
| sean-k-mooney | ok that should do it ^ | 20:58 |
| *** spotz_ is now known as spotz | 23:34 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!