| *** pojadhav- is now known as pojadhav | 05:57 | |
| opendevreview | Andre Aranha proposed openstack/tempest master: Add multinode FIPS job to the experimental queue https://review.opendev.org/c/openstack/tempest/+/826580 | 13:16 |
|---|---|---|
| *** pojadhav is now known as pojadhav|afk | 14:02 | |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Apr 12 15:00:34 2022 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting | 15:01 |
| kopecmartin | agenda ^^ | 15:01 |
| frickler | \o | 15:01 |
| gmann | o/ | 15:01 |
| lpiwowar | hi o/ | 15:01 |
| soniya29 | hello | 15:02 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:02 |
| kopecmartin | none announcements from my side | 15:03 |
| kopecmartin | #topic Zed Priority Items progress | 15:03 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-zed-priority | 15:04 |
| kopecmartin | this ^^ is the etherpad for the priority items for the current cycle | 15:04 |
| kopecmartin | it's based on the discussions we've had during the PTG | 15:04 |
| kopecmartin | feel free to edit/add anything i forgot | 15:04 |
| kopecmartin | btw, is it only me who can't open any etherpad page in firefox? | 15:05 |
| lpiwowar | I have no issue opening the etherped | 15:06 |
| kopecmartin | lol, uncaught exception - ReferenceError: chrome is not defined | 15:06 |
| lpiwowar | Hmmm ... That's weird | 15:06 |
| kopecmartin | i guess the firefox available for mac is missing something , meh | 15:06 |
| lpiwowar | Yes, i guess | 15:07 |
| kopecmartin | anyway, check the priority items , see whether you're ok with them | 15:07 |
| kopecmartin | #topic OpenStack Events Updates and Planning | 15:09 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-zed-ptg | 15:09 |
| kopecmartin | agenda from the ptg ^^ | 15:09 |
| kopecmartin | gmann: where can we publish the recordings from our sessions? | 15:10 |
| kopecmartin | is there a specific service openstack uses? | 15:10 |
| gmann | kopecmartin: I think I have that and I will send to you | 15:10 |
| kopecmartin | okey, thanks | 15:10 |
| gmann | publishing I am not sure but we can send it on ML from google drive or so | 15:10 |
| kopecmartin | yeah, that's what i meant by publishing - where to upload that, preferably | 15:11 |
| kopecmartin | ack | 15:11 |
| kopecmartin | #topic Gate Status Checks | 15:12 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade+OR+project:openstack/hacking) | 15:12 |
| kopecmartin | no changes there | 15:12 |
| kopecmartin | oh, i've used the old link :/ .. hacking shouldn't be part of that link anymore | 15:13 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:13 |
| kopecmartin | periodic stable | 15:13 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&pipeline=periodic-stable | 15:13 |
| kopecmartin | periodic master | 15:13 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:13 |
| * kopecmartin checking out the timeout in tempest-all job | 15:14 | |
| kopecmartin | hmm, tests passed | 15:15 |
| kopecmartin | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tempest/playbooks/devstack-tempest.yaml@master] | 15:15 |
| kopecmartin | seems this failed that job | 15:15 |
| kopecmartin | anyway, it seems random but let's watch out if it happens again | 15:16 |
| kopecmartin | #topic Distros check | 15:16 |
| kopecmartin | centos 8/9 stream | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=tempest-full-py3-centos-8-stream&job_name=devstack-platform-centos-8-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:16 |
| kopecmartin | fedora | 15:17 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0 | 15:17 |
| kopecmartin | openEuler | 15:17 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-20.03-SP2+&skip=0 | 15:17 |
| kopecmartin | debian | 15:17 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0 | 15:17 |
| kopecmartin | centos 8 stream still failing | 15:18 |
| kopecmartin | centos 9 stream looks good | 15:18 |
| kopecmartin | debian too | 15:18 |
| kopecmartin | fedora and epenEuler are 50-50 | 15:18 |
| kopecmartin | #topic Sub Teams highlights (Sub Teams means individual projects under QA program) | 15:19 |
| kopecmartin | Changes with Review-Priority == +1 | 15:19 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade+OR+project:openstack/hacking) | 15:19 |
| kopecmartin | no changes there | 15:20 |
| kopecmartin | #topic Open Discussion | 15:20 |
| kopecmartin | anything for the open discussion? | 15:20 |
| frickler | hacking also doesn't have RP+1 ;) | 15:20 |
| kopecmartin | ack, i'll update this link too | 15:21 |
| frickler | maybe if someone is interested in building jammy images they can help with diskimage-builder | 15:21 |
| frickler | currently failing for some unknown (to me) reason https://review.opendev.org/c/openstack/diskimage-builder/+/836228 | 15:22 |
| frickler | I've tested locally and devstack deploys fine. though there seem to be issues with eventlet and py3.10 | 15:22 |
| frickler | actually, is fedora using py3.10 already? maybe the failures there could be related then? | 15:25 |
| frickler | I know we use fedora for tox-py310 jobs | 15:25 |
| frickler | yep, seems so: PYTHON3_VERSION=3.10 https://0040485a3ab53b03d16a-74c6967b9c388ca77dd0a83c1c662326.ssl.cf2.rackcdn.com/837207/1/check/devstack-platform-fedora-latest/3b8d081/job-output.txt | 15:26 |
| gmann | humm, interesting | 15:27 |
| kopecmartin | it failed on uploading image, well timeouted | 15:28 |
| frickler | will be interesting to see whether jammy shows the same failure rate when we can finally run it | 15:28 |
| kopecmartin | yeah | 15:30 |
| kopecmartin | anything else? | 15:32 |
| kopecmartin | #topic Bug Triage | 15:32 |
| kopecmartin | new etherpad for this cycle | 15:32 |
| kopecmartin | #link https://etherpad.openstack.org/p/qa-bug-triage-zed | 15:32 |
| kopecmartin | gmann: btw, this looks ready | 15:33 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/833750 | 15:33 |
| gmann | kopecmartin: +1, thanks | 15:34 |
| gmann | will check after meeting | 15:34 |
| kopecmartin | ack, thanks | 15:34 |
| kopecmartin | this is all from my side | 15:34 |
| kopecmartin | anything else from anyone? | 15:34 |
| frickler | kopecmartin: do you keep those bug number in some kind of spreadsheet? | 15:34 |
| frickler | might be interesting to make some graphs from them like per cycle | 15:35 |
| frickler | bug counts | 15:35 |
| kopecmartin | it's just in the etherpad | 15:35 |
| kopecmartin | i can copy the data from there to a spreadsheet | 15:35 |
| kopecmartin | why not | 15:35 |
| gmann | I think it is counted from LP only | 15:35 |
| gmann | that is what I used to do and out in etherpad | 15:36 |
| frickler | if we wouldn't just be thinking to retire ethercalc, I would've propose to track the numbers there instead | 15:37 |
| kopecmartin | i see | 15:38 |
| kopecmartin | ok, we can use google spreadsheet | 15:38 |
| kopecmartin | the UI is nicer :) | 15:39 |
| frickler | but it's not open ... but yeah | 15:39 |
| kopecmartin | ok, let me try to copy the bug numbers from the etherpads to something else where we can generate some graphs | 15:39 |
| kopecmartin | thank you everyone, see you next week | 15:40 |
| kopecmartin | #endmeeting | 15:40 |
| opendevmeet | Meeting ended Tue Apr 12 15:40:30 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:40 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2022/qa.2022-04-12-15.00.html | 15:40 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2022/qa.2022-04-12-15.00.txt | 15:40 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2022/qa.2022-04-12-15.00.log.html | 15:40 |
| frickler | ah, I almost forgot, I'll be off next week. or actually tomorrow until the 20th | 15:40 |
| kopecmartin | frickler: ack | 15:42 |
| kopecmartin | gmann: also this patrole change https://review.opendev.org/c/openstack/patrole/+/835743 | 15:57 |
| gmann | kopecmartin: ack | 15:58 |
| gmann | it is back to back 4 meeting for me this morning, will do after I finish those | 15:58 |
| kopecmartin | of course, i didn't mean to rush you, i'm just going through everything ... | 15:59 |
| *** pojadhav|afk is now known as pojadhav | 16:14 | |
| opendevreview | Luigi Toscano proposed openstack/devstack-plugin-ceph master: [DNM][CI] Add CEPHADM_DEPLOY flag to py3 tests https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/834223 | 16:17 |
| dansmith | kopecmartin: do you know why this is not collecting the performance.json file that I'm generating? https://review.opendev.org/c/openstack/devstack/+/837139 | 16:41 |
| dansmith | it's running that section of the post playbook, so it should be generating the actual file, but it doesn't get collected for the log dump | 16:42 |
| *** dmellado_ is now known as dmellado | 16:46 | |
| *** artom_ is now known as artom | 17:01 | |
| opendevreview | Dan Smith proposed openstack/devstack master: WIP: Attempt to dump perf stats after tempest https://review.opendev.org/c/openstack/devstack/+/837139 | 17:09 |
| gmann | dansmith: I think its order of the 'Generate statistics' and 'fetch-devstack-log-di' role. laster one happening first and that time performance.json was not present. https://zuul.opendev.org/t/openstack/build/874e0a219bf24ec18c1884fe9327b777/console | 17:10 |
| dansmith | gmann: okay but I'm running right after "link post-devstack tempest.log" right? so I expected that to be the right time | 17:11 |
| gmann | dansmith: I think add 'Generate statistics' in 'pre_tasks' should work | 17:11 |
| gmann | dansmith: that is after log copy things again. | 17:11 |
| dansmith | oh I assumed "pre" was before the run | 17:12 |
| dansmith | I need this to run after devstack and after tempest | 17:12 |
| dansmith | and yeah, those pre things run super early, so I don't think that's right, unless I'm missing something | 17:13 |
| dansmith | or do you mean pre_tasks in post.yaml ? | 17:13 |
| gmann | not pre.yaml but 'pre_tasks' of that playbook so that in post.yam task run first then roles? | 17:13 |
| gmann | yeah pre_tasks in post.yaml | 17:14 |
| dansmith | huh, okay, why is "link post-devstack tempest.log" running after we've collected logs? | 17:14 |
| opendevreview | Dan Smith proposed openstack/devstack master: WIP: Attempt to dump perf stats after tempest https://review.opendev.org/c/openstack/devstack/+/837139 | 17:14 |
| dansmith | also, where are the roles like fetch-devstack-log-dir defined? because it might be nice to use a role here instead, but I haven't found where those live | 17:16 |
| gmann | dansmith: that is here https://github.com/openstack/devstack/tree/master/roles/fetch-devstack-log-dir | 17:17 |
| * dansmith facepalms | 17:17 | |
| dansmith | sorry :( | 17:17 |
| dansmith | neither google nor codesearch would show me that | 17:18 |
| clarkb | the issue there is they all primarily search file contents not file names/paths | 17:19 |
| gmann | yeah may be dir name are not searched that way. but all devstack and tempest roles are in tree only | 17:19 |
| clarkb | you can ask codesarch to filter by path but you still have to give it a tring to look for and .* isn't quick (in fact I worry I may have just made it very sad trying that) | 17:20 |
| dansmith | ack, well, my grepping suffered the same fate | 17:20 |
| dansmith | I should have tree | grep'd | 17:20 |
| gmann | kopecmartin: sent QA PTG recording in email to you but sorry I lost QA team photo too due to my system crashed during PTG. | 17:22 |
| opendevreview | Benny Kopilov proposed openstack/tempest master: Allows to skip wait for volume create https://review.opendev.org/c/openstack/tempest/+/837603 | 17:38 |
| opendevreview | Benny Kopilov proposed openstack/tempest master: Allows to skip wait for volume create https://review.opendev.org/c/openstack/tempest/+/837603 | 17:41 |
| opendevreview | Merged openstack/patrole master: Update tempest to 30.0.0 https://review.opendev.org/c/openstack/patrole/+/835743 | 17:43 |
| opendevreview | Merged openstack/devstack-plugin-nfs master: Enable volume revert to snapshot NFS tests https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/812716 | 17:58 |
| opendevreview | Luigi Toscano proposed openstack/devstack-plugin-ceph master: [DNM][CI] Add CEPHADM_DEPLOY flag to py3 tests https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/834223 | 18:41 |
| opendevreview | Luigi Toscano proposed openstack/devstack-plugin-ceph master: [DNM][CI] Add CEPHADM_DEPLOY flag to py3 tests https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/834223 | 19:40 |
| dansmith | gmann: yass: https://af03dfc56dd1bea1c6a5-57b719e0009d4036c44d6542bd77bfc6.ssl.cf1.rackcdn.com/837139/11/check/tempest-full-py3/57baa39/controller/logs/performance.json | 20:09 |
| dansmith | gmann: thanks for the help, sorry for being dumb | 20:10 |
| gmann | nice. | 20:10 |
| gmann | np!, we should have better in-file comment/documentation on these things. | 20:10 |
| dansmith | oh, hmm, looks like I'm failing to read the apache log | 20:11 |
| dansmith | ah, it's in access_log not other_vhosts | 20:12 |
| opendevreview | Dan Smith proposed openstack/devstack master: WIP: Gather performance data after tempest https://review.opendev.org/c/openstack/devstack/+/837139 | 20:21 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: Deploy with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 | 20:34 |
| opendevreview | Clark Boylan proposed openstack/tempest master: Stop processing stackviz https://review.opendev.org/c/openstack/tempest/+/837628 | 21:43 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Disable c-bak by default in our CI jobs https://review.opendev.org/c/openstack/devstack/+/837630 | 21:44 |
| clarkb | That first change should be pretty safe to land considering the lack of dstat data. The second one is more for conversation as I think c-bak testing is limited but I could be wrong. I will WIP it and can remove the WIP if that makes sense to proceed with | 21:44 |
| clarkb | dansmith: ^ that sort of fell out of what I was looking at for memory use | 21:45 |
| dansmith | cool | 21:46 |
| johnsom | Lots-o-red on the zuul board. Devstack in having a hard time installing stuff. https://zuul.opendev.org/t/openstack/build/c86a5eafc4a4465bb5c6780892e6565d/log/job-output.txt#11326 | 21:53 |
| clarkb | looks like an issue installing keystone because pbr can't figure out the version. setuptools released 2 days ago | 21:57 |
| clarkb | tox updated yesterday too but devstack installs hsouldn't use tox | 21:57 |
| dansmith | saw a grenade failure like that too | 21:58 |
| johnsom | It's not just keystone, I am seeing other errors | 21:58 |
| dansmith | https://zuul.opendev.org/t/openstack/build/e9ff1ef3a7e84fa2b31a09b9af2a7057/log/job-output.txt#22091 | 21:58 |
| johnsom | One job had that on barbican-client too | 21:58 |
| clarkb | as a sanity check pbr hasn't updated in 2 months | 22:00 |
| clarkb | and pip is a month old | 22:01 |
| clarkb | if I run pip install -e /path/to/keystone locally I can't reproduce with latest setuptools and pip | 22:03 |
| clarkb | with python38 as well which matches the failure | 22:03 |
| clarkb | I'm not sure why this is happening | 22:05 |
| clarkb | is /opt/stack/keystone somehow not a valid package | 22:06 |
| johnsom | Earlier in the file it does a git show and there is a commit message there, so at least at checkout something was there | 22:09 |
| clarkb | https://zuul.opendev.org/t/openstack/build/d6c6c23f329a4d15b5f84e67aad971f8/log/job-output.txt#379-402 THe logging there is weird and seems to indicate we synchronize the repo twice? but later in the log it shows what it checked out to, yup that | 22:09 |
| johnsom | Just for the record, here is one for barbicanclient: https://zuul.opendev.org/t/openstack/build/e57c56b167fa40e78caa1fb1e7d7803b | 22:11 |
| clarkb | I think that may be an artifact of logging. Looking at the rendered console from ansibl eit seems to only show one push per repo | 22:12 |
| clarkb | in these installations we're installing from source so there is no sdist. Installing from source implies that git is present | 22:17 |
| clarkb | however I think if git isn't present we'd get errors like this as well. Or if git isn't able to emit version like things we can parse? | 22:17 |
| clarkb | fungi: ^ do these failures look familiar to you? | 22:19 |
| clarkb | fwiw I'm fairly certain git is installed because we do git pushed to repos on the host to sync up the test repos | 22:19 |
| clarkb | it looks like maybe centos jobs are ok but ubuntu-focal are not? | 22:20 |
| clarkb | http://changelogs.ubuntu.com/changelogs/pool/main/g/git/git_2.25.1-1ubuntu3.3/changelog maybe git updated? I don't know when that latest update actually got releaesd in a package, tryingto figure that out | 22:22 |
| clarkb | looks like git updated 2022-04-12 18:58 on our mirror so image builds after that would include it | 22:25 |
| johnsom | Hmm, dpkg log on an old working run and the failing one have the same git version: 1:2.25.1-1ubuntu3.2 | 22:26 |
| johnsom | Oh, no they don't | 22:26 |
| clarkb | ya looks like our focal images updated about 5 hour sago which is before our mirrors update to the newer git | 22:26 |
| clarkb | but it is possible they update git at runtime too | 22:27 |
| johnsom | Good: git 1:2.25.1-1ubuntu3.2 | 22:27 |
| johnsom | Bad: git 1:2.25.1-1ubuntu3.3 | 22:27 |
| johnsom | The git-man line tricked me | 22:27 |
| clarkb | ah ok so maybe it is related | 22:27 |
| clarkb | I'm trying to reproduce in a focal container now | 22:27 |
| clarkb | hrm not able to reproduce there either and that uses the newer git | 22:28 |
| clarkb | htough the newer git also did the clone. I suppos eit is possible that old git fetches a repo that new git doesn't like. But this seems very unlikel | 22:29 |
| clarkb | oh though that says "add a check to see that the top level directory is owned by the current user" maybe you need to be different users? Does devstack run as zuul but then create stuff for stack? I dunno I'm just throwing ideas out there | 22:30 |
| fungi | that sounds likely | 22:33 |
| fungi | or the initial repo copying results in zuul owning the repos | 22:33 |
| fungi | or it's between the stack and tempest accounts | 22:33 |
| clarkb | I'm going to push a new patchset to my tempest change above and put a hold on tempest-full-py3 | 22:33 |
| clarkb | then we can examine the node directly and try and debug where pbr is failing since my local reproduction all fail so far | 22:34 |
| fungi | i guess you mean your local reproductions all fail to fail? | 22:35 |
| opendevreview | Clark Boylan proposed openstack/tempest master: Stop processing stackviz https://review.opendev.org/c/openstack/tempest/+/837628 | 22:37 |
| clarkb | fungi: yes, but that is because my ubuntu docker container doesn't have multiple users and setting that p to match devstack/tempest/zuul is difficult without holding a node anyway (as I'm not sure whta exactly it looks like) | 22:38 |
| clarkb | I think holding is worthwhile now that we suspect a potential complicated interaction like that. But we'll see | 22:38 |
| opendevreview | Michael Johnson proposed openstack/devstack master: DNM: testing git versions https://review.opendev.org/c/openstack/devstack/+/837634 | 22:39 |
| fungi | agreed, yes | 22:39 |
| johnsom | Not 100% sure that will work, but trying to install the old version over top | 22:39 |
| clarkb | johnsom: I think it will work if apt can find that package version still which might not be the case since old packages get pruned. But we'll see | 22:40 |
| opendevreview | Michael Johnson proposed openstack/devstack master: DNM: testing git versions https://review.opendev.org/c/openstack/devstack/+/837634 | 22:44 |
| clarkb | if it is a difference in file permissions that could explain why tox jobs (which all run as zuul) don't seem to be affected | 22:44 |
| johnsom | Yeah, I don't think that would have worked. This version pulls the package off kernel.org and installs | 22:45 |
| clarkb | assuming that is the cause we could probably just chown as appropriately and move on | 22:45 |
| clarkb | but knowing that is the issue and knowing what to chown to are the first steps | 22:45 |
| clarkb | fungi: ssh root@104.130.253.94 for the node that hsould be held for me | 22:46 |
| ianw | <clarkb> with python38 as well which matches the failure -> isn't it python 3.6? | 22:52 |
| clarkb | ianw: this failure https://zuul.opendev.org/t/openstack/build/6ee7c6851ba0435280428d7c924d3088/log/job-output.txt#7361 was 3.8 | 22:53 |
| ianw | ahh, i was looking at https://zuul.opendev.org/t/openstack/build/c86a5eafc4a4465bb5c6780892e6565d/log/job-output.txt#11326 | 22:54 |
| clarkb | as is https://zuul.opendev.org/t/openstack/build/d6c6c23f329a4d15b5f84e67aad971f8/log/job-output.txt#6978 | 22:54 |
| ianw | ok, at least it's not something dropping 3.6 support, that was my initial worry | 22:54 |
| clarkb | the keystone repo is owned by stack:stack on the held node | 22:56 |
| clarkb | and bash ./stack.sh is running as the stack user | 22:56 |
| clarkb | oh but we probably sudo pip intsall -e because it is a global install | 22:58 |
| clarkb | yes we do | 22:58 |
| fungi | argh | 23:04 |
| fungi | and root != stack | 23:04 |
| clarkb | I've su'd to stack then run `sudo -H LC_ALL=en_US.UTF-8 SETUPTOOLS_USE_DISTUTILS=stdlib http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite python3.8 -m pip install -c /opt/stack/requirements/upper-constraints.txt -e /opt/stack/keystone` and that reproduces. Now I'll try chowning the repo to root:root and rerun | 23:04 |
| clarkb | ok it still fails so maybe not an ownership thing. | 23:05 |
| opendevreview | Michael Johnson proposed openstack/devstack master: DNM: testing git versions https://review.opendev.org/c/openstack/devstack/+/837634 | 23:05 |
| clarkb | Still not clear if this is reltaed to the git update though | 23:05 |
| clarkb | oh wait did my chown not recurse properly? | 23:06 |
| johnsom | Yeah, I forgot the sudo for the package install, so back into the queue | 23:07 |
| clarkb | ok that is it | 23:07 |
| clarkb | I chowned /opt/stack/keystone/keystone but needed to chown /opt/stack/keystone | 23:07 |
| clarkb | it has installed now as the stack user running the sudo pip install above after chowning the repo to root:root | 23:07 |
| clarkb | Still not sure what pbr is doing with git to determine versions that breaks, but this is probably a really good indication that the git update did break it | 23:08 |
| clarkb | and if I chown it back again it fails again. ok now to see if I can trace pbr's git calls | 23:09 |
| fungi | it wants to call git tag, rev-parse, and so on | 23:09 |
| clarkb | ['git', 'rev-parse', '--git-dir'] produces "fatal: unsafe repository ('/opt/stack/keystone' is owned by someone else)\nTo add an exception for this directory, call:\n\n\tgit config --global --add safe.directory /opt/stack/keystone\n" | 23:11 |
| clarkb | I'm not going to try and prescribe a fix now, but I think that tracks it down well. Probably the best thing is to chown the repos | 23:12 |
| clarkb | but I'm not sure if other git operations are performed by devstack itself and that will cause problems later | 23:12 |
| clarkb | I'm off the held node now if anyone else wants to look. My hacked up pbr is in place and keystone is installed, but if you try to reproduce you should be able to just with my extra debuging output from pbr | 23:15 |
| fungi | chown them to root:root? | 23:15 |
| clarkb | then if you chown the repo it will go away | 23:15 |
| clarkb | fungi: yes because the install is done with sudo to become root to do a global install | 23:15 |
| ianw | o add an exception for this directory, call: | 23:15 |
| clarkb | (all the more reason we should be using virtualenvs but what can you do) | 23:15 |
| fungi | that's what i figure you meant, just making sure | 23:15 |
| ianw | git config --global --add safe.directory %s | 23:15 |
| clarkb | ianw: yup thats in the string above too. The problem with that is I don't think we should be modifying user's global git configs in devstack | 23:16 |
| clarkb | but maybe that is ok if we say it is a destructive ci tool | 23:16 |
| clarkb | (which I guess it really is) | 23:16 |
| johnsom | Should we open a pbr bug to track this/collect information? | 23:16 |
| clarkb | johnsom: I don't think so. It isnlt a pbr bug | 23:16 |
| clarkb | maybe a bug against git? | 23:16 |
| johnsom | lol | 23:16 |
| fungi | well, this was intentional, cve came across oss-sec ml today about it | 23:17 |
| clarkb | fungi: yup the rare case when it is ok to break users | 23:17 |
| fungi | in short, if you run any git operations in a git tree owned by another user who has put malicious content in the config, then git will silently run arbitrary commands of their choosing as your account | 23:17 |
| fungi | and since it's become popular to just have your shell prompt or editor or file browser run git commands in the background as you roam around the system, it can happen without you even realizing you've called git | 23:18 |
| clarkb | but this isn't a pbr issue. pbr depends on git and git changes its behavior in a non backward compatible way to address a security issue. I think filing a bug against pbr sets an expectations that pbr maintainers (fungi myself etc) to fix it | 23:19 |
| fungi | right | 23:19 |
| fungi | i'm just saying, git upstream will consider the report #notabug | 23:19 |
| johnsom | yeah, I was just trying to find a place to collect info and track. | 23:19 |
| clarkb | johnsom: I think devstack, devstack is failing doing what it does as what it does is no longer valid (run an install out of a git repo owned by another user) | 23:20 |
| johnsom | Right, I don't think git is going to act here. adding "--unsecure" would be .... silly | 23:20 |
| clarkb | and maybe we make a note in PBR somewhere indicating that modern git has invalidated this use case | 23:20 |
| clarkb | (which transitively affects PBR) | 23:20 |
| ianw | i feel like there is a possibility we install a lot of things from zuul checkouts (zuul:zuul I guess) as non-zuul users ... | 23:23 |
| clarkb | ianw: I wonder if there is a flag that we can set on the command line per git command. That isn't really useful for pbr as it shouldn't assume a repo the user wants to install is trusted , though maybe that is implied since it will run python things too | 23:23 |
| clarkb | ianw: yes, though all of the the tox stuff seems to reliably use zuul for both the execution and the file ownership. Probably most things that use virtualenvs are ok | 23:23 |
| ianw | is that true that we shouldn't assume the repo is trusted? | 23:26 |
| clarkb | ianw: thats what I'm wondering about now that I've written it. If you are running a pip install against a repo there is an implicit trust there | 23:26 |
| ianw | i mean you're telling pip "install from this directory" ... if you don't trust it ... | 23:27 |
| clarkb | because you are asking to run python out of that repo to make it happen | 23:27 |
| clarkb | so ya if there is a flag that pbr could set that might work. I don't think we should do it via global static config but instead on a per git invocation basis | 23:27 |
| ianw | right, it's distinct from having a bash PS1 that does git things and accidentally cd'ing into it | 23:27 |
| clarkb | and maybe with a flag that enforces the stronger rules in case people want that | 23:28 |
| clarkb | https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables doesn't look like we can set env vars that map to arbitrary config values though | 23:29 |
| johnsom | FYI, not a surprise, but my DNM patch that re-installs git 2.25.1-1ubuntu3.2 does in fact get through the keystone install without issue. | 23:29 |
| clarkb | so would need to be a command line flag? | 23:29 |
| ianw | " or via the command line option `-c safe.directory=<path>`." | 23:31 |
| clarkb | ianw: oh where'd you find that? that didn't seem to be in my debugging output from git | 23:32 |
| clarkb | (but maybe I just missed it) | 23:32 |
| ianw | git show 8959555cee7 :) but it's in b/Documentation/config/safe.txt | 23:32 |
| clarkb | ah thanks | 23:32 |
| ianw | also the commands it's running, and the way it's running them, is probably immune to injections via files named "`now-i-own-you.sh`" | 23:34 |
| ianw | it's not going to be tricked into exec-ing any git output i wouldn't have thought | 23:35 |
| johnsom | This config setting is only respected when specified in a system or global config, not when it is specified in a repository config or via the command line option `-c safe.directory=<path>`. | 23:35 |
| ianw | which seems to be the security issue | 23:35 |
| johnsom | https://github.com/git/git/commit/8959555cee7ec045958f9b6dd62e541affb7e7d9 for those curious | 23:37 |
| ianw | bah, indeed. i should have read that sentence from the start | 23:37 |
| clarkb | oh ok so making a workaround in pbr wouldn't be possible | 23:38 |
| clarkb | we could document the behavior in pbr docs and indicate the settings to ste or to chown though | 23:38 |
| clarkb | in that case i htink we should make a devstack bug since this needs to be fixed at the pip install point | 23:38 |
| johnsom | +1 | 23:38 |
| johnsom | I guess I started this thread, so I will open one. (unless one of you has already started one) | 23:40 |
| clarkb | I have not . Idon't think I'm logged into lp either (though does devstakc use lp?) | 23:40 |
| clarkb | I'm happy for you to do it :) | 23:40 |
| clarkb | and then email to the discuss list is probably also a good idea so that w edon't have duplicate debugging overnight from europe :) | 23:41 |
| johnsom | Yeah, it's LP | 23:41 |
| ianw | functions-common/git_clone seems like the place to do it, but i feel like there might be other places too. probably plugins doing things | 23:42 |
| clarkb | ianw: maybe even in the pip_install function | 23:42 |
| clarkb | before running pip chown to root then back again | 23:42 |
| clarkb | that seems hacky but is likely to catch most cases | 23:43 |
| ianw | that seems horrific tbh | 23:43 |
| clarkb | or maybe its time to resurrect the global virtualenv install and not do root installs | 23:43 |
| ianw | most would probably say it fits right in with devstack/global pip/setuptools workarounds etc. :) | 23:44 |
| clarkb | but I ran out of steam ttrying to make that work the last time I pushed on it (for the pip cannot unintsall this dependency problem) | 23:44 |
| fungi | i think the real problem becomes local installs where the repo needs to be owned by root for devstack to do installs, but otherwise gets operated in by a different account | 23:45 |
| fungi | so probably can't just leave it root-owned | 23:47 |
| ianw | yeah, i wouldn't think too many things would be modifying /opt/devstack/<repo> post install | 23:48 |
| fungi | i guess devstack could clone to yet another temporary path, chown that, install from it, and clean it back up | 23:48 |
| clarkb | ya post intsall seems very unlikely | 23:48 |
| clarkb | one exception maybe grenade | 23:48 |
| fungi | i don't mean in ci jobs | 23:48 |
| clarkb | but I think we still use two copies of the repo for that rather than doing a checkuot | 23:48 |
| clarkb | fungi: oh right local dev iterations | 23:49 |
| clarkb | so ya we probably do need to switch ownership back or use a temporary shallow clone or something | 23:49 |
| clarkb | (temporary shallow lcone may break pbr though) | 23:49 |
| fungi | shallow would, yeah | 23:49 |
| clarkb | fungi: any idea if there are multiple CVEs for this? the one on the ubuntu package update specifically indicates git for windows which may give people the wrong idea | 23:51 |
| clarkb | er it indicates that on mitre.org | 23:51 |
| opendevreview | Ian Wienand proposed openstack/devstack master: chown checkout out repos for install to root https://review.opendev.org/c/openstack/devstack/+/837636 | 23:51 |
| ianw | ^ that's just something to get started ... not an ending point .... | 23:51 |
| johnsom | https://bugs.launchpad.net/devstack/+bug/1968798 | 23:52 |
| fungi | clarkb: that's the only cve i know about, the git authors used windows examples because everyone uses windows these days | 23:52 |
| johnsom | Please feel free to add your $0.02 | 23:52 |
| clarkb | fungi: heh ok | 23:52 |
| clarkb | johnsom: I think that covers it well. Thanks! | 23:53 |
| johnsom | +1 working on an email now | 23:54 |
| fungi | clarkb: i'm guessing it's that they figure windows users need more hand-holding, while *nix users can read between the lines | 23:54 |
| ianw | what about group ownership? if root and stack share a group, and it has permissions, is that sufficient? | 23:55 |
| clarkb | ianw: ya will be good to see that change go green indicating we have a sufficient workaround | 23:55 |
| fungi | clarkb: the example was taken from the commit message for the patch, authored by one of the git-for-windows maintainers | 23:56 |
| clarkb | ianw: let me hop onto my held machine and try to check that | 23:56 |
| ianw | it calls "is_path_owned_by_current_user" | 23:57 |
| ianw | git-compat-util.h:#define is_path_owned_by_current_user is_path_owned_by_current_uid | 23:57 |
| ianw | it does a lstat on the path then checks "st.st_uid == geteuid" | 23:58 |
| ianw | so it really is a strict "owned by this user" | 23:58 |
| ianw | i feel like we might be on the leading edge of a lot of general annoyance over this :/ | 23:59 |
| clarkb | ya usermod -a -G stack root then logging out and back in just to be double sure it applied and checking with groups to confirm then switching back over to stack and running the sudo pip install while the repo is stack:stack owned still fails which confirms your analysis | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!