| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : remove deprecated NoOp function https://review.opendev.org/c/openstack/project-config/+/875804 | 00:18 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/project-config master: acls : handle submit requirements in normalise tool https://review.opendev.org/c/openstack/project-config/+/875992 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : convert NoBlock to submit-requirements https://review.opendev.org/c/openstack/project-config/+/875993 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : handle key / values with multiple = https://review.opendev.org/c/openstack/project-config/+/875994 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Update Review-Priority to submit-requirements https://review.opendev.org/c/openstack/project-config/+/875995 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Convert remaining AnyWithBlock to submit requirements https://review.opendev.org/c/openstack/project-config/+/875996 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : disallow "function" https://review.opendev.org/c/openstack/project-config/+/875997 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : submit-requirements for deprecated NoOp function https://review.opendev.org/c/openstack/project-config/+/875804 | 05:48 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : add submit requirements to NoBlock labels https://review.opendev.org/c/openstack/project-config/+/875993 | 05:48 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : handle key / values with multiple = https://review.opendev.org/c/openstack/project-config/+/875994 | 05:48 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Update Review-Priority to submit-requirements https://review.opendev.org/c/openstack/project-config/+/875995 | 05:48 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Convert remaining AnyWithBlock to submit requirements https://review.opendev.org/c/openstack/project-config/+/875996 | 05:48 |
| *** jpena|off is now known as jpena | 08:27 | |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add the main NebulOuS repo https://review.opendev.org/c/openstack/project-config/+/876054 | 10:15 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add the main NebulOuS repo https://review.opendev.org/c/openstack/project-config/+/876054 | 10:31 |
| opendevreview | Merged openstack/openstack-zuul-jobs master: openstack-tox-functional: pre-install python packages https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/875732 | 13:45 |
| dansmith | dpawlik7: around? I'm looking for some field in opensearch that indicates what provider a job ran on.. is that present somewhere or possible? | 15:51 |
| dpawlik7 | dansmith: hey, yes I'm here | 15:52 |
| dpawlik7 | you mean on which host the job was executed? | 15:52 |
| dansmith | well, host might provide that information, but I mean which provider (i.e. rax, etc) | 15:53 |
| dansmith | (I don't see hostname either) | 15:53 |
| dpawlik7 | are you on the logstash-* index? | 15:54 |
| dansmith | I mean, it says _index=logstash-logs-2023-03-02 | 15:55 |
| dansmith | ah on the top left, yeah | 15:55 |
| dansmith | logstash-* | 15:55 |
| dpawlik7 | so each doc should contain "hosts_id" | 15:56 |
| dansmith | yeah that field seems empty | 15:57 |
| dpawlik7 | and that's only what logscraper can take | 15:57 |
| dpawlik7 | could you share _id or build_uuid | 15:57 |
| dansmith | dpawlik: I don't see it for *any* build | 15:58 |
| dansmith | I'm not examining one build | 15:58 |
| dpawlik | I know what is your issue | 16:01 |
| dpawlik | <searching> | 16:02 |
| dpawlik | tl;dr I guess you are using Private tenant, not Global | 16:02 |
| dpawlik | you can switch it by clicking on the user menu (right top corner) | 16:03 |
| dansmith | I'm in global | 16:04 |
| dansmith | I just selected and re-confirmed, but global was already selected | 16:04 |
| dansmith | well, after doing that, it seems hung up | 16:05 |
| dansmith | ah there we go | 16:05 |
| dansmith | and how I see host_id | 16:05 |
| dpawlik | could you share the url? | 16:05 |
| dansmith | I don't get it | 16:05 |
| dpawlik | there is something like share and make a "snapshot"of your current state I will check what's happening | 16:06 |
| dansmith | okay so hosts_id is now present, but it's an opaque identifier | 16:06 |
| dpawlik | oh, so you see the document details now? | 16:06 |
| dansmith | it looks the same as I was seeing before, but hosts_id is now non-empty | 16:07 |
| dansmith | although, | 16:07 |
| dansmith | ah okay I think maybe I see the problem | 16:07 |
| dansmith | I'm looking for build_status=TIMED_OUT situatons, | 16:07 |
| dansmith | and those all seem to not have hosts_id set | 16:08 |
| dpawlik | yhym. So if you saw earlier something that does not contain any hosts_id probably the job got NODE_FAILURE or something else | 16:08 |
| dansmith | https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2w,to:now))&_a=(columns:!(hosts_id),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:TIMED_OUT),type:phrase),query:(match_phrase:(build_status:TIMED_OUT)))),index:'94869730-aea8-11ec-9 | 16:08 |
| dansmith | e6a-83741af3fdcd',interval:d,query:(language:kuery,query:''),sort:!()) | 16:08 |
| dpawlik | and Zuul does not provide any details.In that case, it will push logs what it just knows | 16:08 |
| dansmith | we're having lots of test that run almost to completion, hit TIMED_OUT and logs do get uploaded like normal | 16:09 |
| dansmith | so I'm trying to survey which node providers those are coming from | 16:09 |
| dpawlik | and it seems that the data in Opensearch is correct. Zuul does not provide any info. Example: https://d9812591adc5f2fe5004-b9f7b4910ad16729218670d60db15b93.ssl.cf1.rackcdn.com/875676/1/gate/neutron-tempest-plugin-api-wallaby/c218afc/zuul-info/inventory.yaml | 16:10 |
| dpawlik | aha | 16:10 |
| dpawlik | <wait> | 16:10 |
| dpawlik | so how host_id look like Zuul 8.1.1 https://softwarefactory-project.io/logs/33/27633/1/check/sf-operator-functional-allinone-crc/42ab788/zuul-info/inventory.yaml - there is host_id | 16:16 |
| dansmith | cc clarkb fungi ^ | 16:16 |
| dpawlik | and in the inventory.yaml which I paste that info is in different place | 16:16 |
| dansmith | dpawlik: even still that hosts_id looked like a uuid or something.. how do I turn that into an actual provider? or maybe that's more an infra question | 16:17 |
| dpawlik | dansmith: I will check that thing tomorrow. I'm in the middle of one meeting | 16:17 |
| dpawlik | and I would like to finish today in few minutes | 16:18 |
| dpawlik | I will check logscraper logs related to the build_uuid that is broken | 16:18 |
| dansmith | okay | 16:18 |
| dansmith | dpawlik: thanks | 16:18 |
| dpawlik | why it does not provide that information | 16:18 |
| dpawlik | about how to recognize infra on which cloud provider it was spawned: by the ip address | 16:19 |
| dpawlik | I guess the Opendev infra got network pools that are in each cloud provider | 16:19 |
| dpawlik | each cloud provider gives some dedicated public network,so with CIDR should be easy to recognize | 16:20 |
| dansmith | hrm, seems like it would be nice to have either a hostname, domain, or provider id in there, but maybe that needs to be in inventory.yaml from zuul? | 16:20 |
| dansmith | if I'm trying to figure out which provider has the most timeouts, correlating by IP would be a lot harder :/ | 16:20 |
| dpawlik | and in the log that got "TIMED_OUT" - https://d9812591adc5f2fe5004-b9f7b4910ad16729218670d60db15b93.ssl.cf1.rackcdn.com/875676/1/gate/neutron-tempest-plugin-api-wallaby/c218afc/zuul-info/inventory.yaml there is "cloud: rax" | 16:21 |
| dpawlik | I can add that field to the Opensearch if its necessary | 16:21 |
| dansmith | yeah, could we get .. yeah that :) | 16:21 |
| dansmith | that's what I want | 16:21 |
| dansmith | dpawlik: or maybe provider, which is rax-$region | 16:21 |
| dpawlik | allright. Will provide the patch tomorrow | 16:21 |
| dpawlik | ok | 16:21 |
| dansmith | dpawlik: thanks a lot | 16:21 |
| fungi | dpawlik: no, many of our providers are just shared provider network, no specific allocations for us | 16:22 |
| fungi | you can't easily tell which provider an ip address belongs to without using whois | 16:22 |
| dansmith | yeah, provider or cloud is all we need in opensearch I think | 16:22 |
| clarkb | dpawlik: its not accurate to say zuul provides no info its right there in the file you linked | 16:23 |
| clarkb | nodepool did change the way it names hosts because some systems/clouds/maybe both can't do long hostnames | 16:23 |
| clarkb | so rather than do cloud-region-largenumber its just npXYZ... | 16:23 |
| clarkb | but the information is still there in the inventory file which you linked | 16:24 |
| dpawlik | clarkb: right. I will check why that info was not provided with logsender. | 16:28 |
| dpawlik | need to go. See you | 16:28 |
| fungi | all.hosts.[hostname].nodepool.provider you'll have the provider id included for it | 16:29 |
| fungi | note that the data structure makes it possible for different hosts in the same build to have different providers, but for practical purposes you can probably just grab the first host in the list | 16:30 |
| ade_lee_ | clarkb, hey - you can remove that node that was held | 16:47 |
| clarkb | ade_lee_: thanks I'll get to that as soon as I load my ssh keys | 16:47 |
| clarkb | ade_lee_: out of curiousity did you sort out what the issue was? | 16:47 |
| ade_lee_ | clarkb, fungi fyi - the ubuntu job is now running under fips and is getting to the point where the tempest tests are running, some are failing - but at least they're all running now | 16:48 |
| ade_lee_ | clarkb, yeah - you need to set a special fips-mode setting on both the client and server side for mysql | 16:48 |
| ade_lee_ | so it knows to use fips algorithms when setting up ssl connections | 16:49 |
| ade_lee_ | clarkb, https://review.opendev.org/c/openstack/devstack/+/871606/13/lib/databases/mysql#92 | 16:50 |
| ade_lee_ | I was setting the server, but not the client side -- you need both | 16:50 |
| clarkb | got it | 16:50 |
| *** bhagyashris|ruck is now known as bhagyashris | 16:51 | |
| clarkb | ade_lee_: ok autohold-delete has been issued. It should clean up the node pretty quickly | 16:53 |
| ade_lee_ | clarkb, cool - thanks! | 16:53 |
| fungi | ade_lee_: that's awesome progress! | 16:59 |
| *** hberaud_ is now known as hberaud | 17:08 | |
| *** jpena is now known as jpena|off | 17:47 | |
| *** dpereira_ is now known as dpereira | 17:49 | |
| opendevreview | Felipe Reyes proposed openstack/project-config master: Add Ironic Dashboard charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/876205 | 19:15 |
| opendevreview | Felipe Reyes proposed openstack/project-config master: Add Ironic Dashboard charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/876205 | 19:43 |
| JayF | Is there a guide somewhere about how to move a project from x/ -> official openstack? Just a governance change or are there actual-infrastructural-steps I need to take? | 19:43 |
| JayF | found https://docs.openstack.org/project-team-guide/repository.html -- it has what I need | 19:45 |
| fungi | JayF: yeah, basically rename it in project-config but we need to apply it with a gerrit outage to orchestrate the rename, so add it to the opendev weekly meeting agenda for scheduling since we try to batch them up in order to avoid having gerrit outages too frequently. also you can do governance stuff in parallel with that | 19:46 |
| JayF | OpenStack Bare Metal Program: your one stop shop for emulating BMCs and PDUs with libvirt :D | 19:47 |
| fungi | the tc is generally okay with them being done in any order (adopt officially while still in x/ namespace, or move to openstack/ namespace in gerrit before official adoption is approved) | 19:47 |
| fungi | otherwise it's a sort of catch-22 no matter what order you did it in | 19:48 |
| clarkb | as mentioned in our last meeting tough I want to hold off all renames until all the giteas are replaced | 20:30 |
| clarkb | so it may be a few weeks (which is probably also fine because we don't want to impct openstack's release) | 20:30 |
| fungi | yep | 20:38 |
| JayF | I mean, does that mean the actual-better-move is for me to sit on this for a couple of weeks? | 20:53 |
| JayF | I don't think it's urgent, we have access, we can cut releases that work on new python | 20:53 |
| clarkb | JayF: no, we try to batch up renames (thee is already one listed) so listing yours is a good idea | 20:53 |
| JayF | so for governance/project-config changes, should I be referencing x/ or openstack/ right now? | 20:54 |
| JayF | the instructions exist but are slightly ambiguous | 20:54 |
| clarkb | in governance I would reference the future openstack/ location | 20:54 |
| JayF | in that case; I don't think there are any changes in project-config | 21:02 |
| JayF | x/virtualpdu was already in group: ironic | 21:02 |
| clarkb | you express the rename in project-config by changing the name of the project and then put the rename request on the meeting agenda (that should be documented iirc) | 21:03 |
| JayF | ack, so do the p-c change with openstack/ assuming it'll be merged at the proper time by you all | 21:04 |
| JayF | makes sense | 21:04 |
| clarkb | exactly | 21:04 |
| *** sean-k-mooney1 is now known as sean-k-mooney | 21:06 | |
| fungi | well, i think the governance change may be unable to merge if it references a repository which doesn't exist. probably best to check with the tc but in the past when we've done these they're adopted as x/virtualpdu in governance and then later governance is updated when the repository moves to the openstack/ namespace | 21:15 |
| clarkb | ah | 21:16 |
| JayF | clarkb: lint is ... yes, what fungi said | 21:17 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Ironic program adopting virtualpdu https://review.opendev.org/c/openstack/project-config/+/876231 | 21:55 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Ironic program adopting virtualpdu https://review.opendev.org/c/openstack/project-config/+/876231 | 22:17 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!