| *** zzzeek has quit IRC | 00:15 | |
| *** zzzeek has joined #openstack-glance | 01:11 | |
| *** zzzeek has quit IRC | 01:17 | |
| openstackgerrit | Takashi Kajinami proposed openstack/glance master: Remove osprofiler parameters from api-paste.ini https://review.opendev.org/c/openstack/glance/+/789095 | 01:30 |
|---|---|---|
| *** hamalq has joined #openstack-glance | 01:49 | |
| *** hamalq has quit IRC | 01:54 | |
| *** whoami-rajat has quit IRC | 02:49 | |
| *** hamalq has joined #openstack-glance | 03:50 | |
| *** hamalq has quit IRC | 03:55 | |
| *** ratailor has joined #openstack-glance | 04:40 | |
| *** m75abrams has joined #openstack-glance | 04:51 | |
| *** ratailor_ has joined #openstack-glance | 05:29 | |
| *** ratailor has quit IRC | 05:30 | |
| *** udesale has joined #openstack-glance | 05:34 | |
| *** hamalq has joined #openstack-glance | 05:51 | |
| *** hamalq has quit IRC | 05:56 | |
| *** ratailor__ has joined #openstack-glance | 05:57 | |
| *** ratailor_ has quit IRC | 06:00 | |
| *** ratailor_ has joined #openstack-glance | 06:13 | |
| *** ratailor__ has quit IRC | 06:16 | |
| *** hamalq has joined #openstack-glance | 06:27 | |
| *** hamalq has quit IRC | 06:31 | |
| *** hamalq has joined #openstack-glance | 06:31 | |
| *** k_mouza has joined #openstack-glance | 06:37 | |
| *** k_mouza has quit IRC | 06:41 | |
| *** Luzi has joined #openstack-glance | 07:01 | |
| *** whoami-rajat has joined #openstack-glance | 07:56 | |
| *** hamalq has quit IRC | 08:46 | |
| *** ricolin has joined #openstack-glance | 08:54 | |
| *** hamalq has joined #openstack-glance | 09:36 | |
| *** hamalq has quit IRC | 09:40 | |
| *** ralonsoh_ has joined #openstack-glance | 10:18 | |
| *** ralonsoh has quit IRC | 10:20 | |
| *** gokhani has joined #openstack-glance | 10:27 | |
| *** ratailor has joined #openstack-glance | 11:17 | |
| *** ratailor_ has quit IRC | 11:18 | |
| *** hamalq has joined #openstack-glance | 11:36 | |
| *** hamalq has quit IRC | 11:41 | |
| *** hamalq has joined #openstack-glance | 12:17 | |
| *** hamalq has quit IRC | 12:22 | |
| whoami-rajat | dansmith: so when getting an os-brick connector it traverses through all possible connectors[1], in the nvme connector we execute the command erroring out[2]. the strange thing is that directory should already exist since mkdir was executed in logs[3] | 12:47 |
| whoami-rajat | [1] https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connector.py#L226-L237 | 12:47 |
| whoami-rajat | [2] https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/nvmeof.py#L123-L125 | 12:47 |
| whoami-rajat | [3] https://zuul.opendev.org/t/openstack/build/74c3d110d12140c9ac39c2524b7a2668/log/controller/logs/screen-g-api.txt#14131 | 12:47 |
| *** stand has quit IRC | 13:08 | |
| *** ratailor has quit IRC | 13:24 | |
| openstackgerrit | Gonéri Le Bouder proposed openstack/glance master: doc: update the kvm kw_vif_model driver list https://review.opendev.org/c/openstack/glance/+/789281 | 13:26 |
| dansmith | whoami-rajat: I think that's complaining about the missing 'nvme' command, not /etc/nvme isn't it? | 13:36 |
| dansmith | so maybe we're missing something in devstack to get that installed? | 13:36 |
| whoami-rajat | dansmith: FileNotFoundError: [Errno 2] No such file or directory: 'nvme' | 13:37 |
| dansmith | right, but the command is nvme: "Command: nvme gen-hostnqn | tee /etc/nvme/hostnqn" | 13:38 |
| dansmith | the python side would only know if something was FileNotFound if it's looking for that exact thing.. it doesn't know what tee's error codes mean :) | 13:38 |
| dansmith | also "Exit code: -" means it didn't even run it, I expect, because it couldn't find the binary to run | 13:39 |
| whoami-rajat | ah, right, that can be... | 13:39 |
| whoami-rajat | dansmith: also we don't need nvme installed since the backend used is lvm, it uses the iscsi connector so we don't need the nvme anyway | 13:40 |
| dansmith | ack, so we should probably make brick not log trace after trace for a situation where we don't even need nvme right? :) | 13:40 |
| dansmith | presumably that means operators have this noise in their logs over and over | 13:41 |
| whoami-rajat | dansmith: I'm really not sure why there's a trace since we're not raising anything, just logging a warning https://github.com/openstack/os-brick/blob/4.3.1/os_brick/initiator/connectors/nvmeof.py#L131-L132 | 13:43 |
| dansmith | you probably need LOG.warning(.., exc_info=False) | 13:44 |
| dansmith | I think if you log warning from inside an exception handler you'll get the trace | 13:44 |
| dansmith | I'm not sure where that comes from, but I've seen it before, let me look for an example | 13:46 |
| whoami-rajat | i wasn't aware of that | 13:46 |
| dansmith | there must be something a little more obscure about it, but I'm struggling to remember | 13:47 |
| dansmith | hmm, well, the example I found doesn't have a lot of notes so I think it must have been discussed on IRC | 13:54 |
| dansmith | there's also a report that setting exc_info=False didn't fix the trace in this case, but we ultimately left it in | 13:54 |
| dansmith | so maybe it's something more obscure | 13:54 |
| whoami-rajat | ok, i can propose a try patch and see if the trace is still logged? | 13:58 |
| whoami-rajat | or you can if you want since it's your solution :) | 13:58 |
| dansmith | go ahead if you want.. but I will if you don't want to.. I'm looking through oslo.log right now and there's definitely some magic in there, so I kinda figure *something* is going on | 13:59 |
| whoami-rajat | ok let me propose it real quick | 14:00 |
| *** Luzi has quit IRC | 14:08 | |
| whoami-rajat | dansmith: https://review.opendev.org/c/openstack/os-brick/+/789289 | 14:09 |
| whoami-rajat | wasn't very quick | 14:09 |
| dansmith | whoami-rajat: thanks.. do we need a dnm against glance to poke that or does that show up in jobs that run against brick patches too? | 14:10 |
| whoami-rajat | dansmith: can you provide link of your DNM patch | 14:10 |
| dansmith | ah, yeah let me update it | 14:10 |
| whoami-rajat | dansmith: let me check if the current brick gate... | 14:10 |
| openstackgerrit | Dan Smith proposed openstack/glance master: DNM Test glance_store cinder attach changes https://review.opendev.org/c/openstack/glance/+/788963 | 14:11 |
| dansmith | should repro here ^ at least | 14:11 |
| dansmith | even without the trace, it seems bad for brick to just keep doing that (and even logging a one-line error) on a system where nvme is clearly not available/used | 14:12 |
| whoami-rajat | dansmith: i found this on older brick patch run and it just logs a warning in nova Apr 27 01:19:19.586450 ubuntu-focal-inmotion-iad3-0024322733 nova-compute[100300]: WARNING os_brick.initiator.connectors.nvmeof [None req-be373412-8763-46b9-bd56-74d14e957848 tempest-UpdateMultiattachVolumeNegativeTest-1721339940 tempest-UpdateMultiattachVolumeNegativeTest-1721339940-project] Could not generate host nqn: | 14:14 |
| whoami-rajat | [Errno 2] No such file or directory | 14:14 |
| dansmith | hmm, interesting | 14:15 |
| whoami-rajat | also i assume if nvme command is not found then this should be the trace and not the one seen in glance | 14:16 |
| whoami-rajat | Apr 27 01:19:19.586450 ubuntu-focal-inmotion-iad3-0024322733 nova-compute[100300]: Command: nvme gen-hostnqn | tee /etc/nvme/hostnqn | 14:16 |
| whoami-rajat | Apr 27 01:19:19.586450 ubuntu-focal-inmotion-iad3-0024322733 nova-compute[100300]: Exit code: - | 14:16 |
| whoami-rajat | Apr 27 01:19:19.586450 ubuntu-focal-inmotion-iad3-0024322733 nova-compute[100300]: Stdout: None | 14:16 |
| whoami-rajat | Apr 27 01:19:19.586450 ubuntu-focal-inmotion-iad3-0024322733 nova-compute[100300]: Stderr: None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory | 14:16 |
| whoami-rajat | and glance says FileNotFoundError | 14:16 |
| dansmith | so, I wonder if this is actually privsep logging it and not that place you noted the log.warning | 14:17 |
| *** hamalq has joined #openstack-glance | 14:18 | |
| *** BLZbubba has quit IRC | 14:18 | |
| whoami-rajat | I'm still not sure why it doesn't do it when nova calls brick | 14:21 |
| dansmith | yeah | 14:21 |
| *** hamalq has quit IRC | 14:22 | |
| *** gokhani has quit IRC | 14:43 | |
| *** udesale has quit IRC | 15:51 | |
| *** lbragstad_ is now known as lbragstad | 15:51 | |
| *** hamalq has joined #openstack-glance | 16:00 | |
| *** hamalq has quit IRC | 16:09 | |
| *** hamalq has joined #openstack-glance | 16:14 | |
| *** hamalq_ has joined #openstack-glance | 16:16 | |
| *** hamalq has quit IRC | 16:20 | |
| *** m75abrams has quit IRC | 16:35 | |
| *** gyee has joined #openstack-glance | 16:46 | |
| *** tkajinam has quit IRC | 16:48 | |
| *** k_mouza has joined #openstack-glance | 17:07 | |
| *** k_mouza has quit IRC | 19:36 | |
| *** gyee has quit IRC | 19:53 | |
| *** lbragstad has quit IRC | 19:54 | |
| *** priteau has quit IRC | 19:54 | |
| *** gyee has joined #openstack-glance | 19:55 | |
| *** lbragstad has joined #openstack-glance | 19:55 | |
| *** priteau has joined #openstack-glance | 19:55 | |
| *** k_mouza has joined #openstack-glance | 20:36 | |
| *** k_mouza has quit IRC | 20:41 | |
| *** whoami-rajat has quit IRC | 22:03 | |
| *** ralonsoh_ has quit IRC | 22:09 | |
| *** snapdeal has joined #openstack-glance | 22:14 | |
| *** snapdeal has quit IRC | 22:27 | |
| *** rcernin has joined #openstack-glance | 23:07 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!