Wednesday, 2021-07-07

opendevreviewwangzhenmeng proposed openstack/nova master: Three CPU parameters, mode, model and vendor_id, are added to flavor, some guest os do not support new CPU, so you need to specify a specific CPU when starting.  https://review.opendev.org/c/openstack/nova/+/79698603:31
opendevreviewwangzhenmeng proposed openstack/nova master: Add CPU parameters to flavor  https://review.opendev.org/c/openstack/nova/+/79698603:34
*** abhishekk is now known as akekane|home05:26
*** akekane|home is now known as abhishekk05:26
*** iury|holiday is now known as iurygregory05:53
*** bhagyashris_ is now known as bhagyashris|ruck08:37
*** whoami-rajat is now known as Guest8909:05
*** whoami-rajat__ is now known as whoami-rajat09:05
ralonsohhi folks, do you know what I need to set in libvirt for live-migration?09:24
ralonsohI'm having this error:09:25
ralonsohlibvirt.libvirtError: Secret not found: no secret with matching uuid '70447a0f-443f-4b30-a154-c75706973e08'09:25
sean-k-mooney[m]are you using ceph09:25
sean-k-mooney[m]normally you do not need to set anything in libvirt09:25
ralonsohyes I configured ceph09:26
ralonsohI'll check it again09:26
sean-k-mooney[m]you need to configure the ceph key ring as a secret but the devstack plugin does that for you09:26
ralonsoh(maybe not in the second compute node...)09:26
ralonsohyeah, I think I didn't configure ceph in the second compute node09:27
ralonsohthanks!09:27
sean-k-mooney[m]https://opendev.org/openstack/devstack-plugin-ceph/src/branch/master/devstack/lib/ceph#L24409:28
sean-k-mooney[m]that is what adds the user key ring as a secret09:28
sean-k-mooney[m]so you could just do that or restack the compute09:28
sean-k-mooney[m]you also need to copy over the ceph.confirm09:29
sean-k-mooney[m]*ceph.conf09:29
ralonsohsean-k-mooney[m], btw, I think slaweq made a good point on https://review.opendev.org/c/openstack/neutron-specs/+/799198/3/specs/xena/ovn-auxiliary-port-bridge-live-migration.rst#5109:30
ralonsohI'm checking this now09:30
sean-k-mooney[m]ovn must handel that or we cannot support live migration with ovn09:31
ralonsohI know09:31
sean-k-mooney[m]so from my point of view we should either ignore that and if it cause any issue file it as a bug with ovn or we disable live migration when you use ovn. we are not actually changing the behaviour although we are extending the time its in that state09:33
sean-k-mooney[m]today the port exits on the source and dest for a period of time its just shorter09:33
ralonsohsean-k-mooney[m], in OVN we don't have, for example, the concept of multiple port binding09:34
ralonsohand could be difficult to introduce it09:34
ralonsohcould be possible, just an idea, to monitor this multiple port binding09:35
ralonsohand provide routing for the older one09:35
ralonsohwhen the scr port is deleted, the destination port will prevail09:35
ralonsohthis will also prevent errors if the migration fails09:35
ralonsohanyway, this is not supported09:35
sean-k-mooney[m]this is not related to multiple port bindings09:37
sean-k-mooney[m]this willl happen with just libvirt and ovn without openstack09:37
ralonsohI was making a parallelism09:37
ralonsohwith OVN backend09:37
ralonsohthis is what OVN should support or something similar09:38
sean-k-mooney[m]ovn needs to monitor the packets09:38
ralonsohwhat for?09:38
sean-k-mooney[m]and when it receives new packets on the dest host swap that to to the primary port09:38
ralonsohcould be an idea yes09:39
sean-k-mooney[m]basically if need to activate the dest port when it receives the mac learning RARP packets09:39
ralonsohbut should be aware of the multiple port locations09:39
sean-k-mooney[m]yes09:39
ralonsohin any case, this is very similar to what we have now09:39
sean-k-mooney[m]anyway in post-copy mode the vm today is potentially in this state for several minutes09:39
ralonsohthat means OVN will configure the backend when the VM is unpaused09:40
sean-k-mooney[m]so this is not new to this spec09:40
ralonsohin any case, I didn't think about this problem09:40
ralonsohand now I don't see what's the benefit using this feature09:41
sean-k-mooney[m]ya but its not related to any of the change we are making and it happens already today without them so while we should bring it up with the ovn team09:41
sean-k-mooney[m]its not a blocker for this change09:41
ralonsohbut it is a relevant OVN behavior change we need to document in the spec09:42
sean-k-mooney[m]its not09:42
*** rpittau|afk is now known as rpittau09:42
sean-k-mooney[m]this already happens today09:42
sean-k-mooney[m]so its not a behaviour change09:42
ralonsohright09:43
ralonsohin any case, to make this work properly, we need this OVN change09:43
ralonsohof course09:43
ralonsohthe feature is still valid09:43
ralonsohand the spec09:43
sean-k-mooney[m]perhaps. if ovn is installing teh correct flows on the dest in this sate09:43
sean-k-mooney[m]then the only issue we have is with packets getting sent to the vm right09:44
ralonsohyes09:44
sean-k-mooney[m]ovn proably should be mirroring the packet flow to both ports09:44
ralonsohlet me ask again: how much time could it take to copy the VM to dest?09:45
sean-k-mooney[m]multiple minutes09:45
ralonsohso we can have this network disconnection time, right?09:45
sean-k-mooney[m]with ml2 ovs it wont have downtime09:46
sean-k-mooney[m]the mac learning packets fix this issue09:46
sean-k-mooney[m]i’m not sure how ovn will work09:46
ralonsohOVN will block any traffic from src port09:47
ralonsohbecause the port_binding is changed09:47
sean-k-mooney[m]that is ok09:47
sean-k-mooney[m]i assume it is blocking packets form the dest until the port binding is activated on the dest09:48
ralonsohdo you have 5 mins via meet?09:48
ralonsohor this afternoon09:48
ralonsohwhenever you can09:49
sean-k-mooney[m]are you free at the top of the hour09:49
ralonsohyes09:49
ralonsohping me then09:49
sean-k-mooney[m]i was just going to grab coffee but i’m free then09:49
ralonsohperfect09:49
opendevreviewLee Yarwood proposed openstack/nova master: zuul: Add CentOS 8 stream integrated compute tempest job to gate  https://review.opendev.org/c/openstack/nova/+/79761609:56
sean-k-mooneyralonsoh: https:/meet.google.com/izr-tezp-qab whenever you are free10:03
opendevreviewStephen Finucane proposed openstack/nova master: Remove the redundant code for attach interface  https://review.opendev.org/c/openstack/nova/+/79620810:56
opendevreviewMerged openstack/os-traits master: setup.cfg: Replace dashes with underscores  https://review.opendev.org/c/openstack/os-traits/+/79113310:56
viks_hi, suppose if i have a preconfigured physical system which already have an os installed, can i make it as a openstack instance so that i can control that machine... i.e. assign an ip via neutron, boot, shutdown, restart, inject ssh  etc ?11:36
lyarwoodviks_: https://docs.openstack.org/ironic/latest/11:40
viks_lyarwood:  Thanks... but In ironic it installs OS on the given baremetal... but i do not need to install anything, just add a machine and control it...11:43
lyarwoodviks_: https://docs.openstack.org/ironic/latest/admin/adoption.html I thought it didn't need to do that?11:45
viks_lyarwood: ok.. thanks... i'll check.. i have not gone through it before... 11:46
opendevreviewMerged openstack/nova master: [ironic] Minimize window for a resource provider to be lost  https://review.opendev.org/c/openstack/nova/+/79932712:18
opendevreviewLee Yarwood proposed openstack/nova-specs master: Store and allow libvirt instance device buses and models to be updated  https://review.opendev.org/c/openstack/nova-specs/+/79981112:48
lyarwoodDoes anyone know of an API where we change the response for admins based on a microversion?12:56
lyarwoodOh events, instance faults etc me--12:56
ralonsohhi folks, do you know why if I have two compute node hosts (opentack host list)13:07
ralonsohI can't map the second one to cell1?13:07
ralonsoh"nova-manage cell_v2 discover_hosts", then "nova-manage cell_v2 list_hosts"13:07
ralonsoh+-----------+--------------------------------------+----------+13:07
ralonsoh| Cell Name |              Cell UUID               | Hostname |13:07
ralonsoh+-----------+--------------------------------------+----------+13:07
ralonsoh|   cell1   | db04030a-5cca-439d-b1c8-df1aa8d53d4f |  u20ovn  |13:07
ralonsoh+-----------+--------------------------------------+----------+13:07
ralonsohone is missing13:08
sean-k-mooneyyou need to run discover hosts13:09
sean-k-mooneynova-manage cellv2 discover_hosts13:09
ralonsohI did13:09
sean-k-mooneyfrom the controler13:09
ralonsohyes13:09
sean-k-mooneycan you run that with -v or --debug and past the output somewhere13:10
ralonsohlet me check13:10
ralonsohsean-k-mooney, nothing, no output13:11
sean-k-mooneydoes it show up in openstack hyperviors list13:12
ralonsohyes13:12
sean-k-mooneythe os-host api is deprecated by the way so im not sure it works proerly13:12
ralonsohstack@u20ovn:/opt/stack$ openstack host list13:12
ralonsoh+-----------+-----------+----------+13:12
ralonsoh| Host Name | Service   | Zone     |13:12
ralonsoh+-----------+-----------+----------+13:12
ralonsoh| u20ovn    | scheduler | internal |13:12
ralonsoh| u20ovn    | conductor | internal |13:12
ralonsoh| u20ovn    | conductor | internal |13:12
ralonsoh| u20ovn    | compute   | nova     |13:12
ralonsoh| u20ovn2   | compute   | nova     |13:12
ralonsoh+-----------+-----------+----------+13:12
sean-k-mooneyvery odd indeed13:13
ralonsohdo I need any service on the second node?13:13
ralonsohdevstack@n-cond-cell1.service13:13
stephenfinbauzas: Trivial bugfix here if you have a moment? https://review.opendev.org/c/openstack/nova/+/79400613:13
bauzasstephenfin: I can do13:13
stephenfinta13:14
bauzasstephenfin: same thought than gmann13:14
stephenfinoh, missed that comment13:15
bauzaswow, takes a lot of time to update 13:15
opendevreviewStephen Finucane proposed openstack/nova master: api: Handle invalid volume UUIDs during spawn  https://review.opendev.org/c/openstack/nova/+/79400613:28
stephenfinbauzas: ^13:28
bauzasta, looking13:28
stephenfingmann, lyarwood: either of you care to re-review also. Super trivial ^13:28
opendevreviewStephen Finucane proposed openstack/nova master: docs: Replace 'nova boot' with 'openstack server create'  https://review.opendev.org/c/openstack/nova/+/79400713:29
stephenfinlyarwood: I also addressed your comments on that doc13:29
lyarwoodack looking13:48
*** bhagyashris is now known as bhagyashris|ruck13:59
lyarwoodstephenfin: I'm sure you still need --bootable when creating the volume14:00
stephenfinPossibly in the past, but I tested it locally with a basic DevStack deployment and you didn't14:01
lyarwoodI've just copy pasted from my env, it's failing for me14:02
* stephenfin looks14:02
lyarwoodunless cinder is somehow automatically setting this under a later microversion14:03
lyarwoodor we are somehow updating the bootable flag 14:03
lyarwoodI can't see how this ever worked14:03
stephenfinWhat I was seeing was that the bootable flag would be False in the response returned from the create request, but then True when I did a show call14:04
stephenfinso cinder was changing things under the good14:04
stephenfin*hood14:04
stephenfinI'm pretty sure I traced through cinder to find what was setting it too, but it seems I didn't take notes on the patch like I usually would14:05
lyarwoodweird14:05
lyarwoodoh wait that's with the --bootable flag14:07
lyarwoodhaha nice, I bet the osc command does it in two calls or something weird14:08
stephenfinwdym?14:08
* lyarwood < https://matrix.org/_matrix/media/r0/download/matrix.org/mmvCAyPSjzrTbjzHXATPMYPl/message.txt >14:08
lyarwoodhttps://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=create-a-volume-detail#volumes-volumes - because c-api doesn't take bootable as an arg when creating the volume 14:10
lyarwoodhttps://github.com/openstack/python-openstackclient/blob/4f358367986525625157bee164c694e65573c30c/openstackclient/volume/v2/volume.py#L231-L232 and yeah osc creates and then optionally marks it as bootable if you pass the flag14:12
lyarwoodhttps://github.com/openstack/nova/blob/dfda0c0482acf579452742860cec2bdd64ac616a/nova/block_device.py#L685-L686 is where n-api rejects it btw14:13
stephenfinlyarwood: I'm still seeing the same behavior http://paste.openstack.org/show/807231/14:21
stephenfinSo if I create a volume from an image then the initial response from cinder is bootable='false', but this changes to bootable='true' once the volume is fully created14:22
lyarwoodyeah osc must be doing that14:22
lyarwoodthat makes sense if you've passed --image 14:22
stephenfinNot that I can see14:23
stephenfinand python-cinderclient isn't doing anything too funky either14:23
lyarwoodyeah weird, maybe c-api then, TIL14:23
lyarwoodand I wasn't passing --image so that's why I didn't see it14:24
stephenfinah, yes, I explicitly called that fact out https://review.opendev.org/c/openstack/nova/+/794007/4/doc/source/user/launch-instance-from-volume.rst#26514:24
stephenfinCinder makes a volume bootable when ``--image`` parameter is passed.14:24
*** bhagyashris_ is now known as bhagyashris|ruck14:24
lyarwoodYup apologies14:26
opendevreviewJay Faulkner proposed openstack/nova stable/wallaby: [ironic] Minimize window for a resource provider to be lost  https://review.opendev.org/c/openstack/nova/+/79977214:28
gmannstephenfin: done, thanks for adding test. 14:34
stephenfinta14:36
opendevreviewLee Yarwood proposed openstack/nova-specs master: Add connection_info to responses of the os-volume_attachments API for admins  https://review.opendev.org/c/openstack/nova-specs/+/79985015:22
opendevreviewLee Yarwood proposed openstack/nova-specs master: Add connection_info to responses of the os-volume_attachments API for admins  https://review.opendev.org/c/openstack/nova-specs/+/79985015:44
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] neutron fixture for extended resource request  https://review.opendev.org/c/openstack/nova/+/79430615:48
opendevreviewBalazs Gibizer proposed openstack/nova master: Detect extended_resource_request neutron API extension  https://review.opendev.org/c/openstack/nova/+/79361815:48
opendevreviewBalazs Gibizer proposed openstack/nova master: Reject server create with extended resource req  https://review.opendev.org/c/openstack/nova/+/79361915:50
opendevreviewBalazs Gibizer proposed openstack/nova master: Reject server operations with extended resource req  https://review.opendev.org/c/openstack/nova/+/79362015:50
opendevreviewBalazs Gibizer proposed openstack/nova master: Add same_subtree field to RequestLevelParams  https://review.opendev.org/c/openstack/nova/+/79150315:52
opendevreviewBalazs Gibizer proposed openstack/nova master: Bump min placement microversion to 1.36  https://review.opendev.org/c/openstack/nova/+/79150415:55
opendevreviewBalazs Gibizer proposed openstack/nova master: Support same_subtree in allocation_canadidate query  https://review.opendev.org/c/openstack/nova/+/79150515:57
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] refactor assertPortMatchesAllocation  https://review.opendev.org/c/openstack/nova/+/79245815:57
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] refactor asserts in qos tests  https://review.opendev.org/c/openstack/nova/+/79893015:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Support the new port resource_request format  https://review.opendev.org/c/openstack/nova/+/78720816:01
opendevreviewBalazs Gibizer proposed openstack/nova master: Transfer RequestLevelParams from ports to scheduling  https://review.opendev.org/c/openstack/nova/+/79150616:02
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] ports with both bw and pps resources  https://review.opendev.org/c/openstack/nova/+/79239416:03
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place  https://review.opendev.org/c/openstack/nova/+/79362116:05
*** rpittau is now known as rpittau|afk16:09
opendevreviewLee Yarwood proposed openstack/nova-specs master: Add nova-manage commands to show and refresh connection_info  https://review.opendev.org/c/openstack/nova-specs/+/79962416:15
stephenfinzzzeek: If you've any spare time whatsoever this week, I'd appreciate your attention on an issue I'm having during the sqla-migrate -> alembic switch. Basically, my migration code to create shadow tables doesn't seem to be working with in-memory SQLite databases, but I can't figure out how to fix it16:45
stephenfinzzzeek: I have a pretty minimal reproducer here https://github.com/stephenfin/alembic-sqlite-memory-issue16:45
stephenfinzzzeek: and the series to do the conversion for nova ends here, fwiw https://review.opendev.org/c/openstack/nova/+/799530/16:46
stephenfinzzzeek: I _think_ I'm seeing an issue like the one in the cookbook, but I don't understand the issue well enough yet to say /o\ Any input would be much appreciated https://alembic.sqlalchemy.org/en/latest/cookbook.html#sharing-a-connection-with-a-series-of-migration-commands-and-environments16:47
*** mgoddard- is now known as mgoddard16:54
opendevreviewLee Yarwood proposed openstack/nova-specs master: Store and allow libvirt instance device buses and models to be updated  https://review.opendev.org/c/openstack/nova-specs/+/79981117:06
lyarwoodstephenfin: hope you're happy, I pretty much always forget to manually run pep8 in projects outside of Nova now thanks to pre-commit ;)17:07
sean-k-mooneyhehe same17:20
opendevreviewMerged openstack/nova master: api: Handle invalid volume UUIDs during spawn  https://review.opendev.org/c/openstack/nova/+/79400617:27
zzzeekstephenfin: the issue is a combination of using NullPool with create_engine() combined with the usage of a different connection here: https://github.com/stephenfin/alembic-sqlite-memory-issue/blob/main/foo/migrations/versions/8f2f1571d55b_initial_version.py#L9818:26
zzzeekstephenfin: when you're in a migration, there's a method op.get_bind().  that gives you the current Connection.  use that everywhere for SQL, table reflection, etc18:27
zzzeekstephenfin: I marked it up quite a bit to get you started :)18:34
rm_workanyone seen issues with this error on starting up nova-compute (I'm on wallaby): `greenlet.error: cannot switch to a different thread`19:28
rm_workfull traceback: http://paste.openstack.org/show/807240/19:29
sean-k-mooneyis that in the api19:29
rm_worklog just shows it trying to do heartbeats and then ... this, and the process locks up19:29
rm_worknova-compute service19:29
sean-k-mooneythat should not happen in the compute service19:29
sean-k-mooneywe do use real threas but only to interact with a select number of things liek libvirt19:30
rm_workhmm, no VMs even in this system yet, it's essentially a clean install19:30
rm_workjust trying to start it up for the first time19:30
sean-k-mooneyso i woudl assume this is somehow related to the perodic taks19:30
sean-k-mooney*tasks19:31
sean-k-mooneydo you have anything custom set in you nova.conf19:31
rm_workhmmm... what periodics does nova-compute run? can I disable them to check?19:31
sean-k-mooneye.g. are you trying to use any atypical features19:31
sean-k-mooneyit runs quite a few19:32
sean-k-mooneyi think all or most of them have config options19:32
rm_workIt's hard for me to answer that question -- yes, the conf is probably custom, but I don't know enough about nova's features to know which are atypical19:32
rm_workI've tried to trim it down as much as possible19:32
sean-k-mooneyi was wondering if you were tweakign any oslo concurancy setting related to threading for example19:33
sean-k-mooneyyou coudl try setting https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.run_external_periodic_tasks to false19:33
rm_workah this is a nova-compute configured for ironic19:33
rm_workooo, will try that for testing19:33
rm_workno config in oslo_concurrency besides setting the lock_path19:34
sean-k-mooneyoh its ironic hum19:34
sean-k-mooneyi dont think that shoudl really change things19:35
sean-k-mooneyhttps://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.sync_power_state_interval19:35
sean-k-mooneythere are several "*_interval"19:35
sean-k-mooneyconfig option which contol the perodic tasks19:35
rm_workhmm, added that `run_external_periodic_tasks` to false, didn't help19:35
sean-k-mooneydo you have a more complete errror message19:35
sean-k-mooneyso we can see what it was doing right before the error19:35
rm_workyeah secx19:35
rm_workhttp://paste.openstack.org/show/807242/19:38
sean-k-mooneyoh19:39
sean-k-mooneyits right after the heartbeat19:39
rm_workyeah19:39
sean-k-mooneyone sec19:39
sean-k-mooneytry setting https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L90-L10019:40
sean-k-mooneyto false19:40
rm_workkk19:40
sean-k-mooney[oslo_messaging_rabbit]/heartbeat_in_pthread=false19:40
sean-k-mooneyin the nova.conf19:40
rm_workwell that's ... different19:41
sean-k-mooneyno crash?19:41
rm_workoh derp sorry sec19:41
rm_workit crashed but ... lol19:41
rm_workfucking macbook keyboard19:41
sean-k-mooneyi recently got a ipad pro and have been trying to use it as an ssh clinet19:42
rm_workyeah looks A++19:42
sean-k-mooneyits ... interesting19:42
rm_workseems to have connected up successfully, starts and shows up in a compute service list19:42
rm_workthanks!19:42
rm_workthough, that opt is deprecated it seems ;D19:42
sean-k-mooneywell that option is deprecated for removal and planned to be remvoed this cycle19:43
sean-k-mooneyso we shoudl file a bug and get oslo invovled19:43
sean-k-mooneyto see if they can repoduce this19:43
sean-k-mooneyin theory you should not need to run the heatbeat in a real greenthread by the way19:44
sean-k-mooneythat was to workaround an issue in the api when service are run under uwsgi19:44
sean-k-mooneysince the wsgi server would kill the heatbeat19:45
rm_workhmm19:46
opendevreviewmelanie witt proposed openstack/nova master: Add functional regression test for bug 1853009  https://review.opendev.org/c/openstack/nova/+/69501219:46
opendevreviewmelanie witt proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker  https://review.opendev.org/c/openstack/nova/+/69518719:46
opendevreviewmelanie witt proposed openstack/nova master: Invalidate provider tree when compute node disappears  https://review.opendev.org/c/openstack/nova/+/69518819:46
opendevreviewmelanie witt proposed openstack/nova master: Prevent deletion of a compute node belonging to another host  https://review.opendev.org/c/openstack/nova/+/69480219:47
opendevreviewmelanie witt proposed openstack/nova master: Fix inactive session error in compute node creation  https://review.opendev.org/c/openstack/nova/+/69518919:47
sean-k-mooneyrm_work: you should be able to use that as a workaround for now19:49
sean-k-mooneybut i mentioned it in #openstack-oslo19:49
rm_workyeah, will do. thanks for the help :)19:49
sean-k-mooneyrm_work: if you can file a bug or provide more detail e.g. ubutu20.04 vs centos8 that kind of thing it might help nail this down19:50
rm_workkk19:51
sean-k-mooneyit should be filed against oslo.messaging but you can also add nova if you like althoguh this is not in the nova code19:51
rm_workah k19:54
rm_workposted here: https://bugs.launchpad.net/oslo.messaging/+bug/193493720:40
rm_workI am ... not great at launchpad20:41
rm_workhopefully that's enough to start with20:41
rm_workcan add more details if someone tells me what would be good to add20:41
opendevreviewmelanie witt proposed openstack/nova master: Add functional regression test for bug 1853009  https://review.opendev.org/c/openstack/nova/+/69501221:21
opendevreviewmelanie witt proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker  https://review.opendev.org/c/openstack/nova/+/69518721:21
opendevreviewmelanie witt proposed openstack/nova master: Invalidate provider tree when compute node disappears  https://review.opendev.org/c/openstack/nova/+/69518821:21
opendevreviewmelanie witt proposed openstack/nova master: Prevent deletion of a compute node belonging to another host  https://review.opendev.org/c/openstack/nova/+/69480221:21
opendevreviewmelanie witt proposed openstack/nova master: Fix inactive session error in compute node creation  https://review.opendev.org/c/openstack/nova/+/69518921:21

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!