opendevreview | Minghong Hou proposed openstack/nova master: db: Retrieve VirtualInterface objects by UUID, not address https://review.opendev.org/c/openstack/nova/+/828819 | 01:36 |
---|---|---|
opendevreview | melanie witt proposed openstack/nova master: libvirt: Register defaults for undefined hw image properties https://review.opendev.org/c/openstack/nova/+/800708 | 01:46 |
opendevreview | Minghong Hou proposed openstack/nova master: db: Retrieve VirtualInterface objects by UUID, not address https://review.opendev.org/c/openstack/nova/+/828819 | 06:32 |
*** amoralej|off is now known as amoralej | 07:34 | |
opendevreview | yuval proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606 | 07:39 |
opendevreview | Felix Huettner proposed openstack/nova stable/stein: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829859 | 08:06 |
opendevreview | Felix Huettner proposed openstack/nova stable/rocky: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829860 | 08:07 |
opendevreview | Felix Huettner proposed openstack/nova stable/queens: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829861 | 08:07 |
opendevreview | Elod Illes proposed openstack/nova stable/queens: Fix sphinx doc job https://review.opendev.org/c/openstack/nova/+/830327 | 09:09 |
opendevreview | Fabian Wiesel proposed openstack/nova master: Transport context to all threads https://review.opendev.org/c/openstack/nova/+/827467 | 11:05 |
opendevreview | Balazs Gibizer proposed openstack/nova master: Fix eventlet.tpool import https://review.opendev.org/c/openstack/nova/+/830383 | 11:42 |
chateaulav | gibi: with a multi patch series, like i have. when i go to update, do i rebase back to the driver patch and then make the appropriate changes, or do I have to submit the whole series for review each time even though the objects patch has the first +2? | 11:50 |
chateaulav | i hope that makes sense, just want to make sure i fully understand and am not creating to much work for you guys | 11:51 |
sean-k-mooney | chateaulav: the object patch is the base patch so you dont have to rebase that to work on the following patch | 12:00 |
chateaulav | ok | 12:00 |
sean-k-mooney | you have 2 ways to do it you can write a patch at the end to adress the issues and then do an interactive rebase to squash merge it into the patch its fixign | 12:00 |
sean-k-mooney | our you can start with a interactive rebase and mark the commits you want to alter for editing | 12:01 |
sean-k-mooney | in both cases you tell git to keep the base object patch exactly as is | 12:01 |
chateaulav | ok, good, thats what im doing then. just wanted to make sure | 12:01 |
sean-k-mooney | then when you finish the rebase it will have only modified the following commits and the git review at the end will not update the first review | 12:02 |
sean-k-mooney | you can also pass -R to git review to ensure it does not do any automatic rebases for you | 12:02 |
sean-k-mooney | gibi: did you see ralonsoh comments on https://review.opendev.org/c/openstack/neutron/+/829247 | 12:05 |
sean-k-mooney | gibi: while form a nova perspeictive it looks like the port is fully updated apprently its not fully updated in the db | 12:06 |
opendevreview | yuval proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606 | 12:12 |
gibi | sean-k-mooney: now read ralonsoh's comment | 12:16 |
gibi | sean-k-mooney: I'm not a neutron expert so it can very well be that the change is not persisted | 12:17 |
sean-k-mooney | the fact that what ralonsoh commented is possibel today speaks to a larger problem in neutron IMO but i guess for now you just need to ensure the db version is also updated | 12:17 |
ralonsoh | sean-k-mooney, what other problem? | 12:17 |
sean-k-mooney | gibi: i think its possible perseited but only in memory | 12:17 |
sean-k-mooney | ralonsoh: teh fact that you can partly update a port but later api calls will show it as updated | 12:18 |
sean-k-mooney | even though the db is not | 12:18 |
sean-k-mooney | it suggest there is a problem with caching or how neutron is retriving data | 12:18 |
gibi | ralonsoh: if you happen to have a pointer where shoudl the mac_address be updated in neutron for this to work then please let me know that will help me updating the patch properly | 12:19 |
ralonsoh | sean-k-mooney, no no, this is not the case | 12:19 |
ralonsoh | gibi, yes, let me work on this today (or tomorrow morning) | 12:19 |
ralonsoh | I'll focus on this | 12:19 |
sean-k-mooney | ralonsoh: the unit test are doing a port show to get the port after it was updated, and you said you could see the chage in port show/list | 12:20 |
gibi | ralonsoh: thanks | 12:20 |
sean-k-mooney | eventhough you also said the db record still had the old mac | 12:20 |
sean-k-mooney | so there is clearly a disconenct in my and your understanding | 12:20 |
ralonsoh | sean-k-mooney, yes, this is because you populate the port dictionary (the one that is going to be in the JSON) with the port register info | 12:20 |
ralonsoh | and other related db registers | 12:20 |
ralonsoh | sean-k-mooney, the port dictionary comes from how neutron was implemented 10 years ago | 12:21 |
ralonsoh | without OVOs | 12:21 |
sean-k-mooney | right so you are not just readign the port info form the db | 12:21 |
ralonsoh | the OVO is a DB view, linking several DB registers | 12:21 |
ralonsoh | this method (and most of the _dict_* ones) | 12:22 |
ralonsoh | are only methods to populate in one dict several DB registers data | 12:22 |
sean-k-mooney | but its clearly possibel for the mac to have 2 different values in two places so db consitecy is not enforced | 12:23 |
gibi | OK I think I see. Every time a port is returned it will hit the code I added so that code will apply the MAC from the binding _on the port dict being returned_ | 12:23 |
gibi | but I never apply the MAC from the binding on a port that is being saved to the db | 12:23 |
sean-k-mooney | gibi: ya you coudl validate that by claring the mac form the binding profile | 12:23 |
ralonsoh | sean-k-mooney, no, it is not. But this is why we should review what we populate in the resource dictionaries | 12:23 |
sean-k-mooney | and asserting it does not change | 12:24 |
sean-k-mooney | ralonsoh: well our actully remove them and just use the ovos | 12:24 |
sean-k-mooney | we still ahve some dicts in nova too like the bdms | 12:25 |
sean-k-mooney | but we try to not use dicst where we can avoid it | 12:25 |
ralonsoh | sean-k-mooney, those _dict_* methods are also used by the extensions | 12:25 |
ralonsoh | to populate new data in the resource dictionaries | 12:25 |
ralonsoh | in any case, changing this now is complex and an unnecessary refactor | 12:26 |
sean-k-mooney | we can disagree on unnesesary but it would prevent the possibleity fo havign the db and runtime view getting out of sync in this way | 12:27 |
sean-k-mooney | well or at least reduce it | 12:27 |
sean-k-mooney | so right here at the end https://review.opendev.org/c/openstack/neutron/+/829247/2/neutron/tests/unit/plugins/ml2/test_port_binding.py#776 | 12:28 |
sean-k-mooney | if we just add a call to clear the mac in the binding profile | 12:28 |
sean-k-mooney | and then assert the mac has the new value it should fail | 12:28 |
sean-k-mooney | since it will nolonger triger gibis change and we will get the mac from the db right | 12:29 |
gibi | hm that acutally has an interesting side effect | 12:29 |
gibi | if a port was bound to a PF then unbound from it | 12:29 |
gibi | then that port keeps the MAC of the PF | 12:29 |
ralonsoh | right, and you can also retrieve the port created and stored in the DB | 12:29 |
gibi | so if another port is tried to be bound to the same PF | 12:29 |
gibi | that would cause MAC conflict in neutron | 12:30 |
sean-k-mooney | gibi: because the reset of the mac only resets the mac in the db | 12:30 |
sean-k-mooney | ya unless the bidnign profile is cleared it woudl be a problem | 12:30 |
gibi | I'm not sure how the reset works today, but if we follow ralonsoh's suggestion and persist the MAC in the DB then when the binding is removed we need to clear the DB too | 12:31 |
sean-k-mooney | isnt there already code for that | 12:31 |
sean-k-mooney | for the db part | 12:31 |
sean-k-mooney | that code wont clear the mac in the profile however | 12:31 |
gibi | the profile is cleard by nova | 12:31 |
gibi | that is OPK | 12:31 |
gibi | OK | 12:31 |
gibi | as I said I'm not sure about the reset logic | 12:32 |
sean-k-mooney | ok then the old code for reseting the mac shoudl fix the db version | 12:32 |
sean-k-mooney | https://github.com/openstack/neutron/commit/e603d19939c98b94081bb6a3af8bcb943f7bd2ce | 12:33 |
gibi | ahha OK | 12:34 |
gibi | then that is OK | 12:34 |
sean-k-mooney | ya the detach case is covered | 12:34 |
ralonsoh | sean-k-mooney, https://github.com/openstack/neutron/commit/e603d19939c98b94081bb6a3af8bcb943f7bd2ce | 12:40 |
ralonsoh | this is when the port MAC can be changed | 12:40 |
ralonsoh | of course, in this call you can bind the port and change the MAC | 12:40 |
ralonsoh | the only consideration is the other port with the same MAC | 12:40 |
ralonsoh | because you first bind the new port and then unbind the older one, right? | 12:41 |
sean-k-mooney | no | 12:43 |
sean-k-mooney | there is only 1 port | 12:43 |
sean-k-mooney | we either have multiple prot bidnings | 12:43 |
ralonsoh | right, right | 12:43 |
sean-k-mooney | or we do a update of the single port binding | 12:43 |
sean-k-mooney | https://review.opendev.org/c/openstack/neutron/+/829247/2/neutron/tests/unit/plugins/ml2/test_port_binding.py#775 does that make sense | 12:43 |
ralonsoh | my bad, there is only one DB port, two L1 ports | 12:43 |
sean-k-mooney | yes | 12:43 |
ralonsoh | no because of the implementation of "_update_port_dict_binding" | 12:45 |
sean-k-mooney | i was expecting that call to be wrong | 12:45 |
ralonsoh | in L672 we are calling the API | 12:45 |
ralonsoh | this API will retrieve the DB port and create this dictionary, this is what is returned | 12:45 |
ralonsoh | why? | 12:46 |
sean-k-mooney | right but what i was trying to capture was the logic of unset bidnding:profile['port_mac'] | 12:46 |
sean-k-mooney | to ensure that mac is not coming form the profile but the db | 12:47 |
ralonsoh | ahhhh sorry | 12:47 |
ralonsoh | yes, this is after deleting the port binding | 12:47 |
sean-k-mooney | after deleteing the source one yes | 12:47 |
ralonsoh | so we are reading the port randomly generated mac | 12:47 |
ralonsoh | port = self._create_unbound_port() | 12:48 |
ralonsoh | # neutron generates a MAC for each port created | 12:48 |
ralonsoh | generated_mac = port[port_def.PORT_MAC_ADDRESS] | 12:48 |
sean-k-mooney | with the current code ya but we expect it to keep the host1 pf mac | 12:48 |
sean-k-mooney | it wont now because of the bug | 12:48 |
sean-k-mooney | so i woudl like the test to be writen so that it caught the bug so it cant be regressed going forward | 12:49 |
ralonsoh | sean-k-mooney, when the port is first assigned, the mac is changed, right? | 12:50 |
ralonsoh | before the live migration | 12:50 |
ralonsoh | the Neutron DB port register MAC is changed | 12:50 |
sean-k-mooney | yes | 12:50 |
ralonsoh | ok | 12:50 |
sean-k-mooney | nova does a port update before we bind the port | 12:50 |
sean-k-mooney | the issue is we cant do a port update while it is bound | 12:51 |
ralonsoh | so the case here is to add this exception: to allow to change the MAC of the port even if it is bound | 12:51 |
ralonsoh | but | 12:51 |
ralonsoh | you also need to keep the original MAC | 12:51 |
ralonsoh | right? | 12:51 |
gibi | no | 12:51 |
gibi | we need to be able to define MAC per binding | 12:51 |
gibi | for mutliple binding | 12:51 |
gibi | at least that is how I model this | 12:52 |
ralonsoh | so this is a different feature I think | 12:52 |
gibi | one MAC per binding | 12:52 |
sean-k-mooney | we dont generally care about the orginal mac since we can always just read that form the PF if we need it | 12:52 |
ralonsoh | yes, like multiple port binding | 12:52 |
ralonsoh | ok, do you mind if I spend some time this afternoon or tomorrow thinking about this? | 12:52 |
ralonsoh | I need to finish some patches now | 12:53 |
gibi | ralonsoh: sure | 12:53 |
ralonsoh | thanks!! | 12:53 |
ralonsoh | I'll ping you | 12:53 |
gibi | thanks! | 12:53 |
gibi | I anyhow need to work on the nova side of this too | 12:53 |
gibi | sean-k-mooney: I like the idea to define the MAC when nova creates the binding instead of defining it when nova activates an inactive binding | 12:54 |
gibi | so I'd like to store the MAC in the binding profile | 12:54 |
gibi | it feels clearer | 12:54 |
gibi | as we define the rest of the infor in the profile at the same time like slot | 12:54 |
sean-k-mooney | ack | 12:55 |
sean-k-mooney | but in that case we need to permently keep the mac in the profile yes | 12:55 |
sean-k-mooney | so we shoudl be doing this for first spawn and sriov attach too | 12:55 |
sean-k-mooney | for consitency | 12:55 |
gibi | yes we do it | 12:55 |
gibi | any time we bind a SRIOV PF | 12:56 |
gibi | or create an inactive binding | 12:56 |
sean-k-mooney | ok and the key we use in teh profile for the PF mac is also different form teh key we use for the PF mac in the case of remote managed prots right | 12:56 |
sean-k-mooney | just making sure we dont have a collision there | 12:57 |
gibi | it is differen | 12:57 |
gibi | t | 12:57 |
gibi | the remote managed codepath stores pf_mac_address | 12:57 |
gibi | in the VF profiles | 12:57 |
gibi | my bugfix stores mac_address in the PF profile | 12:57 |
sean-k-mooney | ack | 12:57 |
*** amoralej is now known as amoralej|lunch | 13:08 | |
opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Fix migration with remote-managed ports & add FT https://review.opendev.org/c/openstack/nova/+/829974 | 13:21 |
dmitriis | sean-k-mooney, gibi: ^ found a missing piece related to migration when adding more func tests. Since binding:profile is updated differently during live migration there is a slight change needed to also update new fields. | 13:24 |
dmitriis | I added a test case for live migration, looking into adding resize/evacuate/cold_migrate now as well. | 13:25 |
sean-k-mooney | dmitriis: you will need to update the inactive port bindign with the dest host info | 13:29 |
sean-k-mooney | dmitriis: i assume you are goign to supprot live migration by hot unplugging and hot plugging the VF as we do for vnic_type=direct yes | 13:30 |
sean-k-mooney | in the future we might eb able to read if the vf supprot live migration from sysfs and skip that if it does | 13:30 |
dmitriis | sean-k-mooney: the fix is quite simple https://review.opendev.org/c/openstack/nova/+/829974/2/nova/compute/manager.py#10805 - as you say, it's updating the inactive port binding | 13:31 |
sean-k-mooney | ya although that is adding more coupling to sysfs | 13:32 |
sean-k-mooney | gibi how do you feel about ^ | 13:32 |
dmitriis | yes, I was thinking of storing more in extra_info but this kind of information is more volatile (pf mac, vf num) | 13:33 |
dmitriis | compared to the serial number that we currently store in extra_info | 13:33 |
sean-k-mooney | dmitriis: gibi was going to store it in extra info | 13:33 |
sean-k-mooney | well the pf mac is basically hardcoded in the flash | 13:34 |
gibi | I'm on the side of moving this info the PciDevice object populated by the virt driver | 13:34 |
sean-k-mooney | ya | 13:34 |
sean-k-mooney | we have that object here so we can read if form there | 13:34 |
chateaulav | to set image metadata during a ci tempest, i would most likely have to use a script as there are no variables, correct? | 13:34 |
sean-k-mooney | so really we shoudl put this info in the extra_info dict with properties to access it form teh object | 13:34 |
sean-k-mooney | chateaulav: am it depends if you are writing new test then you woudl do it in the test | 13:35 |
dmitriis | sean-k-mooney: well, the PF itself may be a vport of a NIC switch. I've seen HW with variable number of PFs configurable (per a given flash configuration though) | 13:35 |
sean-k-mooney | but if you are trying to reuse exsitng tempest then you woudl have a pre playbook that generates a local.sh that runs at the end of devstack before tempest is run to update the default image with the properties | 13:35 |
dmitriis | it's *probably* safe to assume that the PF mac won't change across reboots is what I am going for. But we may encounter odd cases too. | 13:36 |
sean-k-mooney | chateaulav: you can also config tempst to use a different image and have devstack download addtional images via the local.conf | 13:36 |
chateaulav | sean-k-mooney: k, thanks | 13:36 |
sean-k-mooney | dmitriis: changing the number of PF would change ther PCI adress which would not be supported if you had existing vms | 13:37 |
sean-k-mooney | we can also update the mac in the db on agent start | 13:37 |
sean-k-mooney | althoguh that wont update the pf mac in the neutron port so maybe we shoudl detect that and cause an error | 13:38 |
sean-k-mooney | or at least a warning | 13:38 |
dmitriis | VF numbers have an offset but they should remain stable unless the number of PFs changes | 13:39 |
sean-k-mooney | in the unlikely case a nic dies we likely want to be able to replace it with another nic | 13:39 |
sean-k-mooney | dmitriis: right which we dont want to supprot | 13:39 |
dmitriis | sean-k-mooney: ok, then I agree in principle that we can store vf_num and pf_mac in extra_info | 13:39 |
dmitriis | not hard to retrieve it from there either | 13:40 |
sean-k-mooney | we are already storing them in the neutron db in the port bidnign | 13:40 |
sean-k-mooney | so we already have the requirement that they be stable | 13:40 |
dmitriis | right | 13:40 |
sean-k-mooney | if we detach a change in the mac we can warn on that in agent start if we want to and tell the operator what the old mac was and the new one so that they know they have to manually update the ports. we could also automaticaly do that i guess but i woudl proably defer that to a followup patch | 13:41 |
dmitriis | ack | 13:43 |
dmitriis | So I think that's the change gibi is working on https://review.opendev.org/c/openstack/nova/+/829248 | 13:46 |
gibi | dmitriis: yes, but note the self -1 :) I realized that I need to change direction | 13:48 |
gibi | to use the PciDevice object | 13:48 |
dmitriis | gibi: ack. I'll focus on adding the remaining func test cases in the short term but I'll be around to help with moving things to PciDevice object | 13:50 |
gibi | dmitriis: thanks | 13:51 |
opendevreview | Felix Huettner proposed openstack/nova stable/queens: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829861 | 13:57 |
opendevreview | Alexey Stupnikov proposed openstack/nova master: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/830010 | 14:06 |
*** amoralej|lunch is now known as amoralej | 14:13 | |
opendevreview | yuval proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606 | 14:33 |
opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | 14:46 |
opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | 14:46 |
sean-k-mooney | gibi: can you respond ot https://review.opendev.org/c/openstack/placement/+/826492/8/placement/lib.py#219 i think that is the only thing im not sure about in your placement series | 14:53 |
sean-k-mooney | i have +w most of the patches before that so they should all be sitting in the ci queue and should merge up to that point later today | 14:54 |
gibi | sean-k-mooney: looking... | 15:03 |
sean-k-mooney | its pretty minor | 15:04 |
sean-k-mooney | just i dont think we are checkign for conflicts properly | 15:04 |
gibi | I have to load context... :) | 15:05 |
sean-k-mooney | like i think we bascially should be flattening the list of required traits and asserting that there is on intersection with forbidden | 15:05 |
sean-k-mooney | im not sure using all does that | 15:05 |
gibi | so the goal is to find a conflict in the request like required=T1,!T1 | 15:11 |
gibi | in the complex case | 15:11 |
gibi | required=in:T1,T2&required=!T2 | 15:11 |
gibi | !T2 is not a conflict | 15:11 |
opendevmeet | gibi: Error: "T2" is not a valid command. | 15:11 |
gibi | but | 15:12 |
sean-k-mooney | well it shoudl still be rejected | 15:12 |
gibi | it is satisfyable | 15:12 |
sean-k-mooney | only if we have t1 | 15:13 |
gibi | yes | 15:13 |
sean-k-mooney | right im not sure we shoudl allow that | 15:13 |
gibi | I think we cannot expect that the client will normalize the query | 15:13 |
sean-k-mooney | i dont think any frobiden traits should over lap with any required even if we in | 15:13 |
gibi | maybe these required triat pieces are coming form different direction in nova | 15:13 |
sean-k-mooney | since we cant make forbidne traits optional | 15:13 |
gibi | one prefilter says I need either T1 or T2 another prefilter says I don't like T2 | 15:14 |
gibi | I don't want nova to do the normalization in this case | 15:14 |
sean-k-mooney | i kind of feel like nova shoudl be preprocessing that rahter then having ot have placment fiture that out | 15:14 |
sean-k-mooney | but i see why you are trying to be more lax | 15:15 |
gibi | I think it is a lot easier to accept it than forcing the client to normalize | 15:15 |
sean-k-mooney | required=in:T1,T2&required=!T2 is logically jsut required=T1 | 15:16 |
gibi | yes | 15:16 |
sean-k-mooney | if we do accept that we shoudl really preprocess that before we generate the sql queries | 15:16 |
gibi | why? | 15:16 |
gibi | I think sql can handle this | 15:16 |
gibi | I even think sql should have the query planner to optimize this out | 15:17 |
sean-k-mooney | i mean it could but i think that will not scale well | 15:17 |
bauzas | sean-k-mooney: thanks for accepting the existing SQL query from gibi | 15:17 |
gibi | * the sql server | 15:17 |
sean-k-mooney | gibi: if that was your orginal intent i guess what you have will do that | 15:17 |
bauzas | sean-k-mooney: as I knew since 10 years, in general the subqueries use the same execution plans than in the inner joins | 15:17 |
bauzas | but we could verify it with the EXPLAIN query | 15:18 |
sean-k-mooney | bauzas: my expirnce is inner joins like that tended to result in more memory usage then subquires | 15:18 |
sean-k-mooney | im not sure that sqligte will optimise to the same degree as mysql in this regard | 15:18 |
gibi | I don't believe either | 15:19 |
gibi | but nobody runs sqlite in production (hopefully :) | 15:19 |
bauzas | sean-k-mooney: well, in general it depends on the DMBS | 15:19 |
sean-k-mooney | ya i was more worreid about our func tests but | 15:19 |
bauzas | sean-k-mooney: but with mysql 5.0 I saw that it was the same | 15:20 |
sean-k-mooney | we likely wont have db quiries that will stress it enough there to matter | 15:20 |
opendevreview | Elod Illes proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606 | 15:20 |
sean-k-mooney | bauzas: lets revisit this in the future when we have data one way or another | 15:21 |
bauzas | ++ | 15:21 |
bauzas | agreed | 15:21 |
gibi | I promise I get back to this and gather some performance data from a mysql env | 15:21 |
sean-k-mooney | bauzas: if you woudl not mind reviwing 826492 that is the only patch that does not have +w | 15:22 |
sean-k-mooney | bauzas: gibi and i were just discussing how we detct conflicts | 15:22 |
bauzas | sean-k-mooney: I could do this after the meeting | 15:22 |
bauzas | btw. | 15:22 |
bauzas | nova meeting in 38 mins | 15:23 |
bauzas | here | 15:23 |
sean-k-mooney | ack for now i think i can accept gibis explation of what they were trying to enable | 15:23 |
gibi | cool | 15:23 |
sean-k-mooney | we can let the db handel the extra complexity for now and we can simplyfy the query either in nova or placment in the futrue if needed | 15:23 |
kashyap | gibi: When you get a minute, do you, or anyone know what's going in this test here? - https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_list_image_filters.py#L108 | 15:23 |
gibi | thank you | 15:23 |
sean-k-mooney | gibi: im just worried that if we say "the db query optimiser will take care of it" too much we could get some nasty surprise in large deplopyments | 15:24 |
kashyap | (I'm trying to debug an upstream TripleO bug from hell that's "critical", and it's hitting the failure at line-128) | 15:24 |
gibi | sean-k-mooney: sure, I will do some performance testing to see if it matters | 15:24 |
gibi | kashyap: looking... | 15:26 |
kashyap | (That's the Tempest bug here: https://bugs.launchpad.net/tripleo/+bug/1959014/) | 15:27 |
gibi | kashyap: boot servers, then snapshot the server and upload the image to glance | 15:27 |
kashyap | Is it _really_ a live snapshot? Where "it" == create_image_from_server() | 15:27 |
kashyap | Is it a live snapshot? /me looks | 15:27 |
kashyap | create_image_from_server() --> create_image() | 15:28 |
gibi | it depends | 15:30 |
kashyap | Hm, I'm combing through the libvirt <-> QEMU logs given that it seems to fail at "live snapshot", but need more time to dig | 15:31 |
sean-k-mooney | kashyap its not always a live snapshot there is a config option that you can enabel. | 15:31 |
gibi | I will try to get to the logic nova does... | 15:31 |
sean-k-mooney | we also fall back to cold snapshots in some cases | 15:31 |
kashyap | sean-k-mooney: Yeah, I realize that | 15:31 |
sean-k-mooney | nova's api does not gurentee that its live | 15:31 |
sean-k-mooney | however if we stop it we will start the vm after the snap shot | 15:32 |
sean-k-mooney | so the pre and post state shoudl be the same | 15:32 |
gibi | kashyap: https://github.com/openstack/nova/blob/28cbcbb3737d544b7f73bd5aec55ccbd40222c95/nova/virt/libvirt/driver.py#L2893-L2922 | 15:35 |
* kashyap clicks | 15:35 | |
gibi | nova logs if it is live of cold snapshot | 15:35 |
kashyap | gibi: Thank you; so this path live snapshot here. (The instance isn't in SHUTDOWN state) | 15:36 |
* kashyap looks the Nova log | 15:36 | |
kashyap | s/looks/looks at/ | 15:37 |
kashyap | That's the Nova log - https://logserver.rdoproject.org/83/38983/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria/67d64d4/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.1.gz | 15:38 |
kashyap | (There's another nova-compute.log.txt.gz in the directory; maybe log rotation) | 15:38 |
kashyap | gibi: Yep, it logs it: | 15:39 |
kashyap | 2022-02-08 17:25:54.917 7 INFO nova.virt.libvirt.driver [req-5a68f755-b20c-41a7-a005-3bcc8b800b28 9afce47c724e4db7958ea7fc1eca1c27 8f305ae0fc324a41971784ef2430d8e3 - default default] [instance: 4262e009-6e83-4d8f-bd24-8a478cecda4a] Beginning live snapshot process | 15:39 |
kashyap | Thank you! | 15:39 |
gibi | kashyap: based on artom's comment in the bug there is also a log about direct snapshot it should be something like "Performing standard snapshot because direct..." | 15:42 |
gibi | direct snasphot only works with RDB I think | 15:42 |
gibi | RBD | 15:42 |
kashyap | As if people know what is a "standard snapshot" :) | 15:43 |
kashyap | I don't see such a message in the above nova-compute log | 15:43 |
kashyap | gibi: So, here it is directly hitting live snapshot process | 15:44 |
kashyap | I'm talking to the libvirt dev to see if we can reverse engineer the failure ... from the QEMU traces | 15:44 |
kashyap | (26MB: https://kashyapc.fedorapeople.org/lp_1959014/QMP_exchange.txt) | 15:44 |
gibi | then maybe artom looked at a different run her https://bugs.launchpad.net/tripleo/+bug/1959014/comments/2 | 15:44 |
kashyap | Yeah, likely. This is a different run where the (QEMU) 'blockdev-del' failure is occuring | 15:45 |
sean-k-mooney | well standard snapshot in this context means a local snapshot of a file done by qemu | 15:50 |
kashyap | I've asked Peter Krempa from libvirt about the log traces; we'll see if we get something | 15:50 |
sean-k-mooney | vs direct which means we are delegating the snapshot to the ceph cluster | 15:50 |
kashyap | sean-k-mooney: Yeah, but "standard" and "direct" are too abstract terms. We should be explicit on what it means. | 15:51 |
kashyap | (But that's a patch for a different day.) | 15:51 |
sean-k-mooney | well perhaps but thse are really intended for us as nova develpers to read | 15:52 |
kashyap | Yeah, even for developers who work on different subsystems it will be far from obvious. | 15:52 |
sean-k-mooney | so there si some expecation of knowing what the code does | 15:53 |
gibi | kashyap: looking at the req-5a68f755-b20c-41a7-a005-3bcc8b800b28 you quoted above, I don't see anything in the logs about the failure, nova logged that the snapshot is done and the image is uploaded https://review.rdoproject.org/zuul/build/67d64d4a4d584eb8be9db0fbc6e18c9d/log/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.1.gz#5705 | 15:53 |
kashyap | gibi: Hmmm, odd; so the guy says the above "reproduces" the problem, not sure what is going on here | 15:54 |
kashyap | But in the libvirtd debug log, I _do_ see | 15:54 |
kashyap | : | 15:54 |
kashyap | 2022-02-08 17:32:24.369+0000: 31580: error : qemuMonitorJSONCheckErrorFull:406 : internal error: unable to execute QEMU command 'blockdev-del': Failed to find node with node-name='libvirt-4- | 15:54 |
kashyap | storage' | 15:55 |
gibi | hm based on the timestamp that is 7 mintutes later | 15:55 |
kashyap | ("blockdev-del" deletes a block device that's added by "blockdev-add") | 15:55 |
kashyap | gibi: Hmm, I see. Then I'm scratching my head here about what's going on from a failure trigger | 15:56 |
sean-k-mooney | that is likely part of the test cleanup | 15:57 |
kashyap | sean-k-mooney: No, from the traceback in the bug description, it is happening at resource_setup() here: | 15:58 |
sean-k-mooney | if its 7 minute after the snapshot | 15:58 |
kashyap | https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_list_image_filters.py#L110 | 15:58 |
opendevreview | Alexey Stupnikov proposed openstack/nova master: Run clean up calls when queued live migration is aborted https://review.opendev.org/c/openstack/nova/+/828570 | 15:58 |
sean-k-mooney | hum why woudl dat do a blockdev-del | 15:59 |
sean-k-mooney | unless the vm coudl not boot | 15:59 |
sean-k-mooney | and it was cleaning up | 15:59 |
kashyap | Yeah, it sounds like a clean-up act somewhere | 16:00 |
gibi | bauzas: ping :) | 16:01 |
opendevreview | Elod Illes proposed openstack/nova stable/queens: [stable-only] Fix sphinx doc job https://review.opendev.org/c/openstack/nova/+/830327 | 16:01 |
bauzas | oh shit | 16:02 |
bauzas | #startmeeting nova | 16:02 |
opendevmeet | Meeting started Tue Feb 22 16:02:13 2022 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:02 |
opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:02 |
opendevmeet | The meeting name has been set to 'nova' | 16:02 |
bauzas | hey folks, sorry the delay | 16:02 |
gmann | o/ | 16:02 |
* bauzas looks at his coffee | 16:02 | |
chateaulav | \o | 16:02 |
bauzas | that's its fault | 16:02 |
bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | 16:03 |
elodilles | o/ | 16:03 |
bauzas | let's a quick meeting since we're close to some deadline :) | 16:03 |
bauzas | let's do* | 16:03 |
bauzas | #topic Bugs (stuck/critical) | 16:03 |
bauzas | #info No Critical bug | 16:03 |
bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 27 new untriaged bugs (-15 since the last meeting) | 16:03 |
bauzas | thanks gibi and others for having looked at the bugs | 16:03 |
bauzas | #help Nova bug triage help is appreciated https://wiki.openstack.org/wiki/Nova/BugTriage | 16:03 |
bauzas | link https://storyboard.openstack.org/#!/project/openstack/placement 27 open stories (+0 since the last meeting) in Storyboard for Placement | 16:03 |
bauzas | #link https://storyboard.openstack.org/#!/project/openstack/placement 27 open stories (+0 since the last meeting) in Storyboard for Placement | 16:04 |
bauzas | voila | 16:04 |
bauzas | any bugs to discuss ? | 16:04 |
bauzas | looks not | 16:04 |
bauzas | again, thanks for the folks who looked at the bugs | 16:04 |
bauzas | #topic Gate status | 16:05 |
bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | 16:05 |
bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status | 16:05 |
bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | 16:05 |
bauzas | so we had a few new gate failures | 16:05 |
bauzas | I saw gibi looking at one | 16:05 |
bauzas | any of them needing to be discussed ? | 16:06 |
gibi | I just did some bug triage and duplicated thing together | 16:06 |
gibi | I did not see a new failure this week | 16:06 |
gibi | * new type of failure | 16:06 |
bauzas | kk | 16:07 |
bauzas | ok, then moving to the next topic | 16:07 |
bauzas | #topic Release Planning | 16:07 |
bauzas | #link https://releases.openstack.org/yoga/schedule.html#y-ff FeatureFreeze in 2 days | 16:07 |
bauzas | #link https://etherpad.opendev.org/p/nova-yoga-blueprint-status Etherpad for blueprints tracking | 16:07 |
bauzas | so the FeatureFreeze will be on Thursday EOB | 16:07 |
bauzas | tbh, that means that nova-cores won't merge any feature change on Friday | 16:08 |
bauzas | unless we got a gate issue | 16:08 |
bauzas | do people have any concerns ? | 16:09 |
gibi | bauzas: you mean no +A on Friday | 16:09 |
bauzas | gibi: sorry yes | 16:09 |
gibi | except if rebase is needed | 16:09 |
bauzas | correct | 16:09 |
gibi | ack | 16:09 |
gibi | works for me | 16:09 |
gmann | ack | 16:10 |
bauzas | do people want to discuss about any blueprint here ? | 16:10 |
gmann | need more reviews on RBAC series https://review.opendev.org/q/topic:bp%252Fpolicy-defaults-refresh-2 | 16:10 |
liorf95 | yes please | 16:10 |
gmann | thanks gibi for reviews | 16:10 |
bauzas | yes, I'll do | 16:10 |
gmann | thanks | 16:10 |
liorf95 | https://blueprints.launchpad.net/nova/+spec/nvmeof-multipath | 16:10 |
bauzas | liorf95: we'll discuss about your bp in the open discussion topic | 16:10 |
sean-k-mooney | liorf95: i think yours is in opendicuss at the end but we coudl bring it up now too | 16:10 |
liorf95 | as you want | 16:11 |
bauzas | liorf95: that's in the agenda :) | 16:11 |
bauzas | anyway | 16:11 |
bauzas | secure RBAC, yes I'll review | 16:11 |
bauzas | we merged a few features before | 16:12 |
bauzas | I'll modify Launchpad on Friday | 16:12 |
chateaulav | just reviews for emulation to ensure nothing bltent sticks out | 16:12 |
bauzas | ok | 16:13 |
bauzas | ok, moving on then | 16:14 |
bauzas | #topic Review priorities | 16:14 |
bauzas | #topic Review priorities | 16:14 |
bauzas | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+label:Review-Priority%252B1 | 16:14 |
bauzas | most of the priorities were merged | 16:14 |
bauzas | nothing to tell here | 16:15 |
bauzas | let's continue to review | 16:15 |
bauzas | #topic Stable Branches | 16:15 |
bauzas | elodilles: your time | 16:15 |
elodilles | info stable/queens gate fix proposed: https://review.opendev.org/830327 (needs to backport to pike as well) | 16:15 |
elodilles | #info stable/xena gate should be better as Artom's workaround was merged (test_tagged_attachment) | 16:15 |
elodilles | #info stable/wallaby still needs the libvirt_disable_apic to merge + it seems the test_tagged_attachment issue is also present | 16:15 |
elodilles | oops | 16:15 |
elodilles | sorry | 16:15 |
elodilles | anyway, this is mostly it ^^^ | 16:16 |
elodilles | we might need the workarounds to stable/victoria too | 16:16 |
elodilles | #info stable/queens gate fix proposed: https://review.opendev.org/830327 (needs to backport to pike as well) | 16:17 |
elodilles | (just to have it in the logs) | 16:17 |
bauzas | :) | 16:18 |
bauzas | thanks elodilles | 16:19 |
elodilles | np | 16:19 |
bauzas | #topic Open discussion | 16:19 |
bauzas | liorf95: (lior Friedman) https://blueprints.launchpad.net/nova/+spec/nvmeof-multipath | 16:20 |
bauzas | liorf95: your turn | 16:20 |
liorf95 | yes please | 16:20 |
bauzas | or I can explain | 16:20 |
bauzas | maybe some of them don't understand the contect | 16:20 |
bauzas | context | 16:20 |
bauzas | so, | 16:20 |
liorf95 | we need this bp to be approved for https://review.opendev.org/c/openstack/nova/+/823941 | 16:20 |
bauzas | basically we had https://review.opendev.org/c/openstack/nova/+/823941 that was reviewed | 16:21 |
bauzas | and stephenfin and sean-k-mooney accepted the change | 16:21 |
bauzas | but, | 16:21 |
bauzas | as you can look at the comments, we saw it was actually a feature | 16:22 |
bauzas | that's why I accepted to have a discussion in this meeting to see whether folks accept to merge this one even if we had not accepted the blueprint before | 16:22 |
bauzas | for the moment, there is a procedural -2 from me | 16:23 |
bauzas | but if folks agree here to have an exception for https://blueprints.launchpad.net/nova/+spec/nvmeof-multipath as a specless bp, then I could remove it | 16:23 |
bauzas | now, the question : is https://blueprints.launchpad.net/nova/+spec/nvmeof-multipath a specless BP and are folks accepting it as a exception ? | 16:24 |
gibi | I've nothing against it. It is just a volume driver so no spec needed. And the change is really small | 16:24 |
bauzas | correct | 16:25 |
bauzas | change is small and only touching a very few places | 16:26 |
sean-k-mooney | i am supprotive fo doign it as a specless blueprint | 16:26 |
bauzas | ok, anyone having concern ? | 16:26 |
sean-k-mooney | its after teh cutoff but the scale of it is small enought that i think an excption can be made | 16:26 |
bauzas | yeah | 16:26 |
opendevreview | Merged openstack/placement master: Extend the RP db query to support any-traits https://review.opendev.org/c/openstack/placement/+/825848 | 16:27 |
opendevreview | Merged openstack/placement master: DB layer should only depend on trait id not names https://review.opendev.org/c/openstack/placement/+/826490 | 16:27 |
bauzas | basically we have deadlines for accepted blueprints because they change lots of things | 16:27 |
opendevreview | Merged openstack/placement master: Enhance doc of _get_trees_with_traits https://review.opendev.org/c/openstack/placement/+/825780 | 16:27 |
opendevreview | Merged openstack/placement master: Extend the RP tree DB query to support any-traits https://review.opendev.org/c/openstack/placement/+/825849 | 16:27 |
opendevreview | Merged openstack/placement master: Add any-traits support for listing resource providers https://review.opendev.org/c/openstack/placement/+/826491 | 16:27 |
bauzas | so we're procedural not for the sake of being it, but for making sure we are sure how this works | 16:27 |
bauzas | here, I don't see a problem | 16:27 |
gibi | I agree | 16:28 |
bauzas | ok so, | 16:29 |
bauzas | #agreed specless BP for https://blueprints.launchpad.net/nova/+spec/nvmeof-multipath as an exception | 16:30 |
bauzas | liorf95: I'll then remove my -2 | 16:30 |
bauzas | and +W it | 16:30 |
bauzas | that's it I had | 16:30 |
liorf95 | Thanks a lot for your efforts here | 16:31 |
bauzas | any other topic to discuss as a last minute ? | 16:33 |
gibi | - | 16:33 |
bauzas | looks not | 16:34 |
bauzas | thanks all | 16:34 |
bauzas | #endmeeting | 16:34 |
opendevmeet | Meeting ended Tue Feb 22 16:34:22 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:34 |
opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-02-22-16.02.html | 16:34 |
opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-02-22-16.02.txt | 16:34 |
opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-02-22-16.02.log.html | 16:34 |
gibi | thanks | 16:34 |
elodilles | thanks bauzas o/ | 16:34 |
liorf95 | Thanks | 16:34 |
gibi | sean-k-mooney: just a reminder you only left +1 on https://review.opendev.org/c/openstack/placement/+/82649 yesterday :) | 16:34 |
chateaulav | thanks o/ | 16:35 |
gmann | thanks | 16:35 |
gibi | kashyap: https://logserver.rdoproject.org/83/38983/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria/67d64d4/logs/undercloud/var/log/tempest/stestr_results.html.gz the tempest logs does not contain either the server uuid or the request id for the snapshot command :/ | 16:38 |
gibi | I cannot correlate the logs :/ | 16:39 |
kashyap | gibi: Good news! | 16:39 |
gibi | oh | 16:39 |
kashyap | gibi: I just talked to Peter Krempa on #virt, he already identified it as a libvirt bug ... | 16:39 |
kashyap | gibi: The problem is this, in short: | 16:39 |
kashyap | [quote] | 16:39 |
kashyap | libvirt has a piece of code which ensures that thh backing image of the reused destination image is added only when finishing the job. On cancellation of the [copy] job, we want to unplug the image, but the backing image was not yet plugged in. | 16:39 |
kashyap | [/quote] | 16:40 |
kashyap | It assumes a bit of libvirt storage knowledge, but Peter is already working on a fix | 16:40 |
kashyap | gibi: I'm filing RHEL bugs to get it backported to CentOS 8 and 9 | 16:41 |
gibi | awesome | 16:41 |
gibi | thanks for the info | 16:41 |
kashyap | gibi: But Peter is again asking what is the test "exactly doing" | 16:44 |
kashyap | Is this exactly it? | 16:44 |
kashyap | "boot servers, then snapshot the server and upload the image to Glance" | 16:45 |
kashyap | Did you mean boot a "server", as opposed to plural? | 16:45 |
gibi | kashyap: create a server, and then create a snapshot | 16:47 |
gibi | kashyap: the test case itself create multiple servers | 16:47 |
kashyap | I see | 16:48 |
kashyap | gibi: https://bugzilla.redhat.com/show_bug.cgi?id=2057067 | 16:57 |
kashyap | gibi: But Peter also says there might be another issue too, because: | 16:57 |
kashyap | ... "this is on a code path that can't propagate errors to the caller" | 16:57 |
kashyap | gibi: Anyway, he posted a patch here: https://listman.redhat.com/archives/libvir-list/2022-February/msg00790.html | 16:57 |
gibi | kashyap: ack. I would need a tripleoo job result where I can correlate logs better to be able to look more into it | 17:19 |
gibi | kashyap: I serched for the QEMU error in the upstream nova job results and there is a lot of hit | 17:20 |
gibi | but I guess it is a generic error message | 17:20 |
gibi | as the QEMU error does not correlate with the tempest test failure | 17:20 |
gibi | I stop for today but happy to help tomorrow if there are things to look at | 17:21 |
*** amoralej is now known as amoralej|off | 17:45 | |
opendevreview | Rajat Dhasmana proposed openstack/nova master: Add support for volume backed server rebuild https://review.opendev.org/c/openstack/nova/+/820368 | 18:53 |
opendevreview | Merged openstack/nova master: Support use_multipath for NVME driver https://review.opendev.org/c/openstack/nova/+/823941 | 18:53 |
opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Fix migration with remote-managed ports & add FT https://review.opendev.org/c/openstack/nova/+/829974 | 19:18 |
opendevreview | Rajat Dhasmana proposed openstack/python-novaclient master: Add parameter to rebuild boot volume https://review.opendev.org/c/openstack/python-novaclient/+/827163 | 19:33 |
whoami-rajat | dansmith, Hey, if you get some time, can you take a look at my feature patches? | 19:37 |
whoami-rajat | nova: https://review.opendev.org/c/openstack/nova/+/820368 | 19:37 |
whoami-rajat | novaclient: https://review.opendev.org/c/openstack/python-novaclient/+/827163 | 19:37 |
dansmith | whoami-rajat: ah I didn't see that you had updated them.. is it working now? | 19:37 |
dansmith | whoami-rajat: ah I didn't see that you had updated them.. is it working now? | 19:39 |
dansmith | whoami-rajat: I assume you've got a tempest patch somewhere (or planned) to poke it | 19:39 |
dansmith | ? | 19:39 |
opendevreview | Merged openstack/placement master: Add any-traits support for allocation candidates https://review.opendev.org/c/openstack/placement/+/826492 | 19:40 |
whoami-rajat | dansmith, just updated everything and yes it is working now. thanks! | 19:40 |
*** tbarron is now known as Guest252 | 19:40 | |
opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | 19:41 |
opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | 19:41 |
whoami-rajat | dansmith, I've mostly tested this manually but i can work on a tempest test, the problem is deadline is close | 19:41 |
whoami-rajat | dansmith, If i can get an FFE or propose the test after feature freeze then it will be great | 19:41 |
dansmith | I mean, I think a tempest test should be part of the requirements here.. was that not in the spec for this? | 19:42 |
dansmith | I would think it'd be easy right? take an existing rebuild test, make it use BFV, and pass the right version on rebuild? | 19:42 |
dansmith | if the tempest test shows it works and just needs some work after the deadline to get it into a merge-able shape, I think that's fine | 19:43 |
whoami-rajat | sure, I haven't looked at existing tests but if there's already one for rebuild, it should be pretty straightforward | 19:44 |
dansmith | yeah, tempest test is in the testing section of the spec... | 19:44 |
whoami-rajat | dansmith, I will try and get something working by tomorrow | 19:48 |
dansmith | ack | 19:48 |
whoami-rajat | thanks! | 19:48 |
opendevreview | Merged openstack/placement master: Remove unused compatibility code https://review.opendev.org/c/openstack/placement/+/826493 | 20:20 |
opendevreview | Merged openstack/placement master: Add microversion 1.39 to support any-trait queries https://review.opendev.org/c/openstack/placement/+/826719 | 20:23 |
gmann | dansmith: rbac series is ready, added release notes also in last patch please check if that is ok https://review.opendev.org/q/topic:bp/policy-defaults-refresh-2+status:open | 20:26 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!