| opendevreview | Michal Arbet proposed openstack/ansible-collection-kolla master: Add Podman support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240 | 06:40 |
|---|---|---|
| SvenKieske | "morning" (it's 11:08 here) :D | 09:08 |
| SvenKieske | has anyone in here ever heard about https://salsa.debian.org/extrepo-team/extrepo-data/-/tree/master/repos/debian ? | 09:09 |
| SvenKieske | that should be the canonical link I guess: https://salsa.debian.org/extrepo-team/extrepo-data | 09:09 |
| SvenKieske | mhm, seems to be a perl script, basically: https://salsa.debian.org/extrepo-team/extrepo | 09:11 |
| kevko | Extrepo is doing same thing as Ansible templates 🤣 | 09:19 |
| kevko | SvenKieske we are using it in our Kolla images | 09:19 |
| SvenKieske | ah, we are? I was not aware of that, so why do we use it for everything? | 09:20 |
| SvenKieske | *not | 09:21 |
| kevko | Because this is Debian stuff | 09:21 |
| SvenKieske | okay, I mean "everything" in the sense: "for everything that wants to use stuff from http://osbpo.debian.net"? | 09:22 |
| SvenKieske | because that is debian stuff, I think? :) | 09:22 |
| kevko | We used to use that repo when we had binary support | 09:24 |
| kevko | But python3-podman is not in stable | 09:24 |
| kevko | So I've built and add to repo which I have control | 09:24 |
| kevko | And it's Openstack repo used for Kolla images | 09:24 |
| SvenKieske | ah okay; I'm still learning about this osbpo server, so bascially anybode can throw their binaries there? | 09:24 |
| SvenKieske | anybody* | 09:25 |
| kevko | Debian policy don't allow new packages to be in stable if it was not migrated from unstable during the release | 09:25 |
| kevko | No | 09:25 |
| kevko | Only me and zigo | 09:25 |
| kevko | As we are Debian developers | 09:25 |
| kevko | And I used to build all Openstack packages some time ago | 09:25 |
| SvenKieske | bottom line is: I really don't like importing key material via insecure http links and I was trying to find a better way..seems it's unnecessary complex :/ | 09:26 |
| kevko | With zigo | 09:26 |
| kevko | As Debian has 2 year cycle ..and Openstack half year ..you need to have separate repo | 09:26 |
| SvenKieske | yeah and zigo pointed me at "extrepo" over in #opendev | 09:27 |
| zigo | SvenKieske: I very much do not agree with you, extrepo is *extremely* simple. | 09:27 |
| SvenKieske | so it seems I'm running in circles | 09:27 |
| SvenKieske | zigo: I was not referring to extrepo with that comment, seems like a fairly simple perl script. just the integration situation in kolla seems complex for not yet truly understood reasons - to me. | 09:28 |
| kevko | SvenKieske: ok, situation is simple, if we want to podman support to be merged in kolla-ansible - we need python3-podman package (debian bookworm don't allow install pip package without venv) to be installed system wide ... but bookworm pool for apt is frozen ... so i built that package for debian and ask zigo to include in his repo ... | 09:33 |
| SvenKieske | sure, I already understood that part :) | 09:33 |
| kevko | maybe it can be uploaded to bookworm backports ..but it takes some time ... | 09:34 |
| kevko | and we don't have a time | 09:34 |
| SvenKieske | my problem is, and I fear that seems to be again a very controversial topic(TM) - where it really should not be - that it would be very very good if we can install software in a secure way. | 09:34 |
| SvenKieske | is that at least a shared goal? Because my understanding is, that it doesn't matter to most, but that might be a misunderstanding. | 09:35 |
| kevko | secure way ? :D so you will be ok if i will create a package with harmfull code and you will download it via https ? :D | 09:37 |
| SvenKieske | and it should be fairly trivial to import a gpg key in a secure way, no? Well I'm actually not really sure, key handling is hard, and outside packaging gpg usage is sharply declining. | 09:37 |
| SvenKieske | kevko: If you're not interested in discussing this topic in a reasonable manner that's fine for me, but than we don't have any discussion. this is not about https or gpg. | 09:37 |
| kevko | The best would be if we will not rely on any external apt/yum repository and have control | 09:38 |
| kevko | we can add package to git repository and install via dpkg ...is this safer for you ? | 09:39 |
| SvenKieske | sure, I'm only commenting on proposed solutions, not on some hypothetical "best solution" that doesn't exist in reality :) so if the "best" solution we can come up with is blindly importing gpg keys over insecure channels straight from the internet, there we go I guess. | 09:40 |
| SvenKieske | it's only marginally more secure - or not at all - then the venerable "curl $foo | sudo bash" | 09:41 |
| SvenKieske | sorry I'm half joking also. it's just really really frustrating. | 09:41 |
| zigo | SvenKieske: The secure way is *SUPER* easy: hardcode the osbpo gpg key in your ansible code. | 09:43 |
| zigo | End of the story. | 09:43 |
| zigo | There's no reason that I change the osbpo key... | 09:43 |
| SvenKieske | I was actually about to suggest that, yeah | 09:43 |
| zigo | I see no issue if you do that, though extrepo is probably nicer. | 09:44 |
| zigo | Also, I'd strongly suggest that you arrange for an osbpo mirror in the OpenStack CI. | 09:44 |
| zigo | I know wikimedia has a mirror, but that'd be the only one... | 09:44 |
| SvenKieske | well yeah, in my dream world the openstack CI would've got a mirror, or at least a pull through cache of everything installed during CI, but that's not the world I live in. | 09:45 |
| kevko | Yep, hardcode gpg is another way | 09:45 |
| kevko | Definitely it's not that world ..and I am afraid that it never will be 🤣 | 09:46 |
| SvenKieske | last time I asked about a simple pypi mirror and it was declined, because it takes too much space and apparently nobody can pay for that. | 09:46 |
| SvenKieske | I wonder where all that openinfra money goes | 09:46 |
| zigo | openinfra money != space sponsored by the CI sponsors | 09:47 |
| SvenKieske | last time I asked about _that_ the disturbing reply was "website redesign" | 09:47 |
| zigo | root@osbpo:/home/ftp/debian# du -sh . | 09:47 |
| zigo | 22G. | 09:47 |
| SvenKieske | yeah, I guess that answer is part of the problem and not part of the solution ;) | 09:47 |
| zigo | We're talking about 22 GB of data for ALL of osbpo, meaning from jessie-kilo all the way to bookworm-bobcat ... | 09:48 |
| SvenKieske | I guess the pypi mirror is more demanding, about some terabytes? also not very much imho, but alas, here we are. | 09:48 |
| zigo | That's a ridiculous SMALL amount of data... | 09:48 |
| SvenKieske | agreed, so it might be worth to pester fungi about it, when he's awake | 09:48 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Revert "CI: Disable bare metal testing on RL9/c9s" https://review.opendev.org/c/openstack/kayobe/+/897260 | 10:11 |
| fungi | SvenKieske: when did you ask for a simple pypi mirror (as opposed to the caching proxies we currently have in the opendev collaboratory), but more importantly, what led you to believe mirroring pypi is "simple"? https://pypi.org/stats/ | 12:05 |
| fungi | (this isn't just guessing either, we used to mirror pypi, but had to stop when actors like tensorflow started putting multi-gigabyte nightly snapshots in it and our servers simply couldn't keep up with the flood of new data) | 12:06 |
| mnasiadka | I had problems mirroring pypi metadata downstream, poor pulp server did not keep up with that nightly :) | 12:07 |
| fungi | also, the budget for the openinfra foundation is public. everyone is free to look at where the money is spent | 12:07 |
| SvenKieske | fungi: I did ask a couple of weeks/month ago why it wasn't mirrored - I think that's the equivalent action of asking it to be mirrored - and as we can see that's only 17.1 TB of data. I don't know every openinfra project, but assuming that maybe not every package in there might be used in openinfra, an incomplete mirror without those tensorflow packages would suffice? | 12:10 |
| SvenKieske | fungi: that was in the context when we had various problems with external pypi mirrors, that ended up reducing upstream mirrors, because those seemed unreliable to the pypi mirror team itself | 12:11 |
| SvenKieske | I still think it's simple, I also recognize openinfra doesn't seem to be able to cough up the necessary resources. so to clarify: I think it is "simple" from the perspective that all it takes are network, i/o etc bandwith. mirroring is not exactly complicated tech. It's "only" hard if you don't have the resources. | 12:12 |
| fungi | we do also mirror a subset of pypi, specifically we pre-build wheels for commonly-used packages that don't already provide them, so that job nodes don't spend extra time rebuilding them on every run | 12:13 |
| SvenKieske | and that's cool :) I don't know the details, maybe that subset could get expanded? I don't know. | 12:14 |
| fungi | right now all the resources we (the opendev collaboratory) have are donated by public cloud providers. attaching 17+tb of block storage to a virtual machine is nontrivial and fragile. putting a mirror of pypi on object storage would require someone to develop a glue layer for that | 12:14 |
| fungi | also, it would be tough to guarantee we as a small team of a few volunteers would be able to run a mirror of pypi that is more stable than pypi itself (even cloud providers suffer major outages form time to time), we'd at best just move where the outages people complain about are | 12:16 |
| SvenKieske | sure, but to give you some more credit: you are way better reachable than upstream :) also the openinfra infrastructure I could even maybe debug myself (to a certain extend). | 12:17 |
| fungi | if we had to run pypi we might not be nearly as available either ;) | 12:17 |
| SvenKieske | tbf I was able to reach the pypi people rather fast, but all they did was not fix the problem but shut off the offending server :D | 12:18 |
| SvenKieske | all I know is that "locally" mirrored repos tend to bring down CI failures by a lot, which might relax otherwise constrained CI resources. do you disagree? | 12:20 |
| SvenKieske | of course "locally" could be a stretch for 17 TB | 12:20 |
| fungi | but anyway, aiui the real question was a few gigabytes for the debian-openstack team's third-party package repository. it seems to me that it would be about the same as how we already mirror the "ubuntu cloud archive" (uca), 22gb is well within the available space on our afs fileservers: https://grafana.opendev.org/d/9871b26303/afs | 12:21 |
| SvenKieske | then again I'd say almost no openstack project probably needs AI frameworks mirrored locally | 12:21 |
| fungi | i'll add osbpo mirroring to tomorrow's opendev sysadmins meeting agenda to double-check with the team, but the request is because kolla wants to use them in ci jobs? | 12:22 |
| fungi | just making sure i have the details right | 12:22 |
| SvenKieske | from what I know we currently already use it somewhere. My initial problem was retrieving gpg keys via unencrypted internet traffic, which let to the suggestion "why don't we just mirror that?" | 12:23 |
| SvenKieske | a classic tale of "why do you solve problem X? solve Y instead!" as is typical in IT xD | 12:24 |
| * SvenKieske feels like a character in the british comedy show "the it crowd" | 12:24 | |
| fungi | why is retrieving gpg keys over an unencrypted channel a problem? is it because you're unable to check the signatures? also why are jobs retrieving signing keys at all? in our (opendev's) jobs, we serialize the public keys we expect to sign things into the job data so it's never retrieved | 12:25 |
| fungi | when writing jobs we look up the public keys, check the signatures on them, then put them into the job so that one ansible task is "add this key to the package manager's keychain" | 12:26 |
| SvenKieske | I didn't come up with these schemes, I only reviewed them :) so please ask kevko - he's probably gone mad from the questions already, and I can't even blame him - because everybody suggests a different method on "how to handle things correctly(TM)" | 12:26 |
| fungi | not "download a key from somewhere on the internet and add it to the keychain" | 12:26 |
| SvenKieske | that was another suggestion if you scroll back, just hardcode the key | 12:27 |
| fungi | it's the safest method, in my opinion | 12:27 |
| fungi | also the most straightforward and least prone to meet with intermittent network issues | 12:28 |
| SvenKieske | again I agree - I don't really care really which option get's used - transmit cryptographic keys via avian carriers - if it can be done securely, that is. | 12:28 |
| SvenKieske | that's a reference to the famous april fools rfc regarding tcp/ip over avian carriers, just in case I'm being taken way to serious :) | 12:29 |
| SvenKieske | still it might be good to mirror this stuff, it would probably also relax some external network traffic and increase build speed (assuming near local connections are faster), but I'm not senior enough on our build chains to comment on potential performance improvements. at least reliability might go up (but I have never witnessed reliability issues with the debian server myself) | 12:31 |
| opendevreview | Jakub Darmach proposed openstack/kolla-ansible stable/yoga: OpenSearch migration - prune old Kibana indices https://review.opendev.org/c/openstack/kolla-ansible/+/897667 | 12:56 |
| opendevreview | Jakub Darmach proposed openstack/kolla-ansible stable/yoga: OpenSearch migration - prune old Kibana indices https://review.opendev.org/c/openstack/kolla-ansible/+/897667 | 13:06 |
| hrw | SvenKieske: there are at least 3 rfc related to avian carriers. And some of them were even implemented. | 13:50 |
| hrw | SvenKieske: when it comes to mirroring pypi - it does not make sense. There are jobs already which fetch, build, mirror, host those python packages which are in openstack/requirements project. | 13:51 |
| hrw | SvenKieske: opendev CI does them for x86-64 and aarch64, for/on debian/centos/ubuntu iirc (been a while since last time I had to look there) | 13:52 |
| hrw | SvenKieske: adding 'yet another repo' was often a struggle. "we need to check is there a space on AFS" was common answer. usually followed by discussion 'what we can remove' | 13:53 |
| hrw | SvenKieske: https://review.opendev.org/c/opendev/system-config/+/881952 for example added Debian 'bookworm' mirroring | 13:54 |
| hrw | SvenKieske: read comments there | 13:55 |
| SvenKieske | hrw: thanks for the information provided. also: I did even review that patch, if you have read it yourself you could have noticed that :) thanks for the reminder. but your statements are at least partly conflicting with fungis above. | 13:57 |
| SvenKieske | hrw: I can't parse this: "when it comes to mirroring pypi - it does not make sense. There are jobs already which fetch, build, mirror, host those python packages which are in openstack/requirements project." it's contradicting | 13:59 |
| hrw | SvenKieske: s/pypi/whole pypi/ | 13:59 |
| SvenKieske | yeah, I already agreed to that if you read the whole discussion. I certainly don't need tensorflow mirrored :) | 14:00 |
| hrw | SvenKieske: wheel cache present on opendev CI does a bit more than just mirroring part of pypi | 14:01 |
| hrw | sorry, some comments I was writing during reading | 14:01 |
| SvenKieske | it's appreciated, though the original problems still persist: network calls can and will fail, introducing CI errors and inevitable performance overhead. | 14:02 |
| * SvenKieske sounds like a broken record at this stage. | 14:02 | |
| hrw | SvenKieske: we do not live in perfectly round world ;( | 14:03 |
| SvenKieske | and of course the tiny problem of supply chain attacks, which every engineer I know who works in software supply chains likes to pretend do not exist :) | 14:04 |
| SvenKieske | hrw: I violently agree :) | 14:04 |
| SvenKieske | I just try everyday to make it a little more round, often failing, but still trying :) | 14:04 |
| hrw | SvenKieske: there are 3rdparty repos we never asked to mirror cause we fetch not that much from them. and they fail from time to time | 14:05 |
| SvenKieske | hrw: sure. I ran CI pipeline for.. idk 3-10 years at some companies. | 14:05 |
| hrw | I build software since 2004 ;D including whole embedded distros at some time | 14:06 |
| SvenKieske | in my round world I have a local mirror of everything in my build graph and only sync the parts with the outer internet that need an update. | 14:06 |
| hrw | SvenKieske: happy person | 14:06 |
| SvenKieske | it's my imaginary round world, please let me be happy at least there :) | 14:07 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Fix podman logs https://review.opendev.org/c/openstack/kolla-ansible/+/893187 | 14:15 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 14:15 |
| greatgatsby | are there any docs for rebooting (after a system upgrade) computes and controllers? I'm looking for things like VM migration strategies, amphora, HA services, etc. | 14:19 |
| mnasiadka | In Kolla? I don't think so | 14:20 |
| greatgatsby | I'm using kolla-ansible, and so far I haven't found anything about rebootin/shutting-down a node. I don't think the answer is to simply issue a `shutdown -r now` and hope for the best, but I can't find any docs that address this. | 14:22 |
| hrw | greatgatsby: compute nodes should be handled by nova so existing instances have a chance of migration | 14:25 |
| jangutter | greatgatsby - for compute nodes (hypervisors) you should follow the nova guide on how to evacuate and disable the compute service on the hypervisor. Note that "evacuate" has got multiple meanings and there's important nuance! | 14:25 |
| jangutter | once the hypervisor is empty and disabled, rebooting it should be fine and won't disrupt your cluster. once the services are healthy, you can re-enable it. | 14:26 |
| jangutter | for the controllers, we usually stop the containers, first stop the keepalived container, verify that the VIP has migrated, then stop the backend containers. | 14:28 |
| SvenKieske | it also depends a bit on your deployment specifics. e.g. if you use ironic you might want to use ironic for server (re)boots, depending on what you want to do. e.g. if you want to shut off a server for maintenance and ironic manages it's power state you should of course do the necessary stuff in ironic (shut off the machine). | 14:28 |
| greatgatsby | Thanks for the replies. Right now I'm building an ansible role for controller/compute shutdown but I feel I'm just making it up as I go | 14:29 |
| jangutter | It's possible to do it more gracefully by telling haproxy to drain the backends, but there's also sql involvement, etc. | 14:29 |
| SvenKieske | fwiw, you _can_ pull the plug if you have a correctly setup ha env. you will lose in flight requests of course, but especially the control plane is pretty robust. | 14:30 |
| greatgatsby | yes, exactly, I'm doing or considering doing things like draining haproxy/rabbitmq/mysql on controllers, etc, just wondering if there's docs or if there's a product I could take some best practises from | 14:30 |
| SvenKieske | I'm also not aware of any docs. It might be nice to write some for some generic/"most of the time right" use cases. | 14:31 |
| greatgatsby | something like that would be great. Just when I (we) think we've covered everything, there's another, "oh, we could also do XYZ" | 14:31 |
| jangutter | It all boils down to the request rate and number of failures you can tolerate. | 14:31 |
| greatgatsby | for us, keeping thing as fault tolerant as possible is top priority. We don't want to disrupt client workloads at all, so we're trying to keep the cluster happy as a controller or compute is shutdown/rebooted | 14:32 |
| SvenKieske | these are the closest links to what you want I think: https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#upgrade-procedure and https://docs.openstack.org/kolla-ansible/latest/user/adding-and-removing-hosts.html | 14:32 |
| jangutter | You can go completely and utterly paranoid, _but_ the system is pretty much designed to handle "unexpected" failures gracefully. | 14:32 |
| SvenKieske | greatgatsby: that again depends on the workload: If your workload is from the "cloud native" kind where k8s does thousands of requests per minute to the openstack api, customers _will_ notice | 14:33 |
| SvenKieske | at least I wasn't able so far to make the APIs sufficiently HA to always answer any request. should be possible in theory with haproxy draining etc. | 14:34 |
| jangutter | Yes - in our case, we tolerate failures of _up to one minute_ but, we have the luxury of very low api rate during certain time periods. | 14:35 |
| SvenKieske | tbh I didn't really try _that_ hard. a cloud api that can't deal with a temporary network failure - which can happen anytime - is broken itself imho. | 14:35 |
| SvenKieske | especially k8s should've retry logic all over the place, but I have seen that fall apart as well. | 14:36 |
| SvenKieske | e.g. if k8s retries an operation that openstack does not support to be retried, and openstack just returns a 400 API error because you can't just retry this api call and then k8s just happily retries again :D | 14:37 |
| greatgatsby | SvenKieske: thanks, the "removing hosts" docs are along the lines of what I'm looking for. Just trying to ensure I've done everything possible to make the whole process as fault tolerant as possible. | 14:38 |
| SvenKieske | for low api call envs it doesn't really matter imho, you can reboot whole clusters without customers noticing. you should only watch out for rabbitmq health and in flight live migrations of VMs | 14:38 |
| greatgatsby | rabbitmq always seems to be the trouble-maker, no matter what we do. | 14:38 |
| SvenKieske | greatgatsby: sure! happy to help. it's always nice to see people caring for this stuff and not just blow up production :D | 14:38 |
| greatgatsby | :-) | 14:39 |
| SvenKieske | well at first you might want to check what's your settings there for HA deployment, because if the cloud is upgraded afaik we still do not set everything up for HA. | 14:39 |
| SvenKieske | I just had one or two cases where that was partly a problem, I still need to write some docs for that. | 14:39 |
| SvenKieske | you might want to check, as a beginning, this: https://docs.openstack.org/kolla-ansible/latest/reference/message-queues/rabbitmq.html#high-availability | 14:40 |
| jangutter | greatgatsby: if you have a test environment, be sure to experiment. It might be counterintuitive, but draining _could_ be more error prone and slower than a reboot on a controller. | 14:40 |
| greatgatsby | yeah we've scraped every rabbitmq HA doc from various OS vendors we could find to try to tune it, still not 100% it's as resilient as possible | 14:40 |
| greatgatsby | jangutter: thanks, never considered that | 14:41 |
| SvenKieske | newer rmq versions are better, on which release are you, if I may ask? | 14:41 |
| greatgatsby | yoga, hopefully zed in the next few months | 14:41 |
| SvenKieske | huh | 14:41 |
| SvenKieske | so the last rmq problems I saw was especially that upgrade :/ | 14:41 |
| SvenKieske | do you know how to monitor/check the rmq queues and exchanges and rmq cluster state? there are never "hard" errors there, but they are pervasive, most of the time easy to fix, when you know where to look | 14:42 |
| SvenKieske | I really need to write those docs :) | 14:43 |
| jangutter | SvenKieske: oooh you triggered me. I've recently had issues in victoria with Neutron exchanges getting "stuck" when adding a new controller to the cluster.... | 14:43 |
| mnasiadka | the rmq gui is a bit useful, rabbitmqctl cluster_status is usually useless and logs are misleading :) | 14:44 |
| SvenKieske | jangutter: well the quickfix is: delete the exchange/queue and restart neutron | 14:44 |
| jangutter | SvenKieske: bwahahaha, that's EXACTLY what my mitigation script does. Do you have a way to detect whether the exchange is broken? | 14:44 |
| greatgatsby | mnasiadka: thanks, yeah, we check the cluster_status periodically and it almost always reports everything is ok, even when the cluster is imploding | 14:44 |
| SvenKieske | mnasiadka: these days you can also list|filter all the queues via cli. it's a little bit easier in some setups because you need to first proxy the gui through ssh most of the time.. | 14:45 |
| mnasiadka | basically I haven't seen a lot of RMQ issues after we implemented HA everywhere | 14:45 |
| opendevreview | Michal Arbet proposed openstack/ansible-collection-kolla master: Add Podman support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240 | 14:45 |
| SvenKieske | jangutter: depends, if you have rabbitmq HA it might be the case that just some queue is out of sync, you could filter for that and simply resync it and if you're lucky it works then :) should be faster then neutron restart. | 14:46 |
| opendevreview | Michal Arbet proposed openstack/ansible-collection-kolla master: Add Podman support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240 | 14:46 |
| SvenKieske | but I've only really experience with problems in HA with the old (pre)3.8 rmq release if memory serves correctly. these were rather problematic. | 14:47 |
| SvenKieske | then again that 3.11 setup from yoga to zed was somehow also butchered.. but it wasn't HA. | 14:47 |
| SvenKieske | they just went nuclear and redeployed the complete rabbitmq service, did also work..so..yeah | 14:48 |
| SvenKieske | sadly you can't debug what is already deleted, would've been interesting, I still wait for feedback from another deployment where they try to recreate the problems in dev. | 14:49 |
| jangutter | So that's good news then, we're planning our migration to yoga + rmq3.9 | 14:49 |
| SvenKieske | pay attention to the erlang version, there was a very recent fix afaik to the erlang release used in yoga | 14:50 |
| SvenKieske | deployment seemed to be completely broken, at least for some users | 14:50 |
| SvenKieske | that one: https://review.opendev.org/c/openstack/kolla/+/897352 | 14:51 |
| jangutter | yeah, we're centos 9 stream, not debian.... we got "different" issues :-p | 14:51 |
| SvenKieske | well you already wrote you use stream :D | 14:52 |
| mnasiadka | jangutter: you deliberately chose stream? uh | 14:53 |
| SvenKieske | mnasiadka: any idea why this could still probably fail? I'm starting to become clueless :/ https://zuul.opendev.org/t/openstack/build/ca7a2a6f88924b458ec32b79d007611b/log/primary/logs/build/000_FAILED_fluentd.log | 14:54 |
| jangutter | mnasiadka: heh, yeah, long story. | 14:54 |
| SvenKieske | I left all the wisdom I could extract in the linked changeset as a comment. the files seem fine, but fail to download from two completely separate cloud providers. | 14:55 |
| mnasiadka | SvenKieske: oh boy, haven't seen that, but I'm not the APT expert here ;) | 14:55 |
| mnasiadka | SvenKieske: tried to built locally somewhere instead OpenDev? | 14:55 |
| SvenKieske | maybe I am only having bad luck and I should try thrice. | 14:55 |
| SvenKieske | mnasiadka: not yet, I always fear that step :D I have PTSD from using devstack in the past.. I guess I could "just" run the kolla build container somehow. | 14:56 |
| kevko | oh, i missed very interesting discussion about gpg and apt :D :D :D | 14:56 |
| SvenKieske | kevko: no no, please let's have an end with that :) now I have problems downloading with apt. seems apt doesn't like me today. | 14:57 |
| kevko | Unable to load bucket: packages.treasuredata.com | 14:57 |
| SvenKieske | I already see me reading apt-get source code to track down this error code, stackoverflow is really not helpful here. | 14:57 |
| kevko | just open the url ? :D | 14:57 |
| SvenKieske | yeah, but when I manually navigate to the file it works? | 14:58 |
| SvenKieske | mhm need to double check the URLs, I mean it's provided by fluentd folks.. | 14:58 |
| kevko | https://td-agent-package-browser.herokuapp.com/lts/5/ubuntu/jammy/dists/jammy | 14:58 |
| kevko | https://s3.amazonaws.com/packages.treasuredata.com/lts/5/ubuntu/jammy/dists/jammy/InRelease | 14:58 |
| kevko | :D | 14:58 |
| kevko | i would say they have something broken :) | 14:59 |
| SvenKieske | oh come on, I hate this | 14:59 |
| SvenKieske | but thanks! | 14:59 |
| SvenKieske | at least I can now go file a bug.. | 14:59 |
| SvenKieske | classic case of staring too long at a problem and missing the forest for the trees.. | 14:59 |
| SvenKieske | mhm, could I just use that aws URL? I highly doubt it.. | 15:00 |
| SvenKieske | but how does this work for anyone? I used the upstream provided URLs, with apt-get, I mean how broken can stuff be? | 15:02 |
| kevko | did you try to build it locally ? | 15:02 |
| mnasiadka | he's afraid :) | 15:03 |
| SvenKieske | ok ok I'll try, but I fear I'll be in a whole new rabbitwhole when trying to run kolla-build manually :D I use it through CI exclusively since 3 years or some such.. | 15:04 |
| kevko | SvenKieske: https://td-agent-package-browser.herokuapp.com/lts/5/debian/bookworm << are u sure that this is repo ? :D | 15:06 |
| kevko | package-browser ? | 15:06 |
| mnasiadka | https://docs.fluentd.org/installation/install-by-deb#step-1-install-from-apt-repository | 15:07 |
| mnasiadka | here are the links to APT repos | 15:07 |
| mnasiadka | (in the script) ;) | 15:07 |
| kevko | no, there is a url of debian package which is installed directly with dpkg :D | 15:10 |
| kevko | :D | 15:10 |
| kevko | aaaa ..that debian package is installing repo probably | 15:10 |
| SvenKieske | yes | 15:12 |
| SvenKieske | I'm fairly certain I extracted the correct URL string from that, but let me double check | 15:13 |
| SvenKieske | mhm it's not matching | 15:15 |
| SvenKieske | it should be https://packages.treasuredata.com/lts/5/ubuntu/jammy/ | 15:15 |
| SvenKieske | the thing is, if you open that link it's a 404 | 15:15 |
| kevko | let me check it :) | 15:15 |
| SvenKieske | I guess that is how I originally ended up with the herokuapp.com link | 15:16 |
| SvenKieske | the deb package from the installer at least installs this sources file in /etc/apt/sources.list.d/ with the above 404 link | 15:17 |
| SvenKieske | maybe that is also some webserver magic which provides a 404 if your user agent doesn't match apt-get | 15:17 |
| kevko | why you are trying to instal apt repo via deb package as they are mentioning in doc ? | 15:19 |
| kevko | i think it's enough to just copy key and render apt sources list | 15:19 |
| frickler | it doesn't matter if the base link has a 404, apt doesn't access that link, but just ./dists/jammy/InRelease behind it | 15:21 |
| frickler | this is s3 storage, not usual web directory browsing | 15:22 |
| SvenKieske | yeah, but I don't do what kevko said. I don't install that deb, I just took the URL, anyway I'll provide the original URL and see if that works | 15:23 |
| opendevreview | Sven Kieske proposed openstack/kolla master: bump td-agent lts from v4 to v5 https://review.opendev.org/c/openstack/kolla/+/894948 | 15:25 |
| SvenKieske | I'm fairly certain it will work, thanks guys :) | 15:28 |
| kevko | SvenKieske: it will | 15:29 |
| kevko | because in that deb package is this | 15:29 |
| kevko | root@pixla:/home/michalarbet/ultimum/git/upstream/kolla# cat /etc/apt/sources.list.d/fluent-lts.sources | 15:29 |
| kevko | Types: deb | 15:29 |
| kevko | URIs: https://packages.treasuredata.com/lts/5/ubuntu/jammy/ | 15:29 |
| kevko | Suites: jammy | 15:29 |
| kevko | Components: contrib | 15:29 |
| kevko | Signed-By: /usr/share/keyrings/fluent-lts-archive-keyring.gpg | 15:29 |
| kevko | (pgp is downloaded from URL in kolla ..so everything should work) | 15:30 |
| SvenKieske | yep :) | 15:30 |
| SvenKieske | I miss the old web where you could browse to any URL and not interact with weird S3 gateways :D | 15:31 |
| SvenKieske | guess I'm getting old | 15:31 |
| kevko | SvenKieske: and you fixed only one occurence | 15:33 |
| kevko | michalarbet@pixla:~/ultimum/git/upstream/kolla$ git show 9e4c14fc3 | grep -i browser | 15:33 |
| kevko | + url: "https://td-agent-package-browser.herokuapp.com/lts/5/debian/bookworm" | 15:33 |
| kevko | + url: "https://td-agent-package-browser.herokuapp.com/lts/5/debian/bookworm" | 15:33 |
| kevko | + url: "https://td-agent-package-browser.herokuapp.com/lts/5/ubuntu/jammy" | 15:33 |
| SvenKieske | ah right | 15:34 |
| SvenKieske | thanks, will follow up | 15:34 |
| SvenKieske | there was also another error I think | 15:36 |
| opendevreview | Sven Kieske proposed openstack/kolla master: bump td-agent lts from v4 to v5 https://review.opendev.org/c/openstack/kolla/+/894948 | 15:37 |
| mnasiadka | have fun guys, I started the extremely interesting season of moving all customers from CentOS 8 Stream to Rocky Linux 9... | 15:38 |
| SvenKieske | that sounds way more fun than fixing wrong mirror urls :D | 15:39 |
| kevko | mnasiadka: it's for kids, try it from centos to debian :) | 15:41 |
| mnasiadka | no thanks, that's why you're using centos libvirt on debian :) | 15:41 |
| kevko | :D :D :D | 15:41 |
| kevko | mnasiadka: yeah, that's the last piece :D ..but this is only one case ..because we adopted very old cloud ... another clouds are just debian :) | 15:42 |
| mnasiadka | yeah, but I remember RH likes to ,,vendor'' their libvirt | 15:42 |
| kevko | yep ! exactly :) | 15:43 |
| mnasiadka | so no live migration from centos to ubuntu/debian | 15:43 |
| kevko | exactly :) | 15:43 |
| greatgatsby | is the rocky 9 base the most tested, or considered the most stable? We're using ubuntu in yoga, but would consider switching if rocky 9 would conform better to the defaults | 15:43 |
| mnasiadka | well, there's the rpm vs deb barricade | 15:43 |
| mnasiadka | you're either rpm or deb lover I guess | 15:44 |
| kevko | yep | 15:44 |
| kevko | greatgatsby: nothing is tested :D | 15:44 |
| greatgatsby | hahahaha | 15:44 |
| mnasiadka | the only thing you tested is tested | 15:44 |
| kevko | greatgatsby: well, we are not running rally or tempest against our various images | 15:45 |
| greatgatsby | so is it better to have the container distro match the baremetal distro? We're an ubuntu shop really | 15:45 |
| kevko | it doesn't matter ..but let's say yes, it's the best option | 15:45 |
| greatgatsby | thanks, ok | 15:45 |
| mnasiadka | matters only when RH backports libvirt features that require kernel features (that they also backport) - so you need to have these in sync - but I don't think it happened recently | 15:47 |
| kevko | for example we are using debian (or some customers ubuntu) ..because of debian, package system, release cycle, python version , and no potentional problems as it is sometimes with rhel derivatives ( closed yum repos and similar) | 15:47 |
| mnasiadka | and matters when you want some security certifications (don't remember which ones) | 15:47 |
| kevko | yep | 15:48 |
| kevko | amen | 15:49 |
| SvenKieske | I remember vividly some ubuntu backport breaking live migrations. personally I like the rpm format more. But I only use debian/ubuntu based openstack installs so far. | 15:50 |
| kevko | sometimes it depends which distro has nicer stickers for your laptop | 15:51 |
| kevko | SvenKieske: when ? | 15:51 |
| SvenKieske | "btw, have you heard of arch linux?" <- this is a meme where I live :D | 15:51 |
| kevko | :d | 15:52 |
| kevko | https://qph.cf2.quoracdn.net/main-qimg-cd9326d4d6a641ca3f0eb83cdbb92d65-lq | 15:53 |
| SvenKieske | kevko: sorry misremembered it was a linux kernel source backport: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1887490 | 15:56 |
| SvenKieske | and canonical even claimed "It's not a regression because there is a workaround"..I'm still angry when reading such hyperbole | 15:57 |
| SvenKieske | here is the relevant part where the regression is noticed: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1887490/comments/53 | 15:58 |
| SvenKieske | you can't just add new cpu flags mid release and expect everything to still work. | 15:58 |
| SvenKieske | just canonical things I guess | 15:58 |
| kevko | RUN LINE_TO_REMOVE=$(grep -in "feature name='mpx'" /usr/share/libvirt/cpu_map/x86_Skylake-Server-noTSX-IBRS.xml | awk -F ':' '{print $1}') && \ | 15:58 |
| kevko | sed -i "${LINE_TO_REMOVE}d" /usr/share/libvirt/cpu_map/x86_Skylake-Server-noTSX-IBRS.xml | 15:58 |
| kevko | this i have in my libvirt for example | 15:58 |
| SvenKieske | lol | 15:59 |
| SvenKieske | poor libvirt I guess :) | 15:59 |
| SvenKieske | yeah all those cpu bugs where quite fun.. and they keep on coming | 15:59 |
| SvenKieske | have you read about the latest where booting with "mitigations=off" miscompiles some stuff? :D | 16:00 |
| kevko | SvenKieske: check this :D | 16:01 |
| kevko | https://paste.openstack.org/show/bYBE7AjrOT6Ls6hol3Gg/ | 16:01 |
| SvenKieske | what is OVMF? | 16:02 |
| SvenKieske | regarding mitigations=off: https://lore.kernel.org/lkml/D99589F4-BC5D-430B-87B2-72C20370CF57@exactcode.com/ | 16:03 |
| kevko | uefi firmware | 16:07 |
| kevko | Open Virtual Machine Firmware (OVMF) | 16:10 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily while we restart it for a combined runtime and platform upgrade | 16:33 | |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Fix podman logs https://review.opendev.org/c/openstack/kolla-ansible/+/893187 | 17:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!