| noonedeadpunk | jrosser_: should we merge https://review.opendev.org/c/openstack/openstack-ansible/+/847668 ? Commit message says part of debugging, but it seems we can indeed land that to unlbock stuff? | 06:13 |
|---|---|---|
| noonedeadpunk | also fwiw I'm mostly off today | 06:13 |
| jrosser_ | oh certainlt yes | 06:14 |
| jrosser_ | talking with #opendev yesterday it is only really us and tripleo being hurt with POST_FAILURE | 06:14 |
| jrosser_ | and extending the timeout lets use see really how long it takes as useful data points | 06:14 |
| jrosser_ | and though a bunch of stuff is now passing, it's very unclear if thats because removing journals/ara has made a difference | 06:15 |
| jrosser_ | or perhaps just the swift backends are faster again | 06:15 |
| opendevreview | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/847704 | 09:36 |
| mgariepy | morning. | 11:16 |
| jrosser_ | o/ hello | 11:16 |
| mgariepy | how are things going this morning ? | 11:17 |
| jrosser_ | seems that the POST_FAILURES thing may have subsided a bit | 11:17 |
| jrosser_ | i have a bunch of semi-related patches to merge, though its unclear if some underlying thing has just got better as well | 11:18 |
| mgariepy | with a lot of moving parts it's often kinda hard to determine what is causing issue | 11:19 |
| mgariepy | i just reviewed the 3hours ones. is there others that need attentions faster than others? | 11:19 |
| mgariepy | otherwise i'll probably spend a couple hours reviewing | 11:19 |
| jrosser_ | i think we need to let those merge then push through a few on stable/yoga | 11:24 |
| jrosser_ | but we will be in zuul-queue fail trouble if we go too quickly on that | 11:24 |
| mgariepy | would it be easier to find / browse logs if we had an html page with all the links ? | 11:25 |
| mgariepy | looking at : https://review.opendev.org/c/openstack/openstack-ansible/+/844817 | 11:25 |
| jrosser_ | not quite following tbh | 11:27 |
| jrosser_ | i was trying to see if we were uploading large amounts of nonsense | 11:27 |
| mgariepy | when debuging patches, often i end up looking for log files in directory. | 11:28 |
| jrosser_ | oh right i see, like a fully expanded tree of all the log files | 11:28 |
| mgariepy | yep | 11:28 |
| jrosser_ | that would be a suggestion for the zuul people i think, but it would be really great | 11:29 |
| mgariepy | it would make the logs file easier to find.. | 11:29 |
| jrosser_ | becasue you can't use the browser back button in any sensible way and i find that really annoying | 11:29 |
| jrosser_ | we can't do that in our job as we don't know what the upload URL will be | 11:30 |
| mgariepy | yueah in the zuul page you always endup in root dir if going back. and the index directory is not fun when needing look at multiple files. | 11:30 |
| mgariepy | we can probably use relative paths. | 11:30 |
| jrosser_ | sometimes i use the 'raw' link as that goes to an old-school directory browser | 11:31 |
| mgariepy | i use the raw ones when i need multiple files since it's annoying otherwise. | 11:32 |
| jrosser_ | we do have ~2000 lines in that tree output though :) | 11:34 |
| jrosser_ | that could turn into a different kind of unwieldy | 11:34 |
| *** dviroel|out is now known as dviroel | 11:37 | |
| mgariepy | indeed. 2000 lines is quite a long page :D haha | 11:39 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/stein: Remove periodic jobs https://review.opendev.org/c/openstack/openstack-ansible/+/847966 | 12:46 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Set zuul post-timeout to 3 hours https://review.opendev.org/c/openstack/openstack-ansible/+/847668 | 13:37 |
| *** dviroel is now known as dviroel|biab | 13:52 | |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Switch ARA report type from html to database for openstack-ansible jobs https://review.opendev.org/c/openstack/openstack-ansible/+/847652 | 14:18 |
| *** dviroel|biab is now known as dviroel | 14:21 | |
| opendevreview | Merged openstack/openstack-ansible master: Set zuul post-timeout to 3 hours https://review.opendev.org/c/openstack/openstack-ansible/+/847797 | 14:33 |
| opendevreview | Merged openstack/openstack-ansible master: Exclude /var/log/journal from log collection https://review.opendev.org/c/openstack/openstack-ansible/+/847836 | 14:34 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/yoga: Exclude /var/log/journal from log collection https://review.opendev.org/c/openstack/openstack-ansible/+/847827 | 14:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Exclude /var/log/journal from log collection https://review.opendev.org/c/openstack/openstack-ansible/+/847828 | 14:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/wallaby: Exclude /var/log/journal from log collection https://review.opendev.org/c/openstack/openstack-ansible/+/847829 | 14:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Set zuul post-timeout to 3 hours https://review.opendev.org/c/openstack/openstack-ansible/+/847990 | 14:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/wallaby: Set zuul post-timeout to 3 hours https://review.opendev.org/c/openstack/openstack-ansible/+/847991 | 14:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_manila stable/yoga: Create backends when running against manila_share https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/847992 | 14:44 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_manila stable/xena: Create backends when running against manila_share https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/847993 | 14:45 |
| *** dviroel is now known as dviroel|afk|lunch | 14:52 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic stable/yoga: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/846777 | 15:40 |
| jrosser_ | mgariepy: these are the things which are important for release parentproject:openstack/openstack-ansible branch:^stable/yoga status:open | 15:41 |
| jrosser_ | argh | 15:41 |
| jrosser_ | https://review.opendev.org/q/parentproject:openstack/openstack-ansible+branch:%255Estable/yoga+status:open+ | 15:41 |
| jrosser_ | unless you spot anything else which needs a backport | 15:42 |
| mgariepy | ok i'll check these this afternoon, i'm with kids today. | 15:43 |
| *** dviroel|afk|lunch is now known as dviroel | 16:23 | |
| spatel | what is going on here? boostrapping new deployment and hit this - https://paste.opendev.org/show/byO9uh7Hh7bTTpPZGNdk/ | 16:52 |
| spatel | server certificate verification failed. CAfile: none CRLfile | 16:53 |
| spatel | This is ubuntu 20.04 | 16:53 |
| jrosser_ | what if you wget or curl the url | 16:54 |
| spatel | no cert error on curl https://opendev.org/openstack/ansible-collections-openstack/ | 16:56 |
| spatel | re-running bootstrap again and see | 16:57 |
| spatel | same error failed again with same error https://paste.opendev.org/show/byO9uh7Hh7bTTpPZGNdk/ | 16:58 |
| spatel | jrosser_ check this out - https://paste.opendev.org/show/bf03iQiXuppINrufcWls/ | 16:59 |
| jrosser_ | sanity check - you've done apt update / dist upgrade before this? | 17:00 |
| spatel | I am running 20.04.1 | 17:01 |
| spatel | i did apt update | 17:01 |
| spatel | why do i need to dist upgrade? | 17:01 |
| mgariepy | apt upgrade as well ? | 17:01 |
| jrosser_ | `apt policy ca-certificates` ? | 17:02 |
| spatel | apt-get install ca-certificates | 17:02 |
| spatel | works for me | 17:02 |
| spatel | why don't we add "apt-get install ca-certificates" in boostrap script to make sure we have latest certs | 17:03 |
| jrosser_ | so you didnt have that? | 17:03 |
| jrosser_ | apt policy would have told you exactly what was going on | 17:04 |
| jrosser_ | too late now | 17:04 |
| spatel | i don't know how that missed out | 17:04 |
| mgariepy | do you run in a stripped off container? | 17:04 |
| spatel | This is brand new deployment | 17:04 |
| jrosser_ | if you really did boot from an image of Ubuntu 20.04.1 then i am not surprised it doesnt work | 17:05 |
| jrosser_ | the first thing you need to do on there is apt update / apt dist-upgrade to get a modern ca-certificates (as well as everything else) | 17:05 |
| spatel | we have 20.04.1 in kickstart which i didn't upgrade yet | 17:05 |
| damiandabrowski[m] | PS. we have fixed it: https://opendev.org/openstack/openstack-ansible-openstack_hosts/commit/8c4ef001c148dc16c980bfc7ea695311e6dfd612 | 17:06 |
| jrosser_ | it does not have the root CA that letsencrypt is using | 17:06 |
| mgariepy | https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/deploymenthost.html | 17:06 |
| mgariepy | step 2. | 17:06 |
| spatel | does apt dist-upgrade will change 20.04 to 22.10 ? | 17:06 |
| mgariepy | no | 17:06 |
| mgariepy | do-release-upgrade would | 17:06 |
| spatel | oh ok so dist-upgrade is safe to do | 17:07 |
| jrosser_ | safe is the wrong word, essential would be better :) | 17:07 |
| mgariepy | apt upgrade will install newer pkg without installing new pkg (ex: new dependency) dist-upgrade will upgrade and install new dependency as well. | 17:07 |
| spatel | jrosser_ copy that | 17:08 |
| mgariepy | where did you clone the opesntack-ansible repo from ? github ? | 17:08 |
| spatel | I used https://github.com | 17:09 |
| spatel | assuming opendev and github both in sync | 17:10 |
| mgariepy | they should | 17:11 |
| mgariepy | it does explains why your initial clone worked. | 17:11 |
| spatel | Yeah! | 17:12 |
| spatel | Does anyone using SmartNIC in their deployment for offloading stuff? | 17:15 |
| spatel | I am planning to buy 2 Mellanox nic to play in lab for learning | 17:16 |
| spatel | I want to test ASAP specifically | 17:17 |
| opendevreview | Merged openstack/openstack-ansible master: Do not try to run etcd in serial mode https://review.opendev.org/c/openstack/openstack-ansible/+/847362 | 17:18 |
| jrosser_ | you can do ASAP with a connectx-6dx | 17:31 |
| jrosser_ | that would be my choice today for a not-smartnic to do ASAP | 17:32 |
| jrosser_ | dmsimard: o/ hello | 17:32 |
| spatel | jrosser_ connectx-6dx is very costly for lab :( | 17:36 |
| jrosser_ | well do you really mean a smartnic | 17:36 |
| spatel | how about this one - https://www.fs.com/products/119649.html | 17:37 |
| jrosser_ | this might be better connectx-6-lx | 17:40 |
| jrosser_ | https://nvdam.widen.net/s/nqbftvdlh7/connectx-6-lx-en-card-product-brief-1731006 | 17:40 |
| jrosser_ | "Full hardware offload for OVS data plane" | 17:41 |
| jrosser_ | but i think you need to see really carefully what they actually mean by that | 17:41 |
| jrosser_ | if it is just east<>west L2 | 17:41 |
| jrosser_ | or any support for offload of north<>south L3 | 17:41 |
| jrosser_ | and if security groups work at all | 17:42 |
| jrosser_ | etc etc | 17:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/yoga: Do not try to run etcd in serial mode https://review.opendev.org/c/openstack/openstack-ansible/+/848002 | 17:44 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Do not try to run etcd in serial mode https://review.opendev.org/c/openstack/openstack-ansible/+/848003 | 17:44 |
| spatel | that is where i am confused so looking for answers from folks who already using them | 17:49 |
| spatel | I was thinking i can deploy those kind of card on network node to make them gateway for all VMs | 17:50 |
| jrosser_ | you would need the L3 offload to be working for that | 17:50 |
| jrosser_ | it would be excellent if that did work | 17:51 |
| spatel | that is the plan to make network node stronger and perfect gateway for all traffic | 17:55 |
| jrosser_ | tbh i have never seen any documentation of that | 17:59 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon stable/yoga: Fix ALLOWED_HOSTS https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/847653 | 18:03 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_horizon stable/xena: Fix ALLOWED_HOSTS https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/848004 | 18:05 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_horizon stable/wallaby: Fix ALLOWED_HOSTS https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/848005 | 18:05 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Return facts gathering to pre_tasks for masakari https://review.opendev.org/c/openstack/openstack-ansible/+/847260 | 18:07 |
| spatel | If you Offload whole OVS switch to NIC then it should use NIC for packet processing instead of CPU correct? - https://docs.nvidia.com/networking/display/TAN10/ASAP+OVS+Offload | 18:10 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Return facts gathering to pre_tasks for masakari https://review.opendev.org/c/openstack/openstack-ansible/+/848006 | 18:10 |
| *** akahat|ruck is now known as akahat|out | 18:11 | |
| jrosser_ | spatel: well, there were two things not really done when i looked at this before, security groups and any form of nat | 18:12 |
| spatel | Full OVS offload only support in OVN because OVN use virtual router in flow (doesn't need any namespace like lagacy design) | 18:12 |
| jrosser_ | but that was a long tie ago | 18:12 |
| jrosser_ | you'd have to try it, or maybe ask in the neutron channel if anyone is doing it | 18:12 |
| spatel | Mellanox doc saying it support NAT/firewall | 18:12 |
| jrosser_ | yes but thats not the same as neutron/OVS/OVN programming rules that are offloadable | 18:13 |
| jrosser_ | there was always a functionality gap that has been slowly closing over a very long time | 18:13 |
| jrosser_ | and it's quite possible that with OVN it just all magically works | 18:14 |
| jrosser_ | like in jamesdenton document here https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch-asap.html | 18:14 |
| jrosser_ | `Hardware offloading is not yet compatible with the openvswitch firewall driver. To ensure flows are offloaded, port security must be disabled` | 18:15 |
| spatel | I believe full OVS offloading only supported by OVN because its pure flow based function | 18:15 |
| jrosser_ | this is a great topic for more lab work and a blog post :) | 18:15 |
| spatel | hmm I am on it.. i really want to prove that so hunting for hardware :) | 18:16 |
| jrosser_ | the newer hardware gets incrementally better, larger flow tables, more offloads etc | 18:16 |
| jrosser_ | thats why i'd get the most recent that you can | 18:16 |
| spatel | its freaking costly so i may need approval from my boss :( | 18:18 |
| jrosser_ | try and find a price for the 6-Lx | 18:18 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Fix facts gathering for zun https://review.opendev.org/c/openstack/openstack-ansible/+/847261 | 18:23 |
| lowercase | BTW, if you ever accidentally upgrade the database up a version, you can hack the api version by changing the service number in the python code which will trick the module working on a future api version but old code. | 18:30 |
| lowercase | or you know... don't do a nova-manage db upgrade until you are ready to upgrade versions >.> | 18:31 |
| opendevreview | Merged openstack/openstack-ansible-os_manila stable/yoga: Create backends when running against manila_share https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/847992 | 20:57 |
| *** dviroel is now known as dviroel|out | 21:19 | |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Exclude /var/log/journal from log collection https://review.opendev.org/c/openstack/openstack-ansible/+/847827 | 21:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!