| opendevreview | Julia Kreger proposed openstack/ironic stable/victoria: CI: Lower test VM memory by 200MB https://review.opendev.org/c/openstack/ironic/+/821140 | 00:45 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/ironic stable/victoria: CI: Lower test VM memory by 400MB https://review.opendev.org/c/openstack/ironic/+/821140 | 00:45 |
| * TheJulia sighs | 00:45 | |
| opendevreview | Verification of a change to openstack/ironic master failed: Ensure desired permissions on all written pxe files https://review.opendev.org/c/openstack/ironic/+/809295 | 00:47 |
| opendevreview | Verification of a change to openstack/ironic master failed: Move place_loaders_for_boot to boot driver __init__ https://review.opendev.org/c/openstack/ironic/+/819729 | 00:47 |
| arne_wiebalck | Good morning, Ironic! | 07:37 |
| rpittau | good morning ironic! o/ | 07:37 |
| janders | good morning arne_wiebalck rpittau and Ironic o/ | 07:48 |
| rpittau | hey janders :) | 07:48 |
| dtantsur | morning folks | 08:06 |
| hgy | hey arne_wiebalck rpittau janders dtantsur and Ironic o/ | 08:08 |
| arne_wiebalck | hey rpittau hgy dtantsur o/ | 08:12 |
| rpittau | hey arne_wiebalck hgy dtantsur :) | 08:13 |
| jingvar | \0 | 08:23 |
| jingvar | I repeate my question | 08:41 |
| jingvar | bifrost stable xena | 08:41 |
| jingvar | exception Validate network | 08:42 |
| jingvar | when - itf_netaddr1 | ipaddr('network') != itf_netaddr2 | ipaddr('network') | 08:42 |
| jingvar | itf_netaddr1: "{{ itf_infos['address'] }}/{{ itf_infos['netmask'] }}" | 08:42 |
| jingvar | itf_netaddr2: "{{ itf_infos['network'] }}/{{ itf_infos['netmask'] }}" | 08:42 |
| jingvar | how it works? | 08:43 |
| rpittau | if anyone has a minute please check those 3 patches before we ask for the bugfix release, thanks! https://review.opendev.org/c/openstack/ironic-python-agent/+/820032 https://review.opendev.org/c/openstack/ironic-python-agent/+/820997 https://review.opendev.org/c/openstack/ironic-python-agent/+/820324 | 08:56 |
| arne_wiebalck | rpittau: on the last one, I am confused: why do we remove the job now if that is supposed to be removed only after yoga (sorry if that is obvious :)) | 09:09 |
| jingvar | it was missed broadcast | 09:24 |
| arne_wiebalck | mnasiadka: seems like I was wrong, we only run with the first of the two patches which turns partx -u into partx -a | 09:32 |
| arne_wiebalck | mnasiadka: IIRC, there were two parts ... | 09:33 |
| mnasiadka | arne_wiebalck: well, yesterday I realised that I used IPA from master (since Kayobe uses ironic-python-agent-builder branch:master by default) - when I used the proper branch (wallaby) + my patch in progress, everything seems to be working. | 09:33 |
| mnasiadka | So sorry for finding out new issues like a moron ;-) | 09:33 |
| arne_wiebalck | heh | 09:34 |
| arne_wiebalck | wallaby has the two patches, so all seems good ... ? | 09:35 |
| rpittau | arne_wiebalck: that job was added because of the initial TC announcement where they decided to remove py3.6 testing from yoga, but after a discussion it was decided to leave it, so that job is not necessary anymore | 09:36 |
| mnasiadka | arne_wiebalck: yeah, all good for now, I just need to add unit tests for my patch - I've redeployed like 5 servers, no issues seen anymore. :) | 09:37 |
| arne_wiebalck | rpittau: oh, ok, thanks! | 09:37 |
| arne_wiebalck | mnasiadka: nice! | 09:37 |
| opendevreview | Merged openstack/ironic-python-agent master: Burn-in: Add SMART self test to disk burn-in https://review.opendev.org/c/openstack/ironic-python-agent/+/819505 | 09:38 |
| opendevreview | Merged openstack/ironic master: CI: Lower test VM memory by 200MB https://review.opendev.org/c/openstack/ironic/+/821047 | 09:55 |
| rpittau | recheck all the things! | 09:58 |
| opendevreview | Verification of a change to openstack/ironic master failed: Move place_loaders_for_boot to boot driver __init__ https://review.opendev.org/c/openstack/ironic/+/819729 | 10:21 |
| *** redrobot6 is now known as redrobot | 10:23 | |
| opendevreview | Verification of a change to openstack/ironic master failed: Ensure desired permissions on all written pxe files https://review.opendev.org/c/openstack/ironic/+/809295 | 10:27 |
| opendevreview | Merged openstack/ironic-python-agent master: Instruct qemu-img to write image zeros to disk. https://review.opendev.org/c/openstack/ironic-python-agent/+/820997 | 11:00 |
| opendevreview | Merged openstack/sushy master: Use only Yoga tests https://review.opendev.org/c/openstack/sushy/+/820322 | 11:22 |
| opendevreview | Merged openstack/ironic-python-agent master: Use only Yoga tests https://review.opendev.org/c/openstack/ironic-python-agent/+/820324 | 11:40 |
| opendevreview | Merged openstack/ironic-python-agent master: Burn-in: Add options for named log files https://review.opendev.org/c/openstack/ironic-python-agent/+/820032 | 11:54 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic-inspector bugfix/10.9: Update .gitreview for bugfix/10.9 https://review.opendev.org/c/openstack/ironic-inspector/+/821209 | 13:08 |
| opendevreview | Verification of a change to openstack/ironic master failed: Ensure desired permissions on all written pxe files https://review.opendev.org/c/openstack/ironic/+/809295 | 13:51 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Ignore fake nodes in the power sync loop https://review.opendev.org/c/openstack/ironic/+/821215 | 13:59 |
| dtantsur | Trying to enable sqlite foreign keys in Ironic. 15 unit tests failed. Oh well. | 14:09 |
| opendevreview | Merged openstack/ironic-inspector bugfix/10.9: Update .gitreview for bugfix/10.9 https://review.opendev.org/c/openstack/ironic-inspector/+/821209 | 14:12 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Enable foreign keys in SQLite https://review.opendev.org/c/openstack/ironic/+/821219 | 14:14 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: [WIP] Install pip package in dib based images https://review.opendev.org/c/openstack/bifrost/+/821223 | 14:32 |
| TheJulia | good morning | 14:49 |
| TheJulia | dtantsur: I saw the metal3 mailing list post question w/r/t performance | 14:50 |
| TheJulia | you should be able to use the benchmark scripts I put up | 14:50 |
| TheJulia | 10,000 nodes, and then try to retrieve them | 14:51 |
| TheJulia | It wouldn't be API based, but it would give you an idea of object -> lower level of the api | 14:51 |
| TheJulia | dtantsur: and only 15 failures is impressive | 14:52 |
| dtantsur | heh | 15:01 |
| dtantsur | good morning TheJulia | 15:01 |
| dtantsur | I've prepared my own script based on fake-hardware | 15:02 |
| TheJulia | post it to the folder! | 15:02 |
| TheJulia | someone may find it useful | 15:02 |
| TheJulia | or we can use it to cry w/r/t overhead | 15:02 |
| dtantsur | https://paste.opendev.org/show/811581/ it's very rough :) | 15:02 |
| TheJulia | yeah, creation is super slow with our object model and overhead... and sqllite is also slow when inserting new rows | 15:03 |
| dtantsur | not that slow with WAL, I think | 15:04 |
| TheJulia | I think it is 64k block space extensions... | 15:04 |
| TheJulia | but it has been *ages* | 15:04 |
| TheJulia | like... I dealt with Computer Room Air Conditioners last time I was banging my head into sqllite performance | 15:04 |
| TheJulia | we used a backup tool which stored everything in sqllite and then exposed an rpc service to interact with the tool | 15:19 |
| TheJulia | And we had so many backup servers and so many tapes per day we actually had to write a tool | 15:19 |
| TheJulia | reviews on https://review.opendev.org/q/Id8c0e4830011ca2fa526df461ed5b9b01f769cf9 would be appreciated | 15:21 |
| TheJulia | err, I didn't put that on wallaby | 15:22 |
| * TheJulia fixes | 15:22 | |
| dtantsur | I guess sqlite has developed a lot recently. WAL, as an example. | 15:23 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/wallaby: CI: Lower test VM memory by 400MB https://review.opendev.org/c/openstack/ironic/+/821233 | 15:24 |
| TheJulia | I *think* we saw that as a way to help performance | 15:24 |
| TheJulia | and I think it was around then, but I don't think we had any way to turn it on | 15:24 |
| TheJulia | nor any knobs around the db | 15:24 |
| TheJulia | we actually asked if we could back it with postgres or mysql at one point because we were a huge postgres shop and we were told nope. | 15:25 |
| TheJulia | initialization was hard coded into the app or something like that | 15:25 |
| * TheJulia twitches | 15:25 | |
| TheJulia | oh w/r/t neutron failures, it looks like a different fix is now is now in review which should help our CI failures | 15:28 |
| opendevreview | Julia Kreger proposed openstack/ironic bugfix/18.1: Explicitly pin upper sushy version https://review.opendev.org/c/openstack/ironic/+/820927 | 15:33 |
| TheJulia | dtantsur: ^^ i think that might fix things with the bugfix branch pin | 15:33 |
| dtantsur | great! | 15:39 |
| rpittau | zuul is down \o/ | 15:45 |
| TheJulia | oh awesome | 15:46 |
| rpittau | https://zuul.opendev.org/ is giving service unavailable | 15:46 |
| TheJulia | does this mean I can stop focusing on making branches work reliably? | 15:46 |
| TheJulia | happened ysterday | 15:46 |
| rpittau | TheJulia: no, please, and thanks btw, 18.1 is very much needed :) | 15:46 |
| TheJulia | I figured :) | 15:47 |
| TheJulia | I'm *really* hoping our OOM counts start to drop over the next 24 hours | 15:47 |
| TheJulia | If I can get the patches merged to xena, victoria, wallaby the world may end up being a happier place again | 15:48 |
| TheJulia | https://review.opendev.org/c/openstack/ironic/+/821103 <-- another memory related patch. Threading afaik is on my default, so two workers is kind of overkill | 15:49 |
| rpittau | about that, TheJulia dtantsur are there any outstanding changes before asking for the release of ironic and bifrost ? | 15:49 |
| TheJulia | and each worker is 60-90 mb | 15:49 |
| TheJulia | https://review.opendev.org/c/openstack/bifrost/+/820390 | 15:50 |
| rpittau | I'm looking at stevebaker[m] patch chain for example, not sure will be able to squeeze that in | 15:50 |
| TheJulia | I doub't it | 15:51 |
| TheJulia | I don't think any of that is pressing for now() | 15:51 |
| rpittau | ok | 15:51 |
| rpittau | oh the dependencies on that merged, good | 15:51 |
| dtantsur | the changes from https://review.opendev.org/c/openstack/bifrost/+/819649/ to https://review.opendev.org/c/openstack/bifrost/+/820479/ should be either merged together or delayed together IMO | 15:51 |
| TheJulia | dtantsur: what is your preference? | 15:51 |
| TheJulia | if no, then I think we can just go ahead and release master branch | 15:52 |
| dtantsur | It depends on the cycles of the reviewers :) it won't be a blocker for me if we don't merge it right away | 15:52 |
| dtantsur | I'm honestly not sure if anybody consumes Bifrost releases (and why) | 15:52 |
| TheJulia | some people do | 15:52 |
| TheJulia | mostly those who don't understand they can likely just use master branch | 15:52 |
| rpittau | we can consider to stop releasing bugfix branches for bifrost | 15:53 |
| TheJulia | eh, *shrug* | 15:54 |
| TheJulia | its a cheap release | 15:54 |
| * TheJulia says release | 15:54 | |
| dtantsur | I have in fact considered that | 15:54 |
| dtantsur | the CI on bugfix branches quickly becomes unmanageable | 15:54 |
| rpittau | (menatime, zuul is back) | 15:54 |
| TheJulia | https://memegenerator.net/instance/85937895/winter-is-coming-brace-yourselves-a-release-is-coming | 15:56 |
| rpittau | :) | 15:56 |
| TheJulia | dtantsur: it is, but I suspect in part because we tend not to quickly pin things like... requirements | 15:56 |
| TheJulia | and it is easy to forget | 15:56 |
| TheJulia | rpittau: are you doing the release honors? | 15:56 |
| TheJulia | or should I? | 15:56 |
| rpittau | I'm taking care of that, yeah | 15:57 |
| TheJulia | dtantsur: but definitely worth a retrospective during a midcycle | 15:57 |
| rpittau | I fixed the python-pip problem in bifrost dib ipa, but something else broken there :/ | 15:57 |
| TheJulia | :( | 15:59 |
| rpittau | ok, going to open release requests for bifrost and ironic | 16:00 |
| TheJulia | rpittau: I just read your reply to ianw | 16:02 |
| TheJulia | removed my +2 to let you two sort through the questions | 16:03 |
| rpittau | yep | 16:03 |
| TheJulia | just ping me when required | 16:03 |
| rpittau | btw the workaround is https://review.opendev.org/c/openstack/bifrost/+/821223 | 16:03 |
| opendevreview | Verification of a change to openstack/ironic master failed: Ensure desired permissions on all written pxe files https://review.opendev.org/c/openstack/ironic/+/809295 | 16:17 |
| TheJulia | hi metalsmith job, why you pause for almost exactly 17 minutes. | 16:30 |
| TheJulia | oh, nvmd | 16:30 |
| TheJulia | provisioned | 16:30 |
| TheJulia | hmm, I think metalsmith is hitting io bounds issues | 16:35 |
| opendevreview | Julia Kreger proposed openstack/metalsmith master: CI: Extend metalsmith CI job timeout https://review.opendev.org/c/openstack/metalsmith/+/821242 | 16:41 |
| TheJulia | ^^ should make metalsmith happier | 16:41 |
| dtantsur | yeah, wanted to do the same | 16:41 |
| TheJulia | Just a little bit | 16:42 |
| opendevreview | Arne Wiebalck proposed openstack/ironic-python-agent master: [WIP] Burn-in: Dynamic network pairing https://review.opendev.org/c/openstack/ironic-python-agent/+/821244 | 16:42 |
| TheJulia | this last failure, and the one prior I looked at were just starting to do final cleaning | 16:42 |
| TheJulia | so that *should* cover underlying cloud io performance deviation nicely | 16:42 |
| arne_wiebalck | I tested ^^ with 12 nodes launching network burn-in in parallel, they all found a partner dynamically (so, no more need to configure the pairs upfront). Early reviews/comments appreciated :) | 16:49 |
| TheJulia | w/r/t 18.1 bugfix branch, I'm cool if we start to nuke jobs from it since we obviously don't need to run everything on it | 16:51 |
| TheJulia | rpittau: hint hint ^^ | 16:52 |
| TheJulia | arne_wiebalck: ooooohhh ahhhh | 16:52 |
| arne_wiebalck | TheJulia: :) | 16:57 |
| dtantsur | TheJulia: I need to remember what I wrote about bugfix branch lifetime | 16:57 |
| TheJulia | dtantsur: uhh... it is *past* it's expected life | 16:57 |
| dtantsur | yeah | 16:57 |
| TheJulia | and if memory serves, we knew/agreed it would be okay to do so | 16:57 |
| TheJulia | since we knew we would hit problems like we have | 16:58 |
| dtantsur | I guess we do use it in a downstream product | 16:58 |
| arne_wiebalck | TheJulia: I do not have too many free nodes at the moment, would like to see this with 100 nodes or so :) | 16:58 |
| dtantsur | do we, rpittau? | 16:58 |
| dtantsur | (one would expect that I can remember which branches we use.. nope) | 16:58 |
| TheJulia | I don't expect it, I can barely keep track of 7 backports of one patch and then 4 of another, and 3 of another | 16:58 |
| dtantsur | yeah, we use it in OCP 4.9. hmm. | 17:00 |
| TheJulia | but you don't need to test BFV | 17:00 |
| dtantsur | absolutely not | 17:00 |
| TheJulia | nor any of the fully integrated stacks | 17:00 |
| TheJulia | which are most risky of breaking | 17:01 |
| dtantsur | I guess I could live with standalone jobs, maybe also redfish-vmedia one | 17:01 |
| TheJulia | yup | 17:01 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic-python-agent bugfix/8.3: Update .gitreview for bugfix/8.3 https://review.opendev.org/c/openstack/ironic-python-agent/+/821246 | 17:18 |
| rpittau | TheJulia, dtantsur, sorry, kind of in the middle of 8 things at the same time, we do need 18.1 to work well, but we can definitely remove some integration jobs, that's anyway "just" a bugfix branch | 17:19 |
| dtantsur | Why, WHY does Python 3 in CentOS depend on python3-pip. WHY. | 17:42 |
| rpittau | OSError: [Errno 101] ENETUNREACH ok..... | 17:46 |
| opendevreview | Merged openstack/ironic-python-agent bugfix/8.3: Update .gitreview for bugfix/8.3 https://review.opendev.org/c/openstack/ironic-python-agent/+/821246 | 17:48 |
| rpittau | alright, time to rest I guess, see you tomorrow! o/ | 17:51 |
| dtantsur | same o/ | 17:51 |
| opendevreview | Julia Kreger proposed openstack/sushy stable/train: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821252 | 17:52 |
| opendevreview | Julia Kreger proposed openstack/sushy stable/ussuri: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821253 | 17:53 |
| opendevreview | Julia Kreger proposed openstack/sushy stable/wallaby: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821254 | 17:53 |
| opendevreview | Julia Kreger proposed openstack/sushy stable/victoria: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821255 | 17:53 |
| opendevreview | Julia Kreger proposed openstack/sushy stable/xena: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821256 | 17:54 |
| opendevreview | Julia Kreger proposed openstack/sushy master: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821257 | 17:54 |
| TheJulia | ripping it all out | 17:54 |
| TheJulia | going to start fresh :( | 17:54 |
| dtantsur | ouch :( | 17:56 |
| TheJulia | yup | 17:56 |
| TheJulia | I've got a good idea of how to do it, but we're going to need a "yes, session auth worked in the past flag" | 17:56 |
| TheJulia | and then, and only then, detonate/raise errors to the client if we can't reauth | 17:56 |
| dtantsur | fair | 17:56 |
| dtantsur | okay, I should go for real, have a good evening (nearly wrote "weekend", but no. not yet.) | 17:57 |
| TheJulia | ++ | 17:57 |
| TheJulia | I might take tomorrow off | 17:57 |
| TheJulia | or do a half day or something | 17:57 |
| dtantsur | very sensible | 17:57 |
| TheJulia | my brain is fried fighting CI | 17:57 |
| dtantsur | I'll be out next Friday, then for 12 days from Thursday | 17:57 |
| TheJulia | "Hi, what did you do this week?" "Fought CI!" | 17:57 |
| dtantsur | \o/ | 17:57 |
| dtantsur | okay, see you tomorrow or on Monday | 17:58 |
| TheJulia | "it was a bitter battle, but in the end, reviews and fixes are still needed :(" | 17:58 |
| TheJulia | You too if I don't wake up with coffee tomorrow | 17:58 |
| opendevreview | Arne Wiebalck proposed openstack/ironic master: [doc] Add parameters for burn-in log files https://review.opendev.org/c/openstack/ironic/+/821272 | 18:22 |
| arne_wiebalck | bye everyone, see you tomorrow o/ | 18:26 |
| *** zbitter is now known as zaneb | 18:54 | |
| *** sshnaidm is now known as sshnaidm|afk | 19:05 | |
| stevebaker[m] | good morning | 19:14 |
| opendevreview | Verification of a change to openstack/ironic master failed: Ensure desired permissions on all written pxe files https://review.opendev.org/c/openstack/ironic/+/809295 | 19:37 |
| opendevreview | Verification of a change to openstack/ironic master failed: Move place_loaders_for_boot to boot driver __init__ https://review.opendev.org/c/openstack/ironic/+/819729 | 19:37 |
| opendevreview | Verification of a change to openstack/ironic master failed: Write initial grub config on startup https://review.opendev.org/c/openstack/ironic/+/815580 | 19:37 |
| opendevreview | Merged openstack/sushy stable/wallaby: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821254 | 19:54 |
| opendevreview | Merged openstack/sushy master: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821257 | 19:55 |
| opendevreview | Merged openstack/sushy stable/ussuri: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821253 | 20:24 |
| opendevreview | Merged openstack/sushy stable/victoria: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821255 | 20:24 |
| opendevreview | Merged openstack/sushy stable/train: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821252 | 20:24 |
| opendevreview | Merged openstack/sushy stable/xena: Revert "Raise an AccessError with SessionService init" https://review.opendev.org/c/openstack/sushy/+/821256 | 20:24 |
| opendevreview | Merged openstack/metalsmith master: CI: Extend metalsmith CI job timeout https://review.opendev.org/c/openstack/metalsmith/+/821242 | 21:08 |
| opendevreview | Julia Kreger proposed openstack/sushy master: Fix session authentication issues https://review.opendev.org/c/openstack/sushy/+/821324 | 21:48 |
| opendevreview | Julia Kreger proposed openstack/sushy master: Fix session authentication issues https://review.opendev.org/c/openstack/sushy/+/821324 | 21:52 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!