*** rcernin has quit IRC | 00:07 | |
*** rcernin has joined #openstack-ironic | 00:09 | |
*** etingof has quit IRC | 00:42 | |
*** sthussey has quit IRC | 00:51 | |
*** jesusaur has joined #openstack-ironic | 01:12 | |
*** tiendc has joined #openstack-ironic | 01:20 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP ipv6 job https://review.openstack.org/613169 | 01:35 |
---|---|---|
*** dustinc has quit IRC | 01:40 | |
openstackgerrit | inspurericzhang proposed openstack/ironic-tempest-plugin master: Update hacking version to latest https://review.openstack.org/629347 | 02:14 |
openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP ipv6 job https://review.openstack.org/613169 | 02:48 |
*** rloo has quit IRC | 02:51 | |
*** Qianbiao has joined #openstack-ironic | 03:00 | |
*** gyee has quit IRC | 03:10 | |
openstackgerrit | QianBiao Ng proposed openstack/ironic-specs master: Add spec for Huawei iBMC driver https://review.openstack.org/628724 | 03:35 |
openstackgerrit | Merged openstack/sushy-tools master: Add configuration file to sushy-emulator https://review.openstack.org/616178 | 03:48 |
*** dustinc has joined #openstack-ironic | 04:11 | |
*** dustinc has quit IRC | 04:22 | |
*** dustinc has joined #openstack-ironic | 04:24 | |
*** dustinc has quit IRC | 04:35 | |
*** dustinc has joined #openstack-ironic | 04:36 | |
*** dustinc has quit IRC | 04:43 | |
*** arne_wiebalck_ has joined #openstack-ironic | 04:53 | |
*** hwoarang has joined #openstack-ironic | 04:56 | |
*** lifeless_ has joined #openstack-ironic | 04:57 | |
*** baha has joined #openstack-ironic | 05:01 | |
*** dnuka has joined #openstack-ironic | 05:02 | |
*** jesusaur has quit IRC | 05:02 | |
*** hwoarang_ has quit IRC | 05:02 | |
*** irclogbot_1 has quit IRC | 05:02 | |
*** trungnv has quit IRC | 05:02 | |
*** arne_wiebalck has quit IRC | 05:02 | |
*** chason has quit IRC | 05:02 | |
*** lifeless has quit IRC | 05:02 | |
*** arne_wiebalck_ is now known as arne_wiebalck | 05:02 | |
dnuka | good morning o/ | 05:02 |
dnuka | good morning arne_wiebalck :) | 05:02 |
*** irclogbot_1 has joined #openstack-ironic | 05:04 | |
*** jesusaur has joined #openstack-ironic | 05:09 | |
*** dustinc has joined #openstack-ironic | 05:20 | |
*** dustinc has quit IRC | 05:29 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP ipv6 job https://review.openstack.org/613169 | 05:34 |
openstackgerrit | Kaifeng Wang proposed openstack/ironic master: Expose conductors: api-ref https://review.openstack.org/622862 | 06:16 |
*** e0ne has joined #openstack-ironic | 06:22 | |
*** e0ne has quit IRC | 06:23 | |
*** pcaruana has joined #openstack-ironic | 07:06 | |
*** moshele has joined #openstack-ironic | 07:10 | |
*** etingof has joined #openstack-ironic | 07:17 | |
*** e0ne has joined #openstack-ironic | 07:18 | |
*** e0ne has quit IRC | 07:22 | |
arne_wiebalck | dnuka: good morning! | 07:23 |
dnuka | o/ | 07:23 |
*** rcernin has quit IRC | 07:26 | |
*** hjensas has quit IRC | 07:31 | |
*** e0ne has joined #openstack-ironic | 07:47 | |
*** rpittau has joined #openstack-ironic | 07:54 | |
rpittau | good morning ironic! o/ | 07:54 |
dnuka | good morning rpittau :) | 07:54 |
rpittau | hi dnuka :) | 07:54 |
iurygregory | good morning everyone o/ | 08:35 |
rpittau | hey iurygregory :) | 08:35 |
dnuka | morning iurygregory | 08:35 |
*** e0ne has quit IRC | 08:47 | |
* iurygregory really love the TimeoutException =( | 08:51 | |
*** S4ren has joined #openstack-ironic | 09:00 | |
*** dougsz has joined #openstack-ironic | 09:13 | |
*** hjensas has joined #openstack-ironic | 09:13 | |
* etingof feels that being in hurry and running openstack do not always play well... ;) | 09:23 | |
etingof | good morning everybody o/ | 09:23 |
dnuka | morning etingof | 09:23 |
etingof | dnuka, so did `tox -r` help? | 09:24 |
iurygregory | morning etingof o/ | 09:24 |
dnuka | etingof, yes :) | 09:24 |
etingof | awesome | 09:24 |
rpittau | hi etingof :) | 09:24 |
*** moshele has quit IRC | 09:24 | |
dnuka | folks, I cannot push changes to gerrit, "Problem running 'git remote update gerrit'" | 09:26 |
iurygregory | can you check your .gitreview file in the repo? | 09:27 |
etingof | is it what `git review` ends up with? | 09:27 |
dnuka | ack | 09:27 |
dnuka | Problem running 'git remote update gerrit' | 09:28 |
dnuka | Fetching gerrit | 09:28 |
dnuka | dnuka@review.openstack.org: Permission denied (publickey). | 09:28 |
dnuka | fatal: Could not read from remote repository. | 09:28 |
iurygregory | well you key seem to have wrong permisions O.o | 09:28 |
dnuka | well, everything worked yestreday :) | 09:29 |
dnuka | *yesterday | 09:29 |
etingof | can it be that ssh key pass phrase has expired in the ssh agent? | 09:29 |
* etingof would run `ssh -v dnuka@review.openstack.org` to see where auth fails | 09:30 | |
dnuka | ack | 09:30 |
dnuka | Permission denied :( | 09:31 |
iurygregory | a workaround woulb be build a new key and add to gerrit =) | 09:32 |
etingof | so `git config -l` should tell you the exact ssh command which git uses to access remote repo | 09:32 |
etingof | then dnuka should make sure that that ssh command (including -p <port> perhaps) works out alone | 09:33 |
dnuka | etingof, iurygregory got it :) | 09:33 |
etingof | along the way ssh should tell dnuka which ssh key(s) it is trying to use and if they get rejected | 09:33 |
dnuka | ack | 09:34 |
*** robbbe has joined #openstack-ironic | 09:36 | |
etingof | if dnuka suspects his local repo is somehow misconfigured, he could clone the upstream repo somewhere (e.g. in /tmp) and (if cloning works out) see if `git config -l` differs... | 09:36 |
*** moshele has joined #openstack-ironic | 09:37 | |
dnuka | etingof, got it | 09:37 |
*** derekh has joined #openstack-ironic | 09:38 | |
dnuka | etingof, cloning works! and git config -l` output differs | 09:41 |
etingof | dnuka, the configuration is stored in .git/config so you can experiment with these files | 09:42 |
dnuka | ack | 09:42 |
*** zhangfei has joined #openstack-ironic | 09:43 | |
etingof | dnuka, just make sure not to accidentally lose the work you've done but not yet pushed upstream | 09:43 |
iurygregory | at least what is in gerrit is not lost =D | 09:43 |
dnuka | etingof, almost forgot that :) | 09:44 |
iurygregory | git review -d <3 | 09:44 |
dnuka | iurygregory, yes :) | 09:44 |
* rpittau looks at the clock, it's coffee time! | 09:48 | |
dnuka | :) | 09:49 |
iurygregory | coffee \o/ | 09:49 |
* etingof can't get his clock reading in the morning unless initial coffeination is over | 09:52 | |
dnuka | :D | 09:52 |
*** S4ren has quit IRC | 09:53 | |
rpittau | lol | 09:57 |
*** dnuka is now known as dnuka|brb | 10:02 | |
*** dtantsur|afk is now known as dtantsur | 10:10 | |
dtantsur | morning ironic | 10:11 |
*** iurygregory has quit IRC | 10:12 | |
*** dnuka has joined #openstack-ironic | 10:13 | |
dnuka | morning dtantsur | 10:16 |
*** S4ren has joined #openstack-ironic | 10:18 | |
*** e0ne has joined #openstack-ironic | 10:21 | |
rpittau | hi dtantsur :) | 10:21 |
*** iurygregory has joined #openstack-ironic | 10:26 | |
*** stendulker has joined #openstack-ironic | 10:27 | |
*** zhangfei has quit IRC | 10:32 | |
openstackgerrit | Dhanuka Warusadura proposed openstack/sushy master: [WIP] Introduce ``convert_to_date`` https://review.openstack.org/625889 | 10:32 |
openstackgerrit | Dhanuka Warusadura proposed openstack/sushy master: [WIP] Introduce ``convert_to_date`` https://review.openstack.org/625889 | 10:36 |
*** iurygregory has quit IRC | 10:50 | |
*** moshele has quit IRC | 10:57 | |
*** stendulker has quit IRC | 11:11 | |
*** iurygregory has joined #openstack-ironic | 11:16 | |
*** dnuka has quit IRC | 11:16 | |
*** dnuka has joined #openstack-ironic | 11:16 | |
iurygregory | dtantsur, morning o/ | 11:23 |
*** hjensas has quit IRC | 11:24 | |
iurygregory | its normal a node have deploy_interface as None? | 11:26 |
openstackgerrit | Hamdy Khader proposed openstack/python-ironicclient master: Add 'hostname' to port's local link connection https://review.openstack.org/628773 | 11:26 |
rpittau | iurygregory, is that on ironic conf ? | 11:32 |
iurygregory | rpittau, nope when looking at the node info | 11:32 |
rpittau | I think it can be empty | 11:33 |
iurygregory | well according to our guide we should set the node in maintenance but we cant lol | 11:34 |
*** moshele has joined #openstack-ironic | 11:34 | |
iurygregory | openstack baremetal node maintenance set d209a91d-5bea-4901-9809-a06697859494 --reason "Changing driver and/or hardware interfaces" | 11:34 |
iurygregory | No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool for conductor group "". (HTTP 400) | 11:34 |
dtantsur | iurygregory: this is expected. no updates are possible with an invalid driver. | 11:35 |
dtantsur | iurygregory: also deploy_interface=None is expected for classic drivers | 11:35 |
dtantsur | also we may be debugging the same thing :) | 11:36 |
iurygregory | cool | 11:36 |
iurygregory | yep we are =P | 11:36 |
* iurygregory not sure if you saw my in email in the thread lol | 11:36 | |
dtantsur | it's an actual bug in ironic :( | 11:37 |
iurygregory | =( | 11:39 |
iurygregory | so our doc is wrong when we say that we can put the node in maintenance =( | 11:40 |
iurygregory | 'It is not recommended to change the interfaces for active nodes. If absolutely needed, the nodes have to be put in the maintenance mode first' | 11:40 |
dtantsur | iurygregory: no, this is correct. | 11:40 |
dtantsur | it's not a workaround for updating broken nodes though | 11:41 |
iurygregory | oh ok | 11:41 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Fix updating nodes with removed or broken drivers https://review.openstack.org/629447 | 11:45 |
dtantsur | iurygregory: ^^^ | 11:45 |
dtantsur | TheJulia, mgoddard if you're around ^^^ | 11:46 |
iurygregory | dtantsur, cool you rock =D | 11:46 |
mgoddard | dtantsur: I'll take a look | 11:47 |
openstackgerrit | Hamdy Khader proposed openstack/python-ironicclient master: Add is-smartnic port attribute to port command https://review.openstack.org/629449 | 11:51 |
openstackgerrit | Derek Higgins proposed openstack/ironic master: [WIP] Replace use of Q_USE_PROVIDERNET_FOR_PUBLIC https://review.openstack.org/621146 | 11:59 |
*** iurygregory has quit IRC | 12:00 | |
*** dnuka has quit IRC | 12:06 | |
*** hjensas has joined #openstack-ironic | 12:07 | |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic stable/rocky: Fix updating nodes with removed or broken drivers https://review.openstack.org/629451 | 12:10 |
*** dnuka has joined #openstack-ironic | 12:13 | |
*** iurygregory has joined #openstack-ironic | 12:15 | |
*** rpittau is now known as rpittau|lunch | 12:20 | |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allocation API: allow picking random conductor for RPC topic https://review.openstack.org/629250 | 12:23 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allocation API: conductor API (without HA and take over) https://review.openstack.org/628223 | 12:23 |
*** tiendc has quit IRC | 12:27 | |
*** e0ne has quit IRC | 12:30 | |
*** dnuka has quit IRC | 12:37 | |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/networking-baremetal master: Change networking-baremetal to zuulv3/python3 https://review.openstack.org/614026 | 12:39 |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-inspector master: Follow-up Zuulv3 (Ironic jobs rename) https://review.openstack.org/629475 | 13:05 |
*** trown|outtypewww is now known as trown | 13:09 | |
*** jcoufal has joined #openstack-ironic | 13:16 | |
*** rpittau|lunch is now known as rpittau | 13:21 | |
*** e0ne has joined #openstack-ironic | 13:24 | |
*** vkmc has left #openstack-ironic | 13:29 | |
*** Qianbiao has quit IRC | 13:51 | |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic master: Remove dsvm from zuulv3 jobs https://review.openstack.org/629173 | 13:54 |
*** rloo has joined #openstack-ironic | 13:55 | |
*** moshele has quit IRC | 13:57 | |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-inspector master: Follow-up Zuulv3 (Ironic jobs rename) https://review.openstack.org/629475 | 13:58 |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Rename zuulv3 jobs https://review.openstack.org/629494 | 14:07 |
*** baha has joined #openstack-ironic | 14:16 | |
openstackgerrit | Ilya Etingof proposed openstack/sushy master: Add System<->Manager linkage follow up https://review.openstack.org/629500 | 14:19 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allocation API: conductor API (without HA and take over) https://review.openstack.org/628223 | 14:26 |
TheJulia | dtantsur: do we have an elastic recheck in place for the failed to read errors? | 14:32 |
dtantsur | TheJulia: nope. at least not from me. I can look into it. | 14:35 |
dtantsur | (and thanks for approving that bug fix) | 14:35 |
TheJulia | dtantsur: thanks, I wonder just how often we're seeing it | 14:35 |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/networking-generic-switch master: Rename ironic job to match zuulv3 https://review.openstack.org/629508 | 14:35 |
iurygregory | morning TheJulia o/ | 14:36 |
dtantsur | TheJulia: quite often I'm sure | 14:36 |
dtantsur | I wonder if the recent IPA change is involved.. or is it just bionic | 14:36 |
dtantsur | TheJulia: do you have an example elastic recheck update handy? | 14:36 |
TheJulia | dtantsur: i don't sadly, but I am super curious about that change | 14:37 |
rpittau | good morning TheJulia :) | 14:37 |
TheJulia | I never really agreed with it, tbh | 14:37 |
TheJulia | but I wasn't going to block it | 14:37 |
dtantsur | why didn't you agree? | 14:38 |
TheJulia | because we shouldn't make assumptions into the contents we write | 14:41 |
TheJulia | How do we know what is being written has or needs a partition table? | 14:41 |
TheJulia | shouldn't be an issue on any of the platforms really used with ironic though, for most common things | 14:41 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: [WIP] Allocation API: implementation https://review.openstack.org/629260 | 14:42 |
dtantsur | TheJulia: hmm.. maybe you're right.. | 14:43 |
dtantsur | TheJulia: as to elastic-recheck, I cannot even make logstash output me *anything* >_< | 14:43 |
TheJulia | dtantsur: ugh :( | 14:43 |
TheJulia | maybe clarkb has some magical insight there he can point us to? | 14:44 |
TheJulia | re platform stuffs, I am quite altruistic though, not quite practical in that regard, so it is likely practical, but worth digging at that change. I can dig at that with a fresh set of eyes once coffee hits and I'm done with email | 14:45 |
dtantsur | okay, the syntax is funny, but I seem to start figuring it out | 14:46 |
dtantsur | TheJulia: 95 hits in a week for message:"Failed to create config drive on disk" AND message:"test -e" :( | 14:50 |
TheJulia | EEK | 14:50 |
TheJulia | We had a lot of failures late last week, so that could be it | 14:50 |
TheJulia | test -e seems... a little not specific enough because I think it is body contents matching not line level matching | 14:51 |
* TheJulia has no idea how the cluster is indexed | 14:51 | |
dtantsur | TheJulia: well, it seems to find what we need | 14:52 |
dtantsur | now creating a story for this | 14:52 |
*** baha has quit IRC | 14:52 | |
TheJulia | \o/ | 14:53 |
TheJulia | almost done with email, so I can begin deep diving on the failure | 14:54 |
dtantsur | TheJulia: https://storyboard.openstack.org/#!/story/2004744 | 14:54 |
TheJulia | awesome | 14:55 |
iurygregory | yay finally finished the patchs to rename jobs \o/ | 14:58 |
dtantsur | TheJulia: the query shows that at least sometimes it failed on xenial, so we may be able to rule out bionic as a cause | 15:04 |
dtantsur | TheJulia: and happens only with wholedisk+direct jobs.. which make the IPA patch an even bigger suspect | 15:05 |
TheJulia | hmmm | 15:06 |
dtantsur | hmm, wait, the IPA code runs for all image types | 15:06 |
TheJulia | Well, it is clear the person with huawei is awake right now, they've responded to two emails so far this morning | 15:07 |
*** chason_ has joined #openstack-ironic | 15:09 | |
*** mjturek has joined #openstack-ironic | 15:10 | |
dtantsur | TheJulia: hmm, does this actually mean that we should run partprobe (again!) http://logs.openstack.org/47/629447/1/check/ironic-tempest-dsvm-ipa-wholedisk-direct-tinyipa-multinode/65739ae/compute1/logs/ironic-bm-logs/subnode-0_no_ansi_2019-01-09-12:38:08_log.txt.gz#_2019-01-09_12_37_27_550 ? | 15:12 |
TheJulia | That is likely what is happening, but that would only fix the symptom | 15:13 |
* TheJulia wonders | 15:13 | |
dtantsur | and why for love's sake are we using GPT in a BIOS job? Oo | 15:13 |
iurygregory | magic | 15:14 |
TheJulia | oO | 15:14 |
dtantsur | because yes, we do: http://logs.openstack.org/47/629447/1/check/ironic-tempest-dsvm-ipa-wholedisk-direct-tinyipa-multinode/65739ae/compute1/logs/ironic-bm-logs/subnode-0_no_ansi_2019-01-09-12:38:08_log.txt.gz#_2019-01-09_12_37_21_815 | 15:14 |
*** baha has joined #openstack-ironic | 15:14 | |
TheJulia | dtantsur: the image we are writing ? | 15:15 |
dtantsur | I mean, it is supported, but why are we doing it? | 15:15 |
dtantsur | well, yes, that's the explanation | 15:15 |
TheJulia | we recently bumped the version of the image, I suspect it is a silent change | 15:16 |
dtantsur | we recently moved to new version of cirros, can it bring GPT here? /cc derekh | 15:16 |
TheJulia | that is what I'm thinking | 15:16 |
TheJulia | I'm glad we put in the gpt repair logic | 15:17 |
dtantsur | vda: vda1 vda15 | 15:17 |
dtantsur | WUT | 15:17 |
dtantsur | no, srsly WUT | 15:17 |
dtantsur | dear linux.. | 15:17 |
derekh | dtantsur: dunno | 15:18 |
TheJulia | wut^wutwut | 15:19 |
iurygregory | O.o | 15:19 |
dtantsur | I need whiskey. a lot of whiskey. | 15:20 |
iurygregory | after see this i need to agree | 15:20 |
TheJulia | 2 is the config drive, 15 is the /boot volume | 15:21 |
* TheJulia suspects it is too early to open the japanese whiskey | 15:22 | |
dtantsur | yeah, except that, I suspect, the configdrive is now always 2... | 15:22 |
dtantsur | I have some great stuff from Islay from my birthday. But too early even for me.. | 15:22 |
* iurygregory miss Japanese beer. | 15:22 | |
dtantsur | another guess is that we need to insert partprobe into yet another place | 15:23 |
dtantsur | this is a better option for us | 15:23 |
dtantsur | we can even try going down this path first | 15:24 |
TheJulia | looks like it would have to be a whdi to hit this failure | 15:25 |
TheJulia | dtantsur: ++ | 15:26 |
TheJulia | looks like we should just be able to do it in ipa which would cover that case... feels like we have another bug someplace in the path | 15:29 |
dtantsur | who knows... I'm patching ironic-lib for now | 15:30 |
dtantsur | TheJulia: I wonder if https://review.openstack.org/#/c/594922/ may be involved | 15:33 |
patchbot | patch 594922 - ironic-lib - Check GPT table with sgdisk insread of partprobe (MERGED) - 2 patch sets | 15:33 |
TheJulia | So we pulled down the partition image... | 15:33 |
openstackgerrit | Ilya Etingof proposed openstack/sushy master: Add Chassis<->ComputerSystem/Manager linkage https://review.openstack.org/623028 | 15:33 |
TheJulia | err no | 15:34 |
TheJulia | yeah, wholedisk is what it was | 15:34 |
iurygregory | "nat" is correct? | 15:35 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic-lib master: [WIP] Run partprobe after adding a configdrive partition https://review.openstack.org/629547 | 15:36 |
dtantsur | TheJulia: let's see what the CI says bout ^^^ | 15:36 |
TheJulia | ++ | 15:37 |
TheJulia | cirros is not starting out with a partition table :\ | 15:40 |
iurygregory | TheJulia, would be necessary push a change to openstack-infra/project-config since the have the name for one job that is changed? http://codesearch.openstack.org/?q=ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa&i=nope&files=&repos=project-config | 15:45 |
openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: Add memoization to expensive emulator calls https://review.openstack.org/612758 | 15:46 |
NobodyCam | Good Morning and happy hump day | 15:50 |
iurygregory | good morning NobodyCam o/ | 15:51 |
dtantsur | morning NobodyCam | 15:51 |
NobodyCam | O/ iurygregory and dtantsur :) | 15:51 |
rpittau | hi NobodyCam :) | 15:54 |
NobodyCam | Good Morning rpittau :) | 15:55 |
TheJulia | something is super fishy about this failure :( and no sushy involved | 15:56 |
TheJulia | likely, that is performance graphing someone setup at some point... | 15:57 |
TheJulia | doh, its a qcow, not raw image | 15:59 |
TheJulia | okay, after converting | 16:00 |
TheJulia | yes, the 0.4.0 is a gpt filesystem | 16:00 |
dtantsur | \o/ | 16:03 |
dtantsur | so we've lost coverage for BIOS+MBR? :D | 16:03 |
*** pcaruana has quit IRC | 16:03 | |
TheJulia | for wholedisk, yes | 16:03 |
TheJulia | but gpt is fine | 16:03 |
TheJulia | err partition based is fine | 16:04 |
TheJulia | we have to trust the image" and we are doing it in whole disk jobs | 16:04 |
TheJulia | we're just much more resilliant apparently | 16:04 |
dtantsur | yeah, but GPT and MBR are different code paths even for wholedisk images (I think) | 16:04 |
TheJulia | dtantsur: interleaved | 16:04 |
dtantsur | yeah. in quite some spaghetti :) | 16:05 |
TheJulia | indeed | 16:06 |
TheJulia | I'm actually really happy that it passed | 16:06 |
TheJulia | well | 16:06 |
TheJulia | gets through | 16:06 |
TheJulia | it fails for other reasons it seems | 16:06 |
* TheJulia gets out the colander | 16:09 | |
* TheJulia wonders if anyone uses the partial code for local caching for nodes awaiting official deployment orders | 16:13 | |
dtantsur | good question | 16:14 |
*** gyee has joined #openstack-ironic | 16:18 | |
*** moshele has joined #openstack-ironic | 16:24 | |
TheJulia | something is really not adding up for me, maybe I just need more coffee | 16:27 |
dtantsur | something? I'm completely in darkness with this issue :) if "just throw another partprobe" does not work, I dunno.. | 16:28 |
TheJulia | I think that is the only option, starting to understand | 16:30 |
*** moshele has quit IRC | 16:32 | |
* TheJulia opens up parted's source code to confirm theory | 16:32 | |
dtantsur | wow | 16:32 |
*** rnoriega has quit IRC | 16:38 | |
*** radez has quit IRC | 16:38 | |
openstackgerrit | Merged openstack/sushy master: Add System<->Manager linkage follow up https://review.openstack.org/629500 | 16:38 |
TheJulia | sweet... github failed | 16:39 |
TheJulia | oh, its back now :( | 16:39 |
*** ltomasbo has quit IRC | 16:39 | |
*** lucasagomes has quit IRC | 16:39 | |
TheJulia | wow, the way it is written makes my head hurt | 16:40 |
iurygregory | dtantsur, should i trigger the check-experimental in nova since some ironic jobs run only there? (2 jobs are in experimental) https://review.openstack.org/#/c/629504/1/.zuul.yaml | 16:41 |
patchbot | patch 629504 - nova - Rename Ironic jobs - 1 patch set | 16:41 |
*** lucasagomes has joined #openstack-ironic | 16:41 | |
*** rnoriega has joined #openstack-ironic | 16:43 | |
TheJulia | yeah, I suspect my feeling is right | 16:44 |
*** lucasagomes has quit IRC | 16:45 | |
*** lucasagomes has joined #openstack-ironic | 16:50 | |
dtantsur | iurygregory: yeah, you can try it | 16:51 |
dtantsur | TheJulia: so? | 16:51 |
iurygregory | dtantsur, ack tks o/ | 16:51 |
TheJulia | still sifting through the c code | 16:51 |
TheJulia | I just confirmed it should be fsyncing, so it is likely just that, the kernel is not loading the table afterwards. There is a note on the fsync docs about behavior dependant upon filesystem though... | 16:53 |
dtantsur | "udevadm settle" returned: 0 in 38.853s <- impressive | 16:54 |
TheJulia | wow | 16:54 |
* TheJulia saves other words for not online | 16:54 | |
dtantsur | I see. I'm looking at http://logs.openstack.org/47/629547/1/check/ironic-lib-wholedisk-bios-ipmi-direct-tinyipa-src-python2/f7961c8/controller/logs/ironic-bm-logs/node-0_no_ansi_2019-01-09-16:28:34_log.txt.gz#_2019-01-09_16_28_03_502 I cannot see my partprobe. weird. | 16:55 |
TheJulia | because we don't build it in by default afaik | 16:55 |
TheJulia | oh, did we build ipa from scratch with the code? | 16:55 |
dtantsur | TheJulia: well, we used to build IPA in ironic-lib gate at least | 16:56 |
dtantsur | but I have a feeling it got broken. sigh. SIGH. | 16:57 |
TheJulia | ugh | 17:01 |
dtantsur | oh, I see. we hardcode /opt/stack/new in some places. it's no longer the thing.. | 17:03 |
*** e0ne has quit IRC | 17:03 | |
* TheJulia gets out whiskey | 17:03 | |
TheJulia | fsync on a ramdisk is an explicit noop | 17:03 |
TheJulia | it does nothing | 17:03 |
*** robbbe has quit IRC | 17:03 | |
TheJulia | https://github.com/torvalds/linux/blob/6f0d349d922ba44e4348a17a78ea51b7135965b1/fs/ramfs/file-mmu.c | 17:04 |
rpittau | I was going to ask a couple of things but I see the whiskey level is already high enough, it can wait :P | 17:04 |
NobodyCam | rpittau: LOL | 17:04 |
TheJulia | so io data from parted is in the memory cache, and waits upon the system to do lazy writing out since it doesn't appear to do explicit direct io calls | 17:05 |
TheJulia | reading however for the kernel requires a lock from the device, so there is the case without forcing a disk sync that the new partition is in ram, but not yet on disk when we attempt to rescan | 17:06 |
rpittau | bye all! good evening o/ | 17:08 |
NobodyCam | have a good night rpittau | 17:08 |
*** rpittau has quit IRC | 17:08 | |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent master: Recover CI run detection in imagebuild https://review.openstack.org/629574 | 17:09 |
dtantsur | TheJulia: maybe this can help ^^^ but I'm too tired to think more | 17:09 |
TheJulia | dtantsur: I think we need to perform an explicit sync as well to totally eliminate slow disk io being an isue | 17:09 |
dtantsur | TheJulia: in my ironic-lib patch? | 17:10 |
TheJulia | that will force the kernel to go disk by disk syncing out buffers | 17:10 |
TheJulia | dtantsur: yeah, I can do it | 17:10 |
dtantsur | yeah, feel free to update it | 17:10 |
*** dustinc has joined #openstack-ironic | 17:10 | |
TheJulia | k | 17:10 |
TheJulia | coffee first | 17:10 |
* etingof has been wondering about explicit sync | 17:10 | |
dtantsur | see you | 17:10 |
*** dtantsur is now known as dtantsur|afk | 17:11 | |
TheJulia | yeah, seems silly to ahve to do it, but ramdisk the rules are different | 17:11 |
TheJulia | goodnight | 17:11 |
* etingof observed that sysadmins tend to run `sync` before committing into some serious business | 17:12 | |
etingof | felt like a genetic trait | 17:12 |
TheJulia | yeah, provides some guarantee (minus raid controllers lying about io cache flush-out) that things are written to disk | 17:16 |
TheJulia | in case, oh... accidental power button or the like | 17:16 |
TheJulia | I had a database platform I was responsible for the datacenter operations and system administration of a long time ago, and it used a mix of direct io and let the rest of the io be a lazy write to disk. The aspects that were a lazy write are almost always the part that led to corruption | 17:18 |
TheJulia | and it was possible to also crash some of the raid controllers then with massive direct io queues | 17:19 |
TheJulia | soooo many database restores.... | 17:19 |
etingof | so the habit of occasional syncing is a good trait ;) | 17:19 |
*** moshele has joined #openstack-ironic | 17:19 | |
TheJulia | indeed | 17:23 |
openstackgerrit | Merged openstack/networking-generic-switch master: Directly source ML2 plugin for devstack https://review.openstack.org/628708 | 17:28 |
*** moshele has quit IRC | 17:28 | |
TheJulia | Interesting dmitry's wip patch failed pep8 :\ but is fine locally | 17:31 |
TheJulia | oh, no, it did | 17:31 |
TheJulia | wrong line \o/ | 17:31 |
openstackgerrit | Derek Higgins proposed openstack/ironic master: [WIP] Replace use of Q_USE_PROVIDERNET_FOR_PUBLIC https://review.openstack.org/621146 | 17:49 |
*** dougsz has quit IRC | 17:54 | |
*** trown is now known as trown|lunch | 17:55 | |
*** derekh has quit IRC | 17:56 | |
*** MattMan has quit IRC | 18:11 | |
*** MattMan has joined #openstack-ironic | 18:11 | |
*** dustinc has quit IRC | 18:22 | |
*** dustinc has joined #openstack-ironic | 18:23 | |
*** S4ren has quit IRC | 18:30 | |
*** e0ne has joined #openstack-ironic | 18:35 | |
openstackgerrit | Julia Kreger proposed openstack/ironic-lib master: Run sync and partprobe after adding a configdrive partition https://review.openstack.org/629547 | 18:44 |
*** e0ne has quit IRC | 19:02 | |
openstackgerrit | Hamdy Khader proposed openstack/ironic master: Add is_smartnic to Port data model https://review.openstack.org/629640 | 19:04 |
*** e0ne has joined #openstack-ironic | 19:05 | |
*** trown|lunch is now known as trown | 19:08 | |
*** moshele has joined #openstack-ironic | 19:50 | |
*** e0ne has quit IRC | 20:08 | |
*** dustinc has quit IRC | 20:37 | |
*** S4ren has joined #openstack-ironic | 20:51 | |
*** dustinc has joined #openstack-ironic | 20:52 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP ipv6 job https://review.openstack.org/613169 | 21:12 |
TheJulia | Core reviews on https://review.openstack.org/#/c/629574/1 would be appreciated | 21:16 |
patchbot | patch 629574 - ironic-python-agent - Recover CI run detection in imagebuild - 1 patch set | 21:16 |
*** moshele has quit IRC | 21:29 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP ipv6 job https://review.openstack.org/613169 | 21:57 |
*** baha has quit IRC | 22:01 | |
*** trown is now known as trown|outtypewww | 22:10 | |
TheJulia | and core reviews on https://review.openstack.org/#/c/629547/ would also be appreciated | 22:31 |
patchbot | patch 629547 - ironic-lib - Run sync and partprobe after adding a configdrive ... - 2 patch sets | 22:31 |
*** rcernin has joined #openstack-ironic | 22:52 | |
*** jcoufal has quit IRC | 23:03 | |
jidar | can I re-run the undercloud installer (openstack undercloud install) and move the interface over from say, eno3 to eno2 if I messed it up? | 23:22 |
*** dustinc has quit IRC | 23:33 | |
*** dustinc has joined #openstack-ironic | 23:36 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!