| *** goldyfruit___ has joined #openstack-ironic | 00:01 | |
| *** igordc has quit IRC | 00:10 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Lower MTU override https://review.opendev.org/683249 | 00:25 |
|---|---|---|
| *** mrda has quit IRC | 00:28 | |
| *** gyee has quit IRC | 00:28 | |
| *** jcoufal has joined #openstack-ironic | 00:31 | |
| *** mmethot_ has quit IRC | 00:40 | |
| *** mmethot has joined #openstack-ironic | 00:40 | |
| *** kaifeng has joined #openstack-ironic | 00:42 | |
| *** mmethot has quit IRC | 00:45 | |
| *** TxGirlGeek has quit IRC | 00:45 | |
| *** mmethot has joined #openstack-ironic | 00:45 | |
| *** mmethot has quit IRC | 00:46 | |
| *** mmethot has joined #openstack-ironic | 00:47 | |
| *** prometheanfire has left #openstack-ironic | 00:53 | |
| *** mmethot has quit IRC | 00:55 | |
| *** mmethot_ has joined #openstack-ironic | 00:55 | |
| *** rloo has quit IRC | 01:15 | |
| *** whoami-rajat has joined #openstack-ironic | 01:17 | |
| *** mrda has joined #openstack-ironic | 01:35 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: wip: if wholedisk set boot dev to disk https://review.opendev.org/683262 | 01:52 |
| openstackgerrit | Ghanshyam Mann proposed openstack/ironic master: [train][goal] Define new 'ironic-tempest-ipv6-only' job https://review.opendev.org/682692 | 01:53 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: Don't try to put a bootloader in place when bootable https://review.opendev.org/679333 | 01:54 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: Don't try to put a bootloader in place when bootable https://review.opendev.org/679333 | 01:55 |
| *** ricolin has joined #openstack-ironic | 01:55 | |
| TheJulia | dtantsur|afk: Ive decomposed two failures... and counted. MTU errors like... 70% of the time. | 01:56 |
| TheJulia | well, two failed changes... maybe 3 | 01:56 |
| TheJulia | Also a boot_option insanity bug | 01:56 |
| TheJulia | details on the etherpad | 01:56 |
| *** jcoufal has quit IRC | 01:56 | |
| * TheJulia goes and drinks now | 01:56 | |
| TheJulia | dtantsur|afk: oh, https://review.opendev.org/683249 appears to resolve our MTU issues | 01:57 |
| patchbot | patch 683249 - ironic - Lower MTU override - 2 patch sets | 01:57 |
| *** dsneddon has quit IRC | 02:04 | |
| *** dsneddon has joined #openstack-ironic | 02:17 | |
| *** dsneddon has quit IRC | 02:22 | |
| kaifeng | morning TheJulia, welcome back :) | 02:29 |
| *** dsneddon has joined #openstack-ironic | 02:45 | |
| *** mkrai_ has joined #openstack-ironic | 02:45 | |
| *** mkrai_ has quit IRC | 02:50 | |
| *** mkrai has joined #openstack-ironic | 02:57 | |
| *** ricolin_ has joined #openstack-ironic | 03:04 | |
| *** bobmel has joined #openstack-ironic | 03:05 | |
| *** ricolin has quit IRC | 03:06 | |
| openstackgerrit | Merged openstack/ironic master: CI: move libvirt images to /opt for standalone and multinode jobs https://review.opendev.org/681446 | 03:06 |
| *** bobmel has quit IRC | 03:10 | |
| *** TxGirlGeek has joined #openstack-ironic | 03:27 | |
| *** gkadam has joined #openstack-ironic | 03:58 | |
| *** gkadam has quit IRC | 03:58 | |
| *** dsneddon has quit IRC | 03:59 | |
| *** dsneddon has joined #openstack-ironic | 04:00 | |
| *** dsneddon has quit IRC | 04:05 | |
| *** rh-jelabarre has quit IRC | 04:32 | |
| *** _mmethot_ has joined #openstack-ironic | 04:41 | |
| *** mmethot_ has quit IRC | 04:41 | |
| *** TxGirlGeek has quit IRC | 05:07 | |
| *** tzumainn has quit IRC | 05:08 | |
| *** TxGirlGeek has joined #openstack-ironic | 05:11 | |
| *** pcaruana has joined #openstack-ironic | 05:28 | |
| *** dsneddon has joined #openstack-ironic | 05:34 | |
| *** dsneddon has quit IRC | 05:40 | |
| *** pcaruana has quit IRC | 05:53 | |
| *** ricolin_ is now known as ricolin | 05:54 | |
| *** jawad_axd has joined #openstack-ironic | 05:58 | |
| *** jawad_ax_ has joined #openstack-ironic | 06:02 | |
| *** jawad_axd has quit IRC | 06:03 | |
| *** TxGirlGeek has quit IRC | 06:08 | |
| arne_wiebalck | Good morning, ironic! | 06:30 |
| *** trident has quit IRC | 06:49 | |
| *** trident has joined #openstack-ironic | 07:01 | |
| *** trident has quit IRC | 07:07 | |
| *** bobmel has joined #openstack-ironic | 07:07 | |
| *** bobmel has quit IRC | 07:13 | |
| *** trident has joined #openstack-ironic | 07:17 | |
| *** yolanda has quit IRC | 07:21 | |
| *** yolanda has joined #openstack-ironic | 07:23 | |
| *** rcernin has quit IRC | 07:29 | |
| *** rpittau|afk is now known as rpittau | 07:31 | |
| rpittau | good morning ironic! o/ | 07:31 |
| rpittau | TGIF! | 07:31 |
| *** yolanda has quit IRC | 07:32 | |
| *** dsneddon has joined #openstack-ironic | 07:36 | |
| *** e0ne has joined #openstack-ironic | 07:46 | |
| *** yolanda has joined #openstack-ironic | 07:47 | |
| *** rachit7 has joined #openstack-ironic | 07:49 | |
| *** iurygregory has joined #openstack-ironic | 07:51 | |
| *** e0ne has quit IRC | 07:59 | |
| iurygregory | morning all | 08:00 |
| iurygregory | its friday \o/ | 08:00 |
| rachit7 | iurygregory: morning | 08:02 |
| *** dsneddon has quit IRC | 08:07 | |
| *** alexmcleod has joined #openstack-ironic | 08:14 | |
| rpittau | hey iurygregory :) | 08:14 |
| mgoddard | morning | 08:22 |
| *** e0ne has joined #openstack-ironic | 08:24 | |
| rpittau | hey mgoddard :) | 08:25 |
| iurygregory | mgoddard, o/ | 08:25 |
| mgoddard | hi iurygregory & rpittau | 08:25 |
| *** dsneddon has joined #openstack-ironic | 08:34 | |
| *** e0ne has quit IRC | 08:42 | |
| *** mkrai has quit IRC | 08:51 | |
| *** ociuhandu has joined #openstack-ironic | 08:52 | |
| *** dtruong has quit IRC | 08:54 | |
| *** problem_v has quit IRC | 08:54 | |
| *** rcernin has joined #openstack-ironic | 08:55 | |
| kaifeng | morning all, hooray friday! | 08:58 |
| *** dsneddon has quit IRC | 09:01 | |
| *** mkrai has joined #openstack-ironic | 09:05 | |
| *** dsneddon has joined #openstack-ironic | 09:06 | |
| mkrai | rpittau, mgoddard iurygregory rachit7 good morning! | 09:10 |
| *** dsneddon has quit IRC | 09:10 | |
| iurygregory | morning kaifeng and mkrai | 09:10 |
| mkrai | Do we have any patch to fix ironic gate failure? | 09:11 |
| mkrai | it seems bad, almost all request failed :( | 09:11 |
| iurygregory | aww the mtu one failed also =( | 09:12 |
| *** pcaruana has joined #openstack-ironic | 09:12 | |
| mgoddard | morning mkrai | 09:15 |
| *** igordc has joined #openstack-ironic | 09:15 | |
| mkrai | iurygregory, Yes I just checked | 09:15 |
| *** igordc has quit IRC | 09:20 | |
| *** derekh has joined #openstack-ironic | 09:27 | |
| *** dtantsur|afk is now known as dtantsur | 09:27 | |
| dtantsur | morning ironic | 09:27 |
| dtantsur | TheJulia: thanks! | 09:28 |
| *** ociuhandu has quit IRC | 09:30 | |
| *** ociuhandu_ has joined #openstack-ironic | 09:30 | |
| iurygregory | morning dtantsur | 09:33 |
| rpittau | hey dtantsur :) | 09:33 |
| openstackgerrit | pengyuesheng proposed openstack/networking-baremetal master: Update the constraints url https://review.opendev.org/683338 | 09:35 |
| *** igordc has joined #openstack-ironic | 09:39 | |
| dtantsur | TheJulia: could you point me to the place where you see local boot issues with RAID? | 09:41 |
| arne_wiebalck | TheJulia: Thanks for submitting the Forum sessions. I put basically all input into the PTG Planning etherpad ... not sure if some of the stuff there should better be a session. | 09:48 |
| *** rcernin has quit IRC | 09:49 | |
| *** e0ne has joined #openstack-ironic | 09:54 | |
| etingof | rpioso, o/ by way of addressing bdodd's comments, I've rebased my floppy & idrac.boot patches. however I am not pushing them yet because I thought may be you want to push your version instead? | 09:56 |
| openstackgerrit | Mark Goddard proposed openstack/networking-generic-switch master: Allow network name format to be configured https://review.opendev.org/683187 | 09:59 |
| *** e0ne has quit IRC | 10:02 | |
| *** ociuhandu_ has quit IRC | 10:07 | |
| *** ociuhandu has joined #openstack-ironic | 10:07 | |
| *** ociuhandu has quit IRC | 10:14 | |
| *** ociuhandu has joined #openstack-ironic | 10:18 | |
| *** ociuhandu has quit IRC | 10:22 | |
| *** rachit7 has quit IRC | 10:26 | |
| *** ociuhandu has joined #openstack-ironic | 10:32 | |
| *** ociuhandu has quit IRC | 10:33 | |
| *** ociuhandu has joined #openstack-ironic | 10:34 | |
| *** ociuhandu has quit IRC | 10:39 | |
| *** ociuhandu has joined #openstack-ironic | 10:40 | |
| *** mkrai has quit IRC | 10:52 | |
| *** mkrai_ has joined #openstack-ironic | 10:52 | |
| *** pcaruana has quit IRC | 10:56 | |
| *** e0ne has joined #openstack-ironic | 11:03 | |
| openstackgerrit | Merged openstack/ironic-python-agent-builder master: CI: run a DIB job and make the inspector job non-voting https://review.opendev.org/682648 | 11:06 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: [WIP] Allow retrying PXE boot if it takes too long https://review.opendev.org/683127 | 11:06 |
| *** dsneddon has joined #openstack-ironic | 11:06 | |
| *** bobmel has joined #openstack-ironic | 11:10 | |
| dtantsur | folks, another small improvement for the CI: https://review.opendev.org/#/c/681166/ | 11:11 |
| patchbot | patch 681166 - ironic-tempest-plugin - Skip the manual cleaning test if software RAID tes... - 1 patch set | 11:11 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: DNM try breaking PXE boot https://review.opendev.org/683131 | 11:11 |
| *** rachit7 has joined #openstack-ironic | 11:11 | |
| openstackgerrit | Merged openstack/networking-baremetal master: Update the constraints url https://review.opendev.org/683338 | 11:13 |
| *** bobmel has quit IRC | 11:15 | |
| *** lucasagomes has joined #openstack-ironic | 11:19 | |
| *** e0ne has quit IRC | 11:24 | |
| *** igordc has quit IRC | 11:26 | |
| *** ociuhandu has quit IRC | 11:27 | |
| *** pcaruana has joined #openstack-ironic | 11:29 | |
| *** dsneddon has quit IRC | 11:31 | |
| *** rh-jelabarre has joined #openstack-ironic | 11:32 | |
| *** pcaruana has quit IRC | 11:42 | |
| *** pcaruana has joined #openstack-ironic | 11:42 | |
| *** dsneddon has joined #openstack-ironic | 11:42 | |
| *** dmellado has quit IRC | 11:45 | |
| *** dsneddon has quit IRC | 11:47 | |
| *** dmellado has joined #openstack-ironic | 11:48 | |
| *** dsneddon has joined #openstack-ironic | 12:01 | |
| *** rachit7 has quit IRC | 12:02 | |
| *** dsneddon has quit IRC | 12:06 | |
| *** ociuhandu has joined #openstack-ironic | 12:06 | |
| *** ociuhandu has quit IRC | 12:06 | |
| *** ociuhandu has joined #openstack-ironic | 12:07 | |
| *** goldyfruit___ has quit IRC | 12:12 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allow retrying PXE boot if it takes too long https://review.opendev.org/683127 | 12:14 |
| *** rloo has joined #openstack-ironic | 12:34 | |
| *** ociuhandu has quit IRC | 12:40 | |
| *** ociuhandu has joined #openstack-ironic | 12:45 | |
| *** ociuhandu has quit IRC | 12:45 | |
| *** ociuhandu has joined #openstack-ironic | 12:46 | |
| *** mkrai_ has quit IRC | 12:46 | |
| *** mkrai has joined #openstack-ironic | 12:47 | |
| *** rcernin has joined #openstack-ironic | 12:56 | |
| *** ociuhandu has quit IRC | 12:57 | |
| *** ociuhandu has joined #openstack-ironic | 12:57 | |
| openstackgerrit | raphael.glon proposed openstack/ironic-python-agent master: Software raid: default partition table type change https://review.opendev.org/674803 | 13:07 |
| *** dsneddon has joined #openstack-ironic | 13:08 | |
| *** ociuhandu has quit IRC | 13:09 | |
| *** jtwill98 has quit IRC | 13:12 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: DNM try breaking PXE boot https://review.opendev.org/683131 | 13:12 |
| *** ociuhandu has joined #openstack-ironic | 13:13 | |
| *** dsneddon has quit IRC | 13:13 | |
| *** ociuhandu has quit IRC | 13:16 | |
| *** ociuhandu has joined #openstack-ironic | 13:16 | |
| *** jawad_ax_ has quit IRC | 13:16 | |
| *** jawad_axd has joined #openstack-ironic | 13:17 | |
| *** jawad_axd has quit IRC | 13:22 | |
| TheJulia | dtantsur: the change default boot option to local change. It keeps getting rechecked when the underlying defect in the selection logic is still broken | 13:22 |
| dtantsur | TheJulia: sorry, maybe I'm too sleepy.. how does it cause random failures in the CI? | 13:22 |
| TheJulia | It doesn't, I was literally going down the list of most recently touched patches with failures | 13:23 |
| TheJulia | Focus on MTU and fortnebula | 13:23 |
| * TheJulia needs coffee and more coffee | 13:23 | |
| dtantsur | ah, I see | 13:24 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allow retrying PXE boot if it takes too long https://review.opendev.org/683127 | 13:26 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: DNM try breaking PXE boot https://review.opendev.org/683131 | 13:26 |
| *** goldyfruit___ has joined #openstack-ironic | 13:28 | |
| *** tzumainn has joined #openstack-ironic | 13:32 | |
| rloo | if anyone has any opinions on how to proceed with https://review.opendev.org/#/c/674803. Which seems to be basically undoing an earlier patch that landed in August. | 13:33 |
| patchbot | patch 674803 - ironic-python-agent - Software raid: default partition table type change - 10 patch sets | 13:33 |
| *** goldyfruit_ has joined #openstack-ironic | 13:35 | |
| *** goldyfruit___ has quit IRC | 13:35 | |
| TheJulia | rloo: I feel like if arne +2'ed it, he is comfortable with it merging. Do you know the patch it undoes in August, there was some back and forth on fixes | 13:36 |
| *** goldyfruit___ has joined #openstack-ironic | 13:36 | |
| rloo | TheJulia: should we delete the release note from the original patch in August -- since this one basically undoes that feature, so the release note doesn't make sense any more. | 13:37 |
| TheJulia | rloo: your going to need to link the original patch from august | 13:37 |
| rloo | TheJulia: i suspect there is one line from that original patch, that is still in this new patch. the rest was deleted. | 13:37 |
| rloo | TheJulia: https://review.opendev.org/#/c/675275/. sorry, was in the review. | 13:38 |
| patchbot | patch 675275 - ironic-python-agent - Software raid: mbr/gpt partition table alternative (MERGED) - 2 patch sets | 13:38 |
| *** goldyfruit_ has quit IRC | 13:39 | |
| dtantsur | morning rloo | 13:41 |
| rloo | happy friday morning dtantsur :) | 13:41 |
| dtantsur | I seem to remember that the older patch was a less invasive part of the patch in question | 13:41 |
| dtantsur | maybe things have changed since then | 13:41 |
| rloo | dtantsur: yup, the code is fairly simple -- no way for user to override. i'm ok with that, but we can't have a release note saying we have a feature, and then remove the feature w/o ... ? | 13:46 |
| dtantsur | fair | 13:46 |
| dtantsur | it must have changed since I last saw it | 13:46 |
| andrein | hi guys, not sure if this is the place to ask questions about diskimage builder, but I'm hoping someone can help me out. I've built my own image based on the centos7 image, however, after my server boots from it the network service is reporting as failed (although everything works correctly). this happens because /etc/sysconfig/network-scripts/ifcfg-eth0 exists, but my interfaces are actually named eno1-4. | 13:47 |
| *** mkrai has quit IRC | 13:48 | |
| andrein | is there any way to convince DIB to remove that file from the image? | 13:48 |
| *** mkrai_ has joined #openstack-ironic | 13:48 | |
| TheJulia | dtantsur: If you look at https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_778/683262/1/check/ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode/77869fb/testr_results.html.gz do you get the same feeling that the job was almost done? | 13:49 |
| *** sthussey has joined #openstack-ironic | 13:49 | |
| dtantsur | andrein: you may want to ask on #openstack-dib | 13:50 |
| andrein | thanks! | 13:50 |
| dtantsur | TheJulia: I've seen such errors. Indeed, everything is seemingly okay. | 13:51 |
| dtantsur | TheJulia: oh, btw, https://review.opendev.org/#/c/682609/ modifies something you've modified before | 13:51 |
| patchbot | patch 682609 - ironic - Do not log an error on heartbeat in deploying/clea... - 3 patch sets | 13:51 |
| TheJulia | xXraphXx: I've left a comment https://review.opendev.org/#/c/674803/10 you might want to take a look. I agree with rloo the prior release note needs to go. It also likely didn't render because it looks like it was created without the hash value for the docs build to be able to construct history. | 13:53 |
| patchbot | patch 674803 - ironic-python-agent - Software raid: default partition table type change - 10 patch sets | 13:53 |
| dtantsur | aha, TheJulia: ERROR ironic.conductor.utils [None req-ec9d3a98-f2be-4c81-84f3-fcf70815ba90 None None] Node d3df3249-5e03-4410-8409-aad904dfcf0b failed deploy step {u'priority': 100, u'interface': u'deploy', u'step': u'deploy', u'argsinfo': None}. Error: IPMI call failed: power on.: IPMIFailure: IPMI call failed: power on. | 13:53 |
| TheJulia | woot! | 13:53 |
| dtantsur | Error: no response from RAKP 3 message\nError: Received an Unexpected RAKP 4 message\nError in open session response message : Unknown (0x6C)\n\nError: Unable to establish IPMI v2 / RMCP+ session\n'.: ProcessExecutionError: Unexpected error while running command. | 13:53 |
| dtantsur | I feel like adding retries for literally everything ironic does >_< | 13:53 |
| TheJulia | dtantsur: we already have retries for ipmi exec operations | 13:54 |
| * TheJulia wrote the selective retry code | 13:54 | |
| dtantsur | mmm, probably not for this one | 13:54 |
| TheJulia | maybe instead, we burn ipmi with fire | 13:54 |
| TheJulia | no, but perhaps add it to the search list? | 13:54 |
| * dtantsur pokes etingof | 13:54 | |
| * TheJulia gets out a pencil sharpener | 13:55 | |
| *** dsneddon has joined #openstack-ironic | 13:55 | |
| dtantsur | I don't see anything abnormal in the virtualbmc logs, hmm | 13:55 |
| * etingof hides in the cloud | 13:56 | |
| *** belmoreira has quit IRC | 13:56 | |
| dtantsur | TheJulia: a very wild guess: can we have issues with IPMI when crossing between testing nodes? | 13:57 |
| dtantsur | to confirm that we need to check if this error ever happens on non-multinode jobs | 13:57 |
| TheJulia | that is stupidly highly likely | 13:58 |
| rpittau | hey TheJulia welcome back :) | 14:00 |
| *** belmoreira has joined #openstack-ironic | 14:00 | |
| *** dsneddon has quit IRC | 14:00 | |
| dtantsur | TheJulia: do you know how to use http://logstash.openstack.org? | 14:01 |
| TheJulia | https://ef6fea84da73ed48af61-f850a1c88b63080f1e34c13fe4924008.ssl.cf2.rackcdn.com/683262/1/check/ironic-tempest-ipa-partition-redfish-tinyipa/782fb5b/controller/logs/screen-ir-api.txt.gz <-- ouch, the conductor became ready 4 seconds AFTER the devstack script blew up trying to enroll nodes :( | 14:01 |
| dtantsur | ouch | 14:02 |
| dtantsur | do we need a loop waiting for non-empty driver list? | 14:02 |
| TheJulia | basically | 14:02 |
| TheJulia | it only tries six times seemingly in rapid succession | 14:03 |
| TheJulia | quick fix | 14:03 |
| *** mkrai_ has quit IRC | 14:04 | |
| TheJulia | oh no, we're relying upon the client integrated retries | 14:04 |
| TheJulia | hmmm | 14:04 |
| dtantsur | how long did it take to start? | 14:05 |
| TheJulia | started 2:38:35, ready 2:38:54 | 14:05 |
| TheJulia | still kind of a long time, but it looks like the db interactions were part of that. It could have just been slow on the back-end disk. | 14:06 |
| dtantsur | 20 seconds. a lot, but not too much. | 14:06 |
| dtantsur | yep | 14:06 |
| *** openstackgerrit has quit IRC | 14:06 | |
| TheJulia | I've typically seen ~10 seconds, so if we put a small wait, it would be appropriate to give the conductor time to actually fire up | 14:07 |
| dtantsur | looping on `driver list` is what I often recommend people to do | 14:07 |
| TheJulia | and then the odds of failure on enrolling nodes should be greatly reduced | 14:07 |
| TheJulia | if you have some example code for that, we coudl do that, I was thinking just putting sleep 10 before enroll_nodes | 14:07 |
| * TheJulia has no running ironic at the moment | 14:08 | |
| dtantsur | I have devstack here, lemme try something | 14:08 |
| TheJulia | k | 14:08 |
| TheJulia | mgoddard: re: https://review.opendev.org/#/c/683104/2 we don't do beta releases which seems to make your change confusing | 14:10 |
| patchbot | patch 683104 - bifrost - Switch git_branch to stable/train - 2 patch sets | 14:10 |
| TheJulia | *gasp* https://review.opendev.org/#/c/683262/ has a +1 from zuul | 14:11 |
| patchbot | patch 683262 - ironic - wip: if wholedisk set boot dev to disk - 1 patch set | 14:11 |
| mgoddard | TheJulia: hmm, I see your point | 14:12 |
| mgoddard | this is why deployment projects normally cycle-trailing | 14:13 |
| TheJulia | yeah, I'm fine if you want to cut a release and then a minor rev afterwards though | 14:13 |
| mgoddard | I don't mind too much if it doesn't make the first release | 14:13 |
| mgoddard | kolla won't need it for a while | 14:13 |
| TheJulia | And bifrost is reporting ""msg": "Node 4e41df61-84b1-5856-bfb6-6b5f2cd3dd11 reached failure state \"inspect failed\"; the last error is Failed to start inspection: Version requested but version discovery document was not found and allow_version_hack was False"" now anyway | 14:14 |
| TheJulia | I suspect a new version of ansible may be expecting different arguments. | 14:15 |
| *** openstackgerrit has joined #openstack-ironic | 14:16 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: devstack: wait for conductor to start and register itself https://review.opendev.org/683400 | 14:16 |
| dtantsur | TheJulia: like this ^^? | 14:16 |
| TheJulia | or a library has broken something | 14:16 |
| *** whoami-rajat has quit IRC | 14:17 | |
| TheJulia | dtantsur: +2 | 14:17 |
| *** dsneddon has joined #openstack-ironic | 14:17 | |
| TheJulia | Looks like our mtu fix is in the final stages to merge. If everyone would please hold off on rechecks so we can get some additional fixes in, it would be greatly appreciated | 14:19 |
| *** iurygregory_ has joined #openstack-ironic | 14:21 | |
| TheJulia | rloo: mgoddard: if you could review/approve https://review.opendev.org/#/c/683400/ it would be appreicated | 14:21 |
| patchbot | patch 683400 - ironic - devstack: wait for conductor to start and register... - 1 patch set | 14:21 |
| *** iurygregory has quit IRC | 14:21 | |
| * dtantsur hopes he did not make a silly mistake there :) | 14:22 | |
| TheJulia | it looked good to me | 14:22 |
| TheJulia | I'm a little worried about the conditional logic, but it will fail most likely if it is not good | 14:22 |
| *** rcernin has quit IRC | 14:23 | |
| TheJulia | the theme and overall idea is what I +2'ed :) | 14:23 |
| mgoddard | looking | 14:24 |
| dtantsur | the conditional logic is cargo-culted from another place and tested locally | 14:24 |
| TheJulia | k | 14:24 |
| TheJulia | dtantsur: you indicated there was yet another gate fix earlier? | 14:25 |
| rloo | TheJulia: dtantsur: +A. thx! | 14:25 |
| dtantsur | https://review.opendev.org/#/c/683127/ may help with random PXE failures | 14:25 |
| patchbot | patch 683127 - ironic - Allow retrying PXE boot if it takes too long - 6 patch sets | 14:25 |
| mgoddard | dtantsur: we don't have set -e in devstack, right? | 14:26 |
| dtantsur | I don't think we do | 14:26 |
| dtantsur | not entirely sure though, why? | 14:26 |
| mgoddard | just wondering if the driver list exits non-zero if there are no drivers | 14:27 |
| mgoddard | checking | 14:27 |
| dtantsur | I don't think so | 14:27 |
| rloo | it shouldn't. it isn't an error if there are no entries. | 14:28 |
| rloo | but worth checking :) | 14:28 |
| dtantsur | hmmm, I remember something.. lemme try | 14:28 |
| mgoddard | confirmed | 14:28 |
| mgoddard | 0 | 14:28 |
| TheJulia | heh | 14:28 |
| dtantsur | yep, it's okay | 14:29 |
| dtantsur | we had a request to make it a failure, but we refused | 14:29 |
| mgoddard | does it matter that we don't check the hostname? | 14:30 |
| TheJulia | it doesn't matter as long as the conductor is up | 14:30 |
| mgoddard | i.e. is it enough in multinode to just have one conductor up? | 14:30 |
| mgoddard | I guess so | 14:30 |
| dtantsur | they'll get created on one conductor and will rebalance later? | 14:30 |
| TheJulia | they will, I've actually seen it happen | 14:31 |
| dtantsur | ironic magic! | 14:32 |
| mgoddard | TheJulia: -2 for MTU :( | 14:32 |
| dtantsur | okay folks, I'm going to another flat visit, then straight for beers | 14:32 |
| TheJulia | #(#@!*@*#!( | 14:32 |
| dtantsur | :* | 14:32 |
| dtantsur | :( | 14:32 |
| rpittau | dtantsur: enjoy both :) | 14:32 |
| dtantsur | flat visits are rarely fun :) | 14:32 |
| dtantsur | have a great weekend | 14:32 |
| iurygregory_ | same error from before on mtu? | 14:33 |
| TheJulia | any objection if we make the the multinodes non-voting for a few days? | 14:33 |
| dtantsur | we probably have to :( | 14:33 |
| dtantsur | it becomes a pre-release tradition, should we add it to the check-list? | 14:33 |
| TheJulia | Thats kind o fhw hat I'm thinking, they bring their own unique races | 14:33 |
| * dtantsur stops kidding and actually goes | 14:33 | |
| TheJulia | o/ | 14:33 |
| TheJulia | Any other cores have feelings on doing so?!? | 14:33 |
| *** dtantsur is now known as dtantsur|afk | 14:33 | |
| * TheJulia kind of agrees we should just doc it as a normal thing because I think I've had to do it for 3 out of the last six releases?!? | 14:35 | |
| *** rcernin has joined #openstack-ironic | 14:38 | |
| *** ociuhandu has quit IRC | 14:39 | |
| rpittau | TheJulia: it's ok for me about the temp non-voting for multinodes | 14:41 |
| mgoddard | TheJulia: if we are confident these are CI issues and not ironic issues, then I'm +1 | 14:42 |
| xXraphXx | Hello ironic o/. TheJulia, about https://review.opendev.org/#/c/674803/ -> answered, I'm actually a bit lost about the way it should be fixed now (sth like patchset 8 ?)... So, I don't want to block anything: should we just wait for it ? If so we may as well revert https://review.opendev.org/#/c/675275/ | 14:43 |
| patchbot | patch 674803 - ironic-python-agent - Software raid: default partition table type change - 10 patch sets | 14:43 |
| patchbot | patch 675275 - ironic-python-agent - Software raid: mbr/gpt partition table alternative (MERGED) - 2 patch sets | 14:43 |
| xXraphXx | *wait for after Train I mean | 14:45 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Make multinode jobs non-voting https://review.opendev.org/683412 | 14:47 |
| NobodyCam | Good Morning Folks, and OFC... | 14:47 |
| NobodyCam | TGIF! | 14:47 |
| TheJulia | mgoddard: likely a mixed bag of things around race conditions :( | 14:47 |
| *** ociuhandu has joined #openstack-ironic | 14:49 | |
| *** ociuhandu has quit IRC | 14:49 | |
| *** ociuhandu has joined #openstack-ironic | 14:50 | |
| iurygregory_ | something in common with the 2 failures: the host have the disk problem and we are using LIBVIRT_STORAGE_POOL_PATH, when we add this option the storyboard also mentioned that we should lower the concurrency for tempest https://storyboard.openstack.org/#!/story/2006520 | 14:52 |
| *** iurygregory_ is now known as iurygregory | 14:53 | |
| *** TxGirlGeek has joined #openstack-ironic | 14:55 | |
| *** TxGirlGeek has quit IRC | 14:56 | |
| *** TxGirlGeek has joined #openstack-ironic | 14:57 | |
| *** dsneddon has quit IRC | 15:00 | |
| * iurygregory "but maybe not related since it worked on the check (2nd run) " CI why you do this =( | 15:01 | |
| openstackgerrit | raphael.glon proposed openstack/ironic-python-agent master: Software raid: default partition table type change https://review.opendev.org/674803 | 15:02 |
| openstackgerrit | raphael.glon proposed openstack/ironic-python-agent master: Fix doc rendering https://review.opendev.org/683419 | 15:03 |
| *** _mmethot_ has quit IRC | 15:04 | |
| *** _mmethot_ has joined #openstack-ironic | 15:05 | |
| *** bnemec is now known as beekneemech | 15:06 | |
| *** belmoreira has quit IRC | 15:07 | |
| *** gyee has joined #openstack-ironic | 15:07 | |
| *** belmoreira has joined #openstack-ironic | 15:11 | |
| *** tzumainn has quit IRC | 15:11 | |
| rloo | TheJulia: xXraphXx: wrt https://review.opendev.org/#/c/674803/. I don't think/know if we should try to get it into Train. Having said that, if you/we feel that we don't want (or are not sure if we want) https://review.opendev.org/#/c/675275/ -- I would prefer if we reverted that before doing a Train release. | 15:12 |
| patchbot | patch 674803 - ironic-python-agent - Software raid: default partition table type change - 11 patch sets | 15:12 |
| patchbot | patch 675275 - ironic-python-agent - Software raid: mbr/gpt partition table alternative (MERGED) - 2 patch sets | 15:12 |
| openstackgerrit | Will Szumski proposed openstack/tenks master: Add debugging statements to wait for resources loop https://review.opendev.org/659025 | 15:14 |
| openstackgerrit | raphael.glon proposed openstack/ironic-python-agent master: Revert "Software raid: mbr/gpt partition table alternative" https://review.opendev.org/683421 | 15:14 |
| *** trident has quit IRC | 15:14 | |
| xXraphXx | rloo, sure, done. Thanks for your feedback | 15:16 |
| arne_wiebalck | rloo: I agree. xXraphXx: thanks! | 15:18 |
| *** goldyfruit___ has quit IRC | 15:21 | |
| *** trident has joined #openstack-ironic | 15:24 | |
| *** goldyfruit has joined #openstack-ironic | 15:26 | |
| *** trident has quit IRC | 15:29 | |
| openstackgerrit | raphael.glon proposed openstack/ironic-python-agent master: Revert "Software raid: mbr/gpt partition table alternative" https://review.opendev.org/683421 | 15:32 |
| arne_wiebalck | rloo: xXraphXx: This however means we won't have UEFI/gpt support for s/w RAID in Train? | 15:33 |
| xXraphXx | yep | 15:33 |
| xXraphXx | but anyway, these pr would not have been enough by themselves | 15:33 |
| arne_wiebalck | ah, no? | 15:33 |
| arne_wiebalck | not having this support for the first release supporting s/w RAID is a pity though | 15:34 |
| TheJulia | I actually would like to see it | 15:35 |
| TheJulia | I get why, and if we merge the second patch that while I have a slight issue with it, ultimately thats just a bug we could fix after the initial release since bios boot or not uefi firmware loading on netboot is just an edge case of what _should_ be happening | 15:36 |
| arne_wiebalck | I think we have a pretty good idea how we'd like to do it ... xXraphXx: the thing missing is the other patch you've put up? | 15:36 |
| arne_wiebalck | https://review.opendev.org/#/c/674879/ ? | 15:37 |
| patchbot | patch 674879 - ironic-python-agent - image extension, install_bootloader improvements - 10 patch sets | 15:37 |
| arne_wiebalck | or is there still sth else? | 15:37 |
| xXraphXx | that one, and there would also be missing some other small changes like the following wip: https://review.opendev.org/#/c/675672/ (or any esp relocation mechanism, that would allow the softraid to be bootable with uefi) | 15:37 |
| patchbot | patch 675672 - ironic-python-agent - Additional command to build efi partitions - 2 patch sets | 15:37 |
| *** belmoreira has quit IRC | 15:37 | |
| arne_wiebalck | ok, I see | 15:38 |
| xXraphXx | (and remove the raid 1 restriction constraint on the root device) | 15:38 |
| arne_wiebalck | I don't know how you see it, but I think that is too ambitious for Train (given the time left) | 15:39 |
| *** trident has joined #openstack-ironic | 15:40 | |
| arne_wiebalck | I commit myself to help with testing all this, early on in Ussuri. | 15:41 |
| * arne_wiebalck is less selfless than this sounds since we will get some UEFI only hardware in the next hardware delivery | 15:42 | |
| *** rcernin has quit IRC | 15:44 | |
| xXraphXx | arne_wiebalck, yep agree, too ambitious -> softraid still works great for BIOS, and we are close for UEFI. On our side, we use it daily (with the small few amends above), combined with deploy templates, to spawn dynamically built raids based on nova flavors. Works great! So thanks to Ironicers for softraid and deploy steps/deploy templates ! | 15:47 |
| arne_wiebalck | xXraphXx: wait ... you build the RAID via deploy steps? | 15:48 |
| *** iurygregory has quit IRC | 15:52 | |
| xXraphXx | arne_wiebalck, yep. I'm using a mixed driver between ansible + agent deploy driver. It's "mixed" (understand: sometimes so ugly it could hurt eyes :) ), so I'm having troubles extracting/cleaning it to share upstream | 15:54 |
| *** jawad_axd has joined #openstack-ironic | 15:55 | |
| *** tzumainn has joined #openstack-ironic | 15:56 | |
| *** TxGirlGeek has quit IRC | 15:57 | |
| rpittau | bye all, have a great weekend! o/ | 15:57 |
| arne_wiebalck | xXraphXx: that sounds really great (apart from the "hurts eyes" part ;-)! Leveraging deploy steps for s/w was actually one of the next steps I had in mind. Will you be at the summit (to share some details)? | 15:57 |
| *** rpittau is now known as rpittau|afk | 15:57 | |
| *** TxGirlGeek has joined #openstack-ironic | 15:58 | |
| *** jawad_axd has quit IRC | 15:59 | |
| xXraphXx | arne_wiebalck, unfortunately not :/, may be next one. But definitely need to find some time to push it (even as a wip). | 16:02 |
| mgoddard | night all, have a good weekend | 16:03 |
| arne_wiebalck | xXraphXx: that'd be great ... anyway: once I get round to have a look at deploy steps for s/w RAID, I'll bother you :) | 16:04 |
| TheJulia | xXraphXx: arne_wiebalck: so, I've been in a meeting for a while... well a couple meetings. What do we feel the consensus is? | 16:06 |
| xXraphXx | arne_wiebalck, sure, happy to share | 16:06 |
| arne_wiebalck | TheJulia: As getting UEFI/gpt work will require multiple patches, including a wip, and due to lack of time for proper testing, we agreed that it is maybe better to not push this into Train. | 16:09 |
| TheJulia | okay | 16:09 |
| TheJulia | I just wanted to make sure things had not changed from start to end of the entire discussion | 16:09 |
| arne_wiebalck | TheJulia: xXraphXx is almost there, though, and I am happy to help with testing. | 16:10 |
| arne_wiebalck | TheJulia: So, I think this is not very far from adding support for UEFI/gpt in s/w RAID. | 16:10 |
| arne_wiebalck | TheJulia: And since peo | 16:11 |
| arne_wiebalck | TheJulia: And since people picked up s/w RAID before the release, they will also pick this feature up before Ussuri. | 16:11 |
| TheJulia | peo? | 16:11 |
| TheJulia | oh, heh | 16:11 |
| TheJulia | very very very likely | 16:11 |
| * arne_wiebalck is working with a new keyboard | 16:12 | |
| arne_wiebalck | TheJulia: As a bonus xXraphXx uses s/w RAID with deploy steps! | 16:12 |
| TheJulia | I picked up on that :) | 16:12 |
| * TheJulia senses this will be a long weekend of recheck/patch approval | 16:17 | |
| *** pcaruana has quit IRC | 16:19 | |
| arne_wiebalck | bye, everyone, have a good weekend! | 16:20 |
| TheJulia | you too arne_wiebalck | 16:21 |
| openstackgerrit | Merged openstack/ironic-tempest-plugin master: Skip the manual cleaning test if software RAID testing is enabled https://review.opendev.org/681166 | 16:21 |
| *** derekh has quit IRC | 16:42 | |
| *** ociuhandu has quit IRC | 16:43 | |
| rpioso | TheJulia: I'm considering rebasing https://review.opendev.org/682029 | 16:53 |
| patchbot | patch 682029 - ironic - Add `FLOPPY` boot device constant - 1 patch set | 16:53 |
| rpioso | TheJulia: It keeps failing in CI. Would rebasing prospectively help? | 16:53 |
| TheJulia | rpioso: you can rebase I guess... But CI has a high failure rate until a MTU setting fix meges | 16:54 |
| TheJulia | Merges | 16:54 |
| *** igordc has joined #openstack-ironic | 16:55 | |
| TheJulia | rpioso: wait on rebase | 16:55 |
| openstackgerrit | raphael.glon proposed openstack/ironic-python-agent master: Revert "Software raid: mbr/gpt partition table alternative" https://review.opendev.org/683421 | 16:56 |
| rpioso | TheJulia: AFAIU, rebasing wipes out existing approvals. Right? | 16:56 |
| rpioso | TheJulia: Ack on waiting. | 16:56 |
| rpioso | etingof: ^^^ | 16:56 |
| rpioso | TheJulia: Please note idrac Redfish vmedia boot depends on that change. | 16:57 |
| TheJulia | rpioso: understood, so that patch is failing on the mtu issue | 16:58 |
| TheJulia | so once that merges, we can recheck. But we need to wait. | 16:58 |
| rpioso | TheJulia: Gotcha | 16:58 |
| rpioso | TheJulia: Thank you! | 16:58 |
| TheJulia | Sep 20 16:38:16 ubuntu-bionic-fortnebula-regionone-0011601584 kernel: qr-3ff709f4-3b: dropped over-mtu packet: 1400 > 1350 | 16:58 |
| TheJulia | in https://6b681e9843df38a89642-8aefe49572c9ad91eb217e6fb5236bb6.ssl.cf5.rackcdn.com/682029/1/gate/ironic-tempest-ipa-partition-uefi-pxe_ipmitool-tinyipa/063c2e6/controller/logs/syslog.txt.gz | 16:58 |
| *** ociuhandu has joined #openstack-ironic | 17:00 | |
| *** goldyfruit has quit IRC | 17:01 | |
| *** goldyfruit has joined #openstack-ironic | 17:01 | |
| openstackgerrit | Merged openstack/ironic-python-agent-builder master: Update tinycore from 8.x to 9.x https://review.opendev.org/666591 | 17:03 |
| openstackgerrit | Merged openstack/ironic-python-agent-builder master: Remove dib/build.sh https://review.opendev.org/680376 | 17:03 |
| *** ociuhandu has quit IRC | 17:04 | |
| TheJulia | ugh, looks like it is going to be a couple hours before the ironic queue gets to the job in question | 17:06 |
| TheJulia | still want an abort button.... | 17:10 |
| *** dougsz has joined #openstack-ironic | 17:20 | |
| *** e0ne has joined #openstack-ironic | 17:22 | |
| *** goldyfruit has quit IRC | 17:31 | |
| *** dougsz has quit IRC | 17:44 | |
| *** whoami-rajat has joined #openstack-ironic | 17:50 | |
| *** goldyfruit has joined #openstack-ironic | 18:07 | |
| *** henriqueof has joined #openstack-ironic | 18:09 | |
| *** dougsz has joined #openstack-ironic | 18:11 | |
| *** goldyfruit_ has joined #openstack-ironic | 18:14 | |
| *** goldyfruit has quit IRC | 18:17 | |
| *** igordc has quit IRC | 18:20 | |
| *** dsneddon has joined #openstack-ironic | 18:22 | |
| *** igordc has joined #openstack-ironic | 18:26 | |
| *** ricolin has quit IRC | 18:27 | |
| *** ricolin has joined #openstack-ironic | 18:27 | |
| *** dsneddon has quit IRC | 18:27 | |
| *** jistr has quit IRC | 18:28 | |
| *** jistr has joined #openstack-ironic | 18:28 | |
| *** dansmith has quit IRC | 18:29 | |
| *** dansmith has joined #openstack-ironic | 18:29 | |
| *** e0ne has quit IRC | 18:34 | |
| *** e0ne has joined #openstack-ironic | 18:36 | |
| *** e0ne has quit IRC | 18:36 | |
| *** dougsz has quit IRC | 18:36 | |
| *** e0ne has joined #openstack-ironic | 18:37 | |
| *** e0ne has quit IRC | 18:41 | |
| *** alexmcleod has quit IRC | 19:12 | |
| *** igordc has quit IRC | 19:13 | |
| *** ricolin has quit IRC | 19:16 | |
| *** igordc has joined #openstack-ironic | 19:19 | |
| *** lucasagomes has quit IRC | 19:25 | |
| *** dsneddon has joined #openstack-ironic | 19:31 | |
| *** ociuhandu has joined #openstack-ironic | 19:49 | |
| *** whoami-rajat has quit IRC | 20:00 | |
| *** ociuhandu has quit IRC | 20:03 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Fix biforst compatability https://review.opendev.org/683722 | 20:13 |
| *** jtomasek has quit IRC | 20:32 | |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: DNM: testing ci https://review.opendev.org/683729 | 20:58 |
| *** _mmethot_ has quit IRC | 21:05 | |
| *** mmethot_ has joined #openstack-ironic | 21:05 | |
| *** rloo has quit IRC | 21:25 | |
| *** rloo has joined #openstack-ironic | 21:26 | |
| *** rh-jelabarre has quit IRC | 21:43 | |
| *** rh-jelabarre has joined #openstack-ironic | 21:44 | |
| TheJulia | dtantsur|afk: I have htis feeling something with with opesntack sdk has gone sideways with bifrost... specificially in the communication out to ironic-inspector. The conductor is throwing an error now, and yeah. :\ | 22:15 |
| TheJulia | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_1b2/683729/1/check/bifrost-integration-tinyipa-ubuntu-xenial/1b2b68e/logs/ironic/ironic-conductor.log | 22:15 |
| *** stevebaker has joined #openstack-ironic | 22:20 | |
| *** rh-jelabarre has quit IRC | 22:22 | |
| *** goldyfruit_ has quit IRC | 22:42 | |
| *** manjeets has quit IRC | 23:10 | |
| *** patchbot has quit IRC | 23:12 | |
| *** mgoddard has quit IRC | 23:17 | |
| *** patchbot has joined #openstack-ironic | 23:17 | |
| *** mgoddard has joined #openstack-ironic | 23:19 | |
| openstackgerrit | Merged openstack/ironic master: Lower MTU override https://review.opendev.org/683249 | 23:21 |
| *** jawad_axd has joined #openstack-ironic | 23:24 | |
| TheJulia | \o/ | 23:27 |
| TheJulia | and there was joy | 23:27 |
| *** jawad_axd has quit IRC | 23:28 | |
| *** dsneddon has quit IRC | 23:29 | |
| *** sthussey has quit IRC | 23:47 | |
| rpioso | TheJulia: Indeed! I rechecked the change we discussed earlier. | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!