| *** Vorrtex has joined #openstack-dib | 04:13 | |
| *** Vorrtex has quit IRC | 04:22 | |
| *** hwoarang_ has quit IRC | 04:37 | |
| *** hwoarang has joined #openstack-dib | 04:42 | |
| *** hwoarang has quit IRC | 06:57 | |
| *** hwoarang has joined #openstack-dib | 07:04 | |
| *** jangutter has quit IRC | 07:15 | |
| *** jangutter_ has joined #openstack-dib | 07:15 | |
| *** sziviani has joined #openstack-dib | 08:08 | |
| *** priteau has joined #openstack-dib | 08:15 | |
| *** yolanda_ has quit IRC | 09:47 | |
| rm_work | yeah we could PROBABLY use that tonyb | 12:08 |
|---|---|---|
| rm_work | if we're willing to say "this is the official openstack build" or something... I suppose Redhat might not want to support that though if people call them up <_< | 12:09 |
| rm_work | though not sure anyone can actually do that with centos anyway, they'd want a real RHEL amp | 12:09 |
| rm_work | cgoncalves: ^^ comment? | 12:09 |
| cgoncalves | rm_work, RHEL-based amps (read as in OSP) we install the official/supported haproxy 1.5, not 1.8 | 12:21 |
| rm_work | ahh :/ | 12:21 |
| rm_work | well that's shitty | 12:21 |
| rm_work | :( | 12:21 |
| rm_work | and no real support for CentOS anyway right? | 12:22 |
| rm_work | so maybe it's fine | 12:22 |
| rm_work | we can just pick whatever one, so if there's one in the openstack ecosystem SOMEWHERE we can pull it | 12:22 |
| cgoncalves | CentOS is community-supported :) | 12:22 |
| *** priteau has quit IRC | 12:42 | |
| cgoncalves | FYI, there's a memory issue with the haproxy version in the openshift repo: https://storyboard.openstack.org/#!/story/2005412 | 12:46 |
| *** mjturek has joined #openstack-dib | 13:07 | |
| *** priteau has joined #openstack-dib | 13:27 | |
| *** mjturek has quit IRC | 13:35 | |
| *** mjturek has joined #openstack-dib | 13:36 | |
| *** Vorrtex has joined #openstack-dib | 13:58 | |
| *** priteau has quit IRC | 14:04 | |
| Vorrtex | cgoncalves clarkb what's the best way to lock down where these steps are located in elements for building one of these images? | 14:50 |
| clarkb | Vorrtex: dib logging (with -x maybe) should be quite verbose and step you through it command by command | 14:51 |
| clarkb | which should show you where things have gone wrong | 14:51 |
| clarkb | The one thing it does that is annoying is only show you the script name not which element it came from iirc | 14:51 |
| clarkb | so if the script name doesn't clear that up you may have to search for where that file lives | 14:52 |
| Vorrtex | Okay, well, its failing on a massive yum install line... which doesn't seem to have a script name attached in the output that I can see? | 14:54 |
| Vorrtex | 2019-04-09 14:52:18.852 | > + yum -v -y install make automake gcc gcc-c++ kernel-devel cryptsetup less iputils acpid libffi-devel logrotate selinux-policy-targeted rsyslog socat curl python-devel dkms git openssh-clients iptables keepalived ipvsadm python2-devel policycoreutils-python libselinux-python which tcpdump policycoreutils ethtool open-vm-tools nmap-ncat traceroute openssh-server openssl-devel screen vconfig gawk | 14:54 |
| Vorrtex | ca-certificates bash-completion selinux-policy irqbalance dbus sos acl psmisc net-tools cloud-init lsof haproxy18 at | 14:54 |
| clarkb | you'll have to scan backwards to get the script name | 14:55 |
| clarkb | it prints on a line of its own iirc | 14:55 |
| clarkb | chances are that is from the pkg_resources element though | 14:55 |
| Vorrtex | That's after scrolling up through all the 4billion lines of yum output ha ha | 14:55 |
| clarkb | the pkg_resources element takes deps defined in each element and combines them together to install them all | 14:55 |
| Vorrtex | dib-run-parts Running /tmp/in_target.d/install.d/01-package-installs | 14:57 |
| Vorrtex | I think that's the failing bit? | 14:57 |
| clarkb | ya and I bet that is in elements/pkg_resources/install.d/01-package-installs | 14:57 |
| clarkb | is it failing because one of the packages listed isn't valid in the ppc repo? | 14:58 |
| clarkb | if so then you'll have to track back to which element adds it as a dep | 14:58 |
| clarkb | pkg-map and package-installs.yaml files would be the files to look in I think if that is the issue | 14:59 |
| Vorrtex | lol, nope, its failing with a 404 looking in a location that doesn't have a ppc64le architecture | 14:59 |
| Vorrtex | There's about 15 or 20 packages with this output: | 14:59 |
| Vorrtex | 2019-04-09 14:52:24.287 | > glibc-headers-2.17-260.el7_6.3.ppc64le: failed to retrieve glibc-headers-2.17-260.el7_6.3.ppc64le.rpm from paas | 14:59 |
| Vorrtex | 2019-04-09 14:52:24.287 | > error was [Errno 14] HTTP Error 404 - Not Found | 14:59 |
| clarkb | oh is it still installing the paas repo? | 15:00 |
| Vorrtex | And if you navigate to the URL provided further up the stack trace, its just an x86_64 architecture | 15:00 |
| clarkb | (I thought that was going to get commented out, but maybe there is a second paas repo configuration) | 15:00 |
| Vorrtex | no, I don't know why its saying "from paas", but the URLs don't have paas in them at all. | 15:01 |
| Vorrtex | http://mirror.centos.org/centos/7/updates/ppc64le/Packages/sos-3.6-16.el7.centos.noarch.rpm | 15:01 |
| Vorrtex | That's the failing URL | 15:01 |
| Vorrtex | well, one of them. | 15:01 |
| Vorrtex | pick up and place any of the failing packages at the URL and it wouldn't matter, since ppc64le is not there. | 15:01 |
| Vorrtex | clarkb either way, I can't seem to find anything at "elements/pkg_resources". Is that supposed to be in diskimage-builder repo? | 15:02 |
| clarkb | Vorrtex: yes part of disk image builder's included elements and it is called "package-installs" | 15:03 |
| cgoncalves | Vorrtex, hey. you might want to use http://mirror.centos.org/altarch/7/updates/ | 15:03 |
| Vorrtex | *I* didn't set that URL, I'm trying to FIND where that URL is set... lol | 15:03 |
| Vorrtex | cgoncalves but thanks for the updated URL | 15:03 |
| cgoncalves | meaning playing tricks with DIB to pick http://mirror.centos.org/altarch/ as base for non x86 arches | 15:04 |
| Vorrtex | cgoncalves if I can find where that URL is set, I can make a patch to detect non-x86 arches and do "the right thing"... which of course, begs the question why it has to be in an "altarch" designation in the first place, since it already has a directory that filters by arch, but hey! I can't reform everyone to do the right thing! ;) | 15:05 |
| clarkb | Vorrtex: centos-minimal/yum.repos.d/yum.repo maybe | 15:05 |
| clarkb | hrm no you don't use centos-minimal | 15:05 |
| cgoncalves | not sure if http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/rpm-distro/pre-install.d/01-override-yum-arch is the right place. currently on a meeting, sorry | 15:06 |
| Vorrtex | clarkb my original question was more geared toward "set me up with some tools so I don't have to hop in this channel, drop a stack trace, and let you guys troubleshoot locations for me." lol. I appreciate how helpful you guys are, but I would love to be self sufficient. | 15:06 |
| clarkb | Vorrtex: the drop a stack trace equivalent is to add `bash` to the script that is failing. You'll then get a shell and be able to poke around | 15:07 |
| Vorrtex | So in this case, I still can't find package-installs anywhere | 15:07 |
| clarkb | Vorrtex: openstack/diskimage-builder/diskimage_builder/elements/package-installs | 15:07 |
| cgoncalves | diskimage_builder/elements/centos7/pre-install.d/01-set-centos-mirror | 15:07 |
| Vorrtex | clarkb you're actually killing me... My problem is finding the script that's failing... lol | 15:07 |
| cgoncalves | Vorrtex, ^. so you may want to add a if-then-else condition there | 15:08 |
| clarkb | The way dib operates is via a collection of elements in an elements path. When you run dib you select elements out of that path and a series of run parts scripts are merged together then run in sequence | 15:08 |
| clarkb | install.d is the "install" phase and all install.d scripts from all selected elements get merged together and run parts | 15:08 |
| clarkb | identifying what element is running a thing is definitely the first step in debugging. Hopefully ^ helps in the spelunking | 15:10 |
| clarkb | unfortunately the logging doesn't identify which element a script originated from, but does identify the script | 15:13 |
| Vorrtex | This is the least helpful script ever... lol. | 15:15 |
| Vorrtex | diskimage-builder/diskimage_builder/elements/package-installs/install.d/01-package-installs | 15:16 |
| Vorrtex | All it does is an install of a temp file. | 15:16 |
| Vorrtex | I'm guessing that's what you're talking about the aggregated list of packages needing installed. | 15:16 |
| clarkb | ya. And looks like the extra-data.d script creates the tmpfile | 15:17 |
| Vorrtex | Yep. | 15:17 |
| clarkb | Vorrtex: thinking about debugging this specific problem I would drop a bash into the start of that script, then look in /etc/yum to find which file defines that repo then search your dib log for what writes that file | 15:21 |
| Vorrtex | yeah, landed on the same flow a little later. Now I'm working on the if/else. | 15:23 |
| Vorrtex | Sorry for the runaround... I don't think I was very clear with my expectations nor my requests. Thanks for the help, clarkb , I do appreciate it. | 15:23 |
| clarkb | no worries. I think this identifies it would be really helpful to better identify the source script in the logs | 15:24 |
| clarkb | (which is good feedback) | 15:25 |
| Vorrtex | I mean, even without debug I could find that script name, but not where to find it specifically. I'm not sure how you could add better "tracing", but it could help if possible. | 15:29 |
| clarkb | we could potentially print the full rooted path to the script | 15:29 |
| Vorrtex | That might be excessive as a default, but part of debug, I vote yes. | 15:31 |
| * Vorrtex cries in "vote doesn't really matter" | 15:31 | |
| Vorrtex | cgoncalves do I need to reinstall DIB to pick up this centos config change? I dropped a bash into this distribution mirror file change and it didn't stop me there, nor did my if/else get evaluated./ | 15:34 |
| cgoncalves | Vorrtex, yes | 15:35 |
| Vorrtex | cgoncalves thanks | 15:35 |
| *** openstackgerrit has joined #openstack-dib | 15:50 | |
| openstackgerrit | Pedro Alvarez Piedehierro proposed openstack/diskimage-builder master: Fix Fedora aarch64 image location https://review.openstack.org/651283 | 15:50 |
| *** mjturek has quit IRC | 16:05 | |
| Vorrtex | cgoncalves so oddly enough a script was already updating to use "altarch", but nothing was uncommenting them. So I added a change to uncomment them... (probably not the right change, but it " | 16:11 |
| Vorrtex | "worked"). However, looks like its still failing on a different mirror list... lol, so RIP for that. | 16:11 |
| cgoncalves | one down, more to go! | 16:11 |
| Vorrtex | Story of the development life, amirite? | 16:12 |
| *** priteau has joined #openstack-dib | 16:13 | |
| clarkb | Vorrtex: as a side note we (openstack-infra) use the centos-minimal element which builds a new image from scratch using yum. I have no idea if that would be more reliable or not. We did it to have more control over the installed package set overtime. | 16:18 |
| Vorrtex | I don't know if that will work for the octavia amphora... That's what I'm trying to build on ppc64le | 16:19 |
| clarkb | you might have to supply your own element that lists out additional packages to install if the amphora elements depend on the implicit packages included in the normal cloud image | 16:20 |
| clarkb | but other than that I would expect it to work. However I've never tested it on ppc :) | 16:20 |
| Vorrtex | Sure, you mean other packages should I use centos-minimal, right? | 16:21 |
| clarkb | correct | 16:22 |
| clarkb | for example ssh isn't installed by default in that case | 16:22 |
| clarkb | neither is vim which always bugs me :P | 16:23 |
| Vorrtex | lol, I would argue vim being absent makes sense, since "minimal" to me would denote "only the required things" and the lack of fluff. | 16:23 |
| Vorrtex | While I love me some vim (use it every day), it is "fluff" ha ha | 16:24 |
| *** mjturek has joined #openstack-dib | 16:51 | |
| *** priteau has quit IRC | 17:04 | |
| Vorrtex | Alright, stuck again... I'm still getting these package installation errors, but I've verified the repofiles are all accurate... where else should I look? | 19:00 |
| Vorrtex | clarkb cgoncalves ? Any ideas? | 19:15 |
| clarkb | Vorrtex: sorry in a meeting now | 19:15 |
| Vorrtex | Alright, no big. I'm about to be in a meeting in 15 mins anyway | 19:15 |
| Vorrtex | clarkb when you get a minute: https://gist.github.com/tvardema/5bb00d0d327b6a026edbddd0b5953d3d | 19:35 |
| Vorrtex | The paas repo file is "correct", but for whatever reason those URLs are still wrong. | 19:35 |
| cgoncalves | Vorrtex, you're pointing to http://mirror.centos.org/centos/7/updates/ppc64le/Packages/ which does not exist | 19:47 |
| cgoncalves | http://mirror.centos.org/centos/7/updates/ppc64le/Packages/glibc-devel-2.17-260.el7_6.3.ppc64le.rpm: [Errno 14] HTTP Error 404 - Not Found | 19:47 |
| cgoncalves | point it to http://mirror.centos.org/altarch/7/updates/ppc64le/Packages/ | 19:47 |
| cgoncalves | going AFK for the day | 19:49 |
| Vorrtex | cgoncalves No, I know its pointing to the wrong one, but I've verified all the repofiles at /etc/yum.repos.d/*.repo are correct | 19:49 |
| Vorrtex | I'm asking where else I can look to find where it might be wrong URL. | 19:50 |
| Vorrtex | I got a suggestion to look for a hardcoded one in the Dockerfile. | 19:50 |
| *** mjturek has quit IRC | 20:18 | |
| Vorrtex | hey clarkb is there a specific base Dockerfile or something used that you know of? | 20:22 |
| clarkb | I dont think there is any docker invol ed | 20:23 |
| Vorrtex | clarkb good to know, bad to find out... I'm not sure where else to look. Like I mentioned above, all the repofiles are correctly using altarch, so I don't know what step is failing here. | 20:24 |
| clarkb | does yum have a debug mode that might explain its choices better? | 20:25 |
| Vorrtex | potentially? I'll check. | 20:25 |
| Vorrtex | it already does verbose output for the yum install. | 20:36 |
| *** Vorrtex has quit IRC | 20:47 | |
| openstackgerrit | Pierre Riteau proposed openstack/diskimage-builder master: Increase size of EFI system partition (again) https://review.openstack.org/641702 | 21:41 |
| *** hwoarang has quit IRC | 23:54 | |
| *** hwoarang has joined #openstack-dib | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!