| *** salv-orlando has joined #openstack-infra | 00:00 | |
| *** jroll has joined #openstack-infra | 00:00 | |
| rm_work | err | 00:02 |
|---|---|---|
| clarkb | ok I've got to go spend time with the kids now. I'm operating under the assumption that I'll pick up the bandersnatch upgrade in the morning and will hold the lock until then | 00:02 |
| rm_work | k i'll figure that out | 00:02 |
| clarkb | rm_work: https://review.openstack.org/#/c/561049/7/playbooks/legacy/neutron-lbaasv2-dsvm-api-l7octavia/run.yaml all those exported vars there control how devstack-gate acts | 00:03 |
| rm_work | ah ok so I can just do it there instead, k | 00:03 |
| rm_work | no big deal then | 00:03 |
| clarkb | #status log PyPi mirror updating is on pause while we sort out updating bandersnatch in order to blacklist large packages that keep filling our mirror disk volumes. | 00:04 |
| openstackstatus | clarkb: finished logging | 00:04 |
| *** salv-orlando has quit IRC | 00:05 | |
| *** yamamoto has joined #openstack-infra | 00:06 | |
| rm_work | so can I just prepend "DEVSTACK_GATE_" to an export var and have it show up? | 00:06 |
| clarkb | rm_work: no, those only work if devstack-gate has explicit support for that var | 00:06 |
| clarkb | rm_work: there is a passthrough var though thatI'm trying to lookup the name for | 00:07 |
| rm_work | ok | 00:08 |
| clarkb | rm_work: DEVSTACK_LOCALCONF looks like | 00:08 |
| rm_work | kk thanks!!! | 00:08 |
| *** warlord77[m] has quit IRC | 00:08 | |
| *** masayukig has quit IRC | 00:09 | |
| *** aspiers[m] has quit IRC | 00:09 | |
| *** dmueller[m] has quit IRC | 00:09 | |
| *** yamamoto has quit IRC | 00:11 | |
| *** jroll has quit IRC | 00:16 | |
| *** jroll has joined #openstack-infra | 00:17 | |
| *** slaweq has joined #openstack-infra | 00:17 | |
| *** dingyichen has joined #openstack-infra | 00:18 | |
| *** zhangfei has joined #openstack-infra | 00:19 | |
| ianw | now i think about it, the xenial pause means we probably haven't tried to build it using pip10, so there might be things hiding. i'll test that out now | 00:21 |
| *** kbaikov[m] has quit IRC | 00:21 | |
| *** ilpianista_ has quit IRC | 00:21 | |
| *** slaweq has quit IRC | 00:21 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove installed packages before pip install https://review.openstack.org/561479 | 00:22 |
| *** jamesmcarthur has joined #openstack-infra | 00:25 | |
| *** HeOS has quit IRC | 00:27 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Better zuul debug info debugging info https://review.openstack.org/561758 | 00:28 |
| fungi | ianw: yes, odds are we don't know whether pip 10 will actually prevent us from building xenial images, much less which jobs will subsequently break if we manage to build them | 00:31 |
| clarkb | I checked the filestamp on the emergency hosts file and it was last updated on friday | 00:32 |
| clarkb | Looking at irc logs on friday I think that may have been for a different host that pabelanger asked dmsimard about | 00:32 |
| clarkb | no mention of mirror-update at least | 00:32 |
| clarkb | and with that I am actually afk now | 00:32 |
| *** aeng_ has quit IRC | 00:32 | |
| ianw | clarkb: ttyl, i won't volunteer to take over because i think i've got enough to fiddle with pip10 stuff atm :/ | 00:33 |
| *** caphrim007_ has joined #openstack-infra | 00:33 | |
| clarkb | ya should be fine ti wait for my morning | 00:34 |
| *** kiennt26 has joined #openstack-infra | 00:35 | |
| *** caphrim007 has quit IRC | 00:35 | |
| fungi | anybody happen to have tried launch-node.py with --environment set to a non-default value? | 00:36 |
| fungi | looks like it installs your environment in /opt/system-config/foo but then the script tries to find /opt/system-config/production/manifests/site.pp on the server and dies because it's not there | 00:37 |
| *** caphrim007_ has quit IRC | 00:37 | |
| fungi | fatal: [survey01.openstack.org]: FAILED! => {"changed": false, "failed": true, "msg": "Manifest file /opt/system-config/production/manifests/site.pp not found."} | 00:38 |
| fungi | and when i ssh into the server, indeed there is a /opt/system-config/fungi but no /opt/system-config/production (presumably because i passed --environment=fungi) | 00:39 |
| fungi | happening in TASK [puppet : run puppet] | 00:39 |
| ianw | fungi: don't believe i've ever used that. i feel like i might have run some things by hand with different environments on the host | 00:41 |
| rm_work | is it possible to see any logs besides the console, until the end of the zuul run? | 00:43 |
| fungi | ianw: well, it _used_ to work, but it's been a while since i tried it so no idea how long ago it regressed | 00:43 |
| fungi | rm_work: not currently as far as i know, but the zuul team have designed the finger protocol backing the log streamer so that it can map streams to additional log files in the future | 00:44 |
| rm_work | hmmm k | 00:45 |
| rm_work | actually maybe what I want is a specific file (the devstack local.conf) so possibly a security risk if they allow arbitrary file access :P | 00:45 |
| fungi | right, it would end up needing to be specific files mapped by the job definition i think, not just any old file on the node | 00:47 |
| *** wolverin_ has quit IRC | 00:48 | |
| *** wolverineav has joined #openstack-infra | 00:48 | |
| rm_work | yeah | 00:49 |
| rm_work | though maybe that one would be a candidate | 00:49 |
| *** aeng_ has joined #openstack-infra | 00:49 | |
| *** tjgresha has quit IRC | 00:50 | |
| fungi | sure, the devstack base job could have a set of interesting files it mapped to the streamer and include that one, for example | 00:50 |
| fungi | hard to say exactly what that would look like until someone starts on the implementation though | 00:50 |
| *** jbadiapa has quit IRC | 00:51 | |
| fungi | but it's definitely a known wishlist item and will likely eventually happen (how soon probably depends on how interested someone is in writing the code) | 00:52 |
| *** jbadiapa has joined #openstack-infra | 00:53 | |
| ianw | rm_work: you can always add your own "cat"; sometimes that and a well placed "exit 1" can be good in early iterations | 00:56 |
| *** aeng_ has quit IRC | 00:57 | |
| *** aeng_ has joined #openstack-infra | 00:58 | |
| mtreinish | infra-root: if you get a sec can you check the mqtt_statsd daemon on firehose01.o.o, I think it might be stuck | 00:58 |
| fungi | lookin' | 00:58 |
| fungi | Active: inactive (dead) since Sat 2018-04-14 02:14:21 UTC; 2 days ago | 00:59 |
| fungi | Process: 20757 ExecStart=/usr/local/bin/mqtt_statsd /etc/mqtt_statsd.yaml (code=killed, signal=TERM) | 01:00 |
| mtreinish | yeah, that's consistent with the graphs on: http://grafana.openstack.org/dashboard/db/mosquitto-status | 01:00 |
| *** andreas_s has joined #openstack-infra | 01:00 | |
| mtreinish | hmm, something sent it SIGTERM? | 01:00 |
| *** Goneri has joined #openstack-infra | 01:01 | |
| *** mriedem has quit IRC | 01:01 | |
| *** jbadiapa has quit IRC | 01:01 | |
| fungi | `sudo journalctl --unit mqtt_statsd.service` reports -- No entries -- | 01:01 |
| mtreinish | fungi: it might be worth checking the germqtt service and lpmqtt service on that host too, I haven't seen messages on either topic either (even though that's not definitive) | 01:02 |
| mtreinish | hmm, no logs | 01:02 |
| mtreinish | although I can't remember if the service code actually prints anything or not :p | 01:02 |
| *** jbadiapa has joined #openstack-infra | 01:02 | |
| fungi | dmesg has nothing since march 16, so not an oom | 01:02 |
| fungi | lpmqtt seems to be running but germqtt does not | 01:03 |
| mtreinish | fungi: is there any info in the sudo logs or full journalctl about if something ran systemctl stop mqtt_statsd | 01:03 |
| mtreinish | hmm, did something kill germqtt too? | 01:03 |
| fungi | looks like germqtt died 2018-04-14 02:13:21 UTC | 01:04 |
| mtreinish | roughly the same time | 01:05 |
| fungi | one minute prior to mqt-statsd | 01:05 |
| *** andreas_s has quit IRC | 01:05 | |
| fungi | er, to mqtt_statsd | 01:05 |
| fungi | systemd mentions the journal having been rotated. i'll see if i can figure out how to tell it to look in the older journal | 01:05 |
| *** eernst has quit IRC | 01:07 | |
| *** yamamoto has joined #openstack-infra | 01:07 | |
| clarkb | there likely isnt an older journal | 01:07 |
| clarkb | look in syslog | 01:07 |
| clarkb | on ubuntu syslog should have longer retention | 01:08 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Don't use pip internals https://review.openstack.org/561659 | 01:08 |
| fungi | clarkb: thanks, will dig | 01:08 |
| *** eernst has joined #openstack-infra | 01:09 | |
| clarkb | fungi: its a fixed size circular buffer thing iirc | 01:09 |
| *** eernst has joined #openstack-infra | 01:09 | |
| clarkb | but with rsyslog installed we should be writing there too with more typical syslog retention | 01:09 |
| *** cshastri has joined #openstack-infra | 01:10 | |
| *** eernst has quit IRC | 01:10 | |
| *** eernst has joined #openstack-infra | 01:10 | |
| fungi | yep, found it | 01:10 |
| *** hongbin_ has joined #openstack-infra | 01:12 | |
| *** yamamoto has quit IRC | 01:13 | |
| *** eernst has quit IRC | 01:14 | |
| rm_work | ianw: ahh yeah actually that is great advice :P | 01:15 |
| rm_work | I should do that | 01:15 |
| *** aeng_ has quit IRC | 01:16 | |
| mtreinish | fungi: any indication there as to why it was killed? | 01:16 |
| *** esberglu has quit IRC | 01:16 | |
| *** aeng has joined #openstack-infra | 01:17 | |
| mtreinish | we should also probably restart both manually since I guess puppet doesn't ensure they're running | 01:18 |
| fungi | mtreinish: it looks like _i_ logged in and was restarting services | 01:18 |
| * fungi checks irc logs from that time | 01:19 | |
| mtreinish | that was late on friday right? That's about when we were looking at the first attempt at re-enabling the tls websockets port | 01:20 |
| mtreinish | I don't remember anything about the mqtt consuming services from then though | 01:20 |
| fungi | yep, the timing lines up with the irc discussion | 01:21 |
| *** xinliang has quit IRC | 01:23 | |
| *** shu-mutow-AWAY is now known as shu-mutow | 01:24 | |
| *** stakeda has joined #openstack-infra | 01:27 | |
| fungi | bizarre, i don't recall stopping those but my command history and the sudo log both indicate i did | 01:28 |
| *** yamahata has quit IRC | 01:29 | |
| mtreinish | hmm, well whatever. Clearly nothing is depending on either otherwise someone or something would have complained about it not publishing since friday | 01:30 |
| mtreinish | I'm just glad I was randomly clicking links in my draft firehose blog post to check them and saw that the mosquitto graphs on grafana were flat | 01:31 |
| openstackgerrit | Ghanshyam Mann proposed openstack-dev/hacking master: Release notes for 1.1.0 https://review.openstack.org/561129 | 01:32 |
| anteaya | awww, he was one of my favourite people: https://www.thestar.com/entertainment/2018/04/16/night-court-star-harry-anderson-dies-at-age-65.html | 01:33 |
| fungi | yeah, he was amazing | 01:33 |
| mordred | OH NO | 01:34 |
| anteaya | :( | 01:34 |
| fungi | also john astin as his father on night court... it was good to see astin get some work at that point in his career | 01:34 |
| anteaya | I never saw the character of his father | 01:35 |
| *** wolverineav has quit IRC | 01:35 | |
| *** wolverineav has joined #openstack-infra | 01:36 | |
| *** jbadiapa has quit IRC | 01:36 | |
| *** aeng has quit IRC | 01:36 | |
| fungi | mtreinish: looks like i may have initially restarted mqtt_statsd instead of mosquitto because i pulled it from my command history and was in a friday night state of confusion. ultimately i killed the pid systemctl status was reporting for it and couldn't figure out why mosquitto wouldn't die ;) | 01:37 |
| fungi | mtreinish: as for germqtt, i think it may have crashed when i eventually actually restarted mosquitto | 01:37 |
| *** aeng has joined #openstack-infra | 01:37 | |
| fungi | there's a backtrace in syslog from it | 01:37 |
| mtreinish | fungi: haha, ok. Well that's what we get for working too late on a friday night | 01:38 |
| *** jbadiapa has joined #openstack-infra | 01:38 | |
| mtreinish | fungi: can you pastebin that? I would expect it to handle a missing broker without issue | 01:38 |
| mtreinish | so it's likely a bug in germqtt | 01:38 |
| anteaya | found one: https://www.youtube.com/watch?v=LWz-tBU6_L0 | 01:39 |
| *** Goneri has quit IRC | 01:41 | |
| *** janki has joined #openstack-infra | 01:42 | |
| *** aeng has quit IRC | 01:43 | |
| *** aeng has joined #openstack-infra | 01:44 | |
| *** wolverineav has quit IRC | 01:44 | |
| *** aeng has quit IRC | 01:45 | |
| *** aeng has joined #openstack-infra | 01:46 | |
| fungi | mtreinish: yeah, i'm working to interleave the sudo log entries for context | 01:48 |
| fungi | mtreinish: looking at these in context now it's not entirely clear what happened to germqtt... looks like it may have spontaneously freaked out http://paste.openstack.org/show/719334/ | 01:54 |
| mtreinish | woah, yeah something doesn't look quite right there | 01:56 |
| mtreinish | I wouldn't be expecting paho to launch all those threads like that on failure reconnect | 01:57 |
| fungi | well, mqtt_statsd went nuts reconnecting, and when it stopped germqtt complained once then fell over | 01:58 |
| *** jamesmcarthur has quit IRC | 01:58 | |
| mtreinish | oops, yeah was reading the wrong service there | 01:58 |
| *** jamesmcarthur has joined #openstack-infra | 01:59 | |
| fungi | oh | 01:59 |
| *** liujiong has joined #openstack-infra | 01:59 | |
| fungi | mqtt_statsd had 8 threads all complain about a connection refused | 01:59 |
| fungi | and then germqtt (one single thread) do the same 6 seconds later | 02:00 |
| mtreinish | that makes a bit more sense, the paho subscriber model is to spawn a listener in a separate thread so it's not blocking (even though in the mqtt_statsd code we have it blcok on messages anyway iirc) | 02:01 |
| mtreinish | so I could see on a reconnect it's spawning a new thread while the old one which failed is shutdown (hopefully it's not leaking) | 02:01 |
| *** myoung_ has joined #openstack-infra | 02:01 | |
| *** rnoriega_ has joined #openstack-infra | 02:01 | |
| *** weshay_ has joined #openstack-infra | 02:01 | |
| *** pabelanger_ has joined #openstack-infra | 02:02 | |
| *** rasca_ has joined #openstack-infra | 02:02 | |
| *** otherwiseguy_ has joined #openstack-infra | 02:02 | |
| mtreinish | but it should have waited longer, because iirc the default keepalive is 60 seconds, but it looks like it only waited 1 there | 02:02 |
| *** _ari__ has joined #openstack-infra | 02:02 | |
| *** rasca has quit IRC | 02:02 | |
| rm_work | hmmmm... so I did https://review.openstack.org/#/c/561049/8/playbooks/legacy/neutron-lbaasv2-dsvm-api-l7octavia/run.yaml@45 | 02:02 |
| rm_work | that doesn't seem to be right? | 02:02 |
| mtreinish | but on the germqtt side that's a bit more concerning, it doesn't look we retried at all on it, which definitely doesn't seem right | 02:02 |
| *** haleyb has quit IRC | 02:03 | |
| *** weshay has quit IRC | 02:03 | |
| *** _ari_ has quit IRC | 02:03 | |
| *** haleyb has joined #openstack-infra | 02:03 | |
| *** pabelanger has quit IRC | 02:03 | |
| *** rasca has joined #openstack-infra | 02:03 | |
| *** pabelang- has joined #openstack-infra | 02:03 | |
| mtreinish | I'll have to investigate tomorrow when I get a chance, I might not be setting the right flag in paho, or use a different call path. Or we might need to add the retry logic in manually | 02:03 |
| *** wolverineav has joined #openstack-infra | 02:03 | |
| *** weshay has joined #openstack-infra | 02:03 | |
| *** otherwiseguy has quit IRC | 02:04 | |
| *** _ari_ has joined #openstack-infra | 02:04 | |
| *** rnoriega- has joined #openstack-infra | 02:04 | |
| *** otherwiseguy has joined #openstack-infra | 02:04 | |
| fungi | mtreinish: yeah, so looks like something happened to cause mqtt_statsd to get connection refusal on 8 threads at startup (all at once) and for seemingly unrelated reasons germqtt gets a connection refusal a few seconds later... did mqtt_statsd overwhelm mosquitto causing it to momentarily block germqtt, and then we failed to catch/retry in germqtt leading it to die? | 02:04 |
| *** jamesmcarthur has quit IRC | 02:05 | |
| *** pabelang- is now known as pabelanger | 02:05 | |
| *** jamesmcarthur has joined #openstack-infra | 02:05 | |
| *** myoung has quit IRC | 02:05 | |
| *** rnoriega has quit IRC | 02:06 | |
| pabelanger | clarkb: ianw fungi: yah, I've been trying to repair debian mirror, for jessie / stretch security. Hoping to finish that off in the morning | 02:06 |
| *** weshay_ has quit IRC | 02:06 | |
| *** _ari__ has quit IRC | 02:06 | |
| *** otherwiseguy_ has quit IRC | 02:06 | |
| *** rnoriega_ has quit IRC | 02:06 | |
| *** pabelanger_ has quit IRC | 02:06 | |
| *** rasca_ has quit IRC | 02:07 | |
| ianw | pabelanger: afs errors? | 02:07 |
| fungi | ianw: reprepro misconfig | 02:07 |
| pabelanger | ianw: no, mixed packages on jessie / stretch | 02:07 |
| *** myoung_ has quit IRC | 02:07 | |
| clarkb | pabelanger: do we need to have puppet disabled for that? or can we just hold the debian repo lock? | 02:07 |
| *** wolverineav has quit IRC | 02:08 | |
| *** myoung has joined #openstack-infra | 02:08 | |
| pabelanger | clarkb: I've manually edited some files in the reprepro for debian, but we can remove it from emergency if needed | 02:08 |
| mordred | fungi, clarkb: in https://review.openstack.org/#/c/549303/1/pbr/siblings.py I actually completely refactored the code to use pkg_resources (which is a real api) | 02:08 |
| pabelanger | I know you're working on bandersnatch | 02:09 |
| *** yamamoto has joined #openstack-infra | 02:09 | |
| fungi | mordred: a related curiosity, why pkg_resources instead of packaging? they both seem to have a similar requirement parser method though the api in packaging seems mildly cleaner | 02:10 |
| mordred | fungi: I think clarkb's patch is great for now until we extract that code into a utility - I've been poking at some other options for that rather than just as a patch to pbr itself - but haven't gotten all the way to making a solid proposal | 02:10 |
| mtreinish | fungi: yeah that's what it looks like. Although I'm not sure how 8 connections like that in a second could overwhelm mosquitto it's handling a lot more connections like that in normal operation | 02:10 |
| fungi | is it that pkg_resources is more likely to already be present? | 02:10 |
| mordred | fungi: yah, it's that | 02:10 |
| mordred | fungi: I *know* pkg_resources will be there | 02:10 |
| mordred | packaging I can never remember if it's real or not yet | 02:10 |
| fungi | makes sense, especially in the case of pbr | 02:11 |
| mordred | but it's definitely not a decision around one being better than the other or anything | 02:11 |
| fungi | packaging is where the pip maintainers have committed to expose other currently internal pip apis in the future as they're requested | 02:12 |
| mordred | gotcha | 02:12 |
| fungi | i expect pkg_resources is slower moving or frozen-ish which makes packaging more attractive for that | 02:12 |
| *** yamamoto has quit IRC | 02:15 | |
| *** camunoz has quit IRC | 02:16 | |
| *** kiennt26 has quit IRC | 02:18 | |
| *** trown|outtypewww has quit IRC | 02:18 | |
| mordred | clarkb: btw - I owe you a sepc buthaven't had time to write it yet and will be in all-day-meeting again tomorrow - I haven't forgotten aoutit -and hopefully can get it done later in the week | 02:20 |
| *** dpawlik has joined #openstack-infra | 02:24 | |
| *** dpawlik has quit IRC | 02:29 | |
| *** caphrim007 has joined #openstack-infra | 02:32 | |
| *** gcb has joined #openstack-infra | 02:33 | |
| *** dave-mcc_ has quit IRC | 02:45 | |
| *** jbadiapa has quit IRC | 02:46 | |
| *** kiennt26 has joined #openstack-infra | 02:47 | |
| *** jbadiapa has joined #openstack-infra | 02:49 | |
| *** kiennt26 has quit IRC | 02:54 | |
| *** nicolasbock has quit IRC | 02:55 | |
| *** jchhatbar has joined #openstack-infra | 03:00 | |
| *** zhangfei has quit IRC | 03:01 | |
| *** kiennt26 has joined #openstack-infra | 03:02 | |
| *** janki has quit IRC | 03:03 | |
| *** jchhatbar is now known as janki | 03:07 | |
| *** kiennt26 has quit IRC | 03:08 | |
| *** psachin has joined #openstack-infra | 03:09 | |
| *** yamamoto has joined #openstack-infra | 03:11 | |
| *** kiennt26 has joined #openstack-infra | 03:11 | |
| *** kiennt26 has quit IRC | 03:13 | |
| *** d0ugal_ has joined #openstack-infra | 03:14 | |
| *** liujiong has quit IRC | 03:14 | |
| *** zhangfei has joined #openstack-infra | 03:14 | |
| *** liujiong has joined #openstack-infra | 03:15 | |
| *** kiennt26 has joined #openstack-infra | 03:16 | |
| *** d0ugal has quit IRC | 03:16 | |
| *** yamamoto has quit IRC | 03:16 | |
| *** caphrim007_ has joined #openstack-infra | 03:18 | |
| *** kiennt26 has quit IRC | 03:21 | |
| *** caphrim007 has quit IRC | 03:22 | |
| *** dsariel has quit IRC | 03:27 | |
| *** annp has joined #openstack-infra | 03:28 | |
| *** ramishra has joined #openstack-infra | 03:35 | |
| *** kiennt26 has joined #openstack-infra | 03:36 | |
| *** liujiong_lj has joined #openstack-infra | 03:38 | |
| *** liujiong has quit IRC | 03:40 | |
| *** jbadiapa has quit IRC | 03:41 | |
| *** lpetrut has joined #openstack-infra | 03:42 | |
| *** jbadiapa has joined #openstack-infra | 03:42 | |
| *** andreas_s has joined #openstack-infra | 03:45 | |
| *** lpetrut has quit IRC | 03:49 | |
| *** lpetrut has joined #openstack-infra | 03:50 | |
| *** andreas_s has quit IRC | 03:50 | |
| *** wolverineav has joined #openstack-infra | 03:55 | |
| *** dklyle has joined #openstack-infra | 03:57 | |
| *** timburke_ has joined #openstack-infra | 04:02 | |
| *** jbryce_ has joined #openstack-infra | 04:02 | |
| *** otherwiseguy has quit IRC | 04:03 | |
| *** ildikov has quit IRC | 04:03 | |
| *** jbryce has quit IRC | 04:03 | |
| *** mrhillsman has quit IRC | 04:03 | |
| *** timburke has quit IRC | 04:03 | |
| *** d0ugal_ has quit IRC | 04:03 | |
| *** jbryce_ is now known as jbryce | 04:03 | |
| *** d0ugal__ has joined #openstack-infra | 04:03 | |
| *** otherwiseguy_ has joined #openstack-infra | 04:03 | |
| *** mrhillsman has joined #openstack-infra | 04:04 | |
| *** ggherdov- has quit IRC | 04:05 | |
| *** andymccr has quit IRC | 04:05 | |
| *** andymccr has joined #openstack-infra | 04:05 | |
| *** pbourke has quit IRC | 04:05 | |
| *** pbourke has joined #openstack-infra | 04:06 | |
| *** udesale has joined #openstack-infra | 04:11 | |
| *** lpetrut has quit IRC | 04:12 | |
| *** yamamoto has joined #openstack-infra | 04:13 | |
| *** haleyb_ has joined #openstack-infra | 04:13 | |
| *** asettle_ has joined #openstack-infra | 04:14 | |
| *** kiennt26 has quit IRC | 04:15 | |
| *** haleyb has quit IRC | 04:16 | |
| *** asettle has quit IRC | 04:16 | |
| *** zer0c00l has quit IRC | 04:16 | |
| *** bstinson has quit IRC | 04:16 | |
| *** bstinson has joined #openstack-infra | 04:17 | |
| *** jchhatbar has joined #openstack-infra | 04:18 | |
| *** yamamoto has quit IRC | 04:18 | |
| *** wolverineav has quit IRC | 04:18 | |
| *** janki has quit IRC | 04:20 | |
| *** annp has quit IRC | 04:21 | |
| *** hongbin_ has quit IRC | 04:21 | |
| *** lpetrut has joined #openstack-infra | 04:21 | |
| *** annp has joined #openstack-infra | 04:21 | |
| *** zer0c00l has joined #openstack-infra | 04:22 | |
| *** claudiub|2 has joined #openstack-infra | 04:30 | |
| *** jamesmcarthur has quit IRC | 04:37 | |
| *** markvoelker has quit IRC | 04:37 | |
| *** lpetrut has quit IRC | 04:40 | |
| *** zhangfei has quit IRC | 04:51 | |
| *** salv-orlando has joined #openstack-infra | 04:55 | |
| *** pgadiya has joined #openstack-infra | 04:58 | |
| *** pgadiya has quit IRC | 04:58 | |
| *** zhangfei has joined #openstack-infra | 05:07 | |
| *** lpetrut has joined #openstack-infra | 05:08 | |
| *** e0ne has joined #openstack-infra | 05:08 | |
| *** e0ne has quit IRC | 05:08 | |
| *** jamesmcarthur has joined #openstack-infra | 05:14 | |
| *** gagehugo has quit IRC | 05:16 | |
| *** dhajare has joined #openstack-infra | 05:18 | |
| *** kbaikov[m] has joined #openstack-infra | 05:18 | |
| *** jamesmcarthur has quit IRC | 05:19 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Do not hardcode zookeeper host to localhost https://review.openstack.org/561729 | 05:23 |
| *** links has joined #openstack-infra | 05:23 | |
| *** vivsoni has quit IRC | 05:27 | |
| *** masayukig has joined #openstack-infra | 05:28 | |
| *** mrhillsman has quit IRC | 05:29 | |
| *** mrhillsman has joined #openstack-infra | 05:30 | |
| *** lpetrut has quit IRC | 05:33 | |
| *** ilpianista_ has joined #openstack-infra | 05:34 | |
| *** dmueller[m] has joined #openstack-infra | 05:35 | |
| *** markvoelker has joined #openstack-infra | 05:38 | |
| *** aeng has quit IRC | 05:38 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove installed packages before pip install https://review.openstack.org/561479 | 05:44 |
| *** aspiers[m] has joined #openstack-infra | 05:44 | |
| *** warlord77[m] has joined #openstack-infra | 05:45 | |
| *** gagehugo has joined #openstack-infra | 05:46 | |
| *** vivsoni has joined #openstack-infra | 05:48 | |
| *** gcb has quit IRC | 05:50 | |
| *** dpawlik has joined #openstack-infra | 05:57 | |
| *** dpawlik has quit IRC | 05:58 | |
| *** shu-mutow has quit IRC | 05:58 | |
| *** dpawlik has joined #openstack-infra | 05:59 | |
| *** vivsoni has quit IRC | 06:00 | |
| *** vivsoni has joined #openstack-infra | 06:01 | |
| *** iranzo has joined #openstack-infra | 06:02 | |
| *** pgadiya has joined #openstack-infra | 06:05 | |
| *** pgadiya has quit IRC | 06:05 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove installed packages before pip install https://review.openstack.org/561479 | 06:10 |
| *** zhangfei has quit IRC | 06:10 | |
| *** hemna_ has quit IRC | 06:11 | |
| *** jchhatbar is now known as janki | 06:12 | |
| *** yamamoto has joined #openstack-infra | 06:16 | |
| *** pcaruana has joined #openstack-infra | 06:16 | |
| *** andreas_s has joined #openstack-infra | 06:17 | |
| *** lpetrut has joined #openstack-infra | 06:19 | |
| *** tobberydberg has quit IRC | 06:21 | |
| *** yamamoto has quit IRC | 06:22 | |
| *** priteau has joined #openstack-infra | 06:23 | |
| *** zhangfei has joined #openstack-infra | 06:24 | |
| *** aojea has joined #openstack-infra | 06:26 | |
| *** lathiat has quit IRC | 06:28 | |
| *** lathiat has joined #openstack-infra | 06:29 | |
| *** pblaho has joined #openstack-infra | 06:31 | |
| *** e0ne has joined #openstack-infra | 06:32 | |
| *** alexchadin has joined #openstack-infra | 06:36 | |
| *** ianychoi has joined #openstack-infra | 06:38 | |
| *** kjackal has quit IRC | 06:38 | |
| *** kjackal has joined #openstack-infra | 06:39 | |
| *** florianf has joined #openstack-infra | 06:48 | |
| *** jchhatbar has joined #openstack-infra | 06:50 | |
| *** slaweq has joined #openstack-infra | 06:50 | |
| *** dhajare has quit IRC | 06:51 | |
| *** janki has quit IRC | 06:53 | |
| *** hashar has joined #openstack-infra | 06:53 | |
| *** pgadiya has joined #openstack-infra | 06:56 | |
| *** pgadiya has quit IRC | 06:56 | |
| *** rcernin has quit IRC | 06:57 | |
| *** jbadiapa has quit IRC | 07:01 | |
| *** jbadiapa has joined #openstack-infra | 07:01 | |
| *** vtapia has joined #openstack-infra | 07:03 | |
| *** jbadiapa has quit IRC | 07:05 | |
| *** jbadiapa has joined #openstack-infra | 07:06 | |
| *** dpawlik has quit IRC | 07:07 | |
| *** numans has quit IRC | 07:08 | |
| *** numans has joined #openstack-infra | 07:08 | |
| *** ramishra_ has joined #openstack-infra | 07:10 | |
| *** yamamoto has joined #openstack-infra | 07:10 | |
| *** ramishra has quit IRC | 07:12 | |
| *** yamamoto has quit IRC | 07:14 | |
| *** numans has quit IRC | 07:14 | |
| *** yamamoto has joined #openstack-infra | 07:14 | |
| openstackgerrit | Lenny Verkhovsky proposed openstack-infra/irc-meetings master: Changed ThirdParty meeting to monthly https://review.openstack.org/561804 | 07:14 |
| *** tesseract has joined #openstack-infra | 07:14 | |
| *** numans has joined #openstack-infra | 07:14 | |
| *** ssbarnea_ has joined #openstack-infra | 07:16 | |
| *** garyk has joined #openstack-infra | 07:18 | |
| *** ralonsoh has joined #openstack-infra | 07:18 | |
| *** aviau has quit IRC | 07:19 | |
| *** ralonsoh has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** numans has quit IRC | 07:21 | |
| *** numans has joined #openstack-infra | 07:21 | |
| *** gongysh has joined #openstack-infra | 07:22 | |
| *** tobberydberg has joined #openstack-infra | 07:23 | |
| *** salv-orlando has quit IRC | 07:28 | |
| *** salv-orlando has joined #openstack-infra | 07:28 | |
| *** dhajare has joined #openstack-infra | 07:28 | |
| *** lpetrut_ has joined #openstack-infra | 07:31 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: networking-bgpvpn: remove networking-odl from required-projects https://review.openstack.org/561811 | 07:31 |
| *** salv-orlando has quit IRC | 07:32 | |
| *** lpetrut has quit IRC | 07:34 | |
| *** yamamoto_ has joined #openstack-infra | 07:39 | |
| *** yamamoto has quit IRC | 07:42 | |
| *** alexchadin has quit IRC | 07:42 | |
| *** alexchadin has joined #openstack-infra | 07:43 | |
| *** kjackal has quit IRC | 07:45 | |
| *** salv-orlando has joined #openstack-infra | 07:45 | |
| *** jogo has quit IRC | 07:48 | |
| *** jpena|off is now known as jpena | 07:52 | |
| ianw | frickler: yeah, have to iterate a bit on those other ones | 07:53 |
| *** pblaho has quit IRC | 07:53 | |
| *** shardy has joined #openstack-infra | 07:54 | |
| *** gagehugo has quit IRC | 07:54 | |
| *** dsariel has joined #openstack-infra | 07:54 | |
| *** pgadiya has joined #openstack-infra | 07:56 | |
| *** pgadiya has quit IRC | 07:57 | |
| *** electrofelix has joined #openstack-infra | 07:57 | |
| *** gongysh has quit IRC | 08:02 | |
| *** gongysh has joined #openstack-infra | 08:03 | |
| *** gongysh has quit IRC | 08:03 | |
| *** lucas-brb is now known as lucasagomes | 08:04 | |
| *** ianychoi has quit IRC | 08:04 | |
| *** jpich has joined #openstack-infra | 08:04 | |
| *** d0ugal__ has quit IRC | 08:05 | |
| *** d0ugal has joined #openstack-infra | 08:05 | |
| *** d0ugal has quit IRC | 08:05 | |
| *** d0ugal has joined #openstack-infra | 08:05 | |
| *** ramishra_ has quit IRC | 08:05 | |
| *** kjackal has joined #openstack-infra | 08:06 | |
| *** amoralej|off is now known as amoralej | 08:08 | |
| *** jtomasek has quit IRC | 08:08 | |
| *** rvba` has quit IRC | 08:08 | |
| *** ianychoi has joined #openstack-infra | 08:10 | |
| *** tmorin has joined #openstack-infra | 08:10 | |
| *** gagehugo has joined #openstack-infra | 08:11 | |
| tmorin | hi infraroot, openstack-zuul-jobs-linters failing with gcc missing header files for fb-re2 python package installation: http://logs.openstack.org/11/561811/1/check/openstack-zuul-jobs-linters/50f0270/job-output.txt.gz#_2018-04-17_08_08_22_504007 | 08:11 |
| tmorin | infraroot: ^^ is this on your radar ? | 08:11 |
| *** jogo has joined #openstack-infra | 08:12 | |
| *** jogo has joined #openstack-infra | 08:12 | |
| *** agopi has quit IRC | 08:12 | |
| *** kjackal_ has joined #openstack-infra | 08:14 | |
| *** tosky has joined #openstack-infra | 08:14 | |
| *** kjackal__ has joined #openstack-infra | 08:15 | |
| *** kjackal has quit IRC | 08:15 | |
| *** kjackal_ has quit IRC | 08:19 | |
| zigo | Guys, http://mirror.dfw.rax.openstack.org/debian/dists/stretch-security/main/binary-amd64/Packages still contains a security mirror for Wheezy. How to fix? | 08:19 |
| *** pcichy has joined #openstack-infra | 08:20 | |
| *** ramishra has joined #openstack-infra | 08:22 | |
| *** HeOS has joined #openstack-infra | 08:23 | |
| *** rnoriega- is now known as rnoriega | 08:23 | |
| *** zoli is now known as zoli|wfh | 08:23 | |
| *** zoli|wfh is now known as zoli | 08:24 | |
| frickler | tmorin: I hadn't seen that one yet, thanks for noticing, I'll take a deeper look soon. btw, the magic word most folks are highlighting on is "infra-root" ;) | 08:26 |
| *** pblaho has joined #openstack-infra | 08:26 | |
| tmorin | frickler: ah... thanks for telling me :) | 08:27 |
| tmorin | frickler: the info would be worth addiing to the channel topic, perhaps | 08:28 |
| frickler | zigo: seems pabelanger tried to fix this recently, but I can't see offhand what it wrong with this https://github.com/openstack-infra/system-config/blob/eb0dde25597cd59f04e9126268e57098bd342166/modules/openstack_project/files/reprepro/debuntu-updates#L41-L47 | 08:32 |
| *** derekh has joined #openstack-infra | 08:32 | |
| *** udesale_ has joined #openstack-infra | 08:32 | |
| *** ramishra_ has joined #openstack-infra | 08:32 | |
| frickler | tmorin: good point, will add that as an item to our meeting tonight | 08:32 |
| *** ramishra1 has joined #openstack-infra | 08:34 | |
| *** TobbeCN has joined #openstack-infra | 08:34 | |
| *** udesale has quit IRC | 08:34 | |
| *** ramishra has quit IRC | 08:36 | |
| *** yamamoto_ has quit IRC | 08:36 | |
| *** yamamoto has joined #openstack-infra | 08:37 | |
| *** efoley has joined #openstack-infra | 08:39 | |
| *** dingyichen has quit IRC | 08:39 | |
| *** ramishra_ has quit IRC | 08:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 08:44 |
| *** namnh has joined #openstack-infra | 08:45 | |
| *** gfidente has joined #openstack-infra | 08:46 | |
| *** gfidente has joined #openstack-infra | 08:46 | |
| *** pcichy has quit IRC | 08:49 | |
| *** quiquell|ruck has joined #openstack-infra | 08:50 | |
| quiquell|ruck | Hi | 08:50 |
| quiquell|ruck | Do you know if this is fixed ? | 08:51 |
| quiquell|ruck | lookup cloudflare.docker.com on 127.0.0.1:53: no such host" ? | 08:51 |
| *** armaan has joined #openstack-infra | 08:51 | |
| *** ralonsoh has joined #openstack-infra | 08:54 | |
| *** TobbeCN has quit IRC | 08:56 | |
| *** TobbeCN has joined #openstack-infra | 08:56 | |
| *** jtomasek has joined #openstack-infra | 08:57 | |
| *** edmondsw has joined #openstack-infra | 08:58 | |
| *** ramishra1 is now known as ramishra | 09:00 | |
| *** xinliang has joined #openstack-infra | 09:01 | |
| *** xinliang has quit IRC | 09:01 | |
| *** xinliang has joined #openstack-infra | 09:01 | |
| *** edmondsw has quit IRC | 09:02 | |
| *** dpawlik has joined #openstack-infra | 09:08 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add emit-job-report role https://review.openstack.org/548425 | 09:13 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-lint job https://review.openstack.org/532083 | 09:17 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-review job https://review.openstack.org/535223 | 09:18 |
| openstackgerrit | Zhao Chao proposed openstack-infra/project-config master: Stop using -python35-jobs-horizon for trove-dashboard https://review.openstack.org/561844 | 09:20 |
| xinliang | ianw: new buiding arm64 nodes has access problem. | 09:24 |
| ianw | xinliang: ubuntu or debian? | 09:25 |
| xinliang | ianw: Both, and when I trigger a job it keep building nodes: we change exteran net link recently. but | 09:25 |
| xinliang | see: http://grafana.openstack.org/dashboard/db/nodepool-linaro | 09:25 |
| *** gongysh has joined #openstack-infra | 09:26 | |
| xinliang | but, I have tested with cirros image on os-jobs project. cirros instance can access by floating ip | 09:26 |
| *** tmorin has quit IRC | 09:26 | |
| ianw | hmm, yeah i changed the dns over | 09:26 |
| *** tmorin has joined #openstack-infra | 09:27 | |
| xinliang | yes, dns seems work. no idea what's the problem | 09:27 |
| xinliang | ianw: the resources is running out due to keep building nodes | 09:27 |
| xinliang | ianw: one thing I found that is the floating ip of ubuntu and debian can't ping. Even I ping from the cirros instance it is failed | 09:29 |
| *** kjackal__ has quit IRC | 09:30 | |
| *** kjackal has joined #openstack-infra | 09:30 | |
| xinliang | ianw: anyother configuration need to ajust? when building the image | 09:31 |
| ianw | hmm, yeah, about 7 nodes building | 09:31 |
| *** udesale__ has joined #openstack-infra | 09:32 | |
| ianw | shade is assigning the ip's ... i wonder if it's cached or something | 09:32 |
| xinliang | as cirros image is ok. maybe the problem is the image itself. It seems ubuntu or debian node's eth0 is not working. | 09:34 |
| *** udesale_ has quit IRC | 09:35 | |
| ianw | urgh, yes this could be the problem we've seen on rax? | 09:36 |
| ianw | [[0;32m OK [0m] Stopped Glean for interface eth0. | 09:39 |
| ianw | it looks like glean is working? | 09:39 |
| ianw | xinliang: i'm deleting ubuntu-xenial-arm64-0000000546 so hosts should revert to the older ubuntu-xenial-arm64-0000000545 image | 09:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Pause arm64 builds https://review.openstack.org/561854 | 09:43 |
| ianw | xinliang: and let's pause new builds while we investigate :/ ^ | 09:44 |
| ianw | ok, couple of new hosts coming up with the older ubuntu-xenial-arm64-1523774825 image | 09:49 |
| ianw | arrrgghhhH!!!! project-config gate is broken | 09:50 |
| ianw | #status log nb03.o.o placed into emergency file and manually applied pause of builds, while project-config gating is broken | 09:52 |
| openstackstatus | ianw: finished logging | 09:53 |
| *** aviau has quit IRC | 09:53 | |
| *** aviau has joined #openstack-infra | 09:53 | |
| zigo | frickler: What we chatted about on IRC was that reprepro was supposed to delete all cruft, but in fact, it doesn't, and it would need to be manually cleaned. | 10:00 |
| zigo | frickler: Are you infra-root, could you do it? | 10:00 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add libre2 for project-config linting https://review.openstack.org/561861 | 10:00 |
| ianw | frickler / tmorin : ^ re the project-config failure | 10:01 |
| AJaeger | ianw: I just finished the same change ;) Let me review yours... | 10:01 |
| AJaeger | ianw: LGTM - thanks | 10:02 |
| *** ramishra_ has joined #openstack-infra | 10:02 | |
| ianw | AJaeger: i guess we should install zuul bindep.txt? but i think that suffices for now | 10:02 |
| AJaeger | ianw: let's try ;) | 10:03 |
| ianw | xinliang: well, that's disappointing, the old images doesn't appear to get a good network either | 10:03 |
| ianw | xinliang: i'm afraid i'm out of time to debug this further at this point, i will email the admins and explain | 10:05 |
| *** ramishra has quit IRC | 10:07 | |
| *** namnh has quit IRC | 10:10 | |
| *** kjackal has quit IRC | 10:13 | |
| AJaeger | ianw: https://review.openstack.org/561861 worked - great. Now waiting for second infra-root to +2A... | 10:13 |
| *** cshastri has quit IRC | 10:24 | |
| *** liujiong_lj has quit IRC | 10:29 | |
| *** eharney has joined #openstack-infra | 10:30 | |
| *** eharney has quit IRC | 10:30 | |
| *** eharney has joined #openstack-infra | 10:30 | |
| *** armaan has quit IRC | 10:31 | |
| *** armaan has joined #openstack-infra | 10:31 | |
| *** boden has joined #openstack-infra | 10:33 | |
| *** kjackal has joined #openstack-infra | 10:33 | |
| *** dpawlik has quit IRC | 10:34 | |
| *** nicolasbock has joined #openstack-infra | 10:37 | |
| *** iranzo has quit IRC | 10:37 | |
| gema | ianw: we will look into it | 10:37 |
| frickler | AJaeger: ianw: oh, that's the failure tmorin posted earlier, great. didn't have time to look into that yet, but +3 | 10:37 |
| gema | does anyone have advice on how to best deal with pip10 requirements install breakages? | 10:37 |
| gema | is this something openstack is dealing with in a coordinated manner | 10:38 |
| frickler | zigo: not much time currently, but it doesn't seem urgent either? so maybe can wait for pabelanger to look at it later | 10:38 |
| gema | nm, found the thread! | 10:38 |
| frickler | gema: I created an etherpad for the actions made by devstack+QA, nothing completely global afaict | 10:39 |
| gema | frickler: can I have the link to it? | 10:39 |
| frickler | gema: https://etherpad.openstack.org/p/pip10-mitigation | 10:39 |
| gema | frickler: thanks! | 10:39 |
| *** stakeda has quit IRC | 10:41 | |
| *** dpawlik has joined #openstack-infra | 10:43 | |
| *** xinliang has quit IRC | 10:44 | |
| *** armaan has quit IRC | 10:46 | |
| *** armaan has joined #openstack-infra | 10:46 | |
| *** armaan has quit IRC | 10:46 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add libre2 for project-config linting https://review.openstack.org/561861 | 10:47 |
| *** armaan has joined #openstack-infra | 10:47 | |
| *** zoli is now known as zoli|wfh | 10:49 | |
| *** zoli|wfh is now known as zoli | 10:49 | |
| *** alexchadin has quit IRC | 10:52 | |
| *** dbecker has joined #openstack-infra | 10:53 | |
| *** eharney has quit IRC | 10:53 | |
| ianw | gema: yeah, it's kind of fix on a case-by-case basis i think :/ | 10:54 |
| gema | ianw: I was wondering what the consensus was, either to stay on pip 9 for a while and move to 10 in rocky or how it was being addressed | 10:55 |
| gema | ianw: imagine supported versions keep old pip and new ones use 10, but I guess I am oversimplifying | 10:56 |
| *** xinliang has joined #openstack-infra | 10:56 | |
| *** xinliang has quit IRC | 10:56 | |
| *** xinliang has joined #openstack-infra | 10:56 | |
| *** efoley has quit IRC | 10:58 | |
| ianw | gema: well, pip is really just a ci deployment issue, not really part of openstack releases | 10:58 |
| gema | ianw: true | 10:58 |
| *** jtomasek has quit IRC | 10:59 | |
| gema | ianw: waiting for jorge to come online and we'll look into your issue | 10:59 |
| *** alexchadin has joined #openstack-infra | 10:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: Pause arm64 builds https://review.openstack.org/561854 | 10:59 |
| *** dbecker has quit IRC | 11:04 | |
| *** cshastri has joined #openstack-infra | 11:06 | |
| mordred | AJaeger: I'm AFK again today - but is there anything I can help with this morning before my meetings start for the day? | 11:08 |
| *** lucasagomes is now known as lucas-hungry | 11:14 | |
| *** tosky has quit IRC | 11:15 | |
| *** tosky has joined #openstack-infra | 11:15 | |
| *** TobbeCN has quit IRC | 11:16 | |
| *** alexchadin has quit IRC | 11:17 | |
| *** TobbeCN has joined #openstack-infra | 11:17 | |
| *** alexchadin has joined #openstack-infra | 11:17 | |
| *** TobbeCN has quit IRC | 11:17 | |
| *** TobbeCN has joined #openstack-infra | 11:17 | |
| *** alexchadin has quit IRC | 11:17 | |
| *** alexchadin has joined #openstack-infra | 11:18 | |
| *** alexchadin has quit IRC | 11:18 | |
| *** alexchadin has joined #openstack-infra | 11:19 | |
| *** alexchadin has quit IRC | 11:19 | |
| *** alexchadin has joined #openstack-infra | 11:19 | |
| *** alexchadin has quit IRC | 11:20 | |
| *** alexchadin has joined #openstack-infra | 11:20 | |
| *** alexchadin has quit IRC | 11:20 | |
| *** efoley has joined #openstack-infra | 11:22 | |
| *** ramishra_ has quit IRC | 11:22 | |
| *** zhangfei has quit IRC | 11:23 | |
| *** dhajare has quit IRC | 11:24 | |
| *** ramishra has joined #openstack-infra | 11:24 | |
| *** jaosorior has quit IRC | 11:24 | |
| *** armaan has quit IRC | 11:25 | |
| *** armaan has joined #openstack-infra | 11:25 | |
| *** jaosorior has joined #openstack-infra | 11:27 | |
| *** dhajare has joined #openstack-infra | 11:27 | |
| *** cshastri has quit IRC | 11:30 | |
| *** udesale__ has quit IRC | 11:34 | |
| *** panda|rover|off is now known as panda | 11:35 | |
| *** panda is now known as panda|rover | 11:35 | |
| openstackgerrit | Merged openstack-infra/project-config master: Stop using -python35-jobs-horizon for trove-dashboard https://review.openstack.org/561844 | 11:39 |
| *** jtomasek has joined #openstack-infra | 11:39 | |
| *** TobbeCN has quit IRC | 11:39 | |
| *** TobbeCN has joined #openstack-infra | 11:40 | |
| *** quiquell|ruck is now known as quique|ruck|food | 11:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: networking-bgpvpn: remove networking-odl from required-projects https://review.openstack.org/561811 | 11:41 |
| *** neith has joined #openstack-infra | 11:42 | |
| *** dhajare is now known as dhajare|busy | 11:42 | |
| *** TobbeCN has quit IRC | 11:42 | |
| *** TobbeCN has joined #openstack-infra | 11:43 | |
| *** armaan has quit IRC | 11:44 | |
| *** armaan has joined #openstack-infra | 11:45 | |
| *** jpena is now known as jpena|lunch | 11:45 | |
| zigo | frickler: Its only blocking for adding (non-voting) Debian jobs to the puppet-openstack project, so indeed, it's not that urgent. | 11:46 |
| AJaeger | mordred: sorry, can't help with current status, I was busy myself ;( | 11:47 |
| AJaeger | mordred: https://etherpad.openstack.org/p/pip10-mitigation - start line 42 is something that I'd love you to look at | 11:49 |
| *** alexchadin has joined #openstack-infra | 11:50 | |
| *** jogo has quit IRC | 11:51 | |
| *** amoralej is now known as amoralej|lunch | 11:52 | |
| *** alexchadin has quit IRC | 11:55 | |
| *** ramishra has quit IRC | 11:55 | |
| *** ramishra has joined #openstack-infra | 11:56 | |
| *** aojea has quit IRC | 11:57 | |
| *** alexchadin has joined #openstack-infra | 11:59 | |
| *** trown has joined #openstack-infra | 12:00 | |
| *** panda|rover is now known as panda|rover|lnc | 12:02 | |
| *** slaweq_ has joined #openstack-infra | 12:03 | |
| *** slaweq has quit IRC | 12:06 | |
| *** ldnunes has joined #openstack-infra | 12:06 | |
| *** slaweq has joined #openstack-infra | 12:07 | |
| *** slaweq_ has quit IRC | 12:08 | |
| *** alexchadin has quit IRC | 12:10 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add emit-job-report role https://review.openstack.org/548425 | 12:11 |
| *** quique|ruck|food is now known as quiquell|ruck | 12:13 | |
| *** armaan has quit IRC | 12:14 | |
| *** armaan has joined #openstack-infra | 12:14 | |
| *** jogo has joined #openstack-infra | 12:16 | |
| *** dave-mccowan has joined #openstack-infra | 12:16 | |
| *** armaan has quit IRC | 12:19 | |
| *** lucas-hungry is now known as lucasagomes | 12:19 | |
| *** rfolco|off is now known as rfolco | 12:20 | |
| *** gongysh has quit IRC | 12:20 | |
| *** alexchadin has joined #openstack-infra | 12:21 | |
| *** markvoelker has quit IRC | 12:25 | |
| *** yamamoto has quit IRC | 12:25 | |
| *** markvoelker has joined #openstack-infra | 12:25 | |
| *** yamamoto has joined #openstack-infra | 12:25 | |
| *** yamamoto has quit IRC | 12:26 | |
| *** amoralej|lunch is now known as amoralej | 12:26 | |
| *** yamamoto has joined #openstack-infra | 12:26 | |
| *** armaan has joined #openstack-infra | 12:29 | |
| *** panda|rover|lnc is now known as panda|rover | 12:30 | |
| *** dbecker has joined #openstack-infra | 12:36 | |
| *** rlandy has joined #openstack-infra | 12:36 | |
| *** edmondsw has joined #openstack-infra | 12:37 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Change label for neutron-tempest-ovsfw to "voting" https://review.openstack.org/561930 | 12:44 |
| *** cshastri has joined #openstack-infra | 12:45 | |
| *** jpena|lunch is now known as jpena | 12:47 | |
| *** camunoz has joined #openstack-infra | 12:48 | |
| *** cshastri has quit IRC | 12:50 | |
| *** kgiusti has joined #openstack-infra | 12:51 | |
| *** neiloy has joined #openstack-infra | 12:53 | |
| *** salv-orlando has quit IRC | 12:53 | |
| *** pcichy has joined #openstack-infra | 12:53 | |
| *** salv-orlando has joined #openstack-infra | 12:54 | |
| *** tpsilva has joined #openstack-infra | 12:55 | |
| *** efoley has quit IRC | 12:57 | |
| *** salv-orlando has quit IRC | 12:58 | |
| *** mriedem has joined #openstack-infra | 12:59 | |
| *** efoley has joined #openstack-infra | 13:01 | |
| *** armaan has quit IRC | 13:04 | |
| *** armaan has joined #openstack-infra | 13:04 | |
| *** ianychoi_ has joined #openstack-infra | 13:07 | |
| *** jcoufal has joined #openstack-infra | 13:10 | |
| *** ianychoi has quit IRC | 13:10 | |
| *** jcoufal_ has joined #openstack-infra | 13:14 | |
| *** alexchadin has quit IRC | 13:14 | |
| *** Goneri has joined #openstack-infra | 13:15 | |
| fungi | zigo: frickler: pabelanger is still working on cleanup of the debian/stretch-security mirror, and said he was planning to continue on that today | 13:17 |
| *** jcoufal has quit IRC | 13:17 | |
| zigo | fungi: Great, thanks ! | 13:17 |
| fungi | pabelanger: assuming there are not actually any voting jobs relying on that yet, it may be simplest to just delete all the content in that tree and rerun reprepro under screen with the flock held so cron doesn't fight with you | 13:18 |
| *** eernst has joined #openstack-infra | 13:18 | |
| fungi | actually, even if there are jobs relying on it, they're using the read-only volume so wouldn't see any change until you vos release anyway | 13:19 |
| *** andreykurilin has joined #openstack-infra | 13:20 | |
| fungi | gema: mostly our policy in the past has been that we attempt to provide (and backport) whatever is needed for use with the latest release of pip as soon as it's available | 13:21 |
| *** jcoufal has joined #openstack-infra | 13:21 | |
| fungi | refusing to support newer pip only serves to kicks the can down the road and makes it harder to find the time to transition so we run the risk of never getting around to it | 13:22 |
| fungi | however, in our ci system, we provide projects with the flexibility to adjust their test environment however they want, so if some of them want to downgrade pip before running certain jobs then they have the ability to do that | 13:24 |
| *** armaan has quit IRC | 13:24 | |
| *** jcoufal_ has quit IRC | 13:24 | |
| *** armaan has joined #openstack-infra | 13:24 | |
| *** armaan has quit IRC | 13:25 | |
| *** armaan has joined #openstack-infra | 13:25 | |
| fungi | i personally think we need to stop providing a global install of pip on our ci images entirely, and leave it up to jobs to install pip themselves (from a cache of course, for performance reasons) but that will probably take a little longer to rectify | 13:26 |
| *** armaan has quit IRC | 13:27 | |
| *** armaan has joined #openstack-infra | 13:28 | |
| mtreinish | infra-root: hmm, it looks like mosquitto crashed. I'm not able to establish a connection anymore | 13:29 |
| fungi | checking | 13:29 |
| fungi | Active: active (exited) since Mon 2018-04-16 18:27:28 UTC; 19h ago | 13:30 |
| fungi | Apr 17 06:25:01 firehose01 kernel: [8333467.773948] traps: mosquitto[14869] general protection ip:7f27420f7072 sp:7ffdfea92490 error:0 in libwebsockets.so.7[7f27420e2000+1f000] | 13:31 |
| mtreinish | sigh, I guess we'll need to turn the tls port back off | 13:31 |
| fungi | looks like that's right when /etc/cron.daily ran | 13:31 |
| fungi | could be triggered by log rotation (maybe signal handling?) | 13:31 |
| mtreinish | hmm, that's an interesting thought | 13:32 |
| fungi | i could start it under screen from the command line and see if it gives us any detail on stdout/stderr the next time it dies | 13:32 |
| mtreinish | is there anything in the mosquitto log file? I would expect it to print anything in there (although given our past experience with debugging failures there won't be anything) | 13:33 |
| fungi | 1523946301: Reloading config. | 13:33 |
| fungi | that's the end of the log | 13:34 |
| fungi | fungi@firehose01:~$ date -d @1523946301 | 13:34 |
| fungi | Tue Apr 17 06:25:01 UTC 2018 | 13:34 |
| mtreinish | well that lines up with when it triggered the general protection fault | 13:35 |
| mtreinish | fungi: https://github.com/eclipse/mosquitto/issues/253 | 13:35 |
| *** tmorin has left #openstack-infra | 13:35 | |
| *** efoley has quit IRC | 13:36 | |
| *** armaan has quit IRC | 13:36 | |
| *** armaan has joined #openstack-infra | 13:36 | |
| mtreinish | oh, hah I linked to that in my bug on this before: https://github.com/eclipse/mosquitto/issues/278 | 13:36 |
| *** esberglu has joined #openstack-infra | 13:36 | |
| *** haleyb_ is now known as haleyb | 13:37 | |
| *** aojea has joined #openstack-infra | 13:38 | |
| *** caphrim007_ has quit IRC | 13:38 | |
| *** links has quit IRC | 13:38 | |
| *** yamamoto has quit IRC | 13:39 | |
| *** yamamoto has joined #openstack-infra | 13:39 | |
| pabelanger | fungi: zigo frickler: yah, working on it this morning. hope to be finished | 13:39 |
| *** psachin has quit IRC | 13:40 | |
| fungi | #status log running `mosquitto -v -c /etc/mosquitto/mosquitto.conf` under a root screen session for crash debugging purposes | 13:40 |
| openstackstatus | fungi: finished logging | 13:40 |
| mtreinish | fungi: we're probably gonna have to restart germqtt, mqtt_statsd, and lpmqtt (for the same reason as last night) | 13:41 |
| fungi | mtreinish: interestingly, mqtt_statsd and lpmqtt are still running but germqtt has died | 13:42 |
| fungi | which may actually line up with our observations from the earlier logs | 13:42 |
| *** felipemonteiro has joined #openstack-infra | 13:42 | |
| fungi | i've started germqtt back up again now | 13:43 |
| mtreinish | ok cool, thanks | 13:43 |
| fungi | yup | 13:43 |
| fungi | Apr 17 06:25:26 firehose01 germqtt[18246]: socket.error: [Errno 111] Connection refused | 13:44 |
| *** yamamoto has quit IRC | 13:44 | |
| fungi | it logged that just before systemd recorded the main process exiting | 13:44 |
| mtreinish | ok yeah, I'll look into fixing that germqtt bug at some point today. But, I've got to drive to boston in a little bit so it might not be for a while | 13:45 |
| *** jamesmcarthur has joined #openstack-infra | 13:45 | |
| mtreinish | hah, I bet lpmqtt didn't crash just because it was sitting there idle waiting for email and not pushing anything | 13:46 |
| mtreinish | because iirc the publishing code between them is more or less the same | 13:46 |
| *** felipemonteiro_ has joined #openstack-infra | 13:46 | |
| *** asettle_ is now known as asettle | 13:47 | |
| fungi | oh, entirely probable, agreed | 13:50 |
| *** felipemonteiro has quit IRC | 13:50 | |
| fungi | lpmqtt has been running since march 8 | 13:51 |
| *** udesale has joined #openstack-infra | 13:51 | |
| *** andreas_s has quit IRC | 13:52 | |
| *** garyk has quit IRC | 13:54 | |
| pabelanger | fungi: ianw: fungi: so, I'm a little confused, if I run reprepro update over each distribution, I have no issues. But as soon as we just do update, we get duplicate packages, and deleting them doesn't seem to fix it. | 13:54 |
| pabelanger | fungi: ianw: zigo: I am wondering if our configuration is either wrong is some place, or use just need to do update for each distro update in a for loop | 13:55 |
| *** M4g1c5t0rM has joined #openstack-infra | 13:55 | |
| *** raildo has joined #openstack-infra | 13:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Merge zuul-website-media when publishing site https://review.openstack.org/561749 | 13:57 |
| *** eharney has joined #openstack-infra | 13:57 | |
| *** lpetrut_ has quit IRC | 13:58 | |
| raildo | hey folks, maybe I've made some dumb stuff and looks like I'm with multiple gerrit accounts on my name, so when someone try to add me to a review, it got a message like "<my_account> does not identify a registered user or group" | 14:00 |
| fungi | raildo: it can happen if you change which e-mail address you use to log into ubuntuone | 14:00 |
| raildo | instead of add a new email to an older account I believe that I've created a new one =/ | 14:00 |
| *** yamamoto has joined #openstack-infra | 14:01 | |
| fungi | raildo: since changing your authentication address creates a new openid on login.ubuntu.com and results in a new gerrit account | 14:01 |
| raildo | fungi, yeah, I just realize that after made this =/ Do you if it's possible to fix that somehow? | 14:01 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/germqtt master: Never give up never surrender https://review.openstack.org/561952 | 14:02 |
| mtreinish | fungi: ^^^ | 14:02 |
| fungi | raildo: technically not in any way officially supported by gerrit, but i can perform some direct surgery in mysql | 14:02 |
| mtreinish | although I haven't tested that, I think it should work | 14:02 |
| raildo | fungi, that would be great | 14:02 |
| fungi | mtreinish: nice commit subject ;) | 14:02 |
| fungi | mtreinish: i bet germqtt also takes no prisoners and gives no quarter | 14:02 |
| *** udesale has quit IRC | 14:02 | |
| *** hashar has quit IRC | 14:02 | |
| *** r-daneel has joined #openstack-infra | 14:04 | |
| mtreinish | fungi: heh, well it definitely doesn't at qos 0 :) | 14:04 |
| *** panda|rover is now known as panda|rover|mtg | 14:05 | |
| *** neiloy has quit IRC | 14:09 | |
| pabelanger | okay, running checkpool on debian pool, if that is clean, I have an update to our mirror script that should work | 14:09 |
| *** hongbin_ has joined #openstack-infra | 14:10 | |
| *** belliott has joined #openstack-infra | 14:11 | |
| *** fkautz has quit IRC | 14:11 | |
| *** johnsom has quit IRC | 14:11 | |
| *** patricku has quit IRC | 14:11 | |
| *** xgerman_ has quit IRC | 14:11 | |
| *** dayou has quit IRC | 14:12 | |
| *** johnsom has joined #openstack-infra | 14:12 | |
| *** xgerman_ has joined #openstack-infra | 14:12 | |
| *** patricku has joined #openstack-infra | 14:12 | |
| *** dayou has joined #openstack-infra | 14:13 | |
| *** fkautz has joined #openstack-infra | 14:13 | |
| *** ralonsoh has quit IRC | 14:13 | |
| gema | fungi: ack, makes sense | 14:13 |
| *** bobh has joined #openstack-infra | 14:14 | |
| fungi | #status log deactivated duplicate gerrit accounts 26191 and 27230, and reassigned their openids to older account 8866 | 14:15 |
| openstackstatus | fungi: finished logging | 14:15 |
| fungi | raildo: ^ | 14:15 |
| raildo | fungi, ack, thanks!! | 14:15 |
| *** felipemonteiro_ has quit IRC | 14:16 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:16 | |
| *** camunoz has quit IRC | 14:18 | |
| *** slaweq_ has joined #openstack-infra | 14:18 | |
| *** rpioso|afk is now known as rpioso | 14:18 | |
| mriedem | is this a known issue? http://logs.openstack.org/69/543569/7/check/neutron-grenade/8e683f9/logs/grenade.sh.txt.gz#_2018-04-16_18_50_40_440 | 14:19 |
| mriedem | Currently installed pip version 1 does not meet minimum requirements (>=6). | 14:19 |
| mriedem | https://bugs.launchpad.net/openstack-gate/+bug/1764760 | 14:19 |
| openstack | Launchpad bug 1764760 in OpenStack-Gate "Currently installed pip version 1 does not meet minimum requirements (>=6)." [Undecided,New] | 14:19 |
| *** mikal has quit IRC | 14:19 | |
| *** mikal has joined #openstack-infra | 14:20 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for pip install bug 1764760 https://review.openstack.org/561956 | 14:21 |
| openstack | bug 1764760 in OpenStack-Gate "Currently installed pip version 1 does not meet minimum requirements (>=6)." [Undecided,New] https://launchpad.net/bugs/1764760 | 14:21 |
| *** slaweq has quit IRC | 14:21 | |
| *** neiloy has joined #openstack-infra | 14:23 | |
| *** dmsimard|off is now known as dmsimard | 14:23 | |
| fungi | mriedem: wow, that's an insanely old pip if so. i wonder if devstack assumed pip major versions would never reach double digits | 14:24 |
| mriedem | Y2K?! | 14:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: DNM: test jobs https://review.openstack.org/561961 | 14:27 |
| fungi | mriedem: looks like it is actually splitting on a separator at least http://git.openstack.org/cgit/openstack-dev/devstack/tree/inc/python?h=stable%2Fpike#n335 | 14:28 |
| *** belliott_ has joined #openstack-infra | 14:29 | |
| *** belliott has quit IRC | 14:30 | |
| *** hemna_ has joined #openstack-infra | 14:30 | |
| fungi | it's not a distro package at least, because the python-pip packaged on xenial (platform that job used) is 8.1.1 | 14:31 |
| *** hashar has joined #openstack-infra | 14:31 | |
| mriedem | huh, but it's not in pip freeze output either | 14:33 |
| *** eharney has quit IRC | 14:33 | |
| fungi | pip freeze doesn't report itself | 14:34 |
| fungi | pip list would, if we ran that | 14:34 |
| fungi | or pip --version | 14:34 |
| *** kiennt26_ has joined #openstack-infra | 14:34 | |
| mriedem | so the old side (ocata) installs pip 10.0.0 | 14:34 |
| mriedem | http://logs.openstack.org/69/543569/7/check/neutron-grenade/8e683f9/logs/old/devstacklog.txt.gz#_2018-04-16_18_27_49_455 | 14:34 |
| dmsimard | infra-root: Stepping momentarily off of PTO -- I've removed static.o.o from the emergency file, it was for testing improvements for the ARA middleware implementation. There will be a new release of ARA this week with improvements that I'll want to test before merging https://review.openstack.org/#/c/558688/ | 14:34 |
| dmsimard | fungi: got your ping yesterday about a survey hiera file ? I don't remember playing around these lands | 14:35 |
| fungi | pip freeze intentionally omits some information about lower-level packaging and package-management libraries, under the assumption that those are going to be provided by your environment and so you wouldn't want to inadvertently pin them by reusing the pip freeze output to install another environment later | 14:35 |
| fungi | dmsimard: the question was specifically whether anyone has noticed a regression with launch-node.py --environment (for launching a new server using a non-production, i.e. staging, copy of the system-config repo) | 14:36 |
| mriedem | Currently installed pip version 1 does not meet minimum requirements (>=6). | 14:36 |
| mriedem | so it's parsing 10.0.0 incorrectly | 14:36 |
| dmsimard | fungi: I have not used that feature yet, apologies. | 14:36 |
| fungi | dmsimard: no worries | 14:36 |
| mriedem | although... | 14:37 |
| mriedem | >>> version = '10.0.0' | 14:37 |
| mriedem | >>> version.split('.')[0] | 14:37 |
| mriedem | '10' | 14:37 |
| fungi | mriedem: right, it doesn't appear to be doing version[0] or anything | 14:37 |
| dmsimard | diablo_rojo: #openstack-101 is officially closed and forwarded to #openstack-dev | 14:37 |
| fungi | mriedem: which is what makes this all the more baffling | 14:38 |
| fungi | mriedem: we probably need a dummy devstack change with some additional debugging calls around that line. it may also be specific to the stable/pike branch of devstack (where have you seen it occur so far?) | 14:39 |
| mriedem | i thought maybe it was just a fix that wasn't in pike or ocata, but https://review.openstack.org/#/q/Ic7d0c04d7fa77774ab2d70fb9d11f182becec553 | 14:39 |
| mriedem | i've mostly seen it on stable/pike but logstash shows it happening on all branches | 14:39 |
| fungi | mriedem: and just for grenade, or also other jobs? | 14:40 |
| mriedem | oh heh https://review.openstack.org/#/c/561411/ | 14:40 |
| mriedem | ^ just merged this morning | 14:40 |
| mriedem | so we should be good | 14:40 |
| mriedem | i'm just late to the party | 14:40 |
| fungi | mriedem: oh, haha yet | 14:40 |
| fungi | yep | 14:40 |
| fungi | two hours ago | 14:40 |
| fungi | so we'd have seen it hitting pike changes enqueued up to the past couple hours | 14:41 |
| *** lpetrut_ has joined #openstack-infra | 14:41 | |
| fungi | and that's a hilariously insidious bug (strip instead of split) | 14:41 |
| *** mihalis68 has joined #openstack-infra | 14:44 | |
| *** quiquell|ruck is now known as quiquell|off | 14:46 | |
| *** eharney has joined #openstack-infra | 14:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Merge zuul-website-media when publishing site https://review.openstack.org/561749 | 14:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Merge zuul-website-media when publishing site https://review.openstack.org/561749 | 14:50 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for pip install bug 1764760 https://review.openstack.org/561956 | 14:51 |
| openstack | bug 1764046 in devstack "duplicate for #1764760 installed pip version 1 does not meet minimum requirements" [Undecided,Fix released] https://launchpad.net/bugs/1764046 - Assigned to Illes Elod (elod-illes) | 14:51 |
| *** rkukura has quit IRC | 14:51 | |
| *** jcoufal_ has joined #openstack-infra | 14:52 | |
| *** camunoz has joined #openstack-infra | 14:54 | |
| *** jcoufal has quit IRC | 14:54 | |
| *** salv-orlando has joined #openstack-infra | 14:55 | |
| *** camunoz has quit IRC | 14:56 | |
| *** HeOS has quit IRC | 14:57 | |
| *** camunoz has joined #openstack-infra | 14:58 | |
| clarkb | pabelanger: so I'm good to remove mirror-update from the emergency file now? | 14:58 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoints to get by id summit push notification https://review.openstack.org/561971 | 14:58 |
| *** salv-orlando has quit IRC | 15:00 | |
| *** efried has quit IRC | 15:00 | |
| *** efried has joined #openstack-infra | 15:01 | |
| *** VW has joined #openstack-infra | 15:01 | |
| *** agopi has joined #openstack-infra | 15:02 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoints to get by id summit push notification https://review.openstack.org/561971 | 15:02 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoints to get by id summit push notification https://review.openstack.org/561971 | 15:03 |
| *** TobbeCN has quit IRC | 15:04 | |
| *** salv-orlando has joined #openstack-infra | 15:05 | |
| *** TobbeCN has joined #openstack-infra | 15:05 | |
| *** armaan has quit IRC | 15:05 | |
| *** armaan has joined #openstack-infra | 15:06 | |
| *** cshastri has joined #openstack-infra | 15:07 | |
| clarkb | pabelanger: also did you see ianw's email about xenial on rax? is that something you are still debugging too? | 15:08 |
| clarkb | looks like a potential problem patch was identified so next step is comparing booting of images with and without taht dib patch | 15:09 |
| *** felipemonteiro_ has joined #openstack-infra | 15:10 | |
| *** M4g1c5t0rM has quit IRC | 15:10 | |
| pabelanger | clarkb: yah, can remove mirror-update, I'll push up changes to gerrit | 15:10 |
| clarkb | pabelanger: ok emergency file has had mirror-update removed | 15:11 |
| pabelanger | clarkb: I haven't seen email yet, but will look shortly | 15:11 |
| pabelanger | clarkb: oh, I didn't do that | 15:11 |
| clarkb | ya I'm saying I removed it | 15:11 |
| *** panda|rover|mtg is now known as panda|rover | 15:11 | |
| clarkb | :) | 15:11 |
| pabelanger | ah, parsed it properly now | 15:12 |
| *** TobbeCN has quit IRC | 15:12 | |
| clarkb | on the xenial front that should probably be something that we try to sort out quickly as its affecting some of the pip10 transition (by keeping us on old pip9 images) | 15:13 |
| *** felipemonteiro__ has quit IRC | 15:13 | |
| clarkb | (I doubt I'll have time between bandersnatch and meeting today, but happy to help anyone interested. IIRC in the past I've built a rather minimal image locally as a qcow2 then copied it to a nodepool builder to convert it to vhd then uploaded to rax with shade) | 15:14 |
| hogepodge | Any chance we can get more eyes on this subunit2sql review? https://review.openstack.org/#/c/543836/5 | 15:16 |
| *** yamamoto has quit IRC | 15:17 | |
| fungi | mtreinish: ^ are your concerns with that addressed? | 15:17 |
| *** yamamoto has joined #openstack-infra | 15:17 | |
| *** e0ne has quit IRC | 15:21 | |
| hogepodge | This one too https://review.openstack.org/#/c/537775/ | 15:21 |
| *** eernst has quit IRC | 15:21 | |
| *** eernst has joined #openstack-infra | 15:21 | |
| *** yamahata has joined #openstack-infra | 15:22 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman queries asynchronous in github driver https://review.openstack.org/560044 | 15:24 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 15:24 |
| *** M4g1c5t0rM has joined #openstack-infra | 15:27 | |
| clarkb | for anyone wanting to test the dib stuff I think what I would do is build an image with these elements: ubuntu-minimal, vm, simple-init, infra-package-needs, growroot, devuser that should allow you to ssh in if everything works or use console login if networking doesn't work | 15:28 |
| *** pcaruana has quit IRC | 15:28 | |
| clarkb | you'll want to build an image with latest dib then one with https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=43bc352c5977272ceadc1ca44d96c086cad59f5c reverted | 15:28 |
| *** eernst has quit IRC | 15:28 | |
| clarkb | can use nb0x.openstack.org to convert to vhd and then use shade to upload to rax | 15:28 |
| pabelanger | I'll do that, let me grab coffee | 15:30 |
| *** M4g1c5t0rM has quit IRC | 15:30 | |
| clarkb | cool. puppet is about to start and update bandersnatch for us | 15:30 |
| clarkb | I should get tea before I run the manual mirror sync | 15:31 |
| fungi | i'm popping out for a quick early lunch, not gonna make the same mistake i did yesterday | 15:33 |
| *** lpetrut_ has quit IRC | 15:33 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 15:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: round 2: Fix debian-stretch-security for reprepro https://review.openstack.org/561981 | 15:35 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman queries asynchronous in github driver https://review.openstack.org/560044 | 15:36 |
| pabelanger | clarkb: fungi: ^is our fix for reprepro configuration, but believe you had suggestions how to best do it | 15:36 |
| *** ramishra has quit IRC | 15:36 | |
| clarkb | pabelanger: that should work | 15:38 |
| clarkb | ok I'm running bandersnatch now | 15:39 |
| clarkb | strace says it is running but I don't think we get logs until it is done | 15:42 |
| clarkb | (I'm running run-bandersnatch not bandersnatch directly to make sure as much of teh system works as possible when running it piecemeal to control the publish step) | 15:42 |
| *** kiennt26_ has quit IRC | 15:43 | |
| *** caphrim007 has joined #openstack-infra | 15:50 | |
| *** caphrim007 has quit IRC | 15:50 | |
| *** caphrim007 has joined #openstack-infra | 15:51 | |
| clarkb | seems like it might be doing a full resync ish | 15:51 |
| *** caphrim007 has quit IRC | 15:51 | |
| clarkb | its into the c's now | 15:51 |
| *** caphrim007 has joined #openstack-infra | 15:51 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added endpoint to approve/unpprove push notification https://review.openstack.org/561996 | 15:52 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added endpoint to approve/unpprove push notification https://review.openstack.org/561996 | 15:53 |
| clarkb | there are a lot of cctx packages to inspect | 16:00 |
| *** ihar has joined #openstack-infra | 16:01 | |
| *** TobbeCN has joined #openstack-infra | 16:04 | |
| *** dpawlik has quit IRC | 16:04 | |
| clarkb | we are now on to packages starting with 'd' so it is moving just not incredibly quickly I guess I should do code review | 16:05 |
| *** hashar is now known as hasharAway | 16:05 | |
| *** aojea has quit IRC | 16:07 | |
| *** TobbeCN has quit IRC | 16:09 | |
| *** dsariel has quit IRC | 16:12 | |
| *** benj_ has quit IRC | 16:13 | |
| *** lpetrut_ has joined #openstack-infra | 16:13 | |
| clarkb | pabelanger: re the dib change. Looks like the bootloader install happens at finalise.d/50-something-or-other but the initramfs -u happens at finalise/80-something-or-other | 16:14 |
| clarkb | pabelanger: I wonder if the order of those two things needs to be flipped? | 16:14 |
| *** links has joined #openstack-infra | 16:15 | |
| pabelanger | clarkb: can look in a moment, I'm just trying to boot a new image with the revert | 16:15 |
| *** benj_ has joined #openstack-infra | 16:15 | |
| clarkb | pabelanger: ya definitely start by confirming or denying that change has anything to do with it | 16:15 |
| clarkb | was just poking around under the assumption it will/does | 16:16 |
| pabelanger | ++ | 16:17 |
| *** pcaruana has joined #openstack-infra | 16:18 | |
| *** gyee has joined #openstack-infra | 16:19 | |
| *** jamesmcarthur has quit IRC | 16:21 | |
| *** jamesmcarthur has joined #openstack-infra | 16:22 | |
| *** harlowja has joined #openstack-infra | 16:23 | |
| *** jpich has quit IRC | 16:25 | |
| *** r-daneel_ has joined #openstack-infra | 16:29 | |
| *** r-daneel has quit IRC | 16:30 | |
| *** r-daneel_ is now known as r-daneel | 16:30 | |
| *** yamamoto_ has joined #openstack-infra | 16:30 | |
| *** rkukura has joined #openstack-infra | 16:32 | |
| *** yamamoto has quit IRC | 16:33 | |
| clarkb | into the 'm' package names now | 16:33 |
| *** trown is now known as trown|lunch | 16:35 | |
| *** e0ne has joined #openstack-infra | 16:36 | |
| pabelanger | clarkb: the revert fixed xenial | 16:36 |
| pabelanger | ssh root@104.130.239.94 | 16:36 |
| clarkb | neat! thank you ianw for pointing that out I guess | 16:37 |
| clarkb | pabelanger: based on my investigating dib my hunch is maybe the ordering of grub being setup before initramfs is rewritten is potentially the cause? | 16:37 |
| clarkb | I want to say you typically go the other direction at least with my package manager? | 16:37 |
| clarkb | fungi: ^ you amy know off the top of your head with all your fun arm baord hacking | 16:37 |
| pabelanger | clarkb: yah, I can start to read up on that, test a few builds | 16:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website-media master: Add simulation video https://review.openstack.org/562001 | 16:38 |
| pabelanger | clarkb: do we want to roll back dib on nodepool-builders to the previous version or release with the revert? | 16:38 |
| *** lyarwood has joined #openstack-infra | 16:39 | |
| clarkb | pabelanger: we're likely going to need to roll forward to pick up the centos and fedora and suse pip10 fixes | 16:39 |
| clarkb | sp pinning to previous version likely not a great idea. Ideally we won't break lvm2 users either | 16:39 |
| fungi | clarkb: my arm hacking doesn't help. all the boards i've used so far are stuck using their own proprietary bootloaders | 16:39 |
| clarkb | maybe we can try and sort out a fix (see what ianw thinks when he wakes) and roll forward in a way that accomodates everyone? | 16:39 |
| *** jamesmcarthur has quit IRC | 16:40 | |
| *** pcichy has quit IRC | 16:41 | |
| fungi | clarkb: it's probably a good idea to give the release and requirements teams a heads up that our pypi mirror is running behind, so newly tagged releases aren't going to be appearing immediately | 16:41 |
| *** e0ne has quit IRC | 16:42 | |
| clarkb | fungi: will do | 16:42 |
| *** e0ne has joined #openstack-infra | 16:43 | |
| *** yamahata has quit IRC | 16:48 | |
| *** rkukura has quit IRC | 16:50 | |
| *** jcoufal_ has quit IRC | 16:51 | |
| *** jamesmcarthur has joined #openstack-infra | 16:51 | |
| *** jcoufal has joined #openstack-infra | 16:52 | |
| *** harlowja has quit IRC | 16:52 | |
| *** shardy has quit IRC | 16:53 | |
| *** mihalis68 has quit IRC | 16:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website-media master: Add simulation video https://review.openstack.org/562001 | 16:55 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Revert "debootstrap: Call update-initramfs explicitly" https://review.openstack.org/562004 | 16:55 |
| pabelanger | clarkb: ianw: ^a revert, until we can figure out why | 16:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add simulation video https://review.openstack.org/562005 | 16:55 |
| pabelanger | looking at ordering now | 16:55 |
| openstackgerrit | Merged openstack-infra/system-config master: round 2: Fix debian-stretch-security for reprepro https://review.openstack.org/561981 | 16:56 |
| *** efoley has joined #openstack-infra | 16:57 | |
| *** lucasagomes is now known as lucas-afk | 16:57 | |
| *** harlowja has joined #openstack-infra | 16:57 | |
| *** derekh has quit IRC | 16:58 | |
| *** jamesmcarthur has quit IRC | 17:01 | |
| *** aojea has joined #openstack-infra | 17:04 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Stop mirroring tensorflow and mxnet pypi packages https://review.openstack.org/561694 | 17:04 |
| clarkb | infra-root ^ watching bandersnatch run I noticed mxnet is another machine learning framework that also has multiple hundred megabyte python packages that release often | 17:05 |
| *** dklyle has quit IRC | 17:07 | |
| *** efoley has quit IRC | 17:09 | |
| *** pcichy has joined #openstack-infra | 17:13 | |
| corvus | clarkb: wow i don't even | 17:13 |
| *** jamesmcarthur has joined #openstack-infra | 17:14 | |
| corvus | clarkb: approved, but the parent is, i think, still waiting on your +A | 17:14 |
| *** yamamoto_ has quit IRC | 17:15 | |
| clarkb | corvus: yup once I get bandersnatch running quickly as normal I'll approve those changes and confirm that it all works before releasing the bandersnatch lock. Thanks! | 17:15 |
| *** yamamoto has joined #openstack-infra | 17:16 | |
| *** yamahata has joined #openstack-infra | 17:17 | |
| *** electrofelix has quit IRC | 17:18 | |
| *** jamesmcarthur has quit IRC | 17:18 | |
| prometheanfire | pabelanger: next steps for gentoo still https://review.openstack.org/561495 (and/or https://review.openstack.org/561492 ) | 17:18 |
| *** harlowja has quit IRC | 17:20 | |
| *** jcoufal_ has joined #openstack-infra | 17:22 | |
| *** zoli is now known as zoli|gone | 17:22 | |
| *** zoli|gone is now known as zoli | 17:22 | |
| *** panda|rover is now known as panda|rover|off | 17:24 | |
| *** jcoufal has quit IRC | 17:25 | |
| *** jchhatbar has quit IRC | 17:27 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to delete push notifications https://review.openstack.org/562009 | 17:32 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to delete push notifications https://review.openstack.org/562009 | 17:33 |
| *** links has quit IRC | 17:35 | |
| diablo_rojo | dmsimard, thank you! And thanks to mattoliverau as well! | 17:36 |
| *** wolverineav has joined #openstack-infra | 17:36 | |
| clarkb | still chugging through mxnet | 17:37 |
| clarkb | prometheanfire: sourcing the devstack functions file isn't safe on gentoo? it should be its just bash functions iirc | 17:40 |
| clarkb | seems like getting the called functions to work like https://review.openstack.org/#/c/561492/1/functions-common is better approach? | 17:41 |
| *** r-daneel_ has joined #openstack-infra | 17:42 | |
| *** r-daneel has quit IRC | 17:43 | |
| *** r-daneel_ is now known as r-daneel | 17:43 | |
| *** HeOS has joined #openstack-infra | 17:44 | |
| zxiiro | mordred: You mentioned at XCI meetup that a version of shade was coming that would sync up with the openstack sdk and version dependencies would be simplified. When will that be available? | 17:44 |
| * zxiiro would love to be able to have shade + openstack client working in the same virtualenv | 17:45 | |
| clarkb | zxiiro: do they not work due to dep conflicts? also I think mordred is mostly afk today but should be around tomorrow | 17:46 |
| zxiiro | ah ok. I'll check back tomorrow then. | 17:46 |
| zxiiro | clarkb: yeah there's some dep conflicts that prevent me from being able to have them together. Although it's been awhile since I last checked. | 17:46 |
| clarkb | I would give it a try I want to say I have them in the same venv currently on my desktop | 17:47 |
| *** links has joined #openstack-infra | 17:47 | |
| zxiiro | clarkb: cool. I'll have to check to see what I currently have pinned maybe the latest versions just work now. | 17:48 |
| prometheanfire | clarkb: it's not distro agnostic | 17:48 |
| clarkb | prometheanfire: what is breaking? its just loading bash functions I thought | 17:49 |
| clarkb | prometheanfire: executing the bash functions will likely not work though | 17:49 |
| *** jpena is now known as jpena|off | 17:49 | |
| *** sshnaidm is now known as sshnaidm|off | 17:50 | |
| fungi | zxiiro: i just did `pip install openstacksdk shade` in a fresh virtualenv and they coinstalled successfully. claims to have installed openstacksdk-0.12.0 and shade-1.27.1 plus their respective dependencies | 17:52 |
| zxiiro | fungi: ah great, in hindsight i probably should have did that before asking. Thanks! | 17:52 |
| fungi | you got me curious ;) | 17:53 |
| *** trown|lunch is now known as trown | 17:53 | |
| prometheanfire | clarkb: it was checking if packages were installed | 17:54 |
| clarkb | I'm going to make some late breakfast/early lunch then prep for meeting so likely afkish until meeting time. Bandersnatch is still working through mxnet packages | 17:54 |
| *** cshastri has quit IRC | 17:54 | |
| clarkb | prometheanfire: but not on the source was it? | 17:54 |
| prometheanfire | clarkb: https://review.openstack.org/561492 specifically | 17:54 |
| clarkb | prometheanfire: only when executing the functions? | 17:54 |
| prometheanfire | clarkb: it was on source | 17:54 |
| *** sshnaidm|off has quit IRC | 17:54 | |
| prometheanfire | clarkb: https://review.openstack.org/561495 removes the source | 17:55 |
| clarkb | prometheanfire: sort of, see ianw's comment | 17:55 |
| clarkb | the source is still run just later | 17:55 |
| prometheanfire | good point :| | 17:56 |
| prometheanfire | well, there's always my first patch | 17:56 |
| *** jcoufal_ has quit IRC | 18:02 | |
| *** jcoufal has joined #openstack-infra | 18:02 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 18:02 |
| frickler | woa did someone just unblock release-announce-digest? received a mail with 332 announcements dating at least from 4 weeks back to now | 18:05 |
| gema | is there a team meeting today? | 18:05 |
| frickler | gema: should start in 55 mins | 18:06 |
| *** aojea has quit IRC | 18:06 | |
| gema | frickler: oh, the meeting didn't move in time like the uk! | 18:06 |
| gema | alright , thanks :D | 18:06 |
| *** tesseract has quit IRC | 18:06 | |
| clarkb | I'm realizing that it may have been a good idea to put the package blacklist in before running this. Its gonna hit tensorflow and take forever probably | 18:09 |
| clarkb | oh well, once the blacklist is in place it should go much quicker | 18:10 |
| *** agopi has quit IRC | 18:14 | |
| *** agopi has joined #openstack-infra | 18:15 | |
| *** jcoufal has quit IRC | 18:15 | |
| fungi | gema: it didn't move at all if you set your clock to utc ;) | 18:15 |
| *** mriedem is now known as mriedem_shamwow | 18:16 | |
| gema | fungi: I am painfully aware of that :D | 18:17 |
| * fungi doesn't understand how people in variable timezones survive scheduling | 18:18 | |
| gema | fungi: we only attend meetings 6 months of the year | 18:19 |
| gema | x) | 18:19 |
| fungi | i've also discovered how hard some device, operating system and even plain old clock manufacturers make it to use a non-daylight-encumbered timezone | 18:19 |
| fungi | mostly assumptions that your physical location indicates what timezone you want to use | 18:20 |
| fungi | even better, i've bought clocks that not only insist on changing to daylight time and back every year on certain dates, but then they fail to predict that those dates might change at some point | 18:21 |
| gema | :D | 18:21 |
| gema | fungi: I have kindly asked niedbalski to attend today on my behalf | 18:21 |
| gema | :D | 18:21 |
| fungi | thanks for the heads up! niedbalski (and anyone else) is most welcome to participate | 18:22 |
| *** amoralej is now known as amoralej|off | 18:22 | |
| *** jamesmcarthur has joined #openstack-infra | 18:26 | |
| *** bkero has quit IRC | 18:28 | |
| *** dklyle has joined #openstack-infra | 18:30 | |
| *** bkero has joined #openstack-infra | 18:32 | |
| *** wolverineav has quit IRC | 18:35 | |
| *** wolverin_ has joined #openstack-infra | 18:35 | |
| clarkb | pabelanger: I had an idea over my bagel. Can we just reinstall grub on your confirmed not working instance and reboot it to see if that changes anything? | 18:38 |
| *** apetrich_ has joined #openstack-infra | 18:38 | |
| *** wolverin_ has quit IRC | 18:38 | |
| clarkb | pabelanger: other idea is that some other package is affecting boot in a way that it couldn't with a non updated initramfs | 18:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add simulation video https://review.openstack.org/562005 | 18:38 |
| *** wolverineav has joined #openstack-infra | 18:38 | |
| *** felipemonteiro_ has quit IRC | 18:38 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:39 | |
| *** wolverineav has quit IRC | 18:39 | |
| *** wolverineav has joined #openstack-infra | 18:39 | |
| *** gfidente is now known as gfidente|afk | 18:40 | |
| *** hongbin_ has quit IRC | 18:41 | |
| *** hongbin_ has joined #openstack-infra | 18:41 | |
| *** dsariel has joined #openstack-infra | 18:41 | |
| *** agopi has quit IRC | 18:42 | |
| clarkb | pabelanger: lsinitramfs may help us to see the difference between the two (we can ls the old and new initrds and diff the results) | 18:44 |
| *** dklyle has quit IRC | 18:46 | |
| *** felipemonteiro has joined #openstack-infra | 18:48 | |
| *** felipemonteiro_ has quit IRC | 18:48 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:49 | |
| *** dklyle has joined #openstack-infra | 18:51 | |
| *** felipemonteiro has quit IRC | 18:53 | |
| *** bobh has quit IRC | 18:55 | |
| *** claudiub|2 has quit IRC | 18:56 | |
| *** dklyle has quit IRC | 18:57 | |
| ianw | clarkb / pabelanger : am i reading right that we're narrowing in on initramfs generation? | 18:59 |
| clarkb | ianw: yup | 18:59 |
| ianw | so arm64 is busted too, my current plan was to roll back dib on nb03.o.o (already in emergency) and see if it works | 19:00 |
| clarkb | meeting time | 19:00 |
| *** r-daneel_ has joined #openstack-infra | 19:01 | |
| *** r-daneel has quit IRC | 19:03 | |
| *** r-daneel_ is now known as r-daneel | 19:03 | |
| *** apetrich_ has quit IRC | 19:04 | |
| *** TobbeCN has joined #openstack-infra | 19:09 | |
| esberglu | Is it possible to force CI to | 19:10 |
| esberglu | run two jenkins jobs on the same underlying compute host | 19:10 |
| *** florianf has quit IRC | 19:10 | |
| esberglu | I'm trying to get multinode CI support working (using jenkins 2.46.1, zuul 2.6.0, nodepool 0.3.0) for the PowerVM CI | 19:11 |
| clarkb | esberglu: we are in our team meeting currently but happy to help after we are done (in about 49 minutes) | 19:11 |
| esberglu | clarkb: Sorry! thanks | 19:11 |
| *** e0ne has quit IRC | 19:13 | |
| *** TobbeCN has quit IRC | 19:13 | |
| *** iyamahat has joined #openstack-infra | 19:17 | |
| *** apetrich has quit IRC | 19:18 | |
| *** apetrich has joined #openstack-infra | 19:19 | |
| fungi | esberglu: i haven't touched jenkins in a couple years, but i think you can configure a slave to have more than one executor. however i don't know how you'd configure nodepool and zuul to take advantage of that (i expect they have plenty of built-in assumptions about one-node-one-job) | 19:22 |
| *** pcichy has quit IRC | 19:22 | |
| fungi | esberglu: oh, wait, you said two jobs on the same compute host. i misunderstood and though you wanted two jobs on a single test node | 19:23 |
| fungi | zuul and nodepool don't have any notion of your individual compute hosts. they just ask the nova api to boot nodes and use what it gives them | 19:23 |
| fungi | if nova can boot two instances on the same compute host then nodepool/zuul will happily use them | 19:24 |
| openstackgerrit | Abbas Yazdanpanah proposed openstack-infra/git-review master: `work-in-progress` and `private` workflow in Gerrit 2.15 https://review.openstack.org/559488 | 19:25 |
| esberglu | fungi: Thanks, that was my understanding, just wanted to make sure I wasn't missing something | 19:25 |
| *** agopi has joined #openstack-infra | 19:27 | |
| edmondsw | fungi and that goes for aggregates too? Can't say "I want instance 2 to be in the same aggregate as instance 1" without having specified an aggregate for instance 1? | 19:28 |
| *** r-daneel_ has joined #openstack-infra | 19:29 | |
| fungi | edmondsw: i don't think nodepool comprehends host aggregates unless you expose them through something like an availability zone | 19:30 |
| fungi | but i could be wrong | 19:30 |
| niedbalski | ianw, re: arm64, did you find something useful on the console logs? | 19:30 |
| *** r-daneel has quit IRC | 19:30 | |
| *** r-daneel_ is now known as r-daneel | 19:30 | |
| ianw | niedbalski: thanks a ton for looking into it, was about to send a mail :) not really; but it appears we have a more generic problem with our image generation | 19:31 |
| *** camunoz has quit IRC | 19:32 | |
| niedbalski | ianw, you think might be related to this? | 19:32 |
| ianw | niedbalski: i have rolled back to dib 2.12.1 on nb03 and am rebuilding an image. the current suspect is a change to initramfs generation that came in 2.13.0 | 19:32 |
| niedbalski | ianw, just by looking at libvirt/console side, I don't really have a clue at this moment , will have to do a deeper debug | 19:32 |
| ianw | all it really does is regenerate the initramfs during the build, which a priori you'd think would be safe | 19:32 |
| niedbalski | ianw, yes, I do think should be safe. | 19:33 |
| ianw | if this new build works, then we can pick apart the images and see what's different | 19:33 |
| niedbalski | ianw, are you seeing something similar on any other cloud | 19:34 |
| niedbalski | ? | 19:34 |
| *** jamesmcarthur has quit IRC | 19:34 | |
| *** bobh has joined #openstack-infra | 19:34 | |
| ianw | niedbalski: yes, i think we are now. at first we thought this was a rax only problem, because they setup their network differently, but evidence is suggesting a more generic problem | 19:34 |
| *** imacdonn has quit IRC | 19:35 | |
| ianw | i've been chasing my tail a bit looking at udev rules and systemd activation, when i think the problem is earlier | 19:35 |
| edmondsw | fungi sorry, I meant az | 19:35 |
| *** imacdonn has joined #openstack-infra | 19:35 | |
| niedbalski | ianw, yep, our setup is quite simple, flat network , no neutron l3ha/nor dvr, vxlan encap. | 19:35 |
| *** xinliang has quit IRC | 19:37 | |
| *** jamesmcarthur has joined #openstack-infra | 19:38 | |
| ianw | niedbalski: yeah, from the image point of view, it's that rax don't do dhcp but provide network info in the config-drive. glean should smooth it all over, but now it appears to not work with configdrive OR dhcp | 19:38 |
| niedbalski | ianw, got it. | 19:38 |
| *** xinliang has joined #openstack-infra | 19:39 | |
| *** xinliang has quit IRC | 19:39 | |
| *** xinliang has joined #openstack-infra | 19:39 | |
| niedbalski | ianw, well, drop me a message if the rollback to 2.12.1 doesn't works and you need any further debugging at the cloud side. | 19:39 |
| *** bobh has quit IRC | 19:39 | |
| *** jamesmcarthur has quit IRC | 19:40 | |
| *** jamesmcarthur has joined #openstack-infra | 19:40 | |
| ianw | niedbalski: thanks! | 19:40 |
| niedbalski | ianw, yw, thank you for looking into this. | 19:42 |
| *** iyamahat has quit IRC | 19:45 | |
| edmondsw | fungi so to rephrase... is there a way to say "I want instance 2 to be in the same availability zone as instance 1" without having specified an az? i.e. let nova choose az on the first one, but need both in the same az. | 19:46 |
| *** camunoz has joined #openstack-infra | 19:48 | |
| fungi | edmondsw: i think there is some logic in nodepool to balance across availability zones and keep members of a node group within the same az, but i honestly don't recall (someone may have better info after our meeting ends in another ~10 minutes) | 19:48 |
| fungi | edmondsw: however i _don't_ think there is a capability to let nova pick the az of one node and then request teh rest of the node group from the same az | 19:49 |
| *** mriedem_shamwow is now known as mriedem | 19:49 | |
| fungi | also pretty sure in most cases we've encountered, if you let nova "pick" the az then you just end up with all your nodes in one (the "default") az | 19:49 |
| edmondsw | yeah, I think you're probably right about that | 19:49 |
| *** harlowja has joined #openstack-infra | 19:50 | |
| *** dklyle has joined #openstack-infra | 19:52 | |
| fungi | though maybe the new placement service has magic scheduling solutions to solve that behind the scenes these days | 19:53 |
| *** apetrich has quit IRC | 19:54 | |
| *** apetrich has joined #openstack-infra | 19:55 | |
| *** jamesmcarthur has quit IRC | 19:57 | |
| *** jamesmcarthur has joined #openstack-infra | 19:58 | |
| ihar | we are missing job-output messages in logstash. is it as designed? | 19:59 |
| ihar | we have gate-cleanup-hosts logs but not what was once in console.html | 19:59 |
| clarkb | ihar: can you be more specific, what file (links are helpful) is missing? | 20:00 |
| clarkb | elastic recheck says we are up to date on indexing | 20:00 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 20:01 |
| clarkb | fungi: edmondsw ya if booting subnodes they should all be in the same az and region as teh primary | 20:01 |
| clarkb | (with nodepool) | 20:01 |
| edmondsw | esberglu ^ | 20:02 |
| clarkb | edmondsw: sorry tab complete fail | 20:02 |
| ihar | clarkb: I specifically look for messages from http://logs.openstack.org/17/553617/16/check/neutron-tempest-plugin-dvr-multinode-scenario/42fb158/job-output.txt.gz | 20:02 |
| edmondsw | clarkb it was both of us asking :) | 20:03 |
| ihar | any message would do, but I play with the following query: build_name:"neutron-tempest-plugin-dvr-multinode-scenario" AND tags:console AND message:"SUCCESS" | 20:03 |
| esberglu | clarkb: Okay thank you! | 20:05 |
| dhellmann | I feel like I've seen at least one other occurence of this error with a doc build in the last few days, but I don't know that I've seen a resolution. Does anyone recognize the cause of "You must give at least one requirement to install" as an error in the "neutron-horizon-hack" step of the build-openstack-sphinx-docs job? http://logs.openstack.org/41/561341/2/check/build-openstack-sphinx-docs/3247c74/job-output.txt.gz#_ | 20:07 |
| dhellmann | 2018-04-16_18_31_18_408616 | 20:07 |
| openstackgerrit | Merged openstack-infra/subunit2sql master: Add ability to grab stream processing variables from kwargs https://review.openstack.org/543836 | 20:07 |
| dhellmann | That's on taskflow, which shouldn't require horizon or neutron, so I wonder if that's the cause? | 20:08 |
| dhellmann | taskflow is still using the old tox_install.sh | 20:09 |
| fungi | dhellmann: it boils down to invoking tox_install.sh in such a way that it passes an empty package list to pip install | 20:10 |
| dhellmann | do we have a definitive example of a patch that removes tox_install.sh? | 20:11 |
| dhellmann | or a fix for the script, I guess? whatever's easiest | 20:11 |
| fungi | i don't know one of the top of my head, but someone else in here probably does | 20:11 |
| clarkb | ihar: there is a bug in os-loganalyze it looks like | 20:11 |
| clarkb | ihar: http://logs.openstack.org/17/553617/16/check/neutron-tempest-plugin-dvr-multinode-scenario/42fb158/job-output.txt.gz?level=INFO is what the log workers fetch and index and that is empty | 20:11 |
| clarkb | ihar: I think that means os-loganalyze cannot parse that file | 20:12 |
| dhellmann | fungi : ok. I'll see what I can figure out by comparing against another oslo lib | 20:12 |
| clarkb | ihar: my hunch is that http://logs.openstack.org/17/553617/16/check/neutron-tempest-plugin-dvr-multinode-scenario/42fb158/job-output.txt.gz#_2018-04-17_02_20_54_854351 is at fault | 20:12 |
| dhellmann | thanks | 20:12 |
| clarkb | corvus: ^ any idea why the job information in the log isn't prefixed like all the other console log output? | 20:13 |
| *** gfidente|afk has quit IRC | 20:14 | |
| *** bobh has joined #openstack-infra | 20:14 | |
| *** dsariel has quit IRC | 20:14 | |
| clarkb | bandersnatch is going through all the tensorflow nightly packages now | 20:14 |
| ianw | clarkb: ok, started https://etherpad.openstack.org/p/dib-initramfs-regen-issues . i've got myself confused with that many terminals, builds, vm's etc going to start more methodically there | 20:16 |
| ihar | clarkb: when I look for SUCCESS matches, I see lots of them from cleanup-host, and note that those messages all have _grokparsefailure tag. The SUCCESS lines from the cleanup-host file also don't have proper timestamps. | 20:17 |
| ihar | I am not sure if that's related but now that you mentioned potential parsing issue I thought maybe it is | 20:17 |
| *** agopi has quit IRC | 20:17 | |
| *** camunoz has quit IRC | 20:18 | |
| clarkb | ihar: I think they are likely unrelated. The issue with job-output.txt is it is never being sent to logstash because the ?level=INFO url above has no content | 20:18 |
| clarkb | ihar: _grokparsefailures will be due to logstash configs and processing of the files | 20:18 |
| clarkb | http://logs.openstack.org/17/553617/16/check/neutron-tempest-plugin-dvr-multinode-scenario/42fb158/ara-report/file/9e13fa1a-c2a0-4b16-b07a-704fac0a6b28/#line-8 is what I think is breaking us | 20:19 |
| clarkb | I think because its a multiline log it gets emitted all together to the log annotater and we only get the first line with the timestamp previx | 20:19 |
| clarkb | *prefix | 20:19 |
| clarkb | I'll push up a patch to do a separate debug for each line and we can see if that fixes it | 20:20 |
| ihar | clarkb: great, thanks. please share a link when ready. | 20:20 |
| corvus | clarkb: i believe it's all one line with embedded \ns | 20:21 |
| clarkb | corvus: ya | 20:21 |
| corvus | clarkb: and now i read what you wrote after that :) | 20:21 |
| *** dklyle has quit IRC | 20:21 | |
| openstackgerrit | Merged openstack-infra/subunit2sql master: Add delete by uuid functions https://review.openstack.org/537775 | 20:23 |
| *** agopi has joined #openstack-infra | 20:25 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Render job info consistently https://review.openstack.org/562042 | 20:27 |
| clarkb | corvus: ihar ^ thats a stab at addressing this | 20:27 |
| clarkb | I think it will be self testing too? | 20:27 |
| ianw | anyone want to guess why debian-stretch-arm64 is marked as pause: true on nb03, but it's currently building debian-stretch-arm64? | 20:28 |
| ihar | yes we should check in logstash after results are ready | 20:28 |
| *** rkukura has joined #openstack-infra | 20:29 | |
| clarkb | yup self testing and I need to fix some issues | 20:29 |
| *** yamamoto has quit IRC | 20:29 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Render job info consistently https://review.openstack.org/562042 | 20:29 |
| *** yamamoto has joined #openstack-infra | 20:30 | |
| *** raildo has quit IRC | 20:30 | |
| *** dklyle has joined #openstack-infra | 20:31 | |
| *** lpetrut_ has quit IRC | 20:32 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 20:33 |
| *** yamamoto has quit IRC | 20:34 | |
| *** jamesmcarthur has quit IRC | 20:37 | |
| *** jamesmcarthur has joined #openstack-infra | 20:37 | |
| *** iyamahat has joined #openstack-infra | 20:39 | |
| *** dbecker has quit IRC | 20:40 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 20:40 |
| *** sshnaidm|off has joined #openstack-infra | 20:41 | |
| *** links has quit IRC | 20:42 | |
| *** pcaruana has quit IRC | 20:42 | |
| *** kgiusti has left #openstack-infra | 20:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website-media master: WIP: add captions https://review.openstack.org/562047 | 20:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: WIP: Add captions https://review.openstack.org/562048 | 20:45 |
| corvus | clarkb: qq -- why do we care about the log parser and that "line" ? | 20:46 |
| *** jamesmcarthur has quit IRC | 20:47 | |
| clarkb | corvus: because I think it broke all console log parsing by logstash. the specific issue is os-loganalyze returns no data for eg http://logs.openstack.org/17/553617/16/check/neutron-tempest-plugin-dvr-multinode-scenario/42fb158/job-output.txt.gz?level=INFO | 20:47 |
| clarkb | so we get the job to index the console log, we pull down the empty file and move on | 20:47 |
| corvus | clarkb: should we fix the parser? that sounds like it could be subject to dos later on? | 20:48 |
| *** jamesmcarthur has joined #openstack-infra | 20:48 | |
| clarkb | I think we should do both, first be consistent as much as possible in the output we do control but ya also do our best to fail gracefully in os-loganalyze | 20:48 |
| clarkb | http://logs.openstack.org/42/562042/2/check/openstack-infra-base-integration-centos-7/682c48c/job-output.txt.gz?level=INFO works with my zuul-jobs change so I think that confirms why it is happening | 20:49 |
| *** jtomasek has quit IRC | 20:49 | |
| corvus | clarkb: wfm; we may want to poke pabelanger (i think he was involved in that orig) | 20:49 |
| clarkb | I want to say that os-loganalyze was supposed to pair up lines that it couldn't match with the previous matching line | 20:51 |
| clarkb | I'm looking at that now (to avoid this specific issue) | 20:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 20:58 |
| ianw | pabelanger: did you remove the "new-glean" images in rax? | 20:59 |
| *** trown is now known as trown|outtypewww | 21:01 | |
| pabelanger | ianw: I removed new-glean11 and new-glean-pabelanger (clean up), I didn't think anybody was using them | 21:08 |
| ianw | pabelanger: that's ok, just wanted a known bad initrd comparision | 21:08 |
| pabelanger | clarkb: corvus: wfm too | 21:09 |
| pabelanger | ianw: I can rebuild and upload | 21:09 |
| ianw | pabelanger: it's ok, i rebuilt it. i could not get a vhd i built to boot yesterday | 21:10 |
| *** rfolco is now known as rfolco|off | 21:10 | |
| *** hasharAway has quit IRC | 21:11 | |
| ianw | pabelanger / clarkb : here is a diff between a "good" initrd (from one of our booted xenial images) and a "bad" initrd (that i just built with dib 2.13.0) -> http://paste.openstack.org/show/719416/ | 21:12 |
| pabelanger | ianw: have you seen http://logs.openstack.org/04/562004/1/check/nodepool-functional-py35-redhat-src/447dd57/controller/logs/builds/centos-7-0000000002_log.txt.gz#_2018-04-17_17_25_17_467 | 21:12 |
| ianw | pabelanger: yep, fix ready for review: https://review.openstack.org/561479 | 21:13 |
| pabelanger | ianw: ah, I thought we merged it | 21:13 |
| *** ldnunes has quit IRC | 21:15 | |
| pabelanger | +2, great work | 21:18 |
| ianw | oh ffs, it never ends ... HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers. | 21:20 |
| ianw | it seem nb03 can not get to tarballs.openstack.org | 21:21 |
| ianw | which halts the build as it tries to cache stackviz | 21:21 |
| *** priteau has quit IRC | 21:22 | |
| ianw | 103.20.251.147 - - [17/Apr/2018:21:22:28 +0000] "GET /package-stackviz-element/stackviz-latest.tar.gz HTTP/1.1" 200 0 "-" "Wget/1.17.1 (linux-gnu)" | 21:23 |
| ianw | tarballs.o.o see it start ... | 21:23 |
| *** boden has quit IRC | 21:23 | |
| *** Goneri has quit IRC | 21:23 | |
| *** claudiub|2 has joined #openstack-infra | 21:23 | |
| *** eernst has joined #openstack-infra | 21:23 | |
| *** edmondsw has quit IRC | 21:24 | |
| *** agopi has quit IRC | 21:25 | |
| ianw | and https works, so i suspect something in between | 21:26 |
| *** agopi has joined #openstack-infra | 21:26 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add pip cache cleanup to pip-and-virtualenv https://review.openstack.org/562055 | 21:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Use https for stackviz get https://review.openstack.org/562056 | 21:28 |
| ianw | fungi: ^ that should be ok? | 21:28 |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add pip cache cleanup to pip-and-virtualenv https://review.openstack.org/562055 | 21:30 |
| fungi | ianw: yeah, that should be fine | 21:30 |
| fungi | i smell a transparent web proxy | 21:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website-media master: Add captions https://review.openstack.org/562047 | 21:30 |
| fungi | ianw: or perhaps a really, really great wall | 21:30 |
| *** agopi has quit IRC | 21:30 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add captions https://review.openstack.org/562048 | 21:31 |
| *** wolverineav has quit IRC | 21:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add captions to simulation video https://review.openstack.org/562048 | 21:31 |
| *** wolverineav has joined #openstack-infra | 21:31 | |
| *** eharney has quit IRC | 21:31 | |
| *** wolverineav has quit IRC | 21:32 | |
| *** wolverineav has joined #openstack-infra | 21:32 | |
| *** agopi has joined #openstack-infra | 21:33 | |
| ianw | fungi: yeah, this just stopped working when the upstream networking provider changed, so i guess the wall looks different depending on where you are | 21:34 |
| fungi | with a wall that great, i'm not surprised there are many and varied views from all angles | 21:34 |
| *** ihar has quit IRC | 21:35 | |
| fungi | but yeah, more likely it's the isp running a transparent caching "web acceleration" proxy which is misbehaving/broken | 21:35 |
| *** ihar has joined #openstack-infra | 21:35 | |
| fungi | and it's blind to https so doesn't try to help | 21:35 |
| ianw | yeah, true | 21:37 |
| *** kjackal has quit IRC | 21:38 | |
| ianw | so plymouth and some crypt-boot stuff is missing from the "bad" xenial initrd ... but nothing immediately of consequence to glean i would have thought | 21:39 |
| fungi | yeah, and framebuffer support likely also now unneeded because of dropping plymouth | 21:40 |
| *** dklyle has quit IRC | 21:42 | |
| *** agopi has quit IRC | 21:44 | |
| ianw | presumably plymouth comes in the default kernel, and we don't install the pkgs so it falls out when we regen | 21:45 |
| *** dklyle has joined #openstack-infra | 21:46 | |
| *** VW_ has joined #openstack-infra | 21:47 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Revert "debootstrap: Call update-initramfs explicitly" https://review.openstack.org/562004 | 21:47 |
| *** slaweq_ has quit IRC | 21:48 | |
| *** slaweq has joined #openstack-infra | 21:48 | |
| ianw | pabelanger: ^ we've confirmed? | 21:51 |
| *** VW has quit IRC | 21:51 | |
| ianw | i'm going to copy this "bad" initramfs into a known good host and see if it replicates | 21:51 |
| *** esberglu has quit IRC | 21:51 | |
| ianw | i certainly don't mind reverting, but feels like a big time-bomb until we understand it | 21:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Use https for stackviz get https://review.openstack.org/562056 | 21:51 |
| *** yamamoto has joined #openstack-infra | 21:52 | |
| *** VW_ has quit IRC | 21:52 | |
| *** slaweq has quit IRC | 21:53 | |
| pabelanger | ianw: yah, I've built it this afternoon | 21:55 |
| pabelanger | and rax worked right after launching | 21:55 |
| *** rh-jelabarre has quit IRC | 21:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website-media master: Add simulation video https://review.openstack.org/562001 | 21:58 |
| *** yamamoto has quit IRC | 21:58 | |
| *** VW has joined #openstack-infra | 22:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add simulation video https://review.openstack.org/562005 | 22:01 |
| pabelanger | ianw: what's your thinking of DIB release? something today or wait until we better understand glean failures | 22:04 |
| pabelanger | ianw: is pip10 fixes needed to unblock anything? haven't really been following centos / fedora | 22:04 |
| ianw | pabelanger: well i'd like to merge the pip10 fix and i guess this revert | 22:05 |
| corvus | fungi, ianw: my experience has been that the wall indeed looks different from different providers. | 22:05 |
| ianw | pabelanger: *hopefully* this current nb03 xenial build will work, and i can double-confirm the revert on arm64 | 22:06 |
| pabelanger | ianw: kk | 22:08 |
| clarkb | ok I think I finally understand what is failing in os-loganaylze there. It thinks that file format supports severity but does not | 22:08 |
| clarkb | if it doesn't support severity it seems to output all the lines as expected | 22:08 |
| clarkb | hwoever comparing the output of the testsuite against the output of the actual http server they are acting different (test suite does the right thing) so now I'm extra confused | 22:09 |
| *** felipemonteiro_ has quit IRC | 22:10 | |
| *** jamesmcarthur has quit IRC | 22:11 | |
| *** verdurin has quit IRC | 22:12 | |
| *** yamamoto has joined #openstack-infra | 22:15 | |
| *** verdurin has joined #openstack-infra | 22:15 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/infra-manual master: Update Launchpad mentions to StoryBoard https://review.openstack.org/561753 | 22:17 |
| *** mriedem is now known as mriedem_away | 22:17 | |
| clarkb | I see the problem \o/ | 22:18 |
| clarkb | I'm actually no longer convinced my zuul-jobs change will fix it | 22:18 |
| *** VW has quit IRC | 22:19 | |
| *** yamamoto has quit IRC | 22:20 | |
| *** bobh has quit IRC | 22:22 | |
| *** pcichy has joined #openstack-infra | 22:22 | |
| *** pcichy has quit IRC | 22:23 | |
| clarkb | bandersnatch is still chewing through tensorflow packages | 22:25 |
| *** dklyle has quit IRC | 22:27 | |
| *** eernst has quit IRC | 22:29 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 22:30 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-gerrit master: [DNM] Test puppet-4 puppet-apply jobs https://review.openstack.org/557515 | 22:30 |
| ianw | we have 38 hosts in a sort of active, no-network purgatory on rax/dfw -> http://paste.openstack.org/show/719418/ ... i'm planning to delete them unless i hear otherwise | 22:30 |
| ianw | infra-root: ^ | 22:31 |
| *** dpawlik has joined #openstack-infra | 22:33 | |
| *** salv-orl_ has joined #openstack-infra | 22:33 | |
| *** pcichy has joined #openstack-infra | 22:35 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/os-loganalyze master: Better supports sev checking https://review.openstack.org/562070 | 22:35 |
| *** yamamoto has joined #openstack-infra | 22:35 | |
| clarkb | corvus: ihar ^ that is the actual fix | 22:36 |
| mattoliverau | dmsimard: awesome, thanks for all the openstack-101 work! | 22:36 |
| *** salv-orlando has quit IRC | 22:36 | |
| clarkb | ianw: those all look like nodepool nodes to me. No objections here | 22:36 |
| rm_work | clarkb: so https://review.openstack.org/#/c/561049/12/playbooks/legacy/neutron-lbaasv2-dsvm-api-l7octavia/run.yaml@45 | 22:36 |
| pabelanger | clarkb: ianw: yah, they look old curious why nodepool isn't deleting them. missing meta data? | 22:36 |
| *** dizquierdo has joined #openstack-infra | 22:37 | |
| rm_work | i kinda expected that would show up in http://logs.openstack.org/49/561049/12/check/neutron-lbaasv2-dsvm-api-l7octavia/dc25c44/logs/local.conf.txt.gz ? | 22:37 |
| rm_work | did i do it wrong? | 22:37 |
| *** pcichy has quit IRC | 22:37 | |
| *** wolverin_ has joined #openstack-infra | 22:39 | |
| *** wolverineav has quit IRC | 22:39 | |
| *** eernst has joined #openstack-infra | 22:39 | |
| *** dpawlik has quit IRC | 22:39 | |
| clarkb | rm_work: hrm it might need to be in local.conf format so have a head for localrc? | 22:40 |
| rm_work | ah | 22:40 |
| rm_work | ummm | 22:40 |
| rm_work | k | 22:40 |
| rm_work | just like.... | 22:40 |
| clarkb | rm_work: http://codesearch.openstack.org/?q=DEVSTACK_LOCALCONF&i=nope&files=&repos= has examples for how neutron does it | 22:41 |
| rm_work | yeah | 22:41 |
| rm_work | but LOCAL_CONF is a file | 22:41 |
| clarkb | rm_work: DEVSTACK_LOCAL_CONFIG may be what you want though | 22:41 |
| rm_work | and i don't know what it looks like | 22:41 |
| clarkb | just to be confusing that looks like ^ is assumed to be localrc and passed in that way | 22:42 |
| clarkb | whereas LOCALCONF is local.conf | 22:42 |
| rm_work | ah yeah http://codesearch.openstack.org/?q=DEVSTACK_LOCAL_CONFIG&i=nope&files=&repos= | 22:42 |
| clarkb | rm_work: you'd have to go find their file | 22:42 |
| *** yamamoto has quit IRC | 22:42 | |
| *** rcernin has joined #openstack-infra | 22:42 | |
| *** dizquierdo has quit IRC | 22:44 | |
| clarkb | still going through tensorflow nightlies | 22:45 |
| clarkb | I wonder if they even realize they are doing this | 22:45 |
| *** r-daneel has quit IRC | 22:46 | |
| *** r-daneel has joined #openstack-infra | 22:52 | |
| *** neiloy has quit IRC | 22:55 | |
| *** agopi has joined #openstack-infra | 22:56 | |
| *** hongbin_ has quit IRC | 22:58 | |
| *** eernst has quit IRC | 22:58 | |
| *** yamamoto has joined #openstack-infra | 22:58 | |
| clarkb | ianw: readnig through the initrd diff, cryptroot has an explicit wait for udev to be ready | 23:01 |
| clarkb | ianw: I'm wondering if we maybe that is related | 23:01 |
| clarkb | s/we// | 23:01 |
| *** yamamoto has quit IRC | 23:03 | |
| ianw | clarkb: yeah ... AIUI what really happens with that glean udev rule is that it associates the automatically created .device file with the glean udev rule | 23:06 |
| clarkb | ianw: ya udev ends up triggering the unit for writing out the interface configs for each interface as it comes up | 23:07 |
| clarkb | then we execute the setup networking unit or something | 23:07 |
| ianw | clarkb: that's the thing, udev doesn't really trigger that as such | 23:08 |
| ianw | it just sets up the WANTS relationship between the device and the glean@eth0.service | 23:08 |
| clarkb | ah | 23:08 |
| ianw | glean@ens3.service | 23:08 |
| ianw | ● └─sys-devices-pci0000:00-0000:00:03.0-virtio0-net-ens3.device | 23:08 |
| ianw | i'm presuming that "networking-pre" or something "starts" the device, which then by descent starts glean? | 23:09 |
| ianw | this is where it all gets fuzzy | 23:09 |
| clarkb | ya and glean just writes out configs too | 23:10 |
| clarkb | and then later the network bring up applies the configs | 23:10 |
| *** TobbeCN has joined #openstack-infra | 23:11 | |
| ianw | sort of, maybe ... glean@ service has "ExecStart=/sbin/ifup %I" | 23:11 |
| ianw | it has ExecStartPre to setup the files before that | 23:12 |
| *** rpioso is now known as rpioso|afk | 23:12 | |
| clarkb | ah | 23:12 |
| ianw | this is how i've managed to sink so many hours into still not understanding what's going on :) | 23:13 |
| ianw | but yeah, there are a lot of "udev settle" calls in the diff, and it's gotta be a big suspect for what's happening | 23:15 |
| *** tpsilva has quit IRC | 23:15 | |
| *** TobbeCN has quit IRC | 23:15 | |
| *** HeOS has quit IRC | 23:17 | |
| clarkb | I'm worried I won't be awake when bandersnatch finishes | 23:18 |
| *** r-daneel has quit IRC | 23:18 | |
| *** yamamoto has joined #openstack-infra | 23:19 | |
| *** vtapia has quit IRC | 23:20 | |
| *** claudiub|2 has quit IRC | 23:24 | |
| *** yamamoto has quit IRC | 23:24 | |
| clarkb | ianw: for bandersnatch I'm thinking I'll let it run over night, then in the morning I rerun it to make sure it runs quickly on subsequent run. Then apply blacklist change rerun to make sure its happy, then reenable cron by releasing the lock | 23:27 |
| clarkb | ianw: that seem reasonable to you? | 23:27 |
| *** rlandy is now known as rlandy|bbl | 23:27 | |
| ianw | clarkb: that all seems good to me. in the mean time we're just running off the released version right? | 23:36 |
| clarkb | ianw: ya it running off of 2.2.0 | 23:37 |
| ianw | we do have all that magic that looks at the release file and only vos releases if requirements changed IIRC | 23:37 |
| ianw | i guess that is unaffected | 23:37 |
| clarkb | just manually with the lock held in a screen session | 23:37 |
| *** jamesmcarthur has joined #openstack-infra | 23:38 | |
| *** yamamoto has joined #openstack-infra | 23:40 | |
| *** tosky has quit IRC | 23:41 | |
| *** jamesmcarthur has quit IRC | 23:43 | |
| ianw | ++ | 23:43 |
| *** mriedem_away has quit IRC | 23:44 | |
| *** ssbarnea_ has quit IRC | 23:45 | |
| ianw | | 1c405451-f0bb-4ec7-85ff-c1d794da4df2 | fedora-27-rax-iad-0003297356 | ACTIVE | public=104.130.4.229, 2001:4802:7802:104:be76:4eff:fe20:4c85; private=10.208.224.49 | | 8 GB Performance | | 23:46 |
| ianw | another weird one, f27 nodes where the underlying image has been deleted | 23:46 |
| *** dhajare|busy has quit IRC | 23:46 | |
| *** yamamoto has quit IRC | 23:46 | |
| *** r-daneel has joined #openstack-infra | 23:47 | |
| *** slaweq has joined #openstack-infra | 23:49 | |
| *** Goneri has joined #openstack-infra | 23:52 | |
| pabelanger | ianw: hmm | 23:54 |
| pabelanger | I wonder if they are locked ready in zuul | 23:54 |
| *** slaweq has quit IRC | 23:54 | |
| pabelanger | ianw: http://paste.openstack.org/show/719422/ | 23:55 |
| pabelanger | that likely explains why were are seeing them | 23:55 |
| pabelanger | there is an outstanding issue in zuul, if we do a reconfiguration when a node is locked and building, we lose the node request, and they will remain locked / ready | 23:56 |
| clarkb | ok I'll check on bandersnatch in a couple hours but need to figure out dinner and groceries | 23:56 |
| pabelanger | I've been meaning to write a patch to fix it, but haven't had time | 23:56 |
| *** r-daneel has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!