"net/http: invalid header field value" when pushing files to a container

Error: Post http://unix.socket/1.0/containers/riot-web-20190620-193707/files?path=%2Froot%2Friot-web%2Fsrc%2Fi18n%2Fstrings%2Ffr.json: net/http: invalid header field value "\x00XD \x0f\x00\x00\x00\x0f\x00\x00\x00nux; x86_64; 4.18.0; Ubuntu; 18.04)" for key User-Agent

I’m doing this.

LXCBIN="/snap/bin/lxc"
if ! $LXCBIN file push -r "$TMPDIR"/"$APPNAME" "$CONTAINER"/root/

This is the (web hosted) file it falls over on.

I get a same failure with a different container on the same host pushing different code to the container.

Error: Post http://unix.socket/1.0/containers/signal-desktop-20190620-193657/files?path=%2Froot%2FSignal-Desktop%2Fjs%2Fmodules%2Flogs.js: net/http: invalid header field value "\x00XD \x0f\x00\x00\x00\x0f\x00\x00\x00nux; x86_64; 4.18.0; Ubuntu; 18.04)" for key User-Agent

Any ideas?

Does retrying help?

I’ve just seen us hit that in CI so that’s definitely something new and it’s likely coming from the uname handling we do to generate the user-agent, but that logic hasn’t changed in a long time so quite confused as to how that’s happening.

No, it happens every time. This is a clean install of 18.04 I’m using and clean install of lxd. I’m running a script which ran fine elsewhere (on this machine before a wipe) which was running 19.04 previously.

Assuming this is using the snap, can you show snap info lxd?

$ snap info lxd
name:      lxd
summary:   System container manager and API
publisher: Canonical✓
contact:   https://github.com/lxc/lxd/issues
license:   unset
description: |
  **LXD is a system container manager**

  With LXD you can run hundreds of containers of a variety of Linux
  distributions, apply resource limits, pass in directories, USB devices
  or GPUs and setup any network and storage you want.

  LXD containers are lightweight, secure by default and a great
  alternative to running Linux virtual machines.


  **Run any Linux distribution you want**

  Pre-made images are available for Ubuntu, Alpine Linux, ArchLinux,
  CentOS, Debian, Fedora, Gentoo, OpenSUSE and more.

  A full list of available images can be found here: https://images.linuxcontainers.org

  Can't find the distribution you want? It's easy to make your own images too, either using our
  `distrobuilder` tool or by assembling your own image tarball by hand.


  **Containers at scale**

  LXD is network aware and all interactions go through a simple REST API,
  making it possible to remotely interact with containers on remote
  systems, copying and moving them as you wish.

  Want to go big? LXD also has built-in clustering support,
  letting you turn dozens of servers into one big LXD server.


  **Configuration options**

  Supported options for the LXD snap (`snap set lxd KEY=VALUE`):
   - criu.enable: Enable experimental live-migration support [default=false]
   - daemon.debug: Increases logging to debug level [default=false]
   - daemon.group: Group of users that can interact with LXD [default=lxd]
   - ceph.builtin: Use snap-specific ceph configuration [default=false]
   - openvswitch.builtin: Run a snap-specific OVS daemon [default=false]

  Documentation: https://lxd.readthedocs.io
commands:
  - lxd.benchmark
  - lxd.buginfo
  - lxd.check-kernel
  - lxd.lxc
  - lxd
  - lxd.migrate
services:
  lxd.activate: oneshot, enabled, inactive
  lxd.daemon:   simple, enabled, active
snap-id:      J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking:     stable
refresh-date: yesterday at 19:30 BST
channels:
  stable:        3.14        2019-06-19 (10934) 56MB -
  candidate:     3.14        2019-06-21 (10953) 56MB -
  beta:          ↑
  edge:          git-01b7642 2019-06-21 (10947) 57MB -
  3.0/stable:    3.0.3       2018-11-26  (9663) 53MB -
  3.0/candidate: 3.0.3       2019-01-19  (9942) 53MB -
  3.0/beta:      ↑
  3.0/edge:      git-eaa62ce 2019-02-19 (10212) 53MB -
  2.0/stable:    2.0.11      2018-07-30  (8023) 28MB -
  2.0/candidate: 2.0.11      2018-07-27  (8023) 28MB -
  2.0/beta:      ↑
  2.0/edge:      git-c7c4cc8 2018-10-19  (9257) 26MB -
installed:       3.14                   (10934) 56MB -

Ok, I’ll attempt to reproduce this. I did see that error come up in our CI earlier today but I’ve never seen it hitting consistently. Having it consistently happen would certainly help track down what’s going on.

It smells like a memory problem when retrieving the kernel uname data but I have no idea why that would have started happening now…

You mentioned a script, is that something you could share?

Thanks! I’m pushing an entire git tree into the container, I don’t know that’s a problem. If you clone either Signal-Desktop or riot-web and push that entire folder, that should reproduce it.

The full (awful) script is at http://paste.ubuntu.com/p/cgpkwTSnfs/ - line 8 should be the only thing that needs changing if you plan to run it. It just clones a repo, patches the source and pushes that into a container where it’s built. You probably want to remove the very last line, but that would fail anyway :slight_smile:

Ok, I can reproduce the issue locally now

There we go:

1 Like

Awesome. Obvious next question, is there a channel I can switch to which will have this fix soon? Or will it land in stable soon?

Stable snap just got updated now.

1 Like

Great! That works, thanks.