Commit 469819a3 authored by Peter Maydell's avatar Peter Maydell
Browse files

Merge remote-tracking branch 'remotes/stsquad/tags/pull-ci-updates-210617-2' into staging 



This is mostly Philippe's updates

We add the following cross-compile targets:
  - mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
  - armeb-linux-user

While I was rolling I discovered we could also back out a bunch of the
emdebian hacks as the newly released stretch handles cross compilers
as first class citizens. Unfortunately this also meant I had to drop
the powerpc support as that is no longer in Debian stable.

# gpg: Signature made Wed 21 Jun 2017 15:09:50 BST
# gpg:                using RSA key 0xFBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>"
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-ci-updates-210617-2: (21 commits)
  MAINTAINERS: self-appoint me as reviewer in build/test automation
  MAINTAINERS: add Shippable automation platform URL
  shippable: add mipsel target
  shippable: add armeb-linux-user target
  shippable: be verbose while building docker images
  shippable: do not initialize submodules automatically
  shippable: build using all available cpus
  shippable: use C locale to simplify console output
  docker: add mipsel build target
  docker: add extra libs to s390x target to extend codebase coverage
  docker: add extra libs to arm64 target to extend codebase coverage
  docker: add extra libs to armhf target to extend codebase coverage
  docker: use eatmydata in debian arm64 image
  docker: use eatmydata in debian armhf image
  docker: use eatmydata, install common build packages in base image
  docker: use better regex to generate deb-src entries
  docker: install ca-certificates package in base image
  docker: rebuild image if 'extra files' checksum does not match
  docker: add --include-files argument to 'build' command
  docker: let _copy_with_mkdir() sub_path argument be optional
  ...

Signed-off-by: default avatarPeter Maydell <peter.maydell@linaro.org>
parents 22a9e1fd 32b9ca98

.shippable.yml

+18 −3
Original line number Diff line number Diff line 
language: c

git:

   submodules: false

env:

  global:

    - LC_ALL=C

  matrix:

    - IMAGE=debian-armhf-cross

      TARGET_LIST=arm-softmmu,arm-linux-user

      TARGET_LIST=arm-softmmu,arm-linux-user,armeb-linux-user

    - IMAGE=debian-arm64-cross

      TARGET_LIST=aarch64-softmmu,aarch64-linux-user

    - IMAGE=debian-s390x-cross

      TARGET_LIST=s390x-softmmu,s390x-linux-user

    # mips64el-softmmu disabled due to libfdt problem

    - IMAGE=debian-mipsel-cross

      TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user

build:

  pre_ci:

    - make docker-image-${IMAGE}

    - make docker-image-${IMAGE} V=1

  pre_ci_boot:

    image_name: qemu

    image_tag: ${IMAGE}
@@ -17,5 +24,13 @@ build: 
    options: "-e HOME=/root"

  ci:

    - unset CC

    # some targets require newer up to date packages, for example TARGET_LIST matching

    # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)

    # see the configure script:

    #    error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"

    #    "  (1) Preferred: Install the DTC (libfdt) devel package"

    #    "  (2) Fetch the DTC submodule, using:"

    #    "      git submodule update --init dtc"

    - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc

    - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}

    - make -j2
 
    - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))

MAINTAINERS

+2 −0
Original line number Diff line number Diff line
@@ -1857,12 +1857,14 @@ Build and test automation 
-------------------------

M: Alex Bennée <alex.bennee@linaro.org>

M: Fam Zheng <famz@redhat.com>

R: Philippe Mathieu-Daudé <f4bug@amsat.org>

L: qemu-devel@nongnu.org

S: Maintained

F: .travis.yml

F: .shippable.yml

F: tests/docker/

W: https://travis-ci.org/qemu/qemu

W: https://app.shippable.com/github/qemu/qemu

W: http://patchew.org/QEMU/



Documentation

tests/docker/Makefile.include

+4 −0
Original line number Diff line number Diff line
@@ -51,12 +51,14 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker 
		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \

		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \

		$(if $(NOUSER),,--add-current-user) \

		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\

		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\

		"BUILD","$*")



# Enforce dependancies for composite images

docker-image-debian-armhf-cross: docker-image-debian

docker-image-debian-arm64-cross: docker-image-debian

docker-image-debian-mipsel-cross: docker-image-debian



# Expand all the pre-requistes for each docker image and test combination

$(foreach i,$(DOCKER_IMAGES), \
@@ -107,6 +109,8 @@ docker: 
	@echo '    NOUSER               Define to disable adding current user to containers passwd.'

	@echo '    NOCACHE=1            Ignore cache when build images.'

	@echo '    EXECUTABLE=<path>    Include executable in image.'

	@echo '    EXTRA_FILES="<path> [... <path>]"'

	@echo '                         Include extra files in image.'



# This rule if for directly running against an arbitrary docker target.

# It is called by the expanded docker targets (e.g. make

tests/docker/docker.py

+23 −7
Original line number Diff line number Diff line
@@ -38,6 +38,9 @@ def _text_checksum(text): 
    """Calculate a digest string unique to the text content"""

    return hashlib.sha1(text).hexdigest()



def _file_checksum(filename):

    return _text_checksum(open(filename, 'rb').read())



def _guess_docker_command():

    """ Guess a working docker command or raise exception if not found"""

    commands = [["docker"], ["sudo", "-n", "docker"]]
@@ -52,7 +55,7 @@ def _guess_docker_command(): 
    raise Exception("Cannot find working docker command. Tried:\n%s" % \

                    commands_txt)



def _copy_with_mkdir(src, root_dir, sub_path):

def _copy_with_mkdir(src, root_dir, sub_path='.'):

    """Copy src into root_dir, creating sub_path as needed."""

    dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))

    try:
@@ -154,7 +157,7 @@ class Docker(object): 
        return labels.get("com.qemu.dockerfile-checksum", "")



    def build_image(self, tag, docker_dir, dockerfile,

                    quiet=True, user=False, argv=None):

                    quiet=True, user=False, argv=None, extra_files_cksum=[]):

        if argv == None:

            argv = []
@@ -170,7 +173,8 @@ class Docker(object): 


        tmp_df.write("\n")

        tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %

                     _text_checksum(dockerfile))

                     _text_checksum("\n".join([dockerfile] +

                                    extra_files_cksum)))

        tmp_df.flush()



        self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
@@ -237,6 +241,10 @@ class BuildCommand(SubCommand): 
                            help="""Specify a binary that will be copied to the

                            container together with all its dependent

                            libraries""")

        parser.add_argument("--extra-files", "-f", nargs='*',

                            help="""Specify files that will be copied in the

                            Docker image, fulfilling the ADD directive from the

                            Dockerfile""")

        parser.add_argument("--add-current-user", "-u", dest="user",

                            action="store_true",

                            help="Add the current user to image's passwd")
@@ -270,16 +278,24 @@ class BuildCommand(SubCommand): 
                    print "%s exited with code %d" % (docker_pre, rc)

                    return 1



            # Do we include a extra binary?

            # Copy any extra files into the Docker context. These can be

            # included by the use of the ADD directive in the Dockerfile.

            cksum = []

            if args.include_executable:

                _copy_binary_with_libs(args.include_executable,

                                       docker_dir)

                # FIXME: there is no checksum of this executable and the linked

                # libraries, once the image built any change of this executable

                # or any library won't trigger another build.

                _copy_binary_with_libs(args.include_executable, docker_dir)

            for filename in args.extra_files or []:

                _copy_with_mkdir(filename, docker_dir)

                cksum += [_file_checksum(filename)]



            argv += ["--build-arg=" + k.lower() + "=" + v

                        for k, v in os.environ.iteritems()

                        if k.lower() in FILTERED_ENV_NAMES]

            dkr.build_image(tag, docker_dir, dockerfile,

                            quiet=args.quiet, user=args.user, argv=argv)

                            quiet=args.quiet, user=args.user, argv=argv,

                            extra_files_cksum=cksum)



            rmtree(docker_dir)

tests/docker/dockerfiles/debian-arm64-cross.docker

+16 −2
Original line number Diff line number Diff line
@@ -8,8 +8,22 @@ FROM qemu:debian 
# Add the foreign architecture we want and install dependencies

RUN dpkg --add-architecture arm64

RUN apt update

RUN apt install -yy crossbuild-essential-arm64

RUN apt-get build-dep -yy -a arm64 qemu

RUN DEBIAN_FRONTEND=noninteractive eatmydata \

    apt-get install -y --no-install-recommends \

        crossbuild-essential-arm64

RUN DEBIAN_FRONTEND=noninteractive eatmydata \

    apt-get build-dep -yy -a arm64 qemu



# Specify the cross prefix for this image (see tests/docker/common.rc)

ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-



RUN DEBIAN_FRONTEND=noninteractive eatmydata \

    apt-get install -y --no-install-recommends \

        glusterfs-common:arm64 \

        libbz2-dev:arm64 \

        liblzo2-dev:arm64 \

        libncursesw5-dev:arm64 \

        libnfs-dev:arm64 \

        librdmacm-dev:arm64 \

        libsnappy-dev:arm64 \

        libxen-dev:arm64