The htscodecs version.h is not built unless its .git is present; no instructions to prepare a working tarball

[adamnovak]

The htslib build process needs to build htscodecs/htscodecs/version.h, since it isn't shipped in the htscodecs repository. This is accomplished here:

htslib/Makefile

Lines 578 to 591 in 2ff207b

	# Build the htscodecs/htscodecs/version.h file if necessary
	htscodecs/htscodecs/version.h: force
	@if test -e $(srcdir)/htscodecs/.git && test -e $(srcdir)/htscodecs/configure.ac ; then \
	vers=`cd $(srcdir)/htscodecs && git describe --always --dirty --match 'v[0-9]\.[0-9]*'` && \
	case "$$vers" in \
	v*) vers=$${vers#v} ;; \
	*) iv=`awk '/^AC_INIT\(htscodecs,/ { match($$0, /[0-9]+(\.[0-9]+)*/); print substr($$0, RSTART, RLENGTH) }' $(srcdir)/htscodecs/configure.ac` ; vers="$$iv$${vers:+-g$$vers}" ;; \
	esac ; \
	if ! grep -s -q '"'"$$vers"'"' $@ ; then \
	echo 'Updating $@ : #define HTSCODECS_VERSION_TEXT "'"$$vers"'"' ; \
	echo '#define HTSCODECS_VERSION_TEXT "'"$$vers"'"' > $@ ; \
	fi ; \
	fi
	endif

But if htscodecs doesn't have a .git folder (for example, because we're building from a tarball and not a source checkout), this doesn't do anything and the file is not created.

So if you want to prepare a buildable tarball of htslib, you need to make sure that make htscodecs/htscodecs/version.h has been run before any .git folders are removed.

For vg, we ship tarballs that include a vendored-in htslib. But it turns out our tarballs don't actually build, since we don't perform that step as part of our tarball packaging process.

So I went looking for the official instructions for preparing htslib to be tarballed, and I couldn't find them. The Makefile provides a make distdir, but that doesn't seem to include htscodecs at all, nor does it include os/lzma_stub.h which the build seems to require, so I'm not sure that it is actually in use. The actual htslib release tarballs include these, so they must not use (just) make distdir.

If tarballing up everything in the Git tree except the .git folders doesn't produce a directory tree that can build, there should be instructions in the project documentation on how to prepare the tree to be tarballed, or how to export a working tarball through the build system somehow.

I can by trial and error discover that I need a make htscodecs/htscodecs/version.h, but how will I know when new required generated files are added to the project?

[daviesrob]

Our release tarballs are built using infrastructure in the https://github.com/samtools/c-maint repository. The htscodecs/htscodecs/version.h file gets made here, although the tarball building process does more than this as it also strips out bits of htscodecs that are not needed, builds the htslib configure script, and adds other files used by configure.

It looks like the make distdir target was supposed to do what you want, but may have stopped working after htscodecs was added. We'll take a look.