Makefiles Revisited

Posted on April 9, 2015

Writing a thesis. Writing at home, writing at uni, at work, on planes, on machines half way across the planet with Swedish keyboards attached to them. The amount of computers that have my git repository cloned on it is either impressive or terrifying depending on how you look at it. One positive thing about my tree is that a good 90-95% of my ridiculous figure count has been built in Tikz / PGFPlots. Totally source control friendly, small and portable.

What hasn’t been that nice about it are the few binary dependencies I keep forgetting to track or that nasty little bug that xelatex has wherein it just utterly fails when you try and draw a gradient or any form of shading. To overcome this issue I’ve been running a two step compile process, where figures that require shading are built using pdflatex, then if text is required (for labels or annotations for example), xelatex is called. This allows the figure font to remain homologous with the main text and the pre-rendered shaded geometry can be simply imported with some form of \includegraphics{} command.

However the figure count is now over 50 and climbing, so building all of them on a new machine or propagating broad changes has become a cumbersome task. Ever since writing my first Makefile, I’ve enjoyed making simple ones for menial tasks. Nothing fancy, but useful enough to take the time and understand the syntax just that little bit more. Building these figures was an obvious case for some new Makefile magic.

Here’s what I came up with on a train to Bendigo with two random obnoxious children crawling over me:

    #All files to be built with pdflatex
    PSOURCES := aluminium.tex emegir.tex flourish.tex jjschematic.tex oxygen.tex \\
                                oxygenb.tex qabuum.tex sf.tex sio2.tex
    #All files to be built with xelatex
    XSOURCES := $(filter-out $(PSOURCES), $(wildcard *.tex))
    PTARGETS := $(PSOURCES:.tex=.pdf)
    XTARGETS := $(XSOURCES:.tex=.pdf)
    PDFLATEX := pdflatex --shell-escape --extra-mem-top=10000000 --save-size=80000
    XELATEX := xelatex --shell-escape --extra-mem-top=10000000 --save-size=80000

    .PHONY: all clear rebuild clean distclean

    all: xelatex

    pdflatex: $(PTARGETS)

    xelatex: $(XTARGETS)

    %.pdf: %.tex
            @[ '$<' == '$(findstring $<,$(PSOURCES))' ] && $(PDFLATEX) $< || $(XELATEX) $<
            @[ '$<' == '$(findstring $<,$(PSOURCES))' ] && $(PDFLATEX) $< || $(XELATEX) $<

    #Dependencies
    $(XTARGETS): AGS.pdf BGS.pdf $(PTARGETS)
    $(PTARGETS): sf.tikz

    rebuild: clean all

    clear:
            @-rm -f *.aux *.log *.dat

    clean: clear
            @-rm -f $(PTARGETS) $(XTARGETS)

    distclean: clean

There are a few interesting things in here that weren’t obvious to me at first. For instance the wildcard property allows a list of some in put file type to be made into an output file of the same name. The target %.pdf: %.tex says that all output pdf files will require a corresponding tex file. With my particular case this isn’t very helpful as I’d like only some tex files rendered to pdf using one method and the rest rendered with a completely different method. Another question I had was “What’s the best way to separate the pdflatex and xelatex runs?”

The second issue could have just been two lists that would need to be updated every time a new figure was added, but that seemed to be a little inefficient and defeat the purpose of the makefiles’ automation. The solution comes from this line:

    $(filter-out $(PSOURCES), $(wildcard *.tex))

which takes the list $(wildcard *.tex) and filters (removes) any matches from $(PSOURCES). The wildcard command is necessary here to identify this list as iteratable for the %.tex target later on. As the sources for the pdflatex run are far fewer than those needing the default xelatex treatment, it’s much easier to filter the exceptions at this juncture. New all new files that don’t require special treatment also don’t require any thought when building them.

Separating the two build types became a bit more of a hassle. Initially, I’d implemented something like

    ifeq ($<,$(findstring $<,$(PSOURCES)))
        $(PDFLATEX) $<
        $(PDFLATEX) $<
    else
        $(XELATEX) $<
        $(XELATEX) $<
    endif

for the %.pdf: %.tex target.

    $(findstring $<,$(PSOURCES))

is tasked to find the current tex file during the expansion of the %.tex wildcard ($<) in the $(PSOURCES) list. If the file isn’t in the list this function will return an empty string and the ifeq ($<, "") test will return false. But this never worked and all files ended up trying to compile with pdflatex. Without internet and these kids still crawling over me, I gave up in disgust and stared out the window for the rest of the trip.

Later, back in civilisation, I pulled up the make documentation and found out why this method was bound to fail: make evaluates conditionals when it reads a makefile

A conditional causes part of a makefile to be obeyed or ignored depending on the values of variables. Conditionals can compare the value of one variable to another, or the value of a variable to a constant string. Conditionals control what make actually “sees” in the makefile, so they cannot be used to control shell commands at the time of execution.

In other words, a first pass of the file will test the ifeq condition, then choose that option for the second (and final) pass. When the second pass meets the wildcard expansion it just inserts whatever commands the first pass sends it and that’s all she wrote. So the solution I required here was to export the problem to the shell each wildcard expansion, test the name of the current tex file and only then invoke the correct compile tool.

All in all, this file does quite a good job. I haven’t seen many people discussing their figure build chain online, so if you have an interesting way of making your figures get in contact - I’d be interested to know what you do.


Comment

Converting 4K Content for Samsung UHD Televisions

Early adoption of new technologies is usually mired by complications, but what’s been shitting me lately is the trend manufacturers seem to have of pushing tech to market that doesn’t even support the main draw card of the product. I purchased a Samsung UA55F9000 UHD TV in December 2013: the month they were released, solely for the 4k capability. Fast forward a year later and I’m still fighting with the upper echelons of Samsung tech support concerning the absolute necessity of being able to use my 4k TV to play 4k content.

You see, the manual states the TV will play a .S4ud file: essentially a H.264 video and an AC3 audio stream multiplexed into a TS container. No firmware update since the TVs manufacture has actually allowed this. A second possibility is to use QFHD - four separate HD streams that are then joined together to yield a 4K stream. After quite a conversation with Samsung tech support it was elucidated that this is actually happening in-store when you see these TVs in action with those amazing demos.

So off I went and cut up a few sample files and tested it out - no dice. Two of the four segments were playing, the others were a jaggered mess. After more back and forth with the guys in tech, turns out the demo capability had been broken somewhere in an early firmware patch that was propagated through a number of subsequent versions. Thankfully, this debacle has been fixed and I’m now happily viewing my 4K content in all its glory. Downside is it’s still QFHD, not a nice single TS container.

There’s also a number of caveats with this method, for instance the files must be named UHD_demo_{a,b,c,d}_*.mp4{sub}, where a is the main file (top left), and the b-d files are sub streams.

Long story short, I’ve developed a few scripts to automatically take care of all of these problems. They could use a little cleaning up, and wifi connectivity problems at my new house has seen me jump back to Windows recently - so these are batch files rather than bash scripts. Shouldn’t be too hard to port them yourself though.

UHD 4K is 3840×2160 pixels, although Digital cinema 4K is 4096×2304. Before the resolution standards settled down, a plethora of content out there was letterboxed in some way, but 4096 pixels wide. Considering this, there are three scripts below for each of these three possibilities. Granted, I could set this up to find the resolution first but brevior saltare cum deformibus mulieribus est vita amiright? Call these snippets conv.bat or something to that effect, and run conv inputFile.mp4 (for example) at the CLI. The output will be four files titled with the convention above, where * will be inputFile (to keep things in order, these files output to a folder qfhd and the converted file is archived to a converted folder). Then, you’ll need to put them on a fast USB3 stick and plug it into the TV. You’ll only see the a file, run it and actually enjoy that massive hole you previously burnt into your wallet.

Ffmpeg is the only dependency you’ll need. Alter the rate factor in the script (-crf) to your liking; but I’ve found 15 works just fine.

3840×2160

    @echo off
    set input=%1
    set name=%input:~0,-4%
    echo Processing %input%
    echo Top left corner and sound
    ffmpeg -i %input% -vcodec libx264 -crf 15 -acodec ac3 -vf "crop=iw/2:ih/2:0:0" a.mp4
    move a.mp4 qfhd\UHD_demo_a_%name%.mp4
    echo Bottom left corner
    ffmpeg -i %input% -vcodec libx264 -crf 15 -an -vf "crop=iw/2:ih/2:0:ih/2" b.mp4
    move b.mp4 qfhd\UHD_demo_b_%name%.mp4sub
    echo Top right corner
    ffmpeg -i %input% -vcodec libx264 -crf 15 -an -vf "crop=iw/2:ih/2:iw/2:0" c.mp4
    move c.mp4 qfhd\UHD_demo_c_%name%.mp4sub
    echo Top right corner
    ffmpeg -i %input% -vcodec libx264 -crf 15 -an -vf "crop=iw/2:ih/2:iw/2:ih/2" d.mp4
    move d.mp4 qfhd\UHD_demo_d_%name%.mp4sub
    move %input% converted\
    echo Processing of %input% Complete
    @echo on

4096×2304

    @echo off
    set input=%1
    set name=%input:~0,-4%
    echo Processing %input%
    echo Top left corner and sound
    ffmpeg -i %input% -vcodec libx264 -crf 15 -acodec ac3 -vf "scale=3840:2160,crop=iw/2:ih/2:0:0" a.mp4
    move a.mp4 qfhd\UHD_demo_a_%name%.mp4
    echo Bottom left corner
    ffmpeg -i %input% -vcodec libx264 -crf 15 -an -vf "scale=3840:2160,crop=iw/2:ih/2:0:ih/2" b.mp4
    move b.mp4 qfhd\UHD_demo_b_%name%.mp4sub
    echo Top right corner
    ffmpeg -i %input% -vcodec libx264 -crf 15 -an -vf "scale=3840:2160,crop=iw/2:ih/2:iw/2:0" c.mp4
    move c.mp4 qfhd\UHD_demo_c_%name%.mp4sub
    echo Top right corner
    ffmpeg -i %input% -vcodec libx264 -crf 15 -an -vf "scale=3840:2160,crop=iw/2:ih/2:iw/2:ih/2" d.mp4
    move d.mp4 qfhd\UHD_demo_d_%name%.mp4sub
    move %input% converted\
    echo Processing of %input% Complete
    @echo on

4096×anything

    @echo off
    set input=%1
    set name=%input:~0,-4%
    echo Processing %input%
    echo Top left corner and sound
    ffmpeg -i %input% -vcodec libx264 -crf 15 -acodec ac3 -vf "scale=3840:-1,pad=3840:2160:0:(2160-ih)/2,crop=iw/2:ih/2:0:0" a.mp4
    move a.mp4 qfhd\UHD_demo_a_%name%.mp4
    echo Bottom left corner
    ffmpeg -i %input% -vcodec libx264 -crf 15 -an -vf "scale=3840:-1,pad=3840:2160:0:(2160-ih)/2,crop=iw/2:ih/2:0:ih/2" b.mp4
    move b.mp4 qfhd\UHD_demo_b_%name%.mp4sub
    echo Top right corner
    ffmpeg -i %input% -vcodec libx264 -crf 15 -an -vf "scale=3840:-1,pad=3840:2160:0:(2160-ih)/2,crop=iw/2:ih/2:iw/2:0" c.mp4
    move c.mp4 qfhd\UHD_demo_c_%name%.mp4sub
    echo Top right corner
    ffmpeg -i %input% -vcodec libx264 -crf 15 -an -vf "scale=3840:-1,pad=3840:2160:0:(2160-ih)/2,crop=iw/2:ih/2:iw/2:ih/2" d.mp4
    move d.mp4 qfhd\UHD_demo_d_%name%.mp4sub
    move %input% converted\
    echo Processing of %input% Complete
    @echo on
...
Continue Reading »

Coloured Tables for the Memoir Class

If you’re a fan of \(\LaTeX\) typography, you may have come across the Motion Mountain set of physics textbooks. This long term project has its own class file, which has been tweaked and honed over many years, has seen many of its pages grace many typography showcases (not to mention commercial successes and other recognitions). One particular stylistic choice that piqued my interest was the use of coloured rules inside the \table{} environment.

There colortbl package (available on ...

Continue Reading »