I’ve been thinking about C++ build systems a lot recently.  I like programming in C++ a lot (relatively speaking), but I don’t think it’s news to anyone that the build system leaves a lot to be desired (dependency management, header + .cpp files, etc.).

It all started back when I started watching Handmade Hero.  Rather than using some complex build systems or even an IDE like Visual Studio, Casey Muratori just used a simple build.bat file to call the compiler.  I’m certainly no fan of batch as a language, but the simplicity of the build script along with the ease of understanding exactly how your program is compiled/linked were appealing.  He also basically just compiled 1 single .cpp file (which obviously simplifies things) – often called a “unity” or “single translation unit” build.

The next major piece came when watching videos for Jonathan Blow’s new “Jai” language.  One critical point Jon makes is that we probably should be writing our build scripts in the language we’re programming in – why bother learning some other (and often inferior) language just to build your program?

There is one point where I differ from Jon and Casey – I actually like using Visual Studio a lot.  I don’t necessarily like everything about it (notably the project files), but on the whole, I feel like the Visual Studio IDE makes me more productive compared to other alternative I’ve tried.

Anyway, it’s probably going to be some time before “Jai” or another language supplants C++.  So I was wondering…why not try and get most of the benefits mentioned above in C++?

Goals

Before we go any further, let me explicitly state some of the goals for my idea for a better C++ build system/IDE integration:

• Build via the command line with minimal options to remember.
• Build via Visual Studio.
• Not specify a bunch of files when building.
• Easily understand exactly how a program is built.
• Easily tweak how a program is built.
• Minimize dependencies on non-native tools (outside of the language, operating system, and Visual Studio).
• Not manually manage Visual Studio solution, project, etc. files.
• Easily find all files for a project within Visual Studio.
• Edit files within Visual Studio.
• Debug within Visual Studio.
• Simplicity.

Prior to work put in for this blog post, I played around some with the build.bat and unity build ideas, both from the command line and within Visual Studio (generating solution/project files manually using the IDE).

The Build Script

From the above, the simplest idea for meeting the “build” requirements is the build.bat idea.  Batch files are built into Windows, and the complexity needed to call the compiler isn’t so large that it drives me crazy using the batch language.

All we really need to do to generate such a build script is have a program write out a basic text file, with a few possible parameters tweaked.  Very simple – the text could be easily embedded as strings in a program.

The basic build.bat file Casey Muratori created for Handmade Hero can be used as a starting point.  On tweak I might like to make (haven’t fully settled on this yet) is automatically calling the VCVARSALL batch file in the build script – that way, I don’t have to worry about making sure the appropriate settings are always set.  Unfortunately, one downside to this from my experience is that sometimes you’ll get an error about a command/path being too long if you repeatedly run the build.bat file in the same command prompt Window.  I haven’t run into any problems with this in Visual Studio yet though.

Deciding exactly what compiler flags/settings or other parameters to include in the build.bat file is tricky.  I’m not sure exactly what I personally want yet, but it might be different from what Handmade Hero has.  For now, punting and just having a basic build script for building a debug build is fine.

I might want to actually wrap a lot of the options in the build script in a separate C++ program – that wouldn’t certainly make supporting advanced options/features easier.  Of course, that has the trade-off of being another program people would have to download/build.

On “Unity” Builds

Before moving on, just a few comments on “unity” or “single translation unit” builds.  They’re definitely an interesting idea, with a nice benefit of speeding up build times.

One thing I definitely like about the idea is that you only have to specify 1 file when passed to the compiler – no need to manually manage a bunch of them.

However, I’m not completely sold on the idea, as it sort of requires restructuring how you right code in an unconventional manner.  And while I have no concrete evidence for it, it seems like structuring code/builds this way might complicate re-use in large code bases where you want to use code across multiple projects.  I’d like to still write my code in a more “conventional” C++ way (many separate files, including the less-than-ideal header + .cpp files) to avoid risks in going all-in with a unity build.

How can we get around these conflicts?  Well, Jonathan Blow’s idea of specifying your build in your regular programming language provides some hints.  We can create a separate file that just #includes all of the .cpp files we want to compile – no need to write the rest of your code in an odd way.

This actually works pretty well from my testing, but it isn’t perfect.  In particular, I was hoping that this file with just a bunch of #includes could end with a file extension other than .cpp, but the Visual Studio compiler doesn’t seem to be able to compile files with other extensions (if you know a way around this, let me know).  The idea of not modifying your “other” .h/.cpp files almost may not be 100% feasible – from my testing, I’ve had to make a few modifications/re-order certain things – but the modifications have been relatively minimal so far to a point where it wouldn’t be much of a burden (or maybe no burden at all) if I wanted to return to a “conventional” C++ build model.

It might be nice to generate this .cpp file with just a bunch of #includes automatically one day, but there might be some challenges, and it’s certainly not at the top of my priority list.

Visual Studio Project Files

As mentioned above, I’ve manually created Visual Studio solution/project files for C++ projects using a build.bat + unity build approach.  But I don’t like having to manually add all files in my project or tweak various build settings.

How to solve this problem?  Well, for the most part, the Visual Studio solution/project files can basically just be copied, meaning that we can embed most of the text directly in a simple program.  There are some unique IDs in the files, but at least from my testing, re-using them doesn’t seem to be a problem.

What remains then?  Well, basically just getting a list of files/folders to include in the Visual Studio project and project filters files.  Recursively getting a list of files/folders in a directory…that doesn’t seem too hard, right?  While the Windows API for doing this isn’t super ideal in my opinion, it’s not too complicated.  So doing this should be pretty easy.

Let’s Do This…Well, Already Done…

I was planning to originally write this blog post before I went and tried to implement any of it.  But feeling extra motivated about this recently, I actually went and recently implemented a lot of the basics for what I want (as described above).  And it didn’t take that long – maybe around 3-5 hours total (at night after work).  Given a folder of code and using the quick program I wrote, I can automatically generate a Visual Studio solution/project and build.bat script that actually works (minus project-specific build tweaks).

It’s far from “complete” – I’ve only tested it on one specific project so far, the build.bat script is most definitely insufficient, and there aren’t a lot of options.  I’ll add improvements as I find I need them, but it’s “good enough” now to at least put out there.

The code is available here on GitHub.  I hope it will be helpful to some others out there.  If you have any suggestions, feel free to forward them to me.  Although, since I’m making the code itself public domain, you’re certainly more than welcome to make your own changes to suit your needs.