Searching for NULL: Making hg and git recognize text as text.

Recently, my boss sent me her version of the LaTeX source for the paper we're working on together, which I then proceeded to enter into the Mercurial repository. (Why she herself isn't using Hg is a story for another time, but it's also not that hard to do her commits for her, I just make sure to track which commit is the parent.) I wanted to see what changes she had made, so I did "hg diff" and was informed that I was comparing a binary file. I tried both "traditional" diff and git-diff and both attempted to handle the file as binary. However, I was still able to open the file in a text editor without any problems. I had read various places that tbe BOM on UTF-16 could cause problems and so I made sure that I was saving the file as UTF-8 without BOM (UTF-8 is a must for us since we had some German examples with Umlauts and Esszet ß). I was growing increasingly frustrated and was about to just damn the torpedoes and commit anyway -- the diffs are calculated and stored the same way, regardless of whether or not the file is binary; only the display is adjusted -- when I read that the presence of the NULL byte was one of the ways that a file is determined to be binary. So I found a way to remove NULL and everything worked as desired. Still, I was kinda curious about where exactly there were NULLs in this file. I so searched a bit more and found a way to grep them and it turns out that NULL was being inserted as the very last byte in the file. Whether this was an issue with the text editor on my boss's end or a by-product of encoding 8-bit formats into 7-bits for email -- especially given that her emails are usually encoded in Western ISO 8859-1, which means that two different 8-bit formats were being encoded into 7-bit ASCII -- I don't know. Anyway, here's a summary of ways to deal with NULL in plain text files.

Diffs for LaTeX in Version Control

I use Mercurial to track changes in my LaTeX documents. While there's latexdiff and the older texdiff to produce a conveniently marked up difference document (like Track Changes in Word or OpenOffice), those depend on having both versions available at the same time -- a bit of a pain when using version control. You have to update to the old version, rename it, update to the new version and then compare them -- far from trivial for documents with many files. Well, now there's a convenient utility to do that for you with Mercurial and Git, scm-latexdiff. Check it out.

Best Software Practices for Science / Scientific Computing

I highly recommend this:

http://software-carpentry.org/2012/10/best-practices-for-scientific-computing/

The most important tips that I would like to see my own group use more of are (abridged from link):

1. version control (with modern DVCS, there's no reason not to have even your little scripts under version control)
2. automate repetitive tasks and use the computer to record (command) history (I think these two really go hand in hand with each other and with #1)
3. Don’t repeat yourself (or others).
1. Every piece of data must have a single authoritative representation in the system. 
2. Code should be modularized rather than copied and pasted.
3. Re-use code instead of rewriting it

Copy and pasting leads to the code blocs getting out of sync, i.e., inconsistent analyses. And I can't tell you the number of times I've found a mess of  inconsistent scripts and literally hundreds of gigabytes of duplicated data, with no single copy "authoritative". (Luckily, in the last case, SHA1 revealed that the individual data files were identical; however, each set had a slightly different collection of files...). And if you do right from the beginning, it doesn't even take that much time!

I think all of this can be summarized into two points:

1. Use version control 
2. Use good coding/documentation practices, including modularity