How to get punched in the face by Go’s Standard Library (and arguably my own stupidity)

Jun 24, 2014

Ever have one of those days? A “I forgot to add a semi colon to terminate a line” type of day? Yeah, me too. And today was one of them.

Many developers, when allowing user uploaded data, tend to trust the web browser’s headers a little too much. I prefer to use MIME Type sniffing on the actual binary stream. This gives me a greater level of assurance that I’m not allowing a user to upload a different type of file than the one I desire. Say an executable binary with which to do my application damage.

I’ve been using using the net/http/sniff/DetectContentType function in the Go standard library to do this type of detection on images.

func ValidateImageType(b []byte) (string) {
  m := http.DetectContentType(b)
  switch m { 
  case "image/jpeg":
    return "jpg"
  case "image/png":
    return "png"
  case "image/gif":
    return "gif"
   }
  return ""
}

It works wonderfully.

Today I figured I’d implement mp4 detection so as to detect videos. I snagged a video file off of my Note 2 and created a quick test.

func ValidateVideoType(b []byte) (string) {
  m := http.DetectContentType(b)
  switch m { 
  case "video/mp4":
    return "mp4"
   }
  return ""
}

This didn’t work and I headed over to the documentation. After reading the function’s signature, I was pretty sure I was using it correctly. Suspecting that my mp4 was corrupt I went and opened up the file in a hex editor.

mp4_hexadecimal

Comparing it to the living specification, it appeared correct. But just in case the file was in fact corrupt, I went out and found a test mp4 file from Apple.com. Unfortunately this file also failed.

Knowing that I was using the function correctly, and that the file was intact, I headed over to review the sniff.go source file.

package main

import (
	"encoding/binary"
	"bytes"
	"log"
	"os"
	"io"
	"net/http"

)

func main() {
	var total int
	var ext string
	buf := make([]byte, 512)

	fi, err := os.Open("./sample_mpeg4.mp4")
	if err != nil {
		panic(err)

	}
	defer fi.Close()

	for {
		n, err := fi.Read(buf)
		total = total + n
		if n == 0 {
			break

		}
		if err == io.EOF {
			break

		}
		if err != nil {
			break

		}

		if n == 512 {

			data := buf[:n]
			if len(data) < 8 {
				break

			}

			boxSize := int(binary.BigEndian.Uint32(data[:4]))
			if boxSize%4 != 0 {
				break

			}
			if len(data) < boxSize {
				break

			}
			if !bytes.Equal(data[4:8], []byte("ftyp")) {
				break

			}

			for st := 8; st < boxSize; st+=4 {
				if st == 12 {
					continue

				}
				seg := string(data[st : st+3])
				switch seg {
					case"mp4", "iso", "M4V", "M4P", "M4B":
					ext = "mp4"

				}
				if len(ext) > 1 {
					break

				}

			}

			m := http.DetectContentType(buf)
			log.Println("LOCAL: ", ext)
			log.Println("SOURCE: ", m)

		}

	}

}

Bizarrely, the local version found the desired file type. Wuuuuuuuu?!?

I wondered if it was my OSX environment so I spun up a linux box, downloaded the Go source, and dove into the source code.

Making sure I was at least starting off in the right direction, I ran the bash script go/src/all.bash and all the tests completed successfully. I went to the sniff_test.go file and found this test dataset commented out.

        //{"MP4 video", []byte("\x00\x00\x00\x18ftypmp42\x00\x00\x00\x00mp42isom<\x06t\xbfmdat"), "video/mp4"},

Hmmmmm, that looked interesting. So I uncommented the code and ran the tests again. THEY FAILED! It wasn’t what I wanted, but at this point I’ll take anything.

Knowing I was close I did a recursive grep for the function signature in question to see what was using it.

elvis@lts-linux:~/go/src$ grep -nr "mp4Sig" ./
./pkg/net/http/sniff.go:102:102//mp4Sig(0),
./pkg/net/http/sniff.go:164:type mp4Sig int
./pkg/net/http/sniff.go:166:func (mp4Sig) match(data []byte, firstNonWS int) string {
elvis@lts-linux:~/go/src$

And immediately, my face struck my palm and my head hit my keyboard.

I opened up sniff.go. The EXACT SAME FILE that I’d been staring at all day and uncommented line 102. Saved, quit, and upon running the tests again they all passed successfully.

So there you have it. An unimplemented feature and traversing old VC subtrees leads to a bad day. Make sure you actually review the source you are working on.