Recent Posts

Uncategorized

How to get punched in the face by Go’s Standard Library (and arguably my own stupidity)

Posted on June 23, 2014

Ever have one of those days? A “I forgot to add a semi colon to terminate a line” type of day? Yeah, me too. And today was one of them.

Many developers, when allowing user uploaded data, tend to trust the web browser’s headers a little too much. I prefer to use MIME Type sniffing on the actual binary stream. This gives me a greater level of assurance that I’m not allowing a user to upload a different type of file than the one I desire. Say an executable binary with which to do my application damage.

I’ve been using using the net/http/sniff/DetectContentType function in the Go standard library to do this type of detection on images.

func ValidateImageType(b []byte) (string) {
  m := http.DetectContentType(b)
  switch m { 
  case "image/jpeg":
    return "jpg"
  case "image/png":
    return "png"
  case "image/gif":
    return "gif"
  }
  return ""
}

It works wonderfully.

Today I figured I’d implement mp4 detection so as to detect videos. I snagged a video file off of my Note 2 and created a quick test.

func ValidateVideoType(b []byte) (string) {
  m := http.DetectContentType(b)
  switch m { 
  case "video/mp4":
    return "mp4"
  }
  return ""
}

This didn’t work and I headed over to the documentation. After reading the function’s signature, I was pretty sure I was using it correctly. Suspecting that my mp4 was corrupt I went and opened up the file in a hex editor.

mp4_hexadecimal

Comparing it to the living specification, it appeared correct. But just in case the file was in fact corrupt, I went out and found a test mp4 file from Apple.com. Unfortunately this file also failed.

Knowing that I was using the function correctly, and that the file was intact, I headed over to review the sniff.go source file.

package main
 
import (
  "encoding/binary"
  "bytes"
  "log"
  "os"
  "io"
  "net/http"
)
 
func main() {
  var total int
  var ext string
  buf := make([]byte, 512)
 
  fi, err := os.Open("./sample_mpeg4.mp4")
  if err != nil {
    panic(err)
  }
  defer fi.Close()
 
  for {
    n, err := fi.Read(buf)
    total = total + n
    if n == 0 {
      break
    }
    if err == io.EOF {
      break
    }
    if err != nil {
      break
    }
 
    if n == 512 {
 
      data := buf[:n]
      if len(data) < 8 {
        break
      }
 
      boxSize := int(binary.BigEndian.Uint32(data[:4]))
      if boxSize%4 != 0 {
        break
      }
      if len(data) < boxSize {
        break
      }
      if !bytes.Equal(data[4:8], []byte("ftyp")) {
        break
      }
 
      for st := 8; st < boxSize; st+=4 {
        if st == 12 {
          continue
        }
        seg := string(data[st : st+3])
        switch seg {
          case"mp4", "iso", "M4V", "M4P", "M4B":
            ext = "mp4"
        }
        if len(ext) > 1 {
          break
        }
      }
 
      m := http.DetectContentType(buf)
      log.Println("LOCAL: ", ext)
      log.Println("SOURCE: ", m)
    }
  }
}

Bizarrely, the local version found the desired file type. Wuuuuuuuu?!?

I wondered if it was my OSX environment so I spun up a linux box, downloaded the Go source, and dove into the source code.

Making sure I was at least starting off in the right direction, I ran the bash script go/src/all.bash and all the tests completed successfully. I went to the sniff_test.go file and found this test dataset commented out.

        //{"MP4 video", []byte("\x00\x00\x00\x18ftypmp42\x00\x00\x00\x00mp42isom&lt;\x06t\xbfmdat"), "video/mp4"},

Hmmmmm, that looked interesting. So I uncommented the code and ran the tests again. THEY FAILED! It wasn’t what I wanted, but at this point I’ll take anything.

Knowing I was close I did a recursive grep for the function signature in question to see what was using it.

backwardselvis@lts-linux:~/go/src$ grep -nr "mp4Sig" ./
./pkg/net/http/sniff.go:102:	//mp4Sig(0),
./pkg/net/http/sniff.go:164:type mp4Sig int
./pkg/net/http/sniff.go:166:func (mp4Sig) match(data []byte, firstNonWS int) string {
backwardselvis@lts-linux:~/go/src$

And immediately, my face struck my palm and my head hit my keyboard.

I opened up sniff.go. The EXACT SAME FILE that I’d been staring at all day and uncommented line 102. Saved, quit, and upon running the tests again they all passed successfully.

And despite this residual lame duck feeling, I’d argue that it is better to chase down the truth then let something unknown slip by.

:peace

How To: Remove index.php from owncloud url path

Posted on April 16, 2013

When installing owncloud on your own server, the path looks pretty crappy.

I’d like to change it from this.

https://example.com/owncloud/index.php/apps/files

To this.

https://example.com/owncloud/apps/files

To do this, you need to make sure mod_rewrite is enabled and change the .htaccess in the ROOT directory (example: /var/www/html/owncloud/)

Change it from this.

...snip...
 
RewriteEngine on
RewriteRule .* - [env=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteRule ^.well-known/host-meta /public.php?service=host-meta [QSA,L]
RewriteRule ^.well-known/host-meta.json /public.php?service=host-meta-json [QSA,L]
RewriteRule ^.well-known/carddav /remote.php/carddav/ [R]
RewriteRule ^.well-known/caldav /remote.php/caldav/ [R]
RewriteRule ^apps/calendar/caldav.php remote.php/caldav/ [QSA,L]
RewriteRule ^apps/contacts/carddav.php remote.php/carddav/ [QSA,L]
RewriteRule ^apps/([^/]*)/(.*\.(css|php))$ index.php?app=$1&amp;getfile=$2 [QSA,L]
RewriteRule ^remote/(.*) remote.php [QSA,L]
 
...snip...

Add the one line.

...snip...
 
RewriteEngine on
RewriteRule .* - [env=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteRule ^.well-known/host-meta /public.php?service=host-meta [QSA,L]
RewriteRule ^.well-known/host-meta.json /public.php?service=host-meta-json [QSA,L]
RewriteRule ^.well-known/carddav /remote.php/carddav/ [R]
RewriteRule ^.well-known/caldav /remote.php/caldav/ [R]
RewriteRule ^apps/calendar/caldav.php remote.php/caldav/ [QSA,L]
RewriteRule ^apps/contacts/carddav.php remote.php/carddav/ [QSA,L]
RewriteRule ^apps/([^/]*)/(.*\.(css|php))$ index.php?app=$1&amp;getfile=$2 [QSA,L]
RewriteRule ^remote/(.*) remote.php [QSA,L]
 
RewriteRule ^index.php/(.*) $1 [L]
 
...snip...

Then you will need to restart apache.

You Should Support Clintmichigan’s Album “Coeur d’Alene”

My friend Clint Asay (and relative through marriage) has released his new album titled “Couer d’alene” and I’ve been having a great time listening to it. The album is awesome and I’ve included several tracks (the title track and the reprise are beautiful) into my programming playlists to keep me wee fingers coding during the long stretches.

A little background:

Clint scraped together the cash to record this, then turned around and self mixed and released the album. That is amazing! As someone who also creates through writing and music, I have to give anyone who completes something a giant high-five. That stuff is hard. And don’t get me started about releasing something personal like Clint has done. You are exposing your heart and mind to the internet and the world, which tends to be an extremely cruel mistress.

I’ve linked to his music, please stop by and support him when you get a second.

http://clintmichigan.bandcamp.com/

-peace
jared

Oracle 11g Enterprise Manager Error: Invalid Connection Pool. ERROR = ORA-28000: the account is locked

Problem:

You are trying to access the Oracle Enterprise Manager and are getting the following error.

Error:

Invalid Connection Pool. ERROR = ORA-28000: the account is locked

Explanation:

In my instance, the password for the SYSMAN database user was different than the emctl SYSMAN password. Every refresh I could see the errors messages indicating it was trying to connect, and then the account would become locked as the connection failed.

1) you need to track down the sysman/log directory

cd /u01/app/oracle/product/11.2.0/dbhome_1/example.com_orcl/sysman/log

2) tail the following log file

tail -f emoms.log

3) refresh your web browser and you should see the following error message pop up

Invalid Connection Pool. ERROR = ORA-28000: the account is locked

4) as the oracle user, login as sysdba

sqlplus / as sysdba

5) unlock the account

alter user sysman account unlock;

6) set a new password (don’t do this on production unless you have to)

alter user sysman identified by 'YOUR_NEW_PASSWORD';

7) now go to your command prompt and type

emctl setpasswd dbconsole

8) when it asks for your password, enter your new password

Please enter new repository password: YOUR_NEW_PASSWORD

9) close your web browser, open it and head to your em link

example.com:1160/em/console

10) enter following

User: sysman
Password: YOUR_NEW_PASSWORD

Older Posts