Recent Posts

Uncategorized

Understanding When To Use Channels Or Mutexes In Go

Problem Domain

Community “Share Memory By Communicating”
Interpretation “CHANNEL ALL TEH THINGZ!”

Because many developers come from backgrounds (Php, Ruby, Perl, Python) where unlike Go, concurrency is not a first class citizen, they struggle when learning about it. But they apply themselves and take the time to dig into Go’s concurrency model. And just as they finally feel they’ve come to grips, something painful happens. The developer decides to use their new found super-power (goroutines + channels) for absolutely everything and it suddenly becomes an anti-pattern.

External Confirmation

I’m not the only one who has noticed. In the discussion below Andrew warns about the overuse of channels to keep state in goroutines.

“Hey, concurrency is great, but all this needs is a mutex”

Also, in the following interview between Andrew and Dave. Andrew asks Dave to give an example of when he wrote some bad code and Dave mentions specifically about using too many channels.

“…and I tried to use channels for everything, if you want to talk about worst code…”

Confession

When I created BadActor, I needed it to help me in identify malicious activities against my systems so that I could respond to them. I set out to solve my problem (and gain proficiency with Go) and I envisioned this grand design where everything was a goroutine and communication would happen only through channels and I would never need another mutex ever again. (Hah!)

It sounded great in theory. But the composition was cumbersome and holy crap did the Benchmarks suck. I didn’t think too much of this. I thought that it was just me misunderstanding on how to use channels in Go. So I hunkered down and persisted with my design for a while until I learned better. When I realized my error, I rewrote large portions of my codebase.

Coding Horror

Below is an old commit of BadActor‘s where originally I had architected the exact model that Andrew and Dave spoke about. Many goroutines, each with their own channel(s), combined with message types to define the behavior of the communication. I leave them here so you can hopefully learn from my mistake.

Here is the Actor’s goroutine with its open channel.

func (a *Actor) Run() {
	go func() {
		for {
			select {
			case in := <-a.Incoming:
				a.switchBoard(in)
			default:
				a.overwatch()
			}
		}
	}()
}

And here is the awkward Type checking that Andrew warns about.

func (a *Actor) switchBoard(in *Incoming) {
	switch in.Type {
	case KEEP_ALIVE:
		err := a.rebaseAll()
		in.Outgoing <- &Outgoing{Error: err}
	case INFRACTION:
		err := a.infraction(in.RuleName)
		in.Outgoing <- &Outgoing{Error: err}
	case CREATE_INFRACTION:
		err := a.createInfraction(in.Infraction)
		in.Outgoing <- &Outgoing{Error: err}
	case STRIKES:
		total := a.strikes(in.RuleName)
		in.Outgoing <- &Outgoing{Message: strconv.Itoa(total)}
	case IS_JAILED:
		res := a.isJailed()
		in.Outgoing <- &Outgoing{Message: strconv.FormatBool(res), Error: nil}
	case IS_JAILED_FOR:
		res := a.isJailedFor(in.RuleName)
		in.Outgoing <- &Outgoing{Message: strconv.FormatBool(res), Error: nil}
	}
}

Practical Example – Mutex

If I synchronously need to solve for (x) and I just have one input, then I protect it with a mutex. I do not implement channels and goroutines just to protect the state of some asset.

Example: A cache takes one key (k) to gather one value (x).

Below is a very naive cache and http server I created to help you understand.

We start by creating our main.go file.

$ vi main.go

Then we import the needing packages (thanks goimport).

package main
 
import (
  "fmt"
  "html"
  "log" 
  "net/http"
  "net/url"
  "sync"
)

Next we define our NaiveCache and define some getter/setter methods.

//
// NaiveCache
//
 
var c *NaiveCache
 
type NaiveCache struct {
  mu      sync.Mutex
  storage map[string]string
}
 
func (c *NaiveCache) Value(k string) (string, error) {
  var v string
  c.mu.Lock()
  defer c.mu.Unlock()
  if v, ok := c.storage[k]; ok {
    return v, nil
  }
  return v, fmt.Errorf("Value Not Found")
}
 
func (c *NaiveCache) SetValue(k string, v string) {
  c.mu.Lock()
  c.storage[k] = v
  c.mu.Unlock()
  return
}

Here we implement the server giving it a handler and two helper functions, get() and post().

//
// Server
//
func NaiveCacheHandler(w http.ResponseWriter, r *http.Request) {
  switch r.Method {
  case "GET":
    get(w, r)
  case "POST":
    post(w, r)
  }
}
 
func get(w http.ResponseWriter, r *http.Request) {
  id, err := url.QueryUnescape(r.URL.Query().Get("key"))
  if err != nil {
    w.WriteHeader(http.StatusBadRequest)
    b := []byte(fmt.Sprintf("%v StatusBadRequest", http.StatusBadRequest))
    w.Write(b)
    return
  }
 
  v, err := c.Value(id)
  if err != nil {
    w.WriteHeader(http.StatusNotFound)
    b := []byte(fmt.Sprintf("%v StatusNotFound", http.StatusNotFound))
    w.Write(b)
    return
  }
 
  w.Write([]byte(html.EscapeString(v)))
  return
}
 
func post(w http.ResponseWriter, r *http.Request) {
  k := html.EscapeString(r.FormValue("key"))
  v := html.EscapeString(r.FormValue("value"))
  if len(k) == 0 || len(v) == 0 {
    w.WriteHeader(http.StatusBadRequest)
    b := []byte(fmt.Sprintf("%v StatusBadRequest", http.StatusBadRequest))
    w.Write(b)
    return
  }
 
  c.SetValue(k, v)
  w.WriteHeader(http.StatusOK)
  b := []byte(fmt.Sprintf("%v StatusOK", http.StatusOK))
  w.Write(b)
  return
}

Ending with the main function which gets called upon running our compiled application.

//
// Main
//
func main() {
  // init the cache
  c = &NaiveCache{storage: make(map[string]string)}
 
  // start the server
  http.HandleFunc("/cache", NaiveCacheHandler)
  log.Fatal(http.ListenAndServe(":9090", nil))
}

Run the application.

$ go run main.go

And submit some curl requests to see it work.

$ curl -X GET http://localhost:9090/cache\?key\=somekey
404 StatusNotFound
$ curl --data "key=somekey&value=somevalue" http://localhost:9090/cache
200 StatusOK
$ curl -X GET http://localhost:9090/cache\?key\=somekey
somevalue
$ curl -X GET http://localhost:9090/cache\?key\=somekey
somevalue
$

Closing

Don’t worry about failing, everyone learns when they fail so go ahead and dive in and start building something in Go. But remember, if you feel like you are fighting your design and you notice that you are primarly using channels to protect state, then hey, concurrency is great, but all you need is a mutex.

How to get punched in the face by Go’s Standard Library (and arguably my own stupidity)

Posted on June 23, 2014

Ever have one of those days? A “I forgot to add a semi colon to terminate a line” type of day? Yeah, me too. And today was one of them.

Many developers, when allowing user uploaded data, tend to trust the web browser’s headers a little too much. I prefer to use MIME Type sniffing on the actual binary stream. This gives me a greater level of assurance that I’m not allowing a user to upload a different type of file than the one I desire. Say an executable binary with which to do my application damage.

I’ve been using using the net/http/sniff/DetectContentType function in the Go standard library to do this type of detection on images.

func ValidateImageType(b []byte) (string) {
  m := http.DetectContentType(b)
  switch m { 
  case "image/jpeg":
    return "jpg"
  case "image/png":
    return "png"
  case "image/gif":
    return "gif"
  }
  return ""
}

It works wonderfully.

Today I figured I’d implement mp4 detection so as to detect videos. I snagged a video file off of my Note 2 and created a quick test.

func ValidateVideoType(b []byte) (string) {
  m := http.DetectContentType(b)
  switch m { 
  case "video/mp4":
    return "mp4"
  }
  return ""
}

This didn’t work and I headed over to the documentation. After reading the function’s signature, I was pretty sure I was using it correctly. Suspecting that my mp4 was corrupt I went and opened up the file in a hex editor.

mp4_hexadecimal

Comparing it to the living specification, it appeared correct. But just in case the file was in fact corrupt, I went out and found a test mp4 file from Apple.com. Unfortunately this file also failed.

Knowing that I was using the function correctly, and that the file was intact, I headed over to review the sniff.go source file.

package main
 
import (
  "encoding/binary"
  "bytes"
  "log"
  "os"
  "io"
  "net/http"
)
 
func main() {
  var total int
  var ext string
  buf := make([]byte, 512)
 
  fi, err := os.Open("./sample_mpeg4.mp4")
  if err != nil {
    panic(err)
  }
  defer fi.Close()
 
  for {
    n, err := fi.Read(buf)
    total = total + n
    if n == 0 {
      break
    }
    if err == io.EOF {
      break
    }
    if err != nil {
      break
    }
 
    if n == 512 {
 
      data := buf[:n]
      if len(data) &lt; 8 {
        break
      }
 
      boxSize := int(binary.BigEndian.Uint32(data[:4]))
      if boxSize%4 != 0 {
        break
      }
      if len(data) &lt; boxSize {
        break
      }
      if !bytes.Equal(data[4:8], []byte("ftyp")) {
        break
      }
 
      for st := 8; st &lt; boxSize; st+=4 {
        if st == 12 {
          continue
        }
        seg := string(data[st : st+3])
        switch seg {
          case"mp4", "iso", "M4V", "M4P", "M4B":
            ext = "mp4"
        }
        if len(ext) &gt; 1 {
          break
        }
      }
 
      m := http.DetectContentType(buf)
      log.Println("LOCAL: ", ext)
      log.Println("SOURCE: ", m)
    }
  }
}

Bizarrely, the local version found the desired file type. Wuuuuuuuu?!?

I wondered if it was my OSX environment so I spun up a linux box, downloaded the Go source, and dove into the source code.

Making sure I was at least starting off in the right direction, I ran the bash script go/src/all.bash and all the tests completed successfully. I went to the sniff_test.go file and found this test dataset commented out.

        //{"MP4 video", []byte("\x00\x00\x00\x18ftypmp42\x00\x00\x00\x00mp42isom&lt;\x06t\xbfmdat"), "video/mp4"},

Hmmmmm, that looked interesting. So I uncommented the code and ran the tests again. THEY FAILED! It wasn’t what I wanted, but at this point I’ll take anything.

Knowing I was close I did a recursive grep for the function signature in question to see what was using it.

backwardselvis@lts-linux:~/go/src$ grep -nr "mp4Sig" ./
./pkg/net/http/sniff.go:102:	//mp4Sig(0),
./pkg/net/http/sniff.go:164:type mp4Sig int
./pkg/net/http/sniff.go:166:func (mp4Sig) match(data []byte, firstNonWS int) string {
backwardselvis@lts-linux:~/go/src$

And immediately, my face struck my palm and my head hit my keyboard.

I opened up sniff.go. The EXACT SAME FILE that I’d been staring at all day and uncommented line 102. Saved, quit, and upon running the tests again they all passed successfully.

So there you have it. An unimplemented feature and traversing old VC subtrees leads to a bad day. Make sure you actually review the source you are working on.

:peace

You Should Support Clintmichigan’s Album “Coeur d’Alene”

My friend Clint Asay (and relative through marriage) has released his new album titled “Couer d’alene” and I’ve been having a great time listening to it. The album is awesome and I’ve included several tracks (the title track and the reprise are beautiful) into my programming playlists to keep me wee fingers coding during the long stretches.

A little background:

Clint scraped together the cash to record this, then turned around and self mixed and released the album. That is amazing! As someone who also creates through writing and music, I have to give anyone who completes something a giant high-five. That stuff is hard. And don’t get me started about releasing something personal like Clint has done. You are exposing your heart and mind to the internet and the world, which tends to be an extremely cruel mistress.

I’ve linked to his music, please stop by and support him when you get a second.

http://clintmichigan.bandcamp.com/

-peace
jared

Oracle 11g Enterprise Manager Error: Invalid Connection Pool. ERROR = ORA-28000: the account is locked

Problem:

You are trying to access the Oracle Enterprise Manager and are getting the following error.

Error:

Invalid Connection Pool. ERROR = ORA-28000: the account is locked

Explanation:

In my instance, the password for the SYSMAN database user was different than the emctl SYSMAN password. Every refresh I could see the errors messages indicating it was trying to connect, and then the account would become locked as the connection failed.

1) you need to track down the sysman/log directory

cd /u01/app/oracle/product/11.2.0/dbhome_1/example.com_orcl/sysman/log

2) tail the following log file

tail -f emoms.log

3) refresh your web browser and you should see the following error message pop up

Invalid Connection Pool. ERROR = ORA-28000: the account is locked

4) as the oracle user, login as sysdba

sqlplus / as sysdba

5) unlock the account

alter user sysman account unlock;

6) set a new password (don’t do this on production unless you have to)

alter user sysman identified by 'YOUR_NEW_PASSWORD';

7) now go to your command prompt and type

emctl setpasswd dbconsole

8) when it asks for your password, enter your new password

Please enter new repository password: YOUR_NEW_PASSWORD

9) close your web browser, open it and head to your em link

example.com:1160/em/console

10) enter following

User: sysman
Password: YOUR_NEW_PASSWORD

Older Posts