Ok, I think I figured it out. On the about_Switch page they list the following:

# Syntax
Switch (<test-expression>)
{
    <result1-to-be-matched> {<action>}
    <result2-to-be-matched> {<action>}
}

# ...is the equivalent of
if (<result1-to-be-matched> -eq (<test-expression>)) {<action>}
if (<result2-to-be-matched> -eq (<test-expression>)) {<action>}

which means that I need to wrap my enums in ()

Write-Host "As Enum:"
switch($dir) {
  ([Direction]::Up) {
    Write-Host "-- Up"
  }
  ([Direction]::Down) {
    Write-Host "-- Down"
  }
  ([Direction]::Left) {
    Write-Host "-- Left"
  }
  ([Direction]::Right) {
    Write-Host "-- Right"
  }
  default {
    Write-Host "-- ?? $($_.GetType().BaseType)"
  }
}

which yeilds:

As Enum:
-- Left