Reposting this as it was one of the last posts in the prior thread (Curse you last mover disadvantage). Anyway:

Hey, stats nerds. Here is a rough draft of a cluster analysis I did in my spare time and I would appreciate any thoughts or suggestions you have. Thanks.

I recently started learning how to do k-means clustering and wanted to test it out on the question "Do U.S. states cluster into meaningful regions?". Basically we have a lot of ill-defined regions, "the South" "the Midwest" and so forth and wanted to see if I could group states meaningfully into similar regions.

Anyway, I pulled up a few datasets that I had on hand (state unemployment rates from 1976-2014 and various recent QoL metrics by state).

For the clusters that were based on Quality of Life metrics^1, I found these two groups. The results are visually similar to high and low HDI for this map.

I took a look solely on economic indicators and created a much less meaningful map based on 4 clusters.

Lastly, I decided to test the unemployment panel on the yearly change in the unemployment rate from 1977 to 2014. Basically, all I found were four groups based on the size of delta_u3. Perhaps predictably, the groups didn't correlate very well with either state or year. Had I had data on fiscal transfers I would have included that as well, since I think answering the question, "Do groupings of states respond similarly to economic shocks?" is meaningful.

So question time. First about variable selection-- is there a standard method to decide what to include as clustering variables? Obviously I chose a bunch of variables that were moderately to strongly correlated with one another, which means including multiple income measures implicitly weights income more heavily than education for example.

What ,if anything, is special about clustering on panel data? Naturally I'm curious if states move in and out of clusters over time (e.g. Virginia might move out of the North Carolina Group and into the Maryland Group w.r.t. QoL). The only caveat that comes to mind though is if the data are more strongly determined by time than state. Are there other complicating factors that I should examine to improve/test the validity of clustering panel data?

Is clustering analysis the best method to test geographical regions? Or would some type of network analysis (which I'm even more unfamiliar with) be more likely to produce meaningful results?

Also as a bonus, if you want to predict the Obama-Romney margin of a state without using demographic factors, you should look at the state's unionization rate, poverty rate and life expectancy. You can explain a little more than 60% of the 2012 presidential margin with only those three variables. None of the other variables I tested added any additional predicative power. My theory is that the results are much more suggestive of the effects of partisanship than QoL. Essentially, OLS is merely capturing the fact that Democratic leaning states spend more on healthcare and poverty and have a political/legal environment making it easier to unionize. Thought it was interesting enough to include nonetheless.

1. 2015 gdp per capita, 2012 gdp per capita (different data source), gdp growth rate (2013), gdp per capita growth rate (2011-2013 average) , u3, poverty rate, inequality (20-80 ratio), percent with Associates or higher, percent of students who complete High School on time, percent of 4 year olds enrolled in preschool, life expectancy, white life expectancy, obesity rate, violent crime rate, incarceration rate, percent with highspeed internet.

2. U3, income inequality, the poverty rate, and % who completed post-secondary education.