I found there was no difference in performance between the 256 MB and 512MB models for WordPress. So I have put the website back onto the 256MB RasPi. I found more difference in removing useless plugins and installing WP Fast Cache. I also have Ban Hammer and Revision Control installed.
A real performance gain was by using the Panda Board. It was 2/3rd faster at everything. Even before it was fully finished. Power usage was very similar. I was able to get WP Super Cache working on the Panda Board. So now I know the path to get it working on the website. I will be doing that soon.
It is about time for another backup of the website so I will try to get super cache working after I get it backed up. The last time I tried to get Super Cache working it did the website in and I had to restore from backup!