The most difficult language to learn is the one you have no desire to. This is why high school language classes (in the US) tend to be a joke. They are required by guidelines, no one wants to be there, and the teachers generally suck. I've been to 3 different high schools, in 3 different areas, and they are pretty much all the same.
I took Latin in high school and college so I have a pretty good grasp on that. Contrary to popular belief (also in this thread) being fluent in Latin doesn't help that much with Romantic languages because Latin isn't spoken. I'm learning Spanish now and the only part that Latin helps with is reading.
The biggest problem that I have with learning Spanish currently is that I don't have anyone to speak with. My girlfriend is Colombian but she gets too frustrated to teach me. I try to learn on my own and talk to her, and I get some things right and can get around (probably), but she speaks so darn quickly that I have to ask her to repeat herself a lot.
I have never really cared about Spanish because I'm lazy and I think that Mexican-Spanish is ugly sounding, but now I want/need to learn since her entire family speaks very rough or no English and it would be good to be able to communicate with her family if I want to stick around for a while.