I learned english by playing video games and watching english movies (good old pirating times, when we had only 128k download via two isdn connections).
Anime and manga help me in the same way, with this I get a constant stream of english thrown at me, which ultimately leads to new vocabulary.
And I also started to pick up some japanese too.
It's just like some mentioned before, the more you are exposed to a foreign language, the more you will be able to understand.
Writing and speaking on the other hand will need practice, I get mad at myself on a daily basis, just because I know what I want to say in english, with the right vocabulary at hand, but simply not knowing how to put the words in a grammatically correct order >.<