Ahh, making contact! Sometimes it's like how I'd imagine communing with aliens would be, don't you think? Aside from figuring out the reason to talk to them, there's also figuring out what exactly to say so you don't accidentally cause an issue and ruin your life. After all, in some ways they really DO speak a different language, and I swear some of the things they interpret are way off the mark of the intention.
But back to your situation. The good news is you already have some interactions going on. You may think passing paper is hardly an indication of affection, but everything has to start off somewhere. Since you obviously have a class with this guy, you already have plenty of potential conversation starters: homework ("How stupid is this?"), group work ("How stupid is this??") and basically anything else relating to the class. That's really the best place to start; once you have a solid foundation of idle classroom chat, it's easy enough to move to a different topic, and from there, to something you actually want to talk about.
Another thing that's helpful in moving beyond the "Can I borrow your pencil?"s is having a mutual friend. If you don't already have one but you're truly interested in this person, becoming friends with one of his friends is a good investment. Not only can they include you in conversations they're having with The Guy, but they can also assist you in becoming a topic of conversation with him.
Good luck :)
P.S. If you're like me and enjoy having a script, here's one to try out.
*After a particularly frustrating/stupid/difficult homework assignment*
You: Hey, did you finish last night's homework?
Him: No/Yes/We had homework?
You (if no): Me either! I mean, what was it even asking us to do....blahblahblah.
OR
You (if yes): What did you get for number _____ (or part one, etc.)?
OR
You (if 'we had homework?'): Yeah... if you grab the worksheet now you can do it before she/he collects it. You can borrow my paper if you want.
And viola! You just started a conversation.