"Special? I don't know about that. I mean, I think other people probably would've responded differently, but that's because a lot of people aren't very open minded. I think I'm just different. I like that you like that, though. Obviously I'm not that great at hiding how different I am, so if you didn't like it... well..."

In other words, he's bizarrely and perhaps overly gratified that Ned thinks that he's special. People always tell him he's unusual, but it's almost never a compliment. Coming from Ned, though, it sounds like one. Coming from Ned, being strange doesn't seem like such a bad thing, even if Ned has his own oddities that he's been hiding his whole life.

"And like I said before, I won't tell anyone. I'm a loudmouth, but I'm not going to tell, okay? Just in case you were still worried."