First, this is an excellent question, and there are many different opinions regarding the answer.  The reason for the many opinions is because the scriptures do not give us a "thus saith the LORD" on this subject.  However, based on scriptures, we can identify where Elijah is NOT.

John 3:13  “And no man hath ascended up to heaven, but he that came down from heaven, even the Son of man which is in heaven.”

2 Kings 2:11  And it came to pass, as they still went on, and talked, that, behold, there appeared a chariot of fire, and horses of fire, and parted them both asunder; and Elijah went up by a whirlwind into heaven.

It appears that these two scriptures conflict.  The old testament says Elijah was taken up by a whirlwind into heaven, but the new testament says no man has ascended into heaven.  How do we harmonize this?

If we look at the definition of heaven, we see it can have multiple meanings.  The following are the definitions from Strong's concordance of the word translated "heaven" from the Hebrew and Greek in the John and 2 Kings scriptures above:

08064 shamayim

            from an unused root meaning to be lofty;

            AV-heaven 398, air 21

1) heaven, heavens, sky

1a) visible heavens, sky

1a1) as abode of the stars

1a2) as the visible universe, the sky, atmosphere, etc

1b) Heaven (as the abode of God)

 

3772 ouranos oo-ran-os'

            perhaps from the same as 3735 (through the idea of elevation); the sky;

            AV-heaven 268, air 10, sky 5

1) the vaulted expanse of the sky with all things visible in it

1a) the universe, the world

1b) the aerial heavens or sky, the region where the clouds and the tempests gather, and where thunder and lightning are produced

1c) the sidereal or starry heavens

2) the region above the sidereal heavens, the seat of order of things eternal and consummately perfect where God dwells and other heavenly beings

So 'heaven'  can simply mean the sky, or it can mean God's abode.  Given these possible definitions, we can determine since John 3:13 states that no man hath ascended up to heaven (God's abode, to sit at the right hand of the Father, Acts 2:32-33) except Jesus, then Elijah can not have been taken to God's abode.  However Elijah was taken up into the sky (heaven) by the whirlwind.  Where he went after that is not given to us.  Some say he fell asleep in death awaiting the resurrection, some say he may have been preserved alive in the Garden of Eden (which contains the tree of life, and God put guards to keep it so that Adam could not reenter).  Any reasoning on the subject is purely a matter of opinion, but one thing is certain, Elijah is not in God's abode.