A directory is, semantically speaking, a mapping from file name to inode. This is how the directory tree abstraction is designed, corresponding to the interface between applications and filesystems. Applications can designate files by name and enumerate the list of files in a directory, and each file has a unique designator which is called an “inode”.
How this semantics is implemented depends on the filesystem type. It's up to each filesystem how the directory is encoded. In most Unix filesystems, a directory is a mapping from filenames to inode numbers, and there's a separate table mapping inode numbers to inode data. (The inode data contains file metadata such as permissions and timestamps, the location of file contents, etc.) The mapping can be a list, a hash table, a tree...
You can't see this mapping with Vim. Vim doesn't show the storage area that represents the directory. Linux, like many other modern Unix systems, doesn't allow applications to see the directory representation directly. Directories act like ordinary files when it comes to their directory entry and to their metadata, but not when it comes to their content. Applications read from ordinary file with system calls such as open, read, write, close; for directories there are other system calls: opendir, readdir, closedir, and modifying a directory is done by creating, moving and deleting files. An application like cat uses open, read, close to read a file's content; an application like ls uses opendir, readdir, closedir to read a directory's content. Vim normally works like cat to read a file's content, but if you ask it to open a directory, it works like ls and prints the data in a nicely-formatted way.
If you want to see what a directory looks like under the hood, you can use a tool such as debugfs for ext2/ext3/ext4. Make sure you don't modify anything! A tool like debugfs bypasses the filesystem and can destroy it utterly. The ext2/ext3/ext4 debugfs is safe because it's in read-only mode unless you explicitly allow writing through a command line option.
# debugfs /dev/root
debugfs 1.42.12 (29-Aug-2014)
debugfs: dump / /tmp/root.bin
debugfs: quit
# od -t x1 /tmp/root.bin
You'll see the names of the directory entries in / amidst a bunch of other characters, some unprintable. To make sense of it, you'd need to know the details of the filesystem format.
vimis not showing you the internal directory structure. When you dovim directory/, it behaves like a file browser allowing you to explore the file system. To see the inode number, runls -li.