If you are using that horrid combination of indexOf, split and substring @Dai is talking about, something similar to the following will do it.
var start = this.value.indexOf("[list]"),
ending = this.value.indexOf("[/list]"),
lines = this.value.substring(start + 7, ending - 1).split("\n");
//perform loop on `lines`:
// "<li>" + lines[i] + "</li>"
Essentially, this will get the start position of [list] and [/list] and then get the lines in between. You'll notice that in the substring(), I say start+7 and ending-1. This can be explained by the fact that the start position of your actual lines will be at the index of [list] + the actual length of the string [list] + the enter character. The end subtracts one to remove that enter character. A better way to do this would be to simply add six to start instead of seven, subtract zero and then just get rid of any empty array values, however, either way is fine depending on the exact syntax you're requiring for your users.
Keep in mind that the lines variable will hold all of the lines entered between the two list bbcodes meaning you have to loop through them and then wrap them with <li></li> tags. Also, I suggest validating multiple things such as making sure that start > ending and performing this multiple times so you know every occurrence of [list] instead of just the first.
indexOf/split, andsubstring?split? That seems relevant here.splitcauses the allocation and copying of string contents, which can have a considerable performance impact when processing lots of strings (such as in a logfile parser), and then there's the cost of GC to consider too. Any task that usessplitcan be performed using a regex or state-machine with considerably better performance.