This module provides a general purpose lexer machine.

The user add actions callbacks to the lexer. The longest pattern matched
prefix wins. In case of ties, the pattern with the highest precedence
wins.

The user prepare a backend to use with a lexer. A backend take a list of
action to compile its core. A dedicated stream is used to maintain a buffer,
and read all kind of handles. The read buffer must be sized to fit the
longest possible token.

	let actions: []lex::action = [];
	defer free(actions);

	append(actions, lex::action {
		expr = `"([^\\"]|(\\.))*"`,
		cb = &literal,
		name = "LIT_STR",
		...
	})!;

	const backend = lex::def_backend()!(actions)!; // use default backend (DFA without environment variable)
	defer lex::destroy(backend);

	const rbuf: [os::BUFSZ]u8 = [0...];
	const lexbuf = buffer::init(in, rbuf)!;
	defer io::close(&lexbuf)!;

	const lexer = lex::init(backend, &lexbuf);
	defer lex::finish(&lexer);

An action callback is associated with an regular expression to
match the tokens. The action callbacks are free to initialize tokens as
they please, but the [[scanner]] object provide convenient functions. The
token lexeme and morphene are sliced from the beginning of the buffer.

	fn literal(
		scan: *lex::scanner,
		lexeme: []u8,
		user: nullable *opaque,
	) (size | *lex::token | lex::error) = {
		return lex::scan_token(scan, void, len(lexeme));
	};

This action callback would return a token of the added action type
(ex: "LIT_STR"), with a void value, and lexing the full lexeme pattern
matched string (ex: "foo").

The callback size return values represents the bytes to slice from the
beginning of the buffer. A size alone is used to skip some bytes.

	append(actions, lex::action {
		expr = "( |\t|\n|\r)+",
		cb = &skip,
		...
	})!;

	fn skip(
		scan: *lex::scanner,
		lexeme: []u8,
		user: nullable *opaque,
	) (size | *lex::token | lex::error) = {
		return len(lexeme);
	};

Action callbacks can be used to match hatch symbols, and then to lex the
scanned input manually.

	append(actions, lex::action {
		expr = `\<`,
		cb = &html,
		name = "ID"
		...
	})!;

	fn html(
		scan: *lex::scanner,
		lexeme: []u8,
		user: nullable *opaque,
	) (size | *lex::token | lex::error) = {
		const start = scan.start;
		let brk = 0z;
		for (let i = 0z; true; i += 1) {
			const r = match (buffer::read_rune(scan)?) {
			case io::EOF =>
				break;
			case let this: rune =>
				yield this;
			};
			if (r == '<') {
				brk += 1;
			} else if (r == '>') {
				brk -= 1;
			};
			if (brk == 0) {
				return lex::scan_token(scan, void, i + 1);
			};
		};

		return lex::syntaxf(start, "unclosed HTML literal");
	};
