An Efficient, Probabilistically Sound Algorithm for Segmentation and Word Discovery
BRENT, MICHAEL R.
Primary Author
Cardie, Claire
Editor
Mooney, Raymond
Editor
mixed material
bibliography
Netherlands
Springer Netherlands
1999
monographic
Volume 34, Numbers 1-3
en
English
35p
Machine Learning
This paper presents a model-based, unsupervised algorithm for recovering word boundaries in a
natural-language text from which they have been deleted. The algorithm is derived from a probability model of the
source that generated the text. The fundamental structure of the model is specified abstractly so that the detailed
component models of phonology, word-order, and word frequency can be replaced in a modular fashion. The
model yields a language-independent, prior probability distribution on all possible sequences of all possible words
over a given alphabet, based on the assumption that the inputwas generated by concatenatingwords from a fixed but
unknown lexicon. The model is unusual in that it treats the generation of a complete corpus, regardless of length,
as a single event in the probability space. Accordingly, the algorithm does not estimate a probability distribution
on words; instead, it attempts to calculate the prior probabilities of various word sequences that could underlie the
observed text. Experiments on phonemic transcripts of spontaneous speech by parents to young children suggest
that our algorithm is more effective than other proposed algorithms, at least when utterance boundaries are given
and the text includes a substantial number of short utterances.
08856125
Portal Perguruan Tinggi
EJ00027
My Library
51
2010-05-21 16:12:06
2010-05-21 16:16:58
machine generated